Jonathan Franklin
agent-bench
Compare pi, oh-my-pi and OpenCode using the same local LLM, with independent grading and inspectable reports.
Compare coding-agent harnesses using the same local model.
agent-bench is a Python command-line tool that runs one LAN-hosted LLM through three coding-agent harnesses: pi, oh-my-pi and OpenCode. Each attempt starts in a fresh Docker container, and the resulting patch is evaluated independently in a separate grading container using hidden acceptance tests.
The built-in tasks cover bug fixes, small features, refactoring, security hardening and performance work. Each run produces a machine-readable JSON report and a standalone HTML report you can open offline, with scores, timings, logs and code diffs.
Why I created it
A coding agent is more than the model behind it. Its system prompt, tools and execution loop all influence whether it can turn a request into a working change. I wanted a way to compare those harnesses while keeping the model constant, rather than relying on impressions from a few interactive sessions.
I built agent-bench to make those comparisons inspectable: give the agents the same tasks, test their patches independently, and keep the failures alongside the successes. It’s a practical experiment for evaluating a local AI development setup, not a claim that a small synthetic test suite can rank every agent on real-world software engineering.
Highlights
- Controlled comparisons: one model endpoint, fresh workspaces and consistent container resource limits across harnesses.
- Independent grading: hidden tests check the submitted code after the agent has stopped, rather than trusting its own assessment.
- Transparent reports: correctness, completion, elapsed time, patch size and usage where available, with logs and diffs for investigation.
- Lightweight orchestration: Python 3.11+ and Docker on the host, with no Python runtime package dependencies; designed for llama.cpp’s OpenAI-compatible API.
Status: Experimental alpha.
Get involved
The repository includes setup instructions, contribution guidance and an issue tracker. Try it out, share feedback or contribute a change.