Same model, different harness: benchmarking qwen3.8-27b with Agent Bench

Changing a coding agent does not necessarily mean changing its model. The same LLM can be wrapped in different system prompts, editing tools, context-management strategies and execution loops. Those choices matter: the useful output is not a confident explanation, but a patch that satisfies the task.

I built Agent Bench to make that comparison easier. It runs the same coding tasks through different agent harnesses against one shared model endpoint, then grades the resulting code independently.

For this first look, I ran qwen3.8-27b locally through ninfer, comparing pi, oh-my-pi and OpenCode on 4 October 2026.

The short answer: OpenCode performed best in this run. It passed all five tasks, scored 100/100, and had the lowest mean wall time at 13.2 seconds per attempt. pi and oh-my-pi both scored 97.5/100, passing four tasks completely. This is an early, five-task snapshot—not evidence of a universal winner.

Why benchmark the harness, not just the model?

Model benchmarks help answer whether an LLM can solve a programming problem. Choosing a coding agent asks a slightly different question: can this model, inside this harness, inspect a workspace, use tools, make the right edits and finish the job?

A harness is the machinery around the model: its instructions, tool definitions, file-editing behavior, conversation management and rules for continuing or stopping. A strong model can still miss a requirement; a polished final response can accompany a broken implementation.

Agent Bench turns that into a repeatable experiment:

1. Give each harness an equivalent fresh task workspace and the same task prompt.
2. Point every harness at the same model server with common benchmark settings.
3. Let the agent inspect files, edit code and run the visible tests.
4. Stop the agent container, then grade its submitted production Python files in a separate clean container using hidden acceptance tests.
5. Save scores, timings, checks and diagnostics in JSON and a standalone HTML report.

The hidden tests are not mounted into the agent’s workspace. Grading containers have no network access, and changes to the agent’s tests do not become the acceptance tests. That separation is useful: a claim of success is not the measurement of success.

This is deliberately a model-plus-harness comparison. Native tools were enabled, so tool semantics and system prompts were not identical. Subagent delegation is disabled by the benchmark’s adapter policy, and auxiliary model routing is pinned to the selected model. Some harnesses can still make background calls, so reported primary-stream token usage is not necessarily a complete inference bill.

What was held constant?

The run used Agent Bench 0.1.0, with one attempt per task per harness: five tasks × three harnesses = 15 attempts. Attempts ran serially in a seeded shuffled order, rather than making the harnesses compete for the model server at the same time.

• Model identifier: qwen3.8-27b on the same OpenAI-compatible endpoint.
• Harness versions: pi 0.99.2, oh-my-pi 18.4.8, OpenCode 1.18.34.
• Task selection: the default set, containing all five built-in tasks.
• Client context metadata: 196,608 tokens; maximum response length: 4,096 tokens. The latter is a per-response limit, not a total attempt budget.
• Reasoning configuration: reasoning: false, thinking: off, using the llama-cpp API profile. That profile requests chat_template_kwargs.enable_thinking=false; this records the intended request control, not an independent audit of server internals.
• Agent and grader container limits: two CPUs and 4 GiB RAM; 600-second agent timeout and 60-second grading timeout. These limits do not constrain the model server’s GPU.
• Scheduling seed: 42. This controls attempt order, not model sampling determinism.

The local inference setup

The local model server runs on my Ubuntu 24.04.5 LTS (x86-64) workstation, with:

• CPU: AMD Ryzen 9 9950X3D, with 16 cores and 32 threads.
• GPU: NVIDIA GeForce RTX 5090, with 32 GiB VRAM.
• System memory: 32 GiB.
• Storage: a WD Green SN3000 2 TB NVMe SSD.

These specifications were inspected on the workstation while preparing this post; they are not a hardware snapshot captured with the benchmark run. The agent containers still use the CPU and RAM limits listed above, rather than the workstation’s full resources.

The ninfer systemd service configuration inspected for this post loads qwen3_8_27b_nvfp4.ninfer and supplies these inference-related flags:

--max-context 240000
--kv-capacity 240000
--max-concurrency 2
--kv-dtype fp8
--host-kv-mib 16384
--spec mtp --draft-tokens 3
--lm-head-draft
--vision
--media-live-mib 2048

The filename identifies an NVFP4 model artifact; the service enables FP8 KV storage and MTP speculative decoding with three draft tokens. Vision is enabled, but these tasks only exercise text and code. Although the server allows two concurrent requests, Agent Bench scheduled attempts serially. In my testing with the above ninfer configuration, I found this maximises the available resources whilst still providing vision capabilities. With this configuration I’m able to achieve ~130 tokens/sec.

The 240,000-token server setting and 196,608-token client context metadata are separate values; neither means each task consumed that much context. These are the service settings available when writing this post, not a server snapshot embedded in the results. The report does not establish a weight checksum, ninfer build, run-time GPU specification or effective sampling settings; the hardware summary above comes from a separate inspection. Timings should therefore be read as measurements of this local setup, not portable speed claims about the model.

What do the five tests ask the agents to do?

These are small, synthetic Python repositories with a visible contract and public tests. Each task also has hidden checks that cover more than the obvious happy path. There are 25 hidden test methods across the five tasks, but tasks are weighted equally in the overall score.

1. Bug fix: invoice reconciliation precision

Fix invoice_total(items) so money stays in decimal arithmetic, discounts work, and negative quantities represent refunds. The invoice must round once at the invoice boundary, using half-even rounding, return a fixed two-decimal string and normalize negative zero.

The five hidden methods cover the monetary contract, edge cases and regressions. This is not just “replace a float”: rounding each line would implement a different policy.

2. Small feature: nested configuration overlays

Extend merge_settings(base, override) with recursive dictionary merging and a DELETE sentinel. Lists replace rather than concatenate; None is a normal value, not deletion. The result must be a completely independent deep copy, without mutating or aliasing either input.

Four hidden methods check merge/delete semantics, independence and seeded generated cases. A merge can look correct until a later mutation exposes shared nested objects.

3. System refactor: centralize pricing policy

Extract pricing.line_total(...) and make checkout and reporting delegate to it dynamically. Preserve validation and behavior, including per-line half-even rounding—intentionally different from the invoice task.

Five hidden methods check behavior and architecture. AST inspections look for leftover pricing logic in the clients, and monkeypatching the shared function checks that both callers really delegate. Producing the right total is necessary, but not sufficient for this requested refactor.

4. Security hardening: safe note-file reads

Harden read_note(root, name) against invalid names, traversal, symlinks and non-regular files, while preserving UTF-8 content and newlines. The actual open must be directory-anchored and use O_NOFOLLOW and O_NONBLOCK, followed by fstat; a preliminary path check alone is not enough. File descriptors must close on both success and failure.

Eight hidden methods cover the contract, safe-open behavior, error handling and resource cleanup. This task produced the only correctness failures in the run.

5. Performance: linear rolling event counts

Optimize rolling_counts(timestamps, window) from repeated scanning to O(n) time with O(1) auxiliary state, excluding the returned list. Preserve an exclusive left window boundary, duplicate timestamps and read-only sequence support.

Three hidden methods check exact results, complexity and runtime. Instrumented sequences impose a 12 × n + 100 element-read budget on adversarial inputs, alongside a generous three-second ceiling for 4,000 events. Counting work is less dependent on machine speed than a tight stopwatch threshold.

Results: OpenCode leads this run

The distinction between mean score and full passes matters. An attempt scores 100 × hidden test methods passed / total methods; a full pass requires every hidden check to pass and the harness to complete successfully. A failing parameterized subtest fails its whole test method. Timeouts and execution errors score zero, even if a partial patch has useful diagnostic grading.

For pi and oh-my-pi, the calculation is:

(100 + 100 + 100 + 87.5 + 100) / 5 = 97.5
4 fully passed tasks / 5 attempts = 80% success rate

Pooling all hidden checks would give those harnesses 24/25, or 96%, but that is not Agent Bench’s overall score. Equal task weighting prevents the eight-method security task from outweighing the three-method performance task merely because it has more methods.

All 15 attempts completed without a timeout or execution error. The two failures were ordinary correctness failures, not broken harness runs.

Where did the difference come from?

Every harness passed the invoice fix, configuration feature, pricing refactor and rolling-count optimization. The entire score gap came from one hidden method in the security task: test_descriptors_closed_on_success_and_errors.

The submitted pi and oh-my-pi patches both opened a directory descriptor, then used it to open the note. They closed the note descriptor, but never closed the directory descriptor. The hidden check tracks descriptors returned by os.open and verifies that they are no longer usable after the function returns or raises. Both patches passed seven of the eight security checks, but missed that cleanup requirement.

This is a useful example of why independent acceptance tests matter. Rejecting traversal and symlinks is important, but a hardened function also needs correct resource lifetime management. Neither a plausible implementation nor a passing public test proves the whole contract has been met.

Speed: wall time, not tokens per second

OpenCode averaged 13.2 seconds, pi 15.6 seconds, and oh-my-pi 19.4 seconds. With correctness as the primary measure and wall time as a secondary measure, OpenCode wins both in this sample. pi and oh-my-pi tie on correctness, with pi faster on average.

Averages conceal task-level variation: pi was faster on the invoice fix (6.7 seconds versus OpenCode’s 10.3 seconds), while oh-my-pi’s configuration task took 37.5 seconds. One attempt per task is not enough to tell whether those differences persist. These timings include harness overhead and server behavior; they are not measurements of raw model throughput.

What this does—and does not—tell us

Agent Bench is an experimental alpha. This version has just five built-in benchmark tasks, all synthetic Python exercises. They are useful for catching concrete contract violations, but they do not represent the breadth of production engineering: large repositories, ambiguous requirements, dependency changes, frontend work, long debugging sessions or multi-language systems.

There are other important limits:

• Only one attempt per harness/task. The 80% and 100% success rates describe these five attempts, not estimated probabilities of future success. There are no confidence intervals or statistically reliable pass@k claims here.
• A near-ceiling suite. All harnesses scored at least 97.5. More diverse and difficult tasks are needed to separate their capabilities meaningfully.
• The harness remains part of the experiment. Different native prompts and tools are intentional, but the results do not isolate a single tool or policy as the cause of the difference.
• Hidden checks are not a security boundary. Candidate Python executes in the grading process; the benchmark is not designed to withstand deliberately malicious test inspection or monkeypatching. Architecture checks also encode the particular refactor requested.
• No cost ranking. Token prices were unset, and electricity, hardware amortization and complete background inference usage were not measured. Local inference does not imply zero cost.

My takeaway is narrow but actionable: for these tasks, this model endpoint and these harness versions, OpenCode produced the strongest measured outcome. The shared cleanup failure is at least as interesting as the ranking: it shows a specific requirement that two otherwise successful agents missed.

The next useful experiment would repeat the suite several times, vary scheduling seeds, record the exact server build and model artifact, and add more realistic tasks. Repeating these five tasks would help assess variability, but would not fix the suite’s limited coverage by itself.

Inspect the data or try your own model

The source run is 20261004T070523Z-3662bf45, recorded from 07:05:23 to 07:09:26 UTC on 4 October 2026. The charts and tables above are generated from a public, allowlisted export of its results.json, not hand-entered chart values.

That export includes individual scores, hidden-check counts, wall times, harness versions, task hashes and a SHA-256 fingerprint of the source report. It deliberately omits private endpoints, raw logs, transcripts and candidate patches. The ninfer settings described above are separate experiment notes, not fields captured by Agent Bench.

To run a comparison yourself, start with the Agent Bench README. You need Python 3.11+, Docker and a reachable tool-capable, OpenAI-compatible model server. The CLI runs the harnesses in containers and produces a JSON file plus an offline HTML report. Keep the model deployment fixed, record its settings, and compare working patches, not just convincing answers.