Deepseek Flash Benchmarked Against Official Inference and Opencode Go with Pi and Opencode Harness
Introduction
Does the harness and the serving stack change how well a model builds a web app? I tested it with one model, one task, and two configurations. Arm A ran DeepSeek-V4.1-Flash through the pi harness against the official DeepSeek API. Arm B ran the same model through the opencode harness against the OpenCode Go gateway. Everything else stayed fixed: same task image, same instruction, same judge, same resource limits, same score model.
The result was clear enough to report and narrow enough to qualify. Arm A passed all 4 trials. Arm B passed 2 of 4, and each failure dropped a different item.
Key Takeaways
- Arm A (pi 1.4.0 plus Official Deepseek inference API) scored 1.000 in all 3 primary trials and in the smoke trial: 4 of 4 passes.
- Arm B (opencode 2.2.5 plus Go gateway) scored 0.7, 1.0, 0.7 in the primary trials and 0.3 in smoke: 2 of 4 passes, mean 0.675.
- Arm B failed a different item in each failing trial, which points to unstable build quality rather than one permanent defect.
- Arm B sent 11 to 26 times more input tokens per trial while writing less output, so the harness conversation shape differs a lot even when the model does not.
Test setup: one task, two arms, one model
The task comes from WorkBuddy Bench Web v1.0: workbuddy/support-inbox-triage-L3-040. The agent builds a static support-inbox workbench from scratch and delivers /workspace/www/index.html plus relative assets. Grading covers desktop 1440x900 and mobile 390x844 viewports. The page needs a three-column layout, filter chips, assignment with rollback, a live SLA badge, debounced read marking, shift-click range select, and a virtualized 200-item list that renders at most 30 row nodes.
| Item | Arm A | Arm B |
|---|---|---|
| Harness | pi 1.4.0 | opencode 2.2.5 |
| Model id | deepseek-flash |
deepseek-v4.1-flash |
| Model | DeepSeek-V4.1-Flash | DeepSeek-V4.1-Flash |
| Endpoint | https://api.deepseek.com |
https://opencode.ai/inference/go/openai/v1 |
| Judge | GLM-5.3-Flash (vision capable, not a DeepSeek model) | Same judge |
| Trials | 3 primary plus 1 smoke | 3 primary plus 1 smoke |
Scoring uses a penalty sum: major 0.3, normal 0.2, minor 0.1, fatal 1.0, with max(0, 1 minus penalties). The pass gate is strict at score 1.0 plus the completion condition, so a single failed normal item already fails the case. Runs happened on a 2 vCPU, 4 GB box.
Primary results
Arm A passed every trial. Arm B passed one of three primary trials, with both failures landing at 0.7 (one major miss each).
| Arm | Trial 1 | Trial 2 | Trial 3 | Mean | Pass (score 1.0) |
|---|---|---|---|---|---|
| A: pi plus official | 1.0000 | 1.0000 | 1.0000 | 1.000 | 3/3 |
| B: opencode plus Go | 0.7000 | 1.0000 | 0.7000 | 0.800 | 1/3 |
Smoke trials came first: arm A scored 1.0, arm B scored 0.3. Counting everything:
| Arm | Trials | Mean | Pass | Failed items seen |
|---|---|---|---|---|
| A | 4 | 1.000 | 4/4 | none |
| B | 4 | 0.675 | 2/4 | assignment_updates_state, overdue_filter_strict, workbench_visual_density, resolve_flow, filters_work |
The per-item picture sharpens the story. The task has 12 items: 6 LLM-graded, 2 vision-graded, 4 agent-graded (the agent judge drives a real browser).
| Item | Judge | Severity | Arm A (4 trials) | Arm B (4 trials) |
|---|---|---|---|---|
three_panel_layout |
vision | normal | 4/4 | 4/4 |
workbench_visual_density |
vision | normal | 4/4 | 3/4 |
conversation_data_used |
LLM | major | 4/4 | 4/4 |
filters_work |
LLM | major | 4/4 | 3/4 |
assignment_updates_state |
LLM | major | 4/4 | 3/4 |
resolve_flow |
LLM | major | 4/4 | 3/4 |
sla_badge_live_recompute |
LLM | major | 4/4 | 4/4 |
overdue_filter_strict |
LLM | normal | 4/4 | 3/4 |
optimistic_assign_rollback |
agent | major | 4/4 | 4/4 |
shift_click_range_select |
agent | major | 4/4 | 4/4 |
conversation_list_virtualized |
agent | major | 4/4 | 4/4 |
debounced_read_marking |
agent | normal | 4/4 | 4/4 |
What the two builds looked like
Both builds below are trial 1 output. Arm A (score 1.0) produced a dense three-pane RelayDesk workbench with the queue, chat thread, and customer rail all populated, plus a visible implementation note panel at the bottom right.
Arm B (score 0.7) produced a cleaner but thinner Support Inbox: three panes render, but the center thread shows placeholder copy (“Waiting for an owner”) and the right rail carries a long implementation trail instead of live customer detail.
On mobile, arm A collapsed to a single queue column with search, tag chips, bulk assign, and SLA badges intact at 390px wide.
Arm B kept all three panes side by side on the mobile viewport instead of collapsing to one column. The queue, thread, and customer panels squeeze together, so text and controls read smaller than in arm A. The content is all present, but the layout never switched to a phone shape.
Token and time costs diverged
Each row is one trial. Patch size is the produced diff, and the token counts come from the container logs.
| Arm | Trial | Score | Patch size | Input tokens | Output tokens | Cache read | Wall clock |
|---|---|---|---|---|---|---|---|
| A | 1 | 1.0000 | 162 KB | 13,970 | 85,547 | 2,970,368 | 24 min 52 s |
| A | 2 | 1.0000 | 161 KB | 23,695 | 78,850 | 2,929,792 | 21 min 17 s |
| A | 3 | 1.0000 | 146 KB | 26,581 | 64,153 | 1,806,592 | 23 min 12 s |
| B | 1 | 0.7000 | 155 KB | 286,791 | 49,763 | 9,346,560 | 36 min 35 s |
| B | 2 | 1.0000 | 155 KB | 317,999 | 34,687 | 3,741,952 | 31 min 47 s |
| B | 3 | 0.7000 | 107 KB | 366,996 | 45,604 | 10,594,176 | 27 min 39 s |
Three cautions before anyone cites these numbers.
- Arm B sent 11 to 26 times more input tokens than arm A per trial while arm A wrote more output tokens, so the two harnesses pack context very differently.
- All three trials of one arm ran at the same time on a 2 vCPU box, so wall clock includes contention from sibling trials: compare arms, not minutes.
- Both providers report zero cost in the container logs, so do not use these logs for a cost comparison.
Limits of this test
The sample is small: 4 trials per arm on one task. Do not use these numbers to predict other tasks.
The judge decides 8 of the 12 items. A different judge can change a result. Both arms use the same judge, and the evidence for both is kept.
Two variables changed at the same time: the harness and the serving stack. This test cannot say which of the two caused the difference. A cross test (pi with Go, opencode with official) would separate them.
Arm B used a different model id on a gateway, not the official endpoint. The gateway can route, quantize, or rate-limit a request differently.
The three trials of one arm ran at the same time on 2 vCPU. Contention can change trial duration and can change the result of a timing-sensitive item.
Arm B failed a different item in each failing trial. Treat this as variance in build quality, not one permanent defect.
The box reports 4 GB of RAM against a task ask of 6 GB. A memory limit can remove a trial at random.