Nine of ten.
Both records open.
Anchor and mini-swe-agent each resolved nine tasks in the published Lite-10 artifacts. The different failures and setup limits matter as much as the aggregate.
Anchor9/10Anchor: one unresolved patch. Mini: one empty patch after extraction failed. Both published grader reports record nine resolved tasks.
Ten selected Lite tasks, not the full benchmark. Both records name zai-org/GLM-5.3 on Parasail and Docker-backed task environments. Anchor's published limits are 120 steps / 25M tokens; mini uses its own 250-step / $3 loop. These are not identical budget policies, and fixed sampling parameters do not establish statistical significance or general superiority.
Read the patches.
Not just the score.
| Task | Anchor | mini-swe-agent | Anchor $ | Mini $ |
|---|---|---|---|---|
| django__django-10914 | Resolved | Resolved | $0.2579 | Not reported |
| matplotlib__matplotlib-18869 | Resolved | —No patch | $0.3979 | Not reported |
| mwaskom__seaborn-2848 | Resolved | Resolved | $1.4578 | Not reported |
| pallets__flask-4045 | Resolved | Resolved | $0.4817 | Not reported |
| psf__requests-1963 | Resolved | Resolved | $0.4610 | Not reported |
| pydata__xarray-3364 | Resolved | Resolved | $0.3211 | Not reported |
| pylint-dev__pylint-5859 | Resolved | Resolved | $0.1524 | Not reported |
| pytest-dev__pytest-11143 | Resolved | Resolved | $0.1003 | Not reported |
| sphinx-doc__sphinx-10325 | Resolved | Resolved | $0.5598 | Not reported |
| sympy__sympy-11400 | —Unresolved | Resolved | $0.1263 | Not reported |
| TOTAL / 10 TASKS | 9/10 | 9/10 | $4.3162 | Not reported |
Mini has nine reported step counts (454 in total); the extraction-failed task has no step count. Its complete step aggregate, token usage, and cost are not reported. Task times are rounded here; source records retain milliseconds.
Equal resolution.
Different failure paths.
Anchor: sympy remained unresolved.
The published patch did not satisfy the official tests. The record explains the missing relational printer; a local test pass was not enough.
Mini: matplotlib produced no extracted patch.
The runner records a git index.lock error during patch extraction. This is a pipeline failure/empty patch, not evidence that the model submitted a patch which failed the hidden tests.
No cost-efficiency ratio for this round.
Anchor records $4.3162 and 11.36M tokens. Mini does not report comparable usage. The earlier $0.60 / ≈$1.99 comparison belongs to an older OpenRouter baseline and is not reused here.
Two records.
Inspectable outcomes.
Each arm has results, verdicts, manifests, and grading reports. Anchor also includes engine traces; mini does not emit Anchor's event schema.
Both run records identify Docker-backed tasks and the same direct Parasail model endpoint with temperature 0 and seed 42. They ran at different times and used different step/budget policies; do not describe this as a perfectly controlled single-variable experiment.
Use the setup guide.
Keep your own records.
The commands start new runs on current code. Install eval dependencies and prepare task images first. Changing budgets or SDK/model versions changes the experiment; new outcomes are not guaranteed to match these artifacts.
Environment & grading guide# From the repository root, after eval setup. Cloud/model calls can cost money.
# PARASAIL_API_KEY must be set. Docker must be running and on PATH.
npm run eval --prefix evals -- \
--tasks benchmarks/swe-bench/data/swebench-lite-10.jsonl \
--provider docker --arm anchor --llm parasail \
--max-steps 120 --max-tokens 25000000
# Mini has its own loop/limits; consult the current runner's options.
npm run eval --prefix evals -- \
--tasks benchmarks/swe-bench/data/swebench-lite-10.jsonl \
--provider docker --arm mini --llm parasail