Project benchmark

Aider Polyglot code-editing snapshot

Aider Benchmarks · 69 configurations with published scores. Review each metric and its test conditions separately.

Report identity & conditions

Data as of2026-10-14· Report published—. See individual records for run dates and test conditions.

The benchmark has 225 exercises across C++, Go, Java, JavaScript, Python and Rust. Some runs report 223 or 224 test cases; the actual count is retained for each run.
The observation date is when we retrieved the data. Run dates, samples, reasoning settings, edit formats, tool versions and commits are recorded separately. This leaderboard includes historical runs and is not a current overall model ranking.
Display names and CLI aliases do not establish an exact snapshot, so they are not automatically linked to catalog versions. Dollar costs for a run are not reusable API unit prices, and runs with different settings are not controlled comparisons.

Read the original report ↗

Test sampleLeft metricRight metric

Compare metrics within the same report. Each chart explains its direction and scope.
See the details for full values and run conditions.

Percent correct

Results from the original runs, rather than a general capability score. Dates, reasoning settings, edit formats and samples remain separate.: showing 8 of 63 records with values. See details for configurations and dates.

Chart data · 8 entries

Name	%
gpt-5 (high)	88
gpt-5 (medium)	86.7
o3-pro (high)	84.9
gemini-2.5-pro-preview-06-05 (32k think)	83.1
gpt-5 (low)	81.3
o3 (high)	81.3
grok-4 (high)	79.6
gemini-2.5-pro-preview-06-05 (default think)	79.1

Correct edit format

Valid edit format rate. This is separate from task completion accuracy and is not combined into an overall score.: showing 8 of 63 records with values. See details for configurations and dates.

Chart data · 8 entries

Name	%
Codestral 25.01	100
DeepSeek R1 + claude-3-5-sonnet-20241022	100
Gemini 2.0 Pro exp-02-05	100
gemini-2.0-flash-exp	100
gemini-2.5-pro-preview-06-05 (default think)	100
gemma-3-27b-it	100
gpt-4o-mini-2024-07-18	100
Grok 3 Mini Beta (low)	100

Full results & model versions

69 / 69configurations have published scores in this scope. Scored configurations are listed; missing values remain “Not provided.” Different run configurations of the same model are listed separately.

gpt-4.5-preview

Model reported by the benchmark

Run2025-02-27 · 224problems · Formatdiff· Reasoning levelNot listed
Tools0.75.2.dev· Submissionb462e55-dirty

Percent correct: 44.9%
Correct edit format: 97.3%

Run command

aider --model openai/gpt-4.5-preview

gemini-2.5-flash-preview-05-20 (no think)

Model reported by the benchmark

Run2025-05-26 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.83.3.dev· Submission214b811-dirty

Percent correct: 44%
Correct edit format: 93.8%

Run command

aider --model gemini/gemini-2.5-flash-preview-05-20

gpt-oss-120b (high)

Model reported by the benchmark

Run2025-08-06 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.85.3.dev· Submission1af0e59

Percent correct: 41.8%
Correct edit format: 79.1%

Run command

aider --model openrouter/openai/gpt-oss-120b --reasoning-effort high

Qwen3 32B

Model reported by the benchmark

Run2025-05-08 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.82.4.dev· Submissionaaacee5-dirty, aeaf259

Percent correct: 40%
Correct edit format: 83.6%

Run command

aider --model openrouter/qwen/qwen3-32b

gemini-exp-1206

Model reported by the benchmark

Run2024-12-22 · 225problems · Formatwhole· Reasoning levelNot listed
Tools0.69.2.dev· Submissionb1bc2f8

Percent correct: 38.2%
Correct edit format: 98.2%

Run command

aider --model gemini/gemini-exp-1206

Gemini 2.0 Pro exp-02-05

Model reported by the benchmark

Run2025-02-25 · 225problems · Formatwhole· Reasoning levelNot listed
Tools0.75.2.dev· Submission2fccd47

Percent correct: 35.6%
Correct edit format: 100%

Run command

aider --model gemini/gemini-2.0-pro-exp-02-05

Grok 3 Mini Beta (low)

Model reported by the benchmark

Run2025-04-10 · 225problems · Formatwhole· Reasoning levelNot listed
Tools0.81.2.dev· Submission14ffe77-dirty

Percent correct: 34.7%
Correct edit format: 100%

Run command

aider --model openrouter/x-ai/grok-3-mini-beta

o1-mini-2024-09-12

Model reported by the benchmark

Run2024-12-22 · 225problems · Formatwhole· Reasoning levelNot listed
Tools0.69.2.dev· Submission37df899

Percent correct: 32.9%
Correct edit format: 96.9%

Run command

aider --model o1-mini

gpt-4.1-mini

Model reported by the benchmark

Run2025-04-14 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.81.4.dev· Submissionffb743e-dirty

Percent correct: 32.4%
Correct edit format: 92.4%

Run command

aider --model gpt-4.1-mini

claude-3-5-haiku-20241022

Model reported by the benchmark

Run2024-12-21 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.69.2.dev· Submissiona755079-dirty

Percent correct: 28%
Correct edit format: 91.1%

Run command

aider --model claude-3-5-haiku-20241022

chatgpt-4o-latest (2025-02-15)

Model reported by the benchmark

Run2025-02-15 · 223problems · Formatdiff· Reasoning levelNot listed
Tools0.74.3.dev· Submission108ce18-dirty

Percent correct: 27.1%
Correct edit format: 93.3%

Run command

aider --model chatgpt-4o-latest

QwQ-32B + Qwen 2.5 Coder Instruct

Model reported by the benchmark

Run2025-03-07 · 225problems · Formatarchitect· Reasoning levelNot listed
Tools0.75.3.dev· Submission52162a5

Percent correct: 26.2%
Correct edit format: 100%

Run command

aider --model fireworks_ai/accounts/fireworks/models/qwq-32b --architect

gpt-4o-2024-08-06

Model reported by the benchmark

Run2024-12-30 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.70.1.dev· Submission09ee197-dirty

Percent correct: 23.1%
Correct edit format: 94.2%

Run command

aider --model gpt-4o-2024-08-06

gemini-2.0-flash-exp

Model reported by the benchmark

Run2024-12-22 · 225problems · Formatwhole· Reasoning levelNot listed
Tools0.69.2.dev· Submissionb1bc2f8

Percent correct: 22.2%
Correct edit format: 100%

Run command

aider --model gemini/gemini-2.0-flash-exp

qwen-max-2025-01-25

Model reported by the benchmark

Run2025-01-28 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.72.4.dev· Submissionae7d459

Percent correct: 21.8%
Correct edit format: 90.2%

Run command

OPENAI_API_BASE=https://dashscope-intl.aliyuncs.com/compatible-mode/v1 aider --model openai/qwen-max-2025-01-25

QwQ-32B

Model reported by the benchmark

Run2025-03-06 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.75.3.dev· Submission51d118f-dirty

Percent correct: 20.9%
Correct edit format: 67.6%

Run command

aider --model fireworks_ai/accounts/fireworks/models/qwq-32b

gemini-2.0-flash-thinking-exp-01-21

Model reported by the benchmark

Run2025-01-21 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.72.2.dev· Submission843720a

Percent correct: 18.2%
Correct edit format: 77.8%

Run command

aider --model gemini/gemini-2.0-flash-thinking-exp-01-21

gpt-4o-2024-11-20

Model reported by the benchmark

Run2024-12-30 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.70.1.dev· Submission09ee197-dirty

Percent correct: 18.2%
Correct edit format: 95.1%

Run command

aider --model gpt-4o-2024-11-20

DeepSeek Chat V2.5

Model reported by the benchmark

Run2024-12-21 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.69.2.dev· Submissiona755079-dirty

Percent correct: 17.8%
Correct edit format: 92.9%

Run command

aider --model deepseek/deepseek-chat

Qwen2.5-Coder-32B-Instruct

Model reported by the benchmark

Run2024-12-26 · 225problems · Formatwhole· Reasoning levelNot listed
Tools0.69.2.dev· Submissionb51768b0

Percent correct: 16.4%
Correct edit format: 99.6%

Run command

aider --model openai/Qwen2.5-Coder-32B-Instruct