Project benchmark

Aider Polyglot code-editing snapshot

Aider Benchmarks · 69 configurations with published scores. Review each metric and its test conditions separately.

Report identity & conditions

Data as of2026-10-14· Report published—. See individual records for run dates and test conditions.

The benchmark has 225 exercises across C++, Go, Java, JavaScript, Python and Rust. Some runs report 223 or 224 test cases; the actual count is retained for each run.
The observation date is when we retrieved the data. Run dates, samples, reasoning settings, edit formats, tool versions and commits are recorded separately. This leaderboard includes historical runs and is not a current overall model ranking.
Display names and CLI aliases do not establish an exact snapshot, so they are not automatically linked to catalog versions. Dollar costs for a run are not reusable API unit prices, and runs with different settings are not controlled comparisons.

Read the original report ↗

Test sampleLeft metricRight metric

Compare metrics within the same report. Each chart explains its direction and scope.
See the details for full values and run conditions.

Percent correct

Results from the original runs, rather than a general capability score. Dates, reasoning settings, edit formats and samples remain separate.: showing 8 of 63 records with values. See details for configurations and dates.

Chart data · 8 entries

Name	%
gpt-5 (high)	88
gpt-5 (medium)	86.7
o3-pro (high)	84.9
gemini-2.5-pro-preview-06-05 (32k think)	83.1
gpt-5 (low)	81.3
o3 (high)	81.3
grok-4 (high)	79.6
gemini-2.5-pro-preview-06-05 (default think)	79.1

Correct edit format

Valid edit format rate. This is separate from task completion accuracy and is not combined into an overall score.: showing 8 of 63 records with values. See details for configurations and dates.

Chart data · 8 entries

Name	%
Codestral 25.01	100
DeepSeek R1 + claude-3-5-sonnet-20241022	100
Gemini 2.0 Pro exp-02-05	100
gemini-2.0-flash-exp	100
gemini-2.5-pro-preview-06-05 (default think)	100
gemma-3-27b-it	100
gpt-4o-mini-2024-07-18	100
Grok 3 Mini Beta (low)	100

Full results & model versions

69 / 69configurations have published scores in this scope. Scored configurations are listed; missing values remain “Not provided.” Different run configurations of the same model are listed separately.

o1-2024-12-17 (high)

Model reported by the benchmark

Run2024-12-21 · 224problems · Formatdiff· Reasoning levelNot listed
Tools0.69.2.dev· Submissiona755079-dirty

Percent correct: 61.7%
Correct edit format: 91.5%

Run command

aider --model openrouter/openai/o1

claude-sonnet-4-20250514 (32k thinking)

Model reported by the benchmark

Run2025-05-24 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.83.3.dev· Submissione3cb907

Percent correct: 61.3%
Correct edit format: 97.3%

Run command

aider --model claude-sonnet-4-20250514

claude-3-7-sonnet-20250219 (no thinking)

Model reported by the benchmark

Run2025-02-24 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.74.4.dev· Submission75e9ee6

Percent correct: 60.4%
Correct edit format: 93.3%

Run command

aider --model sonnet

o3-mini (high)

Model reported by the benchmark

Run2025-01-31 · 224problems · Formatdiff· Reasoning levelNot listed
Tools0.72.4.dev· Submissionb0d58d1-dirty

Percent correct: 60.4%
Correct edit format: 93.3%

Run command

aider --model o3-mini --reasoning-effort high

Qwen3 235B A22B diff, no think, Alibaba API

Model reported by the benchmark

Run2025-05-09 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.82.4.dev· Submission91d7fbd-dirty

Percent correct: 59.6%
Correct edit format: 92.9%

Run command

aider --model openai/qwen3-235b-a22b

Kimi K2

Model reported by the benchmark

Run2025-07-17 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.85.3.dev· Submission915ebff-dirty

Percent correct: 59.1%
Correct edit format: 92.9%

Run command

aider --model openrouter/moonshotai/kimi-k2

DeepSeek R1

Model reported by the benchmark

Run2025-01-20 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.71.2.dev· Submission5650697-dirty

Percent correct: 56.9%
Correct edit format: 96.9%

Run command

aider --model deepseek/deepseek-reasoner

claude-sonnet-4-20250514 (no thinking)

Model reported by the benchmark

Run2025-05-24 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.83.3.dev· Submissionef3f8bb-dirty

Percent correct: 56.4%
Correct edit format: 98.2%

Run command

aider --model claude-sonnet-4-20250514

gemini-2.5-flash-preview-05-20 (24k think)

Model reported by the benchmark

Run2025-05-25 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.83.3.dev· Submissiona8568c3-dirty

Percent correct: 55.1%
Correct edit format: 95.6%

Run command

aider --model gemini/gemini-2.5-flash-preview-05-20

DeepSeek V3 (0324)

Model reported by the benchmark

Run2025-03-24 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.78.1.dev· Submission502b863

Percent correct: 55.1%
Correct edit format: 99.6%

Run command

aider --model deepseek/deepseek-chat

Quasar Alpha

Model reported by the benchmark

Run2025-04-04 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.80.5.dev· Submission8a34a6c-dirty

Percent correct: 54.7%
Correct edit format: 98.2%

Run command

aider --model openrouter/openrouter/quasar-alpha

o3-mini (medium)

Model reported by the benchmark

Run2025-01-31 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.72.4.dev· Submission2fb517b-dirty

Percent correct: 53.8%
Correct edit format: 95.1%

Run command

aider --model o3-mini

Grok 3 Beta

Model reported by the benchmark

Run2025-04-10 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.81.2.dev· Submission2dd40fc-dirty

Percent correct: 53.3%
Correct edit format: 99.6%

Run command

aider --model openrouter/x-ai/grok-3-beta

Optimus Alpha

Model reported by the benchmark

Run2025-04-10 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.81.2.dev· Submission532bc45-dirty

Percent correct: 52.9%
Correct edit format: 97.3%

Run command

aider --model openrouter/openrouter/optimus-alpha

gpt-4.1

Model reported by the benchmark

Run2025-04-14 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.81.4.dev· Submission7a87db5-dirty

Percent correct: 52.4%
Correct edit format: 98.2%

Run command

aider --model gpt-4.1

claude-3-5-sonnet-20241022

Model reported by the benchmark

Run2025-01-17 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.71.2.dev· Submission6451d59

Percent correct: 51.6%
Correct edit format: 99.6%

Run command

aider --model claude-3-5-sonnet-20241022

Grok 3 Mini Beta (high)

Model reported by the benchmark

Run2025-04-10 · 225problems · Formatwhole· Reasoning levelNot listed
Tools0.81.3.dev· Submission8ee33da-dirty

Percent correct: 49.3%
Correct edit format: 99.6%

Run command

aider --model xai/grok-3-mini-beta --reasoning-effort high

DeepSeek Chat V3 (prev)

Model reported by the benchmark

Run2024-12-25 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.69.2.dev· Submission0a23c4a-dirty

Percent correct: 48.4%
Correct edit format: 98.7%

Run command

aider --model deepseek/deepseek-chat

gemini-2.5-flash-preview-04-17 (default)

Model reported by the benchmark

Run2025-04-20 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.82.3.dev· Submission7fcce5d-dirty

Percent correct: 47.1%
Correct edit format: 85.3%

Run command

aider --model gemini/gemini-2.5-flash-preview-04-17

chatgpt-4o-latest (2025-03-29)

Model reported by the benchmark

Run2025-03-29 · 225problems · Formatdiff· Reasoning levelNot listed
Tools0.79.3.dev· Submission0decbad

Percent correct: 45.3%
Correct edit format: 64.4%

Run command

aider --model chatgpt-4o-latest