Project benchmark

BFCL V4 function calling, search & memory

BFCL · 伯克利工具调用 · 109 configurations with published scores. Review each metric and its test conditions separately.

Report identity & conditions

Data as of2026-10-19· Report published—. See individual records for run dates and test conditions.

Leaderboard updated2026-04-12

Overall and task scores retain the official results. This report covers function calling, multi-turn interactions, web search and memory, rather than all general capabilities.
FC means native function/tool calling; Prompt uses text generation to express calls. Thinking configurations remain separate. Unstated calling modes are not inferred.
Estimated benchmark costs use USD, latency uses seconds and format sensitivity uses percentage points. They are compared separately and are not API price quotes.
The leaderboard update date is not each configuration’s test date. Organizations, model links and licenses follow the original board; license scope depends on the specific model terms.
Results are cited from the official public BFCL leaderboard. Test questions, answers, model outputs and benchmark implementations are not reproduced.

Left metricRight metric

Compare metrics within the same report. Each chart explains its direction and scope.
See the details for full values and run conditions.

Overall accuracy

The published overall accuracy is retained rather than recalculated from subtasks. Comparisons stay within the same BFCL V4 configuration scope.: showing 8 of 109 records with values. See details for configurations and dates.

Chart data · 8 entries

Name	%
Claude-Opus-4-5-20251101 (FC)	77.47
Claude-Sonnet-4-5-20250929 (FC)	73.24
Gemini-3-Pro-Preview (Prompt)	72.51
GLM-4.6 (FC thinking)	72.38
Grok-4-1-fast-reasoning (FC)	69.57
Claude-Haiku-4-5-20251001 (FC)	68.7
Gemini-3-Pro-Preview (FC)	68.14
o3-2025-04-16 (Prompt)	63.05

Estimated benchmark cost

Estimated total benchmark cost in USD. This is neither cost per task, cost per successful task nor a current API unit price.: showing 8 of 109 records with values. See details for configurations and dates.

Chart data · 8 entries

Name	USD
Qwen3-0.6B (FC)	0.46
Llama-3.1-Nemotron-Ultra-253B-v1 (FC)	0.72
Qwen3-14B (Prompt)	1.35
Granite-4.0-350m (FC)	1.44
Command R7B (FC)	1.5
Qwen3-30B-A3B-Instruct-2507 (Prompt)	1.56
Llama-3.2-1B-Instruct (FC)	1.64
Falcon3-1B-Instruct (FC)	1.72

Full results & model versions

109 / 109configurations have published scores in this scope. Scored configurations are listed; missing values remain “Not provided.” Different run configurations of the same model are listed separately.

GLM-4.6 (FC thinking)

Model reported by the benchmark

Native function calling · FC · 思考模式

Multi-turn overall · original rank 5 名 · 综合准确率原榜第 4

As reportedOrganization: Zhipu AI · License: MIT · Model details ↗

Overall accuracy: 72.38%
Estimated benchmark cost: 4.64 USD
Average latency: 4.34 秒
Latency standard deviation: 7.22 秒

More metrics

P95 latency: 13.5 秒
Non-live AST overall: 87.56%
Non-live simple calls: 74.25%
Non-live multiple-function selection: 95%
Non-live parallel calls: 91.5%
Non-live parallel multiple-function calls: 89.5%
Live AST overall: 80.9%
Live simple calls: 89.53%
Live multiple-function selection: 78.92%
Live parallel calls: 81.25%
Live parallel multiple-function calls: 75%
Multi-turn overall: 68%
Multi-turn base tasks: 74.5%
Multi-turn missing functions: 68%
Multi-turn missing parameters: 63%
Multi-turn long context: 66.5%
Web search overall: 77.5%
Search with summaries: 79%
Search without summaries: 76%
Memory overall: 55.7%
Key-value memory: 43.87%
Vector memory: 56.13%
Recursive-summary memory: 67.1%
Relevant-function detection: 75%
Irrelevant-function detection: 84.96%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

CoALM-8B

Model reported by the benchmark

Multi-turn overall · original rank 73 名 · 综合准确率原榜第 84

As reportedOrganization: UIUC + Oumi · License: Meta Llama 3 Community · Model details ↗

Overall accuracy: 26.81%
Estimated benchmark cost: 25.33 USD
Average latency: 20.36 秒
Latency standard deviation: 73.74 秒

More metrics

P95 latency: 138.04 秒
Non-live AST overall: 84.87%
Non-live simple calls: 69.5%
Non-live multiple-function selection: 93.5%
Non-live parallel calls: 88%
Non-live parallel multiple-function calls: 88.5%
Live AST overall: 66.77%
Live simple calls: 70.54%
Live multiple-function selection: 66.19%
Live parallel calls: 62.5%
Live parallel multiple-function calls: 54.17%
Multi-turn overall: 8%
Multi-turn base tasks: 10%
Multi-turn missing functions: 7%
Multi-turn missing parameters: 8%
Multi-turn long context: 7%
Web search overall: 0%
Search with summaries: 0%
Search without summaries: 0%
Memory overall: 2.8%
Key-value memory: 3.23%
Vector memory: 3.87%
Recursive-summary memory: 1.29%
Relevant-function detection: 87.5%
Irrelevant-function detection: 86.9%
Maximum format variation: 79 百分点
Format variation standard deviation: 34.18 百分点

Qwen3-1.7B (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 65 名 · 综合准确率原榜第 71

As reportedOrganization: Qwen · License: apache-2.0 · Model details ↗

Overall accuracy: 28.41%
Estimated benchmark cost: 4.33 USD
Average latency: 5.12 秒
Latency standard deviation: 7.37 秒

More metrics

P95 latency: 13.35 秒
Non-live AST overall: 82.92%
Non-live simple calls: 70.67%
Non-live multiple-function selection: 92.5%
Non-live parallel calls: 88.5%
Non-live parallel multiple-function calls: 80%
Live AST overall: 74.61%
Live simple calls: 76.74%
Live multiple-function selection: 74.26%
Live parallel calls: 62.5%
Live parallel multiple-function calls: 75%
Multi-turn overall: 11%
Multi-turn base tasks: 15%
Multi-turn missing functions: 6%
Multi-turn missing parameters: 12%
Multi-turn long context: 11%
Web search overall: 2.5%
Search with summaries: 3%
Search without summaries: 2%
Memory overall: 6.02%
Key-value memory: 4.52%
Vector memory: 7.74%
Recursive-summary memory: 5.81%
Relevant-function detection: 81.25%
Irrelevant-function detection: 76.54%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Grok-4-0709 (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 38 名 · 综合准确率原榜第 10

As reportedOrganization: xAI · License: Proprietary · Model details ↗

Overall accuracy: 61.38%
Estimated benchmark cost: 355.17 USD
Average latency: 15.49 秒
Latency standard deviation: 26.22 秒

More metrics

P95 latency: 44.28 秒
Non-live AST overall: 85.38%
Non-live simple calls: 73.5%
Non-live multiple-function selection: 92.5%
Non-live parallel calls: 88.5%
Non-live parallel multiple-function calls: 87%
Live AST overall: 75.57%
Live simple calls: 82.17%
Live multiple-function selection: 73.88%
Live parallel calls: 75%
Live parallel multiple-function calls: 79.17%
Multi-turn overall: 33.88%
Multi-turn base tasks: 44%
Multi-turn missing functions: 19%
Multi-turn missing parameters: 28.5%
Multi-turn long context: 44%
Web search overall: 82%
Search with summaries: 80%
Search without summaries: 84%
Memory overall: 55.91%
Key-value memory: 57.42%
Vector memory: 58.71%
Recursive-summary memory: 51.61%
Relevant-function detection: 87.5%
Irrelevant-function detection: 75.4%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Llama-3.3-70B-Instruct (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 50 名 · 综合准确率原榜第 62

As reportedOrganization: Meta · License: Meta Llama 3 Community · Model details ↗

Overall accuracy: 31.9%
Estimated benchmark cost: 29.54 USD
Average latency: 26.11 秒
Latency standard deviation: 93.22 秒

More metrics

P95 latency: 187.93 秒
Non-live AST overall: 88.02%
Non-live simple calls: 76.08%
Non-live multiple-function selection: 95%
Non-live parallel calls: 90%
Non-live parallel multiple-function calls: 91%
Live AST overall: 76.61%
Live simple calls: 81.4%
Live multiple-function selection: 75.5%
Live parallel calls: 81.25%
Live parallel multiple-function calls: 70.83%
Multi-turn overall: 21.5%
Multi-turn base tasks: 26%
Multi-turn missing functions: 19%
Multi-turn missing parameters: 14.5%
Multi-turn long context: 26.5%
Web search overall: 10%
Search with summaries: 14%
Search without summaries: 6%
Memory overall: 8.17%
Key-value memory: 4.52%
Vector memory: 8.39%
Recursive-summary memory: 11.61%
Relevant-function detection: 100%
Irrelevant-function detection: 53.53%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Qwen3-8B (Prompt)

Model reported by the benchmark

Text prompting · Prompt

Multi-turn overall · original rank 39 名 · 综合准确率原榜第 44

As reportedOrganization: Qwen · License: apache-2.0 · Model details ↗

Overall accuracy: 40.43%
Estimated benchmark cost: 63.95 USD
Average latency: 54.17 秒
Latency standard deviation: 79.9 秒

More metrics

P95 latency: 194.15 秒
Non-live AST overall: 88.56%
Non-live simple calls: 75.25%
Non-live multiple-function selection: 95%
Non-live parallel calls: 94.5%
Non-live parallel multiple-function calls: 89.5%
Live AST overall: 80.09%
Live simple calls: 84.5%
Live multiple-function selection: 78.92%
Live parallel calls: 93.75%
Live parallel multiple-function calls: 75%
Multi-turn overall: 33.38%
Multi-turn base tasks: 41.5%
Multi-turn missing functions: 38.5%
Multi-turn missing parameters: 27%
Multi-turn long context: 26.5%
Web search overall: 13.5%
Search with summaries: 19%
Search without summaries: 8%
Memory overall: 13.12%
Key-value memory: 3.87%
Vector memory: 10.32%
Recursive-summary memory: 25.16%
Relevant-function detection: 75%
Irrelevant-function detection: 82.27%
Maximum format variation: 16.5 百分点
Format variation standard deviation: 5.09 百分点

Granite-20b-FunctionCalling (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 81 名 · 综合准确率原榜第 93

As reportedOrganization: IBM · License: Apache-2.0 · Model details ↗

Overall accuracy: 23.23%
Estimated benchmark cost: 5.23 USD
Average latency: 3.2 秒
Latency standard deviation: 3.43 秒

More metrics

P95 latency: 9.97 秒
Non-live AST overall: 82.35%
Non-live simple calls: 72.92%
Non-live multiple-function selection: 91.5%
Non-live parallel calls: 83.5%
Non-live parallel multiple-function calls: 81.5%
Live AST overall: 58.7%
Live simple calls: 67.83%
Live multiple-function selection: 56.7%
Live parallel calls: 43.75%
Live parallel multiple-function calls: 58.33%
Multi-turn overall: 5.38%
Multi-turn base tasks: 9%
Multi-turn missing functions: 3%
Multi-turn missing parameters: 6.5%
Multi-turn long context: 3%
Web search overall: 0%
Search with summaries: 0%
Search without summaries: 0%
Memory overall: 0%
Key-value memory: 0%
Vector memory: 0%
Recursive-summary memory: 0%
Relevant-function detection: 87.5%
Irrelevant-function detection: 75.13%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Claude-Sonnet-4-5-20250929 (Prompt)

Model reported by the benchmark

Text prompting · Prompt

Multi-turn overall · original rank 97 名 · 综合准确率原榜第 89

As reportedOrganization: Anthropic · License: Proprietary · Model details ↗

Overall accuracy: 24.9%
Estimated benchmark cost: 47.82 USD
Average latency: 3.84 秒
Latency standard deviation: 1.53 秒

More metrics

P95 latency: 6.66 秒
Non-live AST overall: 59.81%
Non-live simple calls: 47.25%
Non-live multiple-function selection: 79.5%
Non-live parallel calls: 53.5%
Non-live parallel multiple-function calls: 59%
Live AST overall: 46.56%
Live simple calls: 73.26%
Live multiple-function selection: 40.17%
Live parallel calls: 56.25%
Live parallel multiple-function calls: 33.33%
Multi-turn overall: 1.62%
Multi-turn base tasks: 2%
Multi-turn missing functions: 0%
Multi-turn missing parameters: 3%
Multi-turn long context: 1.5%
Web search overall: 16%
Search with summaries: 16%
Search without summaries: 16%
Memory overall: 5.38%
Key-value memory: 4.52%
Vector memory: 9.68%
Recursive-summary memory: 1.94%
Relevant-function detection: 37.5%
Irrelevant-function detection: 95.03%
Maximum format variation: 37.5 百分点
Format variation standard deviation: 10.07 百分点

GPT-4.1-2025-04-14 (Prompt)

Model reported by the benchmark

Text prompting · Prompt

Multi-turn overall · original rank 70 名 · 综合准确率原榜第 45

As reportedOrganization: OpenAI · License: Proprietary · Model details ↗

Overall accuracy: 39.38%
Estimated benchmark cost: 145.85 USD
Average latency: 1.2 秒
Latency standard deviation: 3.23 秒

More metrics

P95 latency: 2.53 秒
Non-live AST overall: 88.69%
Non-live simple calls: 78.25%
Non-live multiple-function selection: 93.5%
Non-live parallel calls: 94%
Non-live parallel multiple-function calls: 89%
Live AST overall: 78.9%
Live simple calls: 84.88%
Live multiple-function selection: 77.4%
Live parallel calls: 87.5%
Live parallel multiple-function calls: 75%
Multi-turn overall: 9.75%
Multi-turn base tasks: 10.5%
Multi-turn missing functions: 11%
Multi-turn missing parameters: 8%
Multi-turn long context: 9.5%
Web search overall: 35%
Search with summaries: 40%
Search without summaries: 30%
Memory overall: 21.51%
Key-value memory: 9.68%
Vector memory: 19.35%
Recursive-summary memory: 35.48%
Relevant-function detection: 100%
Irrelevant-function detection: 83.99%
Maximum format variation: 23.5 百分点
Format variation standard deviation: 6.18 百分点

Hammer2.1-3b (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 55 名 · 综合准确率原榜第 68

As reportedOrganization: MadeAgents · License: qwen-research · Model details ↗

Overall accuracy: 29.71%
Estimated benchmark cost: 10.89 USD
Average latency: 11.24 秒
Latency standard deviation: 15.81 秒

More metrics

P95 latency: 47.44 秒
Non-live AST overall: 84.96%
Non-live simple calls: 79.33%
Non-live multiple-function selection: 93.5%
Non-live parallel calls: 86.5%
Non-live parallel multiple-function calls: 80.5%
Live AST overall: 70.54%
Live simple calls: 68.22%
Live multiple-function selection: 71.32%
Live parallel calls: 62.5%
Live parallel multiple-function calls: 66.67%
Multi-turn overall: 16.5%
Multi-turn base tasks: 22%
Multi-turn missing functions: 12.5%
Multi-turn missing parameters: 16%
Multi-turn long context: 15.5%
Web search overall: 0%
Search with summaries: 0%
Search without summaries: 0%
Memory overall: 3.01%
Key-value memory: 2.58%
Vector memory: 3.87%
Recursive-summary memory: 2.58%
Relevant-function detection: 56.25%
Irrelevant-function detection: 86.12%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Claude-Opus-4-5-20251101 (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 4 名 · 综合准确率原榜第 1

As reportedOrganization: Anthropic · License: Proprietary · Model details ↗

Overall accuracy: 77.47%
Estimated benchmark cost: 86.55 USD
Average latency: 4.38 秒
Latency standard deviation: 3.13 秒

More metrics

P95 latency: 7.56 秒
Non-live AST overall: 88.58%
Non-live simple calls: 76.83%
Non-live multiple-function selection: 95.5%
Non-live parallel calls: 93.5%
Non-live parallel multiple-function calls: 88.5%
Live AST overall: 79.79%
Live simple calls: 86.43%
Live multiple-function selection: 78.16%
Live parallel calls: 87.5%
Live parallel multiple-function calls: 75%
Multi-turn overall: 68.38%
Multi-turn base tasks: 81%
Multi-turn missing functions: 64%
Multi-turn missing parameters: 58%
Multi-turn long context: 70.5%
Web search overall: 84.5%
Search with summaries: 84%
Search without summaries: 85%
Memory overall: 73.76%
Key-value memory: 70.97%
Vector memory: 72.9%
Recursive-summary memory: 77.42%
Relevant-function detection: 62.5%
Irrelevant-function detection: 84.72%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Qwen3-0.6B (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 86 名 · 综合准确率原榜第 92

As reportedOrganization: Qwen · License: apache-2.0 · Model details ↗

Overall accuracy: 23.93%
Estimated benchmark cost: 0.46 USD
Average latency: 0.68 秒
Latency standard deviation: 8.45 秒

More metrics

P95 latency: 0.96 秒
Non-live AST overall: 71.79%
Non-live simple calls: 64.17%
Non-live multiple-function selection: 86%
Non-live parallel calls: 67.5%
Non-live parallel multiple-function calls: 69.5%
Live AST overall: 56.62%
Live simple calls: 61.24%
Live multiple-function selection: 56.13%
Live parallel calls: 43.75%
Live parallel multiple-function calls: 37.5%
Multi-turn overall: 3.62%
Multi-turn base tasks: 5.5%
Multi-turn missing functions: 2%
Multi-turn missing parameters: 3%
Multi-turn long context: 4%
Web search overall: 1%
Search with summaries: 1%
Search without summaries: 1%
Memory overall: 8.6%
Key-value memory: 2.58%
Vector memory: 1.94%
Recursive-summary memory: 21.29%
Relevant-function detection: 75%
Irrelevant-function detection: 80.84%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

GPT-5-nano-2025-08-07 (Prompt)

Model reported by the benchmark

Text prompting · Prompt

Multi-turn overall · original rank 102 名 · 综合准确率原榜第 79

As reportedOrganization: OpenAI · License: Proprietary · Model details ↗

Overall accuracy: 27.55%
Estimated benchmark cost: 21.47 USD
Average latency: 10.67 秒
Latency standard deviation: 7.68 秒

More metrics

P95 latency: 23.28 秒
Non-live AST overall: 80.81%
Non-live simple calls: 69.25%
Non-live multiple-function selection: 86%
Non-live parallel calls: 87.5%
Non-live parallel multiple-function calls: 80.5%
Live AST overall: 70.69%
Live simple calls: 76.36%
Live multiple-function selection: 69.71%
Live parallel calls: 68.75%
Live parallel multiple-function calls: 54.17%
Multi-turn overall: 0.75%
Multi-turn base tasks: 1%
Multi-turn missing functions: 1%
Multi-turn missing parameters: 0%
Multi-turn long context: 1%
Web search overall: 13.5%
Search with summaries: 10%
Search without summaries: 17%
Memory overall: 24.52%
Key-value memory: 20.65%
Vector memory: 31.61%
Recursive-summary memory: 21.29%
Relevant-function detection: 93.75%
Irrelevant-function detection: 45.75%
Maximum format variation: 8.5 百分点
Format variation standard deviation: 2.57 百分点

Open-Mistral-Nemo-2407 (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 74 名 · 综合准确率原榜第 78

As reportedOrganization: Mistral AI · License: Proprietary · Model details ↗

Overall accuracy: 27.63%
Estimated benchmark cost: 8.12 USD
Average latency: 1.07 秒
Latency standard deviation: 11.93 秒

More metrics

P95 latency: 1.39 秒
Non-live AST overall: 82.81%
Non-live simple calls: 65.75%
Non-live multiple-function selection: 92.5%
Non-live parallel calls: 88.5%
Non-live parallel multiple-function calls: 84.5%
Live AST overall: 73.8%
Live simple calls: 78.68%
Live multiple-function selection: 72.84%
Live parallel calls: 68.75%
Live parallel multiple-function calls: 66.67%
Multi-turn overall: 7.75%
Multi-turn base tasks: 12.5%
Multi-turn missing functions: 6.5%
Multi-turn missing parameters: 7.5%
Multi-turn long context: 4.5%
Web search overall: 7%
Search with summaries: 9%
Search without summaries: 5%
Memory overall: 10.32%
Key-value memory: 8.39%
Vector memory: 9.68%
Recursive-summary memory: 12.9%
Relevant-function detection: 81.25%
Irrelevant-function detection: 61.77%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

xLAM-2-1b-fc-r (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 33 名 · 综合准确率原榜第 65

As reportedOrganization: Salesforce · License: cc-by-nc-4.0 · Model details ↗

Overall accuracy: 30.44%
Estimated benchmark cost: 2.79 USD
Average latency: 2.84 秒
Latency standard deviation: 2.35 秒

More metrics

P95 latency: 6.52 秒
Non-live AST overall: 69.04%
Non-live simple calls: 64.17%
Non-live multiple-function selection: 82.5%
Non-live parallel calls: 73.5%
Non-live parallel multiple-function calls: 56%
Live AST overall: 55.14%
Live simple calls: 68.22%
Live multiple-function selection: 52.8%
Live parallel calls: 43.75%
Live parallel multiple-function calls: 25%
Multi-turn overall: 36%
Multi-turn base tasks: 45.5%
Multi-turn missing functions: 36%
Multi-turn missing parameters: 37%
Multi-turn long context: 25.5%
Web search overall: 0%
Search with summaries: 0%
Search without summaries: 0%
Memory overall: 3.87%
Key-value memory: 3.87%
Vector memory: 3.87%
Recursive-summary memory: 3.87%
Relevant-function detection: 87.5%
Irrelevant-function detection: 64.47%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Falcon3-10B-Instruct (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 78 名 · 综合准确率原榜第 82

As reportedOrganization: TII UAE · License: falcon-llm-license · Model details ↗

Overall accuracy: 27.01%
Estimated benchmark cost: 52.59 USD
Average latency: 69.27 秒
Latency standard deviation: 92.22 秒

More metrics

P95 latency: 190.96 秒
Non-live AST overall: 85%
Non-live simple calls: 70.5%
Non-live multiple-function selection: 93.5%
Non-live parallel calls: 88.5%
Non-live parallel multiple-function calls: 87.5%
Live AST overall: 75.43%
Live simple calls: 77.13%
Live multiple-function selection: 76.16%
Live parallel calls: 50%
Live parallel multiple-function calls: 41.67%
Multi-turn overall: 6.5%
Multi-turn base tasks: 6.5%
Multi-turn missing functions: 9.5%
Multi-turn missing parameters: 5%
Multi-turn long context: 5%
Web search overall: 1.5%
Search with summaries: 2%
Search without summaries: 1%
Memory overall: 27.53%
Key-value memory: 12.26%
Vector memory: 19.35%
Recursive-summary memory: 50.97%
Relevant-function detection: 93.75%
Irrelevant-function detection: 32.09%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

o4-mini-2025-04-16 (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 27 名 · 综合准确率原榜第 21

As reportedOrganization: OpenAI · License: Proprietary · Model details ↗

Overall accuracy: 53.24%
Estimated benchmark cost: 81.91 USD
Average latency: 3.71 秒
Latency standard deviation: 7.18 秒

More metrics

P95 latency: 9.33 秒
Non-live AST overall: 37.73%
Non-live simple calls: 66.92%
Non-live multiple-function selection: 84%
Non-live parallel calls: 0%
Non-live parallel multiple-function calls: 0%
Live AST overall: 66.1%
Live simple calls: 69.38%
Live multiple-function selection: 67.81%
Live parallel calls: 0%
Live parallel multiple-function calls: 0%
Multi-turn overall: 41.75%
Multi-turn base tasks: 51%
Multi-turn missing functions: 30%
Multi-turn missing parameters: 40.5%
Multi-turn long context: 45.5%
Web search overall: 75.5%
Search with summaries: 75%
Search without summaries: 76%
Memory overall: 34.19%
Key-value memory: 19.35%
Vector memory: 24.52%
Recursive-summary memory: 58.71%
Relevant-function detection: 81.25%
Irrelevant-function detection: 83.91%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

GPT-5-nano-2025-08-07 (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 36 名 · 综合准确率原榜第 24

As reportedOrganization: OpenAI · License: Proprietary · Model details ↗

Overall accuracy: 51.45%
Estimated benchmark cost: 8.79 USD
Average latency: 10.36 秒
Latency standard deviation: 10.37 秒

More metrics

P95 latency: 23.56 秒
Non-live AST overall: 68%
Non-live simple calls: 57%
Non-live multiple-function selection: 64.5%
Non-live parallel calls: 79%
Non-live parallel multiple-function calls: 71.5%
Live AST overall: 59.44%
Live simple calls: 58.91%
Live multiple-function selection: 59.83%
Live parallel calls: 50%
Live parallel multiple-function calls: 54.17%
Multi-turn overall: 34.5%
Multi-turn base tasks: 44%
Multi-turn missing functions: 23.5%
Multi-turn missing parameters: 32.5%
Multi-turn long context: 38%
Web search overall: 72.5%
Search with summaries: 74%
Search without summaries: 71%
Memory overall: 24.73%
Key-value memory: 18.06%
Vector memory: 27.1%
Recursive-summary memory: 29.03%
Relevant-function detection: 75%
Irrelevant-function detection: 89.1%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Grok-4-1-fast-non-reasoning (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 20 名 · 综合准确率原榜第 12

As reportedOrganization: xAI · License: Proprietary · Model details ↗

Overall accuracy: 58.29%
Estimated benchmark cost: 16.27 USD
Average latency: 2.29 秒
Latency standard deviation: 7.31 秒

More metrics

P95 latency: 5.34 秒
Non-live AST overall: 88.13%
Non-live simple calls: 76%
Non-live multiple-function selection: 93%
Non-live parallel calls: 93%
Non-live parallel multiple-function calls: 90.5%
Live AST overall: 77.94%
Live simple calls: 82.95%
Live multiple-function selection: 76.92%
Live parallel calls: 75%
Live parallel multiple-function calls: 70.83%
Multi-turn overall: 46.75%
Multi-turn base tasks: 58%
Multi-turn missing functions: 39.5%
Multi-turn missing parameters: 37.5%
Multi-turn long context: 52%
Web search overall: 75%
Search with summaries: 74%
Search without summaries: 76%
Memory overall: 26.24%
Key-value memory: 20.65%
Vector memory: 20%
Recursive-summary memory: 38.06%
Relevant-function detection: 81.25%
Irrelevant-function detection: 74.09%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Arch-Agent-1.5B

Model reported by the benchmark

Multi-turn overall · original rank 44 名 · 综合准确率原榜第 60

As reportedOrganization: katanemo · License: katanemo-research · Model details ↗

Overall accuracy: 32.14%
Estimated benchmark cost: 2.45 USD
Average latency: 2.38 秒
Latency standard deviation: 4.01 秒

More metrics

P95 latency: 5.3 秒
Non-live AST overall: 82.67%
Non-live simple calls: 72.17%
Non-live multiple-function selection: 92%
Non-live parallel calls: 85.5%
Non-live parallel multiple-function calls: 81%
Live AST overall: 67.73%
Live simple calls: 70.54%
Live multiple-function selection: 67.81%
Live parallel calls: 31.25%
Live parallel multiple-function calls: 58.33%
Multi-turn overall: 26.62%
Multi-turn base tasks: 35.5%
Multi-turn missing functions: 27.5%
Multi-turn missing parameters: 21.5%
Multi-turn long context: 22%
Web search overall: 0%
Search with summaries: 0%
Search without summaries: 0%
Memory overall: 8.17%
Key-value memory: 5.81%
Vector memory: 5.81%
Recursive-summary memory: 12.9%
Relevant-function detection: 75%
Irrelevant-function detection: 74.83%
Maximum format variation: Not provided
Format variation standard deviation: Not provided