Project benchmark

BFCL V4 function calling, search & memory

BFCL · 伯克利工具调用 · 109 configurations with published scores. Review each metric and its test conditions separately.

Report identity & conditions

Data as of2026-10-19· Report published—. See individual records for run dates and test conditions.

Leaderboard updated2026-04-12

Overall and task scores retain the official results. This report covers function calling, multi-turn interactions, web search and memory, rather than all general capabilities.
FC means native function/tool calling; Prompt uses text generation to express calls. Thinking configurations remain separate. Unstated calling modes are not inferred.
Estimated benchmark costs use USD, latency uses seconds and format sensitivity uses percentage points. They are compared separately and are not API price quotes.
The leaderboard update date is not each configuration’s test date. Organizations, model links and licenses follow the original board; license scope depends on the specific model terms.
Results are cited from the official public BFCL leaderboard. Test questions, answers, model outputs and benchmark implementations are not reproduced.

Left metricRight metric

Compare metrics within the same report. Each chart explains its direction and scope.
See the details for full values and run conditions.

Overall accuracy

The published overall accuracy is retained rather than recalculated from subtasks. Comparisons stay within the same BFCL V4 configuration scope.: showing 8 of 109 records with values. See details for configurations and dates.

Chart data · 8 entries

Name	%
Claude-Opus-4-5-20251101 (FC)	77.47
Claude-Sonnet-4-5-20250929 (FC)	73.24
Gemini-3-Pro-Preview (Prompt)	72.51
GLM-4.6 (FC thinking)	72.38
Grok-4-1-fast-reasoning (FC)	69.57
Claude-Haiku-4-5-20251001 (FC)	68.7
Gemini-3-Pro-Preview (FC)	68.14
o3-2025-04-16 (Prompt)	63.05

Estimated benchmark cost

Estimated total benchmark cost in USD. This is neither cost per task, cost per successful task nor a current API unit price.: showing 8 of 109 records with values. See details for configurations and dates.

Chart data · 8 entries

Name	USD
Qwen3-0.6B (FC)	0.46
Llama-3.1-Nemotron-Ultra-253B-v1 (FC)	0.72
Qwen3-14B (Prompt)	1.35
Granite-4.0-350m (FC)	1.44
Command R7B (FC)	1.5
Qwen3-30B-A3B-Instruct-2507 (Prompt)	1.56
Llama-3.2-1B-Instruct (FC)	1.64
Falcon3-1B-Instruct (FC)	1.72

Full results & model versions

109 / 109configurations have published scores in this scope. Scored configurations are listed; missing values remain “Not provided.” Different run configurations of the same model are listed separately.

Nanbeige4-3B-Thinking-2511 (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 15 名 · 综合准确率原榜第 25

As reportedOrganization: Nanbeige · License: apache-2.0 · Model details ↗

Overall accuracy: 51.4%
Estimated benchmark cost: 14.14 USD
Average latency: 13.46 秒
Latency standard deviation: 26.41 秒

More metrics

P95 latency: 37.45 秒
Non-live AST overall: 81.58%
Non-live simple calls: 63.83%
Non-live multiple-function selection: 93.5%
Non-live parallel calls: 84.5%
Non-live parallel multiple-function calls: 84.5%
Live AST overall: 79.42%
Live simple calls: 86.05%
Live multiple-function selection: 78.06%
Live parallel calls: 75%
Live parallel multiple-function calls: 70.83%
Multi-turn overall: 51.12%
Multi-turn base tasks: 58.5%
Multi-turn missing functions: 54%
Multi-turn missing parameters: 45%
Multi-turn long context: 47%
Web search overall: 21.5%
Search with summaries: 31%
Search without summaries: 12%
Memory overall: 36.77%
Key-value memory: 31.61%
Vector memory: 34.19%
Recursive-summary memory: 44.52%
Relevant-function detection: 75%
Irrelevant-function detection: 83.09%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Mistral-Medium-2505

Model reported by the benchmark

Multi-turn overall · original rank 69 名 · 综合准确率原榜第 48

As reportedOrganization: Mistral AI · License: Proprietary · Model details ↗

Overall accuracy: 37.69%
Estimated benchmark cost: 36.51 USD
Average latency: 1.21 秒
Latency standard deviation: 3.5 秒

More metrics

P95 latency: 2.86 秒
Non-live AST overall: 85.33%
Non-live simple calls: 76.33%
Non-live multiple-function selection: 91%
Non-live parallel calls: 88.5%
Non-live parallel multiple-function calls: 85.5%
Live AST overall: 66.03%
Live simple calls: 80.23%
Live multiple-function selection: 62.39%
Live parallel calls: 81.25%
Live parallel multiple-function calls: 62.5%
Multi-turn overall: 9.88%
Multi-turn base tasks: 13.5%
Multi-turn missing functions: 6.5%
Multi-turn missing parameters: 6%
Multi-turn long context: 13.5%
Web search overall: 39%
Search with summaries: 41%
Search without summaries: 37%
Memory overall: 21.72%
Key-value memory: 16.13%
Vector memory: 14.84%
Recursive-summary memory: 34.19%
Relevant-function detection: 75%
Irrelevant-function detection: 74.49%
Maximum format variation: 21.5 百分点
Format variation standard deviation: 5.02 百分点

Hammer2.1-7b (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 46 名 · 综合准确率原榜第 64

As reportedOrganization: MadeAgents · License: cc-by-nc-4.0 · Model details ↗

Overall accuracy: 31.67%
Estimated benchmark cost: 4.99 USD
Average latency: 5.77 秒
Latency standard deviation: 10.29 秒

More metrics

P95 latency: 31.26 秒
Non-live AST overall: 85.5%
Non-live simple calls: 72.5%
Non-live multiple-function selection: 92.5%
Non-live parallel calls: 91%
Non-live parallel multiple-function calls: 86%
Live AST overall: 69.5%
Live simple calls: 66.67%
Live multiple-function selection: 69.99%
Live parallel calls: 75%
Live parallel multiple-function calls: 75%
Multi-turn overall: 23.87%
Multi-turn base tasks: 24%
Multi-turn missing functions: 28.5%
Multi-turn missing parameters: 21.5%
Multi-turn long context: 21.5%
Web search overall: 0%
Search with summaries: 0%
Search without summaries: 0%
Memory overall: 0%
Key-value memory: 0%
Vector memory: 0%
Recursive-summary memory: 0%
Relevant-function detection: 50%
Irrelevant-function detection: 90.12%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

DeepSeek-V3.2-Exp (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 31 名 · 综合准确率原榜第 19

As reportedOrganization: DeepSeek · License: MIT · Model details ↗

Overall accuracy: 54.12%
Estimated benchmark cost: 6.71 USD
Average latency: 5.83 秒
Latency standard deviation: 11.71 秒

More metrics

P95 latency: 10.59 秒
Non-live AST overall: 34.85%
Non-live simple calls: 37.92%
Non-live multiple-function selection: 74%
Non-live parallel calls: 15%
Non-live parallel multiple-function calls: 12.5%
Live AST overall: 53.66%
Live simple calls: 66.28%
Live multiple-function selection: 51.66%
Live parallel calls: 25%
Live parallel multiple-function calls: 25%
Multi-turn overall: 37.38%
Multi-turn base tasks: 41.5%
Multi-turn missing functions: 39.5%
Multi-turn missing parameters: 33.5%
Multi-turn long context: 35%
Web search overall: 69.5%
Search with summaries: 80%
Search without summaries: 59%
Memory overall: 54.19%
Key-value memory: 41.94%
Vector memory: 61.29%
Recursive-summary memory: 59.35%
Relevant-function detection: 37.5%
Irrelevant-function detection: 93.18%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Claude-Sonnet-4-5-20250929 (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 9 名 · 综合准确率原榜第 2

As reportedOrganization: Anthropic · License: Proprietary · Model details ↗

Overall accuracy: 73.24%
Estimated benchmark cost: 43.73 USD
Average latency: 4.31 秒
Latency standard deviation: 4.43 秒

More metrics

P95 latency: 7.27 秒
Non-live AST overall: 88.65%
Non-live simple calls: 72.58%
Non-live multiple-function selection: 95.5%
Non-live parallel calls: 94.5%
Non-live parallel multiple-function calls: 92%
Live AST overall: 81.13%
Live simple calls: 89.53%
Live multiple-function selection: 78.92%
Live parallel calls: 87.5%
Live parallel multiple-function calls: 83.33%
Multi-turn overall: 61.37%
Multi-turn base tasks: 69%
Multi-turn missing functions: 65%
Multi-turn missing parameters: 52.5%
Multi-turn long context: 59%
Web search overall: 81%
Search with summaries: 82%
Search without summaries: 80%
Memory overall: 64.95%
Key-value memory: 54.19%
Vector memory: 57.42%
Recursive-summary memory: 83.23%
Relevant-function detection: 68.75%
Irrelevant-function detection: 86.61%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Command A Reasoning (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 17 名 · 综合准确率原榜第 13

As reportedOrganization: Cohere · License: CC-BY-NC 4.0 License (w/ Acceptable Use Addendum) · Model details ↗

Overall accuracy: 57.06%
Estimated benchmark cost: 3.04 USD
Average latency: 3.44 秒
Latency standard deviation: 4.91 秒

More metrics

P95 latency: 8.39 秒
Non-live AST overall: 86.27%
Non-live simple calls: 73.58%
Non-live multiple-function selection: 93.5%
Non-live parallel calls: 89.5%
Non-live parallel multiple-function calls: 88.5%
Live AST overall: 78.61%
Live simple calls: 80.23%
Live multiple-function selection: 78.35%
Live parallel calls: 75%
Live parallel multiple-function calls: 75%
Multi-turn overall: 50.12%
Multi-turn base tasks: 61.5%
Multi-turn missing functions: 41%
Multi-turn missing parameters: 49.5%
Multi-turn long context: 48.5%
Web search overall: 55.5%
Search with summaries: 65%
Search without summaries: 46%
Memory overall: 28.82%
Key-value memory: 16.13%
Vector memory: 23.87%
Recursive-summary memory: 46.45%
Relevant-function detection: 68.75%
Irrelevant-function detection: 86.75%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Phi-4 (Prompt)

Model reported by the benchmark

Text prompting · Prompt

Multi-turn overall · original rank 85 名 · 综合准确率原榜第 70

As reportedOrganization: Microsoft · License: MIT · Model details ↗

Overall accuracy: 28.79%
Estimated benchmark cost: 8.72 USD
Average latency: 9.49 秒
Latency standard deviation: 26.73 秒

More metrics

P95 latency: 23.02 秒
Non-live AST overall: 69.56%
Non-live simple calls: 74.25%
Non-live multiple-function selection: 89.5%
Non-live parallel calls: 65%
Non-live parallel multiple-function calls: 49.5%
Live AST overall: 60.7%
Live simple calls: 65.5%
Live multiple-function selection: 59.64%
Live parallel calls: 81.25%
Live parallel multiple-function calls: 41.67%
Multi-turn overall: 3.88%
Multi-turn base tasks: 9%
Multi-turn missing functions: 0%
Multi-turn missing parameters: 3.5%
Multi-turn long context: 3%
Web search overall: 4.5%
Search with summaries: 4%
Search without summaries: 5%
Memory overall: 24.73%
Key-value memory: 17.42%
Vector memory: 25.16%
Recursive-summary memory: 31.61%
Relevant-function detection: 50%
Irrelevant-function detection: 87.55%
Maximum format variation: 81.5 百分点
Format variation standard deviation: 23.34 百分点

Claude-Opus-4-5-20251101 (Prompt)

Model reported by the benchmark

Text prompting · Prompt

Multi-turn overall · original rank 56 名 · 综合准确率原榜第 57

As reportedOrganization: Anthropic · License: Proprietary · Model details ↗

Overall accuracy: 33.47%
Estimated benchmark cost: 88.33 USD
Average latency: 3.76 秒
Latency standard deviation: 13.19 秒

More metrics

P95 latency: 5.52 秒
Non-live AST overall: 89.65%
Non-live simple calls: 79.58%
Non-live multiple-function selection: 93.5%
Non-live parallel calls: 93%
Non-live parallel multiple-function calls: 92.5%
Live AST overall: 76.02%
Live simple calls: 84.5%
Live multiple-function selection: 74.17%
Live parallel calls: 81.25%
Live parallel multiple-function calls: 62.5%
Multi-turn overall: 16.12%
Multi-turn base tasks: 20.5%
Multi-turn missing functions: 9%
Multi-turn missing parameters: 21.5%
Multi-turn long context: 13.5%
Web search overall: 13%
Search with summaries: 13%
Search without summaries: 13%
Memory overall: 1.94%
Key-value memory: 1.29%
Vector memory: 1.94%
Recursive-summary memory: 2.58%
Relevant-function detection: 68.75%
Irrelevant-function detection: 90.75%
Maximum format variation: 13 百分点
Format variation standard deviation: 3.65 百分点

Gemini-2.5-Flash (Prompt)

Model reported by the benchmark

Text prompting · Prompt

Multi-turn overall · original rank 53 名 · 综合准确率原榜第 26

As reportedOrganization: Google · License: Proprietary · Model details ↗

Overall accuracy: 50.9%
Estimated benchmark cost: 33.45 USD
Average latency: 3.18 秒
Latency standard deviation: 4.44 秒

More metrics

P95 latency: 6.09 秒
Non-live AST overall: 88.08%
Non-live simple calls: 77.33%
Non-live multiple-function selection: 91.5%
Non-live parallel calls: 96%
Non-live parallel multiple-function calls: 87.5%
Live AST overall: 78.16%
Live simple calls: 87.21%
Live multiple-function selection: 75.97%
Live parallel calls: 81.25%
Live parallel multiple-function calls: 75%
Multi-turn overall: 16.75%
Multi-turn base tasks: 14.5%
Multi-turn missing functions: 16.5%
Multi-turn missing parameters: 17.5%
Multi-turn long context: 18.5%
Web search overall: 62%
Search with summaries: 60%
Search without summaries: 64%
Memory overall: 38.71%
Key-value memory: 13.55%
Vector memory: 47.1%
Recursive-summary memory: 55.48%
Relevant-function detection: 62.5%
Irrelevant-function detection: 91.09%
Maximum format variation: 9 百分点
Format variation standard deviation: 2.45 百分点

Qwen3-235B-A22B-Instruct-2507 (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 21 名 · 综合准确率原榜第 31

As reportedOrganization: Qwen · License: apache-2.0 · Model details ↗

Overall accuracy: 47.99%
Estimated benchmark cost: 2.5 USD
Average latency: 2.57 秒
Latency standard deviation: 2.44 秒

More metrics

P95 latency: 6.27 秒
Non-live AST overall: 37.4%
Non-live simple calls: 40.58%
Non-live multiple-function selection: 36.5%
Non-live parallel calls: 53%
Non-live parallel multiple-function calls: 19.5%
Live AST overall: 68.91%
Live simple calls: 58.53%
Live multiple-function selection: 71.6%
Live parallel calls: 68.75%
Live parallel multiple-function calls: 62.5%
Multi-turn overall: 45.38%
Multi-turn base tasks: 57.5%
Multi-turn missing functions: 35%
Multi-turn missing parameters: 33.5%
Multi-turn long context: 55.5%
Web search overall: 54%
Search with summaries: 57%
Search without summaries: 51%
Memory overall: 23.87%
Key-value memory: 7.1%
Vector memory: 18.71%
Recursive-summary memory: 45.81%
Relevant-function detection: 87.5%
Irrelevant-function detection: 81.73%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

GPT-5.2-2025-12-11 (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 42 名 · 综合准确率原榜第 16

As reportedOrganization: OpenAI · License: Proprietary · Model details ↗

Overall accuracy: 55.87%
Estimated benchmark cost: 85.65 USD
Average latency: 2.23 秒
Latency standard deviation: 9.75 秒

More metrics

P95 latency: 5.26 秒
Non-live AST overall: 81.85%
Non-live simple calls: 72.92%
Non-live multiple-function selection: 88%
Non-live parallel calls: 89%
Non-live parallel multiple-function calls: 77.5%
Live AST overall: 70.39%
Live simple calls: 71.71%
Live multiple-function selection: 70.37%
Live parallel calls: 68.75%
Live parallel multiple-function calls: 58.33%
Multi-turn overall: 28.12%
Multi-turn base tasks: 36.5%
Multi-turn missing functions: 18%
Multi-turn missing parameters: 27.5%
Multi-turn long context: 30.5%
Web search overall: 75.5%
Search with summaries: 78%
Search without summaries: 73%
Memory overall: 45.81%
Key-value memory: 33.55%
Vector memory: 43.23%
Recursive-summary memory: 60.65%
Relevant-function detection: 75%
Irrelevant-function detection: 79.42%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Llama-3.1-Nemotron-Ultra-253B-v1 (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 108 名 · 综合准确率原榜第 108

As reportedOrganization: NVIDIA · License: nvidia-open-model-license · Model details ↗

Overall accuracy: 10%
Estimated benchmark cost: 0.72 USD
Average latency: 1.42 秒
Latency standard deviation: 1.84 秒

More metrics

P95 latency: 2.4 秒
Non-live AST overall: 0%
Non-live simple calls: 0%
Non-live multiple-function selection: 0%
Non-live parallel calls: 0%
Non-live parallel multiple-function calls: 0%
Live AST overall: 0%
Live simple calls: 0%
Live multiple-function selection: 0%
Live parallel calls: 0%
Live parallel multiple-function calls: 0%
Multi-turn overall: 0%
Multi-turn base tasks: 0%
Multi-turn missing functions: 0%
Multi-turn missing parameters: 0%
Multi-turn long context: 0%
Web search overall: 0%
Search with summaries: 0%
Search without summaries: 0%
Memory overall: 0%
Key-value memory: 0%
Vector memory: 0%
Recursive-summary memory: 0%
Relevant-function detection: 0%
Irrelevant-function detection: 100%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Granite-3.1-8B-Instruct (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 76 名 · 综合准确率原榜第 81

As reportedOrganization: IBM · License: Apache-2.0 · Model details ↗

Overall accuracy: 27.1%
Estimated benchmark cost: 9.32 USD
Average latency: 13.23 秒
Latency standard deviation: 31.28 秒

More metrics

P95 latency: 65.19 秒
Non-live AST overall: 78.33%
Non-live simple calls: 67.33%
Non-live multiple-function selection: 92%
Non-live parallel calls: 84%
Non-live parallel multiple-function calls: 70%
Live AST overall: 60.33%
Live simple calls: 58.53%
Live multiple-function selection: 61.82%
Live parallel calls: 18.75%
Live parallel multiple-function calls: 41.67%
Multi-turn overall: 7.5%
Multi-turn base tasks: 11.5%
Multi-turn missing functions: 2%
Multi-turn missing parameters: 7.5%
Multi-turn long context: 9%
Web search overall: 0.5%
Search with summaries: 1%
Search without summaries: 0%
Memory overall: 14.41%
Key-value memory: 9.68%
Vector memory: 7.1%
Recursive-summary memory: 26.45%
Relevant-function detection: 68.75%
Irrelevant-function detection: 79.98%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

ToolACE-2-8B (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 30 名 · 综合准确率原榜第 40

As reportedOrganization: Huawei Noah & USTC · License: Apache-2.0 · Model details ↗

Overall accuracy: 42.44%
Estimated benchmark cost: 24.43 USD
Average latency: 15.95 秒
Latency standard deviation: 40.06 秒

More metrics

P95 latency: 65.26 秒
Non-live AST overall: 87.1%
Non-live simple calls: 73.42%
Non-live multiple-function selection: 91%
Non-live parallel calls: 93%
Non-live parallel multiple-function calls: 91%
Live AST overall: 77.42%
Live simple calls: 71.32%
Live multiple-function selection: 79.39%
Live parallel calls: 68.75%
Live parallel multiple-function calls: 62.5%
Multi-turn overall: 38.38%
Multi-turn base tasks: 49%
Multi-turn missing functions: 28%
Multi-turn missing parameters: 30.5%
Multi-turn long context: 46%
Web search overall: 8.5%
Search with summaries: 13%
Search without summaries: 4%
Memory overall: 18.49%
Key-value memory: 5.81%
Vector memory: 16.13%
Recursive-summary memory: 33.55%
Relevant-function detection: 75%
Irrelevant-function detection: 90.79%
Maximum format variation: 81.5 百分点
Format variation standard deviation: 27.92 百分点

Falcon3-3B-Instruct (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 100 名 · 综合准确率原榜第 104

As reportedOrganization: TII UAE · License: falcon-llm-license · Model details ↗

Overall accuracy: 16.25%
Estimated benchmark cost: 36.7 USD
Average latency: 38.52 秒
Latency standard deviation: 107.47 秒

More metrics

P95 latency: 103.62 秒
Non-live AST overall: 54.62%
Non-live simple calls: 56.5%
Non-live multiple-function selection: 69.5%
Non-live parallel calls: 67%
Non-live parallel multiple-function calls: 25.5%
Live AST overall: 54.48%
Live simple calls: 57.36%
Live multiple-function selection: 54.7%
Live parallel calls: 25%
Live parallel multiple-function calls: 33.33%
Multi-turn overall: 1%
Multi-turn base tasks: 1.5%
Multi-turn missing functions: 0.5%
Multi-turn missing parameters: 0.5%
Multi-turn long context: 1.5%
Web search overall: 1%
Search with summaries: 1%
Search without summaries: 1%
Memory overall: 7.74%
Key-value memory: 6.45%
Vector memory: 8.39%
Recursive-summary memory: 8.39%
Relevant-function detection: 81.25%
Irrelevant-function detection: 32.92%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

o3-2025-04-16 (Prompt)

Model reported by the benchmark

Text prompting · Prompt

Multi-turn overall · original rank 8 名 · 综合准确率原榜第 8

As reportedOrganization: OpenAI · License: Proprietary · Model details ↗

Overall accuracy: 63.05%
Estimated benchmark cost: 234.64 USD
Average latency: 4.83 秒
Latency standard deviation: 7.01 秒

More metrics

P95 latency: 11.7 秒
Non-live AST overall: 81.94%
Non-live simple calls: 74.25%
Non-live multiple-function selection: 89%
Non-live parallel calls: 86.5%
Non-live parallel multiple-function calls: 78%
Live AST overall: 73.21%
Live simple calls: 83.33%
Live multiple-function selection: 70.75%
Live parallel calls: 75%
Live parallel multiple-function calls: 70.83%
Multi-turn overall: 62.25%
Multi-turn base tasks: 68%
Multi-turn missing functions: 63.5%
Multi-turn missing parameters: 54.5%
Multi-turn long context: 63%
Web search overall: 50.5%
Search with summaries: 51%
Search without summaries: 50%
Memory overall: 51.83%
Key-value memory: 33.55%
Vector memory: 50.32%
Recursive-summary memory: 71.61%
Relevant-function detection: 93.75%
Irrelevant-function detection: 83.98%
Maximum format variation: 8.5 百分点
Format variation standard deviation: 2.75 百分点

Qwen3-14B (Prompt)

Model reported by the benchmark

Text prompting · Prompt

Multi-turn overall · original rank 45 名 · 综合准确率原榜第 47

As reportedOrganization: Qwen · License: apache-2.0 · Model details ↗

Overall accuracy: 37.77%
Estimated benchmark cost: 1.35 USD
Average latency: 1.2 秒
Latency standard deviation: 8.5 秒

More metrics

P95 latency: 2.3 秒
Non-live AST overall: 89.46%
Non-live simple calls: 76.83%
Non-live multiple-function selection: 93.5%
Non-live parallel calls: 95.5%
Non-live parallel multiple-function calls: 92%
Live AST overall: 79.35%
Live simple calls: 84.5%
Live multiple-function selection: 78.06%
Live parallel calls: 87.5%
Live parallel multiple-function calls: 75%
Multi-turn overall: 26.13%
Multi-turn base tasks: 16.5%
Multi-turn missing functions: 37.5%
Multi-turn missing parameters: 31%
Multi-turn long context: 19.5%
Web search overall: 10.5%
Search with summaries: 6%
Search without summaries: 15%
Memory overall: 11.18%
Key-value memory: 4.52%
Vector memory: 6.45%
Recursive-summary memory: 22.58%
Relevant-function detection: 81.25%
Irrelevant-function detection: 87.18%
Maximum format variation: 14 百分点
Format variation standard deviation: 3.97 百分点

GPT-4.1-nano-2025-04-14 (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 47 名 · 综合准确率原榜第 58

As reportedOrganization: OpenAI · License: Proprietary · Model details ↗

Overall accuracy: 33.05%
Estimated benchmark cost: 5.66 USD
Average latency: 1.44 秒
Latency standard deviation: 10.84 秒

More metrics

P95 latency: 2.26 秒
Non-live AST overall: 72.98%
Non-live simple calls: 59.92%
Non-live multiple-function selection: 79.5%
Non-live parallel calls: 84%
Non-live parallel multiple-function calls: 68.5%
Live AST overall: 60.77%
Live simple calls: 58.14%
Live multiple-function selection: 61.44%
Live parallel calls: 68.75%
Live parallel multiple-function calls: 54.17%
Multi-turn overall: 23.62%
Multi-turn base tasks: 39.5%
Multi-turn missing functions: 7.5%
Multi-turn missing parameters: 17.5%
Multi-turn long context: 30%
Web search overall: 11%
Search with summaries: 13%
Search without summaries: 9%
Memory overall: 18.92%
Key-value memory: 10.32%
Vector memory: 19.35%
Recursive-summary memory: 27.1%
Relevant-function detection: 93.75%
Irrelevant-function detection: 66%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Command R7B (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 72 名 · 综合准确率原榜第 61

As reportedOrganization: Cohere · License: cc-by-nc-4.0 · Model details ↗

Overall accuracy: 32.07%
Estimated benchmark cost: 1.5 USD
Average latency: 1.38 秒
Latency standard deviation: 2.87 秒

More metrics

P95 latency: 2.69 秒
Non-live AST overall: 80.96%
Non-live simple calls: 67.33%
Non-live multiple-function selection: 89.5%
Non-live parallel calls: 85.5%
Non-live parallel multiple-function calls: 81.5%
Live AST overall: 69.06%
Live simple calls: 62.79%
Live multiple-function selection: 70.94%
Live parallel calls: 43.75%
Live parallel multiple-function calls: 70.83%
Multi-turn overall: 8.25%
Multi-turn base tasks: 12%
Multi-turn missing functions: 0.5%
Multi-turn missing parameters: 10.5%
Multi-turn long context: 10%
Web search overall: 27%
Search with summaries: 43%
Search without summaries: 11%
Memory overall: 5.16%
Key-value memory: 2.58%
Vector memory: 9.68%
Recursive-summary memory: 3.23%
Relevant-function detection: 68.75%
Irrelevant-function detection: 81.65%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Grok-4-1-fast-reasoning (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 11 名 · 综合准确率原榜第 5

As reportedOrganization: xAI · License: Proprietary · Model details ↗

Overall accuracy: 69.57%
Estimated benchmark cost: 17.26 USD
Average latency: 6.74 秒
Latency standard deviation: 12.78 秒

More metrics

P95 latency: 17.57 秒
Non-live AST overall: 88.27%
Non-live simple calls: 77.58%
Non-live multiple-function selection: 93%
Non-live parallel calls: 92.5%
Non-live parallel multiple-function calls: 90%
Live AST overall: 78.46%
Live simple calls: 84.11%
Live multiple-function selection: 77.3%
Live parallel calls: 75%
Live parallel multiple-function calls: 70.83%
Multi-turn overall: 58.87%
Multi-turn base tasks: 70.5%
Multi-turn missing functions: 59.5%
Multi-turn missing parameters: 43%
Multi-turn long context: 62.5%
Web search overall: 82.5%
Search with summaries: 82%
Search without summaries: 83%
Memory overall: 53.98%
Key-value memory: 41.29%
Vector memory: 57.42%
Recursive-summary memory: 63.23%
Relevant-function detection: 81.25%
Irrelevant-function detection: 79.43%
Maximum format variation: Not provided
Format variation standard deviation: Not provided