Project benchmark

BFCL V4 function calling, search & memory

BFCL · 伯克利工具调用 · 109 configurations with published scores. Review each metric and its test conditions separately.

Report identity & conditions

Data as of2026-10-19· Report published—. See individual records for run dates and test conditions.

Leaderboard updated2026-04-12

Overall and task scores retain the official results. This report covers function calling, multi-turn interactions, web search and memory, rather than all general capabilities.
FC means native function/tool calling; Prompt uses text generation to express calls. Thinking configurations remain separate. Unstated calling modes are not inferred.
Estimated benchmark costs use USD, latency uses seconds and format sensitivity uses percentage points. They are compared separately and are not API price quotes.
The leaderboard update date is not each configuration’s test date. Organizations, model links and licenses follow the original board; license scope depends on the specific model terms.
Results are cited from the official public BFCL leaderboard. Test questions, answers, model outputs and benchmark implementations are not reproduced.

Left metricRight metric

Compare metrics within the same report. Each chart explains its direction and scope.
See the details for full values and run conditions.

Overall accuracy

The published overall accuracy is retained rather than recalculated from subtasks. Comparisons stay within the same BFCL V4 configuration scope.: showing 8 of 109 records with values. See details for configurations and dates.

Chart data · 8 entries

Name	%
Claude-Opus-4-5-20251101 (FC)	77.47
Claude-Sonnet-4-5-20250929 (FC)	73.24
Gemini-3-Pro-Preview (Prompt)	72.51
GLM-4.6 (FC thinking)	72.38
Grok-4-1-fast-reasoning (FC)	69.57
Claude-Haiku-4-5-20251001 (FC)	68.7
Gemini-3-Pro-Preview (FC)	68.14
o3-2025-04-16 (Prompt)	63.05

Estimated benchmark cost

Estimated total benchmark cost in USD. This is neither cost per task, cost per successful task nor a current API unit price.: showing 8 of 109 records with values. See details for configurations and dates.

Chart data · 8 entries

Name	USD
Qwen3-0.6B (FC)	0.46
Llama-3.1-Nemotron-Ultra-253B-v1 (FC)	0.72
Qwen3-14B (Prompt)	1.35
Granite-4.0-350m (FC)	1.44
Command R7B (FC)	1.5
Qwen3-30B-A3B-Instruct-2507 (Prompt)	1.56
Llama-3.2-1B-Instruct (FC)	1.64
Falcon3-1B-Instruct (FC)	1.72

Full results & model versions

109 / 109configurations have published scores in this scope. Scored configurations are listed; missing values remain “Not provided.” Different run configurations of the same model are listed separately.

DeepSeek-V3.2-Exp (Prompt + Thinking)

Model reported by the benchmark

Text prompting · Prompt · 思考模式

Multi-turn overall · original rank 22 名 · 综合准确率原榜第 14

As reportedOrganization: DeepSeek · License: MIT · Model details ↗

Overall accuracy: 56.73%
Estimated benchmark cost: 57.75 USD
Average latency: 37.89 秒
Latency standard deviation: 49.56 秒

More metrics

P95 latency: 102.09 秒
Non-live AST overall: 85.52%
Non-live simple calls: 74.08%
Non-live multiple-function selection: 92%
Non-live parallel calls: 89.5%
Non-live parallel multiple-function calls: 86.5%
Live AST overall: 76.02%
Live simple calls: 82.56%
Live multiple-function selection: 74.74%
Live parallel calls: 87.5%
Live parallel multiple-function calls: 54.17%
Multi-turn overall: 44.88%
Multi-turn base tasks: 55%
Multi-turn missing functions: 49%
Multi-turn missing parameters: 27%
Multi-turn long context: 48.5%
Web search overall: 58%
Search with summaries: 64%
Search without summaries: 52%
Memory overall: 44.09%
Key-value memory: 46.45%
Vector memory: 46.45%
Recursive-summary memory: 39.35%
Relevant-function detection: 93.75%
Irrelevant-function detection: 67%
Maximum format variation: 10 百分点
Format variation standard deviation: 2.77 百分点

Qwen3-4B-Instruct-2507 (Prompt)

Model reported by the benchmark

Text prompting · Prompt

Multi-turn overall · original rank 51 名 · 综合准确率原榜第 55

As reportedOrganization: Qwen · License: apache-2.0 · Model details ↗

Overall accuracy: 35.52%
Estimated benchmark cost: 53.66 USD
Average latency: 44.7 秒
Latency standard deviation: 163.79 秒

More metrics

P95 latency: 208.06 秒
Non-live AST overall: 86.44%
Non-live simple calls: 77.25%
Non-live multiple-function selection: 91%
Non-live parallel calls: 88%
Non-live parallel multiple-function calls: 89.5%
Live AST overall: 74.69%
Live simple calls: 77.91%
Live multiple-function selection: 74.17%
Live parallel calls: 81.25%
Live parallel multiple-function calls: 58.33%
Multi-turn overall: 20.5%
Multi-turn base tasks: 24.5%
Multi-turn missing functions: 21.5%
Multi-turn missing parameters: 16%
Multi-turn long context: 20%
Web search overall: 4.5%
Search with summaries: 4%
Search without summaries: 5%
Memory overall: 23.87%
Key-value memory: 12.9%
Vector memory: 14.19%
Recursive-summary memory: 44.52%
Relevant-function detection: 87.5%
Irrelevant-function detection: 75.87%
Maximum format variation: 18 百分点
Format variation standard deviation: 5.22 百分点

mistral-large-2411 (Prompt)

Model reported by the benchmark

Text prompting · Prompt

Multi-turn overall · original rank 61 名 · 综合准确率原榜第 63

As reportedOrganization: Mistral AI · License: Proprietary · Model details ↗

Overall accuracy: 31.84%
Estimated benchmark cost: 232.42 USD
Average latency: 1.82 秒
Latency standard deviation: 7.15 秒

More metrics

P95 latency: 4.08 秒
Non-live AST overall: 83%
Non-live simple calls: 75.5%
Non-live multiple-function selection: 89.5%
Non-live parallel calls: 87%
Non-live parallel multiple-function calls: 80%
Live AST overall: 68.1%
Live simple calls: 83.72%
Live multiple-function selection: 64.01%
Live parallel calls: 93.75%
Live parallel multiple-function calls: 62.5%
Multi-turn overall: 13.75%
Multi-turn base tasks: 20%
Multi-turn missing functions: 5%
Multi-turn missing parameters: 11%
Multi-turn long context: 19%
Web search overall: 20%
Search with summaries: 28%
Search without summaries: 12%
Memory overall: 23.66%
Key-value memory: 16.77%
Vector memory: 30.97%
Recursive-summary memory: 23.23%
Relevant-function detection: 93.75%
Irrelevant-function detection: 38.77%
Maximum format variation: 13.5 百分点
Format variation standard deviation: 3.91 百分点

Gemini-2.5-Flash (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 32 名 · 综合准确率原榜第 15

As reportedOrganization: Google · License: Proprietary · Model details ↗

Overall accuracy: 56.24%
Estimated benchmark cost: 26.36 USD
Average latency: 2.99 秒
Latency standard deviation: 9.22 秒

More metrics

P95 latency: 5.62 秒
Non-live AST overall: 84.96%
Non-live simple calls: 74.33%
Non-live multiple-function selection: 92%
Non-live parallel calls: 94%
Non-live parallel multiple-function calls: 79.5%
Live AST overall: 74.39%
Live simple calls: 85.27%
Live multiple-function selection: 71.7%
Live parallel calls: 81.25%
Live parallel multiple-function calls: 70.83%
Multi-turn overall: 36.25%
Multi-turn base tasks: 41.5%
Multi-turn missing functions: 36%
Multi-turn missing parameters: 32%
Multi-turn long context: 35.5%
Web search overall: 59%
Search with summaries: 59%
Search without summaries: 59%
Memory overall: 41.29%
Key-value memory: 19.35%
Vector memory: 50.32%
Recursive-summary memory: 54.19%
Relevant-function detection: 75%
Irrelevant-function detection: 93.67%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

GPT-5-mini-2025-08-07 (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 43 名 · 综合准确率原榜第 17

As reportedOrganization: OpenAI · License: Proprietary · Model details ↗

Overall accuracy: 55.46%
Estimated benchmark cost: 22.18 USD
Average latency: 8.32 秒
Latency standard deviation: 17.35 秒

More metrics

P95 latency: 19.8 秒
Non-live AST overall: 69.85%
Non-live simple calls: 59.92%
Non-live multiple-function selection: 69%
Non-live parallel calls: 80%
Non-live parallel multiple-function calls: 70.5%
Live AST overall: 58.62%
Live simple calls: 62.02%
Live multiple-function selection: 58.02%
Live parallel calls: 62.5%
Live parallel multiple-function calls: 45.83%
Multi-turn overall: 27.5%
Multi-turn base tasks: 36.5%
Multi-turn missing functions: 17%
Multi-turn missing parameters: 23.5%
Multi-turn long context: 33%
Web search overall: 82%
Search with summaries: 87%
Search without summaries: 77%
Memory overall: 44.3%
Key-value memory: 36.77%
Vector memory: 43.87%
Recursive-summary memory: 52.26%
Relevant-function detection: 62.5%
Irrelevant-function detection: 91.01%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

GPT-4.1-mini-2025-04-14 (Prompt)

Model reported by the benchmark

Text prompting · Prompt

Multi-turn overall · original rank 92 名 · 综合准确率原榜第 67

As reportedOrganization: OpenAI · License: Proprietary · Model details ↗

Overall accuracy: 29.73%
Estimated benchmark cost: 20.52 USD
Average latency: 1.36 秒
Latency standard deviation: 4.5 秒

More metrics

P95 latency: 3.38 秒
Non-live AST overall: 84.6%
Non-live simple calls: 74.92%
Non-live multiple-function selection: 92.5%
Non-live parallel calls: 87.5%
Non-live parallel multiple-function calls: 83.5%
Live AST overall: 74.76%
Live simple calls: 80.62%
Live multiple-function selection: 73.31%
Live parallel calls: 81.25%
Live parallel multiple-function calls: 70.83%
Multi-turn overall: 2.5%
Multi-turn base tasks: 1.5%
Multi-turn missing functions: 4.5%
Multi-turn missing parameters: 2.5%
Multi-turn long context: 1.5%
Web search overall: 4%
Search with summaries: 7%
Search without summaries: 1%
Memory overall: 24.3%
Key-value memory: 20.65%
Vector memory: 13.55%
Recursive-summary memory: 38.71%
Relevant-function detection: 87.5%
Irrelevant-function detection: 73.88%
Maximum format variation: 45 百分点
Format variation standard deviation: 13.33 百分点

GPT-4.1-2025-04-14 (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 29 名 · 综合准确率原榜第 20

As reportedOrganization: OpenAI · License: Proprietary · Model details ↗

Overall accuracy: 53.96%
Estimated benchmark cost: 100.75 USD
Average latency: 1.63 秒
Latency standard deviation: 3.05 秒

More metrics

P95 latency: 4.01 秒
Non-live AST overall: 82.79%
Non-live simple calls: 72.67%
Non-live multiple-function selection: 89%
Non-live parallel calls: 88%
Non-live parallel multiple-function calls: 81.5%
Live AST overall: 69.95%
Live simple calls: 69.38%
Live multiple-function selection: 70.28%
Live parallel calls: 56.25%
Live parallel multiple-function calls: 70.83%
Multi-turn overall: 38.88%
Multi-turn base tasks: 47.5%
Multi-turn missing functions: 32.5%
Multi-turn missing parameters: 32.5%
Multi-turn long context: 43%
Web search overall: 68%
Search with summaries: 67%
Search without summaries: 69%
Memory overall: 23.87%
Key-value memory: 16.13%
Vector memory: 18.06%
Recursive-summary memory: 37.42%
Relevant-function detection: 87.5%
Irrelevant-function detection: 86.52%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Qwen3-0.6B (Prompt)

Model reported by the benchmark

Text prompting · Prompt

Multi-turn overall · original rank 98 名 · 综合准确率原榜第 94

As reportedOrganization: Qwen · License: apache-2.0 · Model details ↗

Overall accuracy: 22.38%
Estimated benchmark cost: 3.65 USD
Average latency: 3.1 秒
Latency standard deviation: 4.32 秒

More metrics

P95 latency: 10.31 秒
Non-live AST overall: 70%
Non-live simple calls: 64%
Non-live multiple-function selection: 78%
Non-live parallel calls: 75%
Non-live parallel multiple-function calls: 63%
Live AST overall: 49.37%
Live simple calls: 57.75%
Live multiple-function selection: 47.77%
Live parallel calls: 37.5%
Live parallel multiple-function calls: 37.5%
Multi-turn overall: 1.38%
Multi-turn base tasks: 1.5%
Multi-turn missing functions: 1.5%
Multi-turn missing parameters: 1.5%
Multi-turn long context: 1%
Web search overall: 0.5%
Search with summaries: 1%
Search without summaries: 0%
Memory overall: 8.39%
Key-value memory: 1.29%
Vector memory: 2.58%
Recursive-summary memory: 21.29%
Relevant-function detection: 75%
Irrelevant-function detection: 82.5%
Maximum format variation: 60.5 百分点
Format variation standard deviation: 24.35 百分点

Qwen3-235B-A22B-Instruct-2507 (Prompt)

Model reported by the benchmark

Text prompting · Prompt

Multi-turn overall · original rank 23 名 · 综合准确率原榜第 23

As reportedOrganization: Qwen · License: apache-2.0 · Model details ↗

Overall accuracy: 52.15%
Estimated benchmark cost: 3.12 USD
Average latency: 2.56 秒
Latency standard deviation: 2.75 秒

More metrics

P95 latency: 7.61 秒
Non-live AST overall: 90.33%
Non-live simple calls: 79.83%
Non-live multiple-function selection: 95%
Non-live parallel calls: 95.5%
Non-live parallel multiple-function calls: 91%
Live AST overall: 78.68%
Live simple calls: 82.95%
Live multiple-function selection: 77.78%
Live parallel calls: 81.25%
Live parallel multiple-function calls: 70.83%
Multi-turn overall: 44.62%
Multi-turn base tasks: 54%
Multi-turn missing functions: 42.5%
Multi-turn missing parameters: 31.5%
Multi-turn long context: 50.5%
Web search overall: 50.5%
Search with summaries: 56%
Search without summaries: 45%
Memory overall: 19.35%
Key-value memory: 12.9%
Vector memory: 11.61%
Recursive-summary memory: 33.55%
Relevant-function detection: 93.75%
Irrelevant-function detection: 78.89%
Maximum format variation: 8 百分点
Format variation standard deviation: 1.95 百分点

BitAgent-Bounty-8B

Model reported by the benchmark

Multi-turn overall · original rank 7 名 · 综合准确率原榜第 36

As reportedOrganization: Bittensor · License: Apache-2.0 · Model details ↗

Overall accuracy: 46.23%
Estimated benchmark cost: 18.02 USD
Average latency: 16.52 秒
Latency standard deviation: 30.73 秒

More metrics

P95 latency: 77.12 秒
Non-live AST overall: 81.6%
Non-live simple calls: 72.42%
Non-live multiple-function selection: 93%
Non-live parallel calls: 83%
Non-live parallel multiple-function calls: 78%
Live AST overall: 93.12%
Live simple calls: 90.31%
Live multiple-function selection: 94.02%
Live parallel calls: 75%
Live parallel multiple-function calls: 95.83%
Multi-turn overall: 62.38%
Multi-turn base tasks: 75%
Multi-turn missing functions: 49.5%
Multi-turn missing parameters: 68%
Multi-turn long context: 57%
Web search overall: 0%
Search with summaries: 0%
Search without summaries: 0%
Memory overall: 1.51%
Key-value memory: 1.29%
Vector memory: 1.29%
Recursive-summary memory: 1.94%
Relevant-function detection: 68.75%
Irrelevant-function detection: 97.48%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Ministral-8B-Instruct-2410 (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 106 名 · 综合准确率原榜第 105

As reportedOrganization: Mistral AI · License: Mistral AI Research License · Model details ↗

Overall accuracy: 11.1%
Estimated benchmark cost: 70.01 USD
Average latency: 82.07 秒
Latency standard deviation: 212.99 秒

More metrics

P95 latency: 568.59 秒
Non-live AST overall: 0%
Non-live simple calls: 0%
Non-live multiple-function selection: 0%
Non-live parallel calls: 0%
Non-live parallel multiple-function calls: 0%
Live AST overall: 0%
Live simple calls: 0%
Live multiple-function selection: 0%
Live parallel calls: 0%
Live parallel multiple-function calls: 0%
Multi-turn overall: 0%
Multi-turn base tasks: 0%
Multi-turn missing functions: 0%
Multi-turn missing parameters: 0%
Multi-turn long context: 0%
Web search overall: 1%
Search with summaries: 2%
Search without summaries: 0%
Memory overall: 4.52%
Key-value memory: 3.87%
Vector memory: 7.1%
Recursive-summary memory: 2.58%
Relevant-function detection: 0%
Irrelevant-function detection: 100%
Maximum format variation: 0 百分点
Format variation standard deviation: 0 百分点

Command A (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 41 名 · 综合准确率原榜第 35

As reportedOrganization: Cohere · License: CC-BY-NC 4.0 License (w/ Acceptable Use Addendum) · Model details ↗

Overall accuracy: 46.49%
Estimated benchmark cost: 91.37 USD
Average latency: 2.09 秒
Latency standard deviation: 7.36 秒

More metrics

P95 latency: 4.94 秒
Non-live AST overall: 87.56%
Non-live simple calls: 75.75%
Non-live multiple-function selection: 93%
Non-live parallel calls: 93.5%
Non-live parallel multiple-function calls: 88%
Live AST overall: 78.53%
Live simple calls: 85.66%
Live multiple-function selection: 76.92%
Live parallel calls: 81.25%
Live parallel multiple-function calls: 70.83%
Multi-turn overall: 29.5%
Multi-turn base tasks: 38%
Multi-turn missing functions: 23%
Multi-turn missing parameters: 32%
Multi-turn long context: 25%
Web search overall: 46.5%
Search with summaries: 60%
Search without summaries: 33%
Memory overall: 16.56%
Key-value memory: 4.52%
Vector memory: 5.16%
Recursive-summary memory: 40%
Relevant-function detection: 81.25%
Irrelevant-function detection: 84.19%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Gemma-3-12b-it (Prompt)

Model reported by the benchmark

Text prompting · Prompt

Multi-turn overall · original rank 79 名 · 综合准确率原榜第 66

As reportedOrganization: Google · License: gemma-terms-of-use · Model details ↗

Overall accuracy: 30.43%
Estimated benchmark cost: 10.77 USD
Average latency: 11.1 秒
Latency standard deviation: 17.17 秒

More metrics

P95 latency: 34.66 秒
Non-live AST overall: 79.44%
Non-live simple calls: 76.25%
Non-live multiple-function selection: 94%
Non-live parallel calls: 91%
Non-live parallel multiple-function calls: 56.5%
Live AST overall: 74.24%
Live simple calls: 85.66%
Live multiple-function selection: 71.89%
Live parallel calls: 87.5%
Live parallel multiple-function calls: 45.83%
Multi-turn overall: 5.75%
Multi-turn base tasks: 6.5%
Multi-turn missing functions: 7.5%
Multi-turn missing parameters: 5%
Multi-turn long context: 4%
Web search overall: 4%
Search with summaries: 4%
Search without summaries: 4%
Memory overall: 27.53%
Key-value memory: 8.39%
Vector memory: 25.16%
Recursive-summary memory: 49.03%
Relevant-function detection: 93.75%
Irrelevant-function detection: 70.29%
Maximum format variation: 67.5 百分点
Format variation standard deviation: 22.41 百分点

Gemini-2.5-Flash-Lite (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 62 名 · 综合准确率原榜第 52

As reportedOrganization: Google · License: Proprietary · Model details ↗

Overall accuracy: 36.87%
Estimated benchmark cost: 7.55 USD
Average latency: 1.18 秒
Latency standard deviation: 8.06 秒

More metrics

P95 latency: 1.67 秒
Non-live AST overall: 86.6%
Non-live simple calls: 70.92%
Non-live multiple-function selection: 90%
Non-live parallel calls: 93.5%
Non-live parallel multiple-function calls: 92%
Live AST overall: 65.8%
Live simple calls: 73.26%
Live multiple-function selection: 63.82%
Live parallel calls: 75%
Live parallel multiple-function calls: 66.67%
Multi-turn overall: 13.5%
Multi-turn base tasks: 20%
Multi-turn missing functions: 1.5%
Multi-turn missing parameters: 15%
Multi-turn long context: 17.5%
Web search overall: 21%
Search with summaries: 26%
Search without summaries: 16%
Memory overall: 20.65%
Key-value memory: 3.87%
Vector memory: 6.45%
Recursive-summary memory: 51.61%
Relevant-function detection: 43.75%
Irrelevant-function detection: 92.5%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Moonshotai-Kimi-K2-Instruct (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 16 名 · 综合准确率原榜第 11

As reportedOrganization: MoonshotAI · License: modified-mit · Model details ↗

Overall accuracy: 59.06%
Estimated benchmark cost: 6.19 USD
Average latency: 6.4 秒
Latency standard deviation: 9.38 秒

More metrics

P95 latency: 13.78 秒
Non-live AST overall: 81.6%
Non-live simple calls: 69.42%
Non-live multiple-function selection: 92%
Non-live parallel calls: 82%
Non-live parallel multiple-function calls: 83%
Live AST overall: 78.68%
Live simple calls: 81.78%
Live multiple-function selection: 78.06%
Live parallel calls: 87.5%
Live parallel multiple-function calls: 66.67%
Multi-turn overall: 50.63%
Multi-turn base tasks: 62%
Multi-turn missing functions: 41%
Multi-turn missing parameters: 44.5%
Multi-turn long context: 55%
Web search overall: 66.5%
Search with summaries: 72%
Search without summaries: 61%
Memory overall: 29.03%
Key-value memory: 21.94%
Vector memory: 20%
Recursive-summary memory: 45.16%
Relevant-function detection: 75%
Irrelevant-function detection: 87.34%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Gemini-3-Pro-Preview (Prompt)

Model reported by the benchmark

Text prompting · Prompt

Multi-turn overall · original rank 10 名 · 综合准确率原榜第 3

As reportedOrganization: Google · License: Proprietary · Model details ↗

Overall accuracy: 72.51%
Estimated benchmark cost: 298.47 USD
Average latency: 12.08 秒
Latency standard deviation: 21.3 秒

More metrics

P95 latency: 32.73 秒
Non-live AST overall: 90.65%
Non-live simple calls: 79.58%
Non-live multiple-function selection: 96%
Non-live parallel calls: 95%
Non-live parallel multiple-function calls: 92%
Live AST overall: 83.12%
Live simple calls: 87.6%
Live multiple-function selection: 81.77%
Live parallel calls: 93.75%
Live parallel multiple-function calls: 87.5%
Multi-turn overall: 60.75%
Multi-turn base tasks: 64.5%
Multi-turn missing functions: 60%
Multi-turn missing parameters: 54.5%
Multi-turn long context: 64%
Web search overall: 80%
Search with summaries: 78%
Search without summaries: 82%
Memory overall: 61.72%
Key-value memory: 59.35%
Vector memory: 62.58%
Recursive-summary memory: 63.23%
Relevant-function detection: 68.75%
Irrelevant-function detection: 85.59%
Maximum format variation: 8.5 百分点
Format variation standard deviation: 1.7 百分点

Falcon3-1B-Instruct (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 109 名 · 综合准确率原榜第 106

As reportedOrganization: TII UAE · License: falcon-llm-license · Model details ↗

Overall accuracy: 11.08%
Estimated benchmark cost: 1.72 USD
Average latency: 5.23 秒
Latency standard deviation: 14.34 秒

More metrics

P95 latency: 11.48 秒
Non-live AST overall: 9.02%
Non-live simple calls: 2.58%
Non-live multiple-function selection: 6%
Non-live parallel calls: 18%
Non-live parallel multiple-function calls: 9.5%
Live AST overall: 2.89%
Live simple calls: 4.26%
Live multiple-function selection: 2.37%
Live parallel calls: 0%
Live parallel multiple-function calls: 12.5%
Multi-turn overall: 0%
Multi-turn base tasks: 0%
Multi-turn missing functions: 0%
Multi-turn missing parameters: 0%
Multi-turn long context: 0%
Web search overall: 0%
Search with summaries: 0%
Search without summaries: 0%
Memory overall: 5.81%
Key-value memory: 5.16%
Vector memory: 7.74%
Recursive-summary memory: 4.52%
Relevant-function detection: 0%
Irrelevant-function detection: 87.3%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

palmyra-x-004 (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 103 名 · 综合准确率原榜第 76

As reportedOrganization: Writer · License: Proprietary · Model details ↗

Overall accuracy: 27.87%
Estimated benchmark cost: 178.15 USD
Average latency: 3.71 秒
Latency standard deviation: 7.62 秒

More metrics

P95 latency: 8.04 秒
Non-live AST overall: 87.46%
Non-live simple calls: 71.33%
Non-live multiple-function selection: 96%
Non-live parallel calls: 93%
Non-live parallel multiple-function calls: 89.5%
Live AST overall: 77.87%
Live simple calls: 79.46%
Live multiple-function selection: 77.97%
Live parallel calls: 56.25%
Live parallel multiple-function calls: 70.83%
Multi-turn overall: 0.38%
Multi-turn base tasks: 0.5%
Multi-turn missing functions: 0%
Multi-turn missing parameters: 0.5%
Multi-turn long context: 0.5%
Web search overall: 2.5%
Search with summaries: 4%
Search without summaries: 1%
Memory overall: 13.12%
Key-value memory: 6.45%
Vector memory: 14.19%
Recursive-summary memory: 18.71%
Relevant-function detection: 81.25%
Irrelevant-function detection: 80.99%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

xLAM-2-8b-fc-r (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 2 名 · 综合准确率原榜第 34

As reportedOrganization: Salesforce · License: cc-by-nc-4.0 · Model details ↗

Overall accuracy: 46.68%
Estimated benchmark cost: 20.92 USD
Average latency: 22.65 秒
Latency standard deviation: 46.92 秒

More metrics

P95 latency: 108.81 秒
Non-live AST overall: 84.58%
Non-live simple calls: 73.83%
Non-live multiple-function selection: 93.5%
Non-live parallel calls: 87.5%
Non-live parallel multiple-function calls: 83.5%
Live AST overall: 67.95%
Live simple calls: 75.58%
Live multiple-function selection: 66.57%
Live parallel calls: 56.25%
Live parallel multiple-function calls: 54.17%
Multi-turn overall: 70%
Multi-turn base tasks: 76%
Multi-turn missing functions: 72%
Multi-turn missing parameters: 65%
Multi-turn long context: 67%
Web search overall: 6.5%
Search with summaries: 11%
Search without summaries: 2%
Memory overall: 13.98%
Key-value memory: 5.81%
Vector memory: 15.48%
Recursive-summary memory: 20.65%
Relevant-function detection: 87.5%
Irrelevant-function detection: 63.28%
Maximum format variation: Not provided
Format variation standard deviation: Not provided

Claude-Haiku-4-5-20251001 (FC)

Model reported by the benchmark

Native function calling · FC

Multi-turn overall · original rank 14 名 · 综合准确率原榜第 6

As reportedOrganization: Anthropic · License: Proprietary · Model details ↗

Overall accuracy: 68.7%
Estimated benchmark cost: 14.23 USD
Average latency: 1.68 秒
Latency standard deviation: 3.92 秒

More metrics

P95 latency: 3.15 秒
Non-live AST overall: 86.5%
Non-live simple calls: 71%
Non-live multiple-function selection: 94%
Non-live parallel calls: 92.5%
Non-live parallel multiple-function calls: 88.5%
Live AST overall: 78.68%
Live simple calls: 83.72%
Live multiple-function selection: 77.59%
Live parallel calls: 75%
Live parallel multiple-function calls: 75%
Multi-turn overall: 53.62%
Multi-turn base tasks: 63.5%
Multi-turn missing functions: 42.5%
Multi-turn missing parameters: 52.5%
Multi-turn long context: 56%
Web search overall: 83.5%
Search with summaries: 86%
Search without summaries: 81%
Memory overall: 54.41%
Key-value memory: 51.61%
Vector memory: 55.48%
Recursive-summary memory: 56.13%
Relevant-function detection: 62.5%
Irrelevant-function detection: 85.11%
Maximum format variation: Not provided
Format variation standard deviation: Not provided