When building GenAI and agentic applications, one of the most pressing and persistent questions is: \u201cWhich local model should I use for tool calling?\u201d<\/em>\u00a0 We kept hearing again and again, from colleagues within Docker and the developer community, ever since we started working on Docker Model Runner<\/strong><\/a>, a local inference engine that helps developers run and experiment with local models.\u00a0<\/p>\n\n It\u2019s a deceptively simple question with a surprisingly nuanced answer. Even when we tried to answer it for a very specific case: \u201cWhat if I just expose 5 simple tools to the model?\u201d<\/em> Our first instinct was to build something quickly and try it out manually.<\/p>\n So we created chat2cart<\/strong><\/a>, an AI-powered shopping assistant that lets users interact via chat to build, modify, and check out a shopping cart. Through a natural conversation, users can discover products, add or remove items, and complete or cancel their purchase, all from the chat interface.<\/p>\n To support testing across different LLMs, we added a model selector that makes it easy to switch between local models (via Docker Model Runner or Ollama) and hosted models using the OpenAI API.<\/p>\n OpenAI\u2019s GPT-4 or GPT-3.5 worked as expected, and the experience was fairly smooth.\u00a0<\/p>\n Called tools when they were needed<\/p>\n Avoided unnecessary tool usage<\/p>\n Handled tool responses naturally<\/p>\n But the local models? That\u2019s where the challenges started to surface.<\/p>\n\n We started experimenting with some of the local models listed on the Berkeley Function-Calling Leaderboard<\/a>. Our goal was to find smaller models, ideally with fewer than 10 billion parameters, so we tested xLAM-2-8b-fc-r and watt-tool-8B. We quickly ran into several recurring issues:<\/p>\n Eager invocation<\/strong>: Tools were being called even for greeting messages like \u201cHi there!\u201d<\/p>\n Wrong tool selection<\/strong>: The model would search when it should have added, or tried to remove when the cart was empty<\/p>\n Invalid arguments<\/strong>: Parameters like product_name or quantity were missing or malformed<\/p>\n Ignored responses<\/strong>: The model often failed to respond to tool output, leading to awkward or incomplete conversations<\/p>\n At this point, it was clear that manual testing wouldn\u2019t scale. Different models failed in different ways, some struggled with invocation logic, while others mishandled tool arguments or responses.\u00a0 Testing was not only slow, but also unreliable. Because these models are non-deterministic, we had to run each scenario multiple times just to get a reliable read on behavior.<\/p>\n We needed a testing setup that was repeatable, measurable, and fast.<\/p>\n\n Our goal wasn\u2019t academic rigor. In a couple of days, we created model-test<\/strong><\/a>, This is a flexible project with the following capabilities<\/p>\n Define real-world test cases<\/strong> with multiple valid tool call sequences<\/p>\n Run them against many models (local & hosted)<\/p>\n Track tool-calling accuracy<\/strong>, tool selection<\/strong>, and latency<\/strong><\/p>\n Log everything<\/strong> for analysis (or eventual fine-tuning)<\/p>\n The core idea behind model-test is simple: simulate realistic tool-using conversations, give the model room to reason and act, and check whether its behavior makes sense.<\/p>\n Each test case includes:<\/p>\n A prompt<\/strong> (e.g. \u201cAdd iPhone to cart\u201d)<\/p>\n The initial cart state<\/strong> (optional)<\/p>\n One or more valid tool-call variants<\/strong>, because there\u2019s often more than one right answer<\/p>\n Here\u2019s a typical case:<\/p>\n In this case, we consider both \u201cjust add \u2018iPhone’\u201d<\/strong> and \u201csearch first, then add the result\u201d<\/strong> as acceptable. Even though \u201ciPhone\u201d isn\u2019t a real product name, we\u2019re fine with it. We weren\u2019t aiming for overly strict precision, just realistic behavior.<\/p>\n Each test case belongs to a test suite. We provide two built-in suites. However, you can run an entire suite, individual test cases, or a selection of multiple test cases. Additionally, you can create your own custom suites to group tests as needed.\u00a0<\/p>\n Simple<\/strong>: Greetings, single-step actions<\/p>\n Complex<\/strong>: Multi-step reasoning and tool chaining<\/p>\n To make tests feel closer to how real agents behave, we simulate an agent loop up to 5 rounds<\/strong>.<\/p>\n Example:<\/p>\n User: \u201cAdd iPhone 5 to cart\u201d<\/em><\/p>\n Model: \u201cLet me search for iPhone 5\u2026\u201d<\/em> Tool: (returns product list)<\/em> Model: \u201cAdding product X to cart\u2026\u201d<\/em> Tool: (updates cart)<\/em> Model: \u201cDone\u201d<\/em> But if the model still wants to keep going after round 5?<\/p>\n That\u2019s it, my friend,\u00a0 test failed<\/strong>. Time\u2019s up.<\/p>\n\n We deliberately avoided designing tests that require perfect predictions.<\/p>\n We didn\u2019t demand that the model always know the exact product name.<\/p>\n What mattered was: did the tool sequence make sense<\/strong> for the intent?<\/p>\n This helped us focus on the kind of reasoning and behavior we actually want in agents, not just perfect token matches.<\/p>\n\n Our test outputs distilled down to a final F1 score, encapsulating three core dimensions:<\/p>\n Metric<\/strong><\/p>\n What it tells us<\/strong><\/p>\n Tool Invocation<\/strong><\/p>\n Did the model realize a tool was needed?<\/p>\n Tool Selection<\/strong><\/p>\n Did it choose the right tool(s) and use them correctly?<\/p>\n Parameter accuracy<\/strong><\/p>\n Whether the tool call arguments were correct?<\/p>\n<\/div>\n The F1 score is the harmonic mean of two things: precision (how often the model made valid tool calls) and recall (how often it made the tool calls it was supposed to).<\/p>\n We also tracked latency, the average runtime in seconds, but that wasn\u2019t part of the F1 calculation; it simply helped us evaluate speed and user experience.<\/p>\n We tested 21 models across 3570 test cases<\/strong> using 210 batch runs.<\/p>\n Hardware<\/strong>: MacBook Pro M4 Max, 128GB RAM Model <\/p>\n F1 Score<\/p>\n gpt-4<\/p>\n 0.974<\/p>\n qwen3:14B-Q4_K_M<\/p>\n 0.971<\/p>\n qwen3:14B-Q6_K<\/p>\n 0.943<\/p>\n claude-3-haiku-20240307<\/p>\n 0.933<\/p>\n qwen3:8B-F16<\/p>\n 0.933<\/p>\n qwen3:8B-Q4_K_M<\/p>\n 0.919<\/p>\n gpt-3.5-turbo<\/p>\n 0.899<\/p>\n gpt-4o<\/p>\n 0.857<\/p>\n gpt-4o-mini<\/p>\n 0.852<\/p>\n claude-3-5-sonnet-20241022<\/p>\n 0.851<\/p>\n llama3.1:8B-F16<\/p>\n 0.835<\/p>\n qwen2.5:14B-Q4_K_M<\/p>\n 0.812<\/p>\n claude-3-opus-20240229<\/p>\n 0.794<\/p>\n llama3.1:8B-Q4_K_M<\/p>\n 0.793<\/p>\n qwen2.5:7B-Q4_K_M<\/p>\n 0.753<\/p>\n gemma3:4B<\/p>\n 0.733<\/p>\n llama3.2:3B_F16<\/p>\n 0.727<\/p>\n llama3grog:7B-Q4_K_M<\/p>\n 0.723<\/p>\n llama3.3:70B.Q4_K_M<\/p>\n 0.607<\/p>\n llama-xlam:8B-Q4_K_M<\/p>\n 0.570<\/p>\n watt-tool:8B-Q4_K_M<\/p>\n 0.484<\/p>\n<\/div>\n Among all models, OpenAI\u2019s GPT-4 came out on top with a tool selection F1 score of 0.974, completing responses in just under 5 seconds on average. While hosted and not the focus of our local model exploration, it served as a reliable benchmark and provided some ground truths.<\/p>\n On the local side, Qwen 3 (14B) delivered outstanding results, nearly matching GPT-4 with a 0.971 F1 score, though with significantly higher latency (~142 seconds per interaction).<\/p>\n If you\u2019re looking for something faster, Qwen 3 (8B) also achieved an F1 score of 0.933, while cutting latency nearly in half (~84 seconds), making it a compelling balance between speed and tool-use accuracy.<\/p>\n Hosted models like Claude 3 Haiku also performed very well, hitting 0.933 F1 with exceptional speed (3.56 seconds average latency), further illustrating the high bar set by cloud-based offerings.<\/p>\n\n Not all models handled tool calling well. The quantized Watt 8B model struggled with parameter accuracy and ended up with a tool selection F1 score of just 0.484. Similarly, the LLaMA-based XLam 8B variant often missed the correct tool path altogether, finishing with an F1 score of 0.570. These models may be suitable for other tasks, but for our structured tool use test, they underdeliver.<\/p>\n\n We also experimented with both quantized<\/strong> and non-quantized<\/strong> variants for some models, and in all cases observed no significant difference<\/strong> in tool-calling behavior or performance. This suggests that quantization is beneficial for reducing resource usage without negatively impacting accuracy or reasoning quality, at least for the models and scenarios we tested.<\/p>\n\n If your goal is maximum tool-calling accuracy, then Qwen 3 (14B) or Qwen 3 (8B) are your best bets, both local, both precise, with the 8B variant being notably faster.<\/p>\n For a good trade-off between speed and performance, Qwen 2.5 stood out as a solid option. It\u2019s fast enough to support real-time experiences, while still maintaining decent tool selection accuracy.<\/p>\n If you need something more lightweight, especially for resource-constrained environments, the LLaMA 3 Groq 7B<\/a> variant offers modest performance at a much lower compute footprint.<\/p>\n\n Our testing confirmed that the Qwen family of models leads the pack among open-source options for tool calling. But as always, there\u2019s a trade-off; you\u2019ll need to balance between accuracy and latency when designing your application<\/p>\n Qwen models dominate<\/strong>: Even the 8B version of Qwen3 outperformed any other local model<\/p>\n Reasoning = latency<\/strong>: Higher-accuracy models take longer, often significantly.<\/p>\n Tool calling is core to almost every real-world GenAI application. Whether you\u2019re building agents or creating agentic workflows, your LLM must know when to act and how. Thanks to this simple framework, \u201cWe don\u2019t know which model to pick\u201d became \u201cWe\u2019ve narrowed it down to three great options, each with clear pros and cons.\u201d<\/p>\n If you\u2019re evaluating models for your agentic applications<\/a>, skip the guesswork. Try model-test<\/a> and make it your own for testing!\u00a0<\/p>\n\n Get an inside look at the design architecture of the Docker Model Runner<\/a>.\u00a0<\/p>\n Explore the story<\/a> behind our model distribution specification<\/p>\n Read our quickstart guide to Docker Model Runner<\/a>.<\/p>\n Find documentation for Model Runner<\/a>.<\/p>\n Subscribe to the Docker Navigator Newsletter<\/a>.<\/p>\n New to Docker? Create an account<\/a>.\u00a0<\/p>\n
We realized we had no definite answer for <\/strong>that. Local LLM models<\/a> offer control, cost-efficiency, and privacy, but when it comes to structured tool use, deciding when and how to act, they can behave very differently. We decided to dig deep and test this properly. We started with manual experimentation, then built a framework to scale our testing. This blog documents that journey and shares which models ranked highest on our tool-calling leaderboard.<\/p>\n\nThe first attempt: Manual testing<\/h2>\n
What went wrong with local models<\/h2>\n
Our second attempt: A scalable testing tool<\/h2>\n
It was: \u201cGive us good-enough answers in 2\u20133 days, not weeks.\u201d<\/em><\/p>\nHow it works<\/h3>\n
\n\u00a0\u00a0“prompt”: “Add iPhone to cart”,
\n\u00a0\u00a0“expected_tools_variants”: [
\n\u00a0\u00a0\u00a0\u00a0{
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0“name”: “direct_add”,
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0“tools”: [{ “name”: “add_to_cart”, “arguments”: { “product_name”: “iPhone” } }]
\n\u00a0\u00a0\u00a0\u00a0},
\n\u00a0\u00a0\u00a0\u00a0{
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0“name”: “search_then_add”,
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0“tools”: [
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{ “name”: “search_products”, “arguments”: { “query”: “iPhone” } },
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{ “name”: “add_to_cart”, “arguments”: { “product_name”: “iPhone 15” } }
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0]
\n\u00a0\u00a0\u00a0\u00a0}
\n\u00a0\u00a0]
\n}\n<\/div>\n\nThe agent loop<\/h3>\n
<\/em><\/p>\n
<\/em><\/p>\n
<\/em><\/p>\n
<\/em><\/p>\n
<\/em> \u2192 Great, test passed!<\/p>\nNot all-or-nothing<\/h3>\n
What We Measured<\/h2>\n
21 models and 3,570 tests later: Which models nailed tool calling?<\/h2>\n
Runner<\/strong>: test-all-models.sh<\/a><\/p>\n\nOverall Rankings (by Tool Selection F1):<\/h3>\n
Top performers<\/h3>\n
Underperformers<\/h3>\n
Quantization<\/h3>\n
Our recommendations<\/h3>\n
What we learned and why this matters<\/h2>\n
Learn more<\/h2>\n