Sources checked
Evaluate selection, arguments, and evidence use
An original evaluation should check tool selection, valid arguments, and use of the returned evidence separately. A correct final answer can otherwise hide a brittle or unnecessary tool call.
What they trained
They had the model propose positions where a tool might help, executed the tools, and kept the calls that improved the model’s ability to predict the following text. The tools were small and the success was measured on their tasks.
The interesting result is that a model can learn when to call out, rather than only calling when a prompt forces it.
The limitation is the same one every tool paper has. The tool’s answer is only as good as the tool.
A calculator helps arithmetic the model was going to miss. A search tool returns whatever the index returns, including the wrong page.
Toolformer does not know which. A learned call can also fire too often and spend a budget the loss function did not price.
What product teams should take
Take the logging habit: the call, the arguments, the result, and whether the final text used the result. Do not take a 2023 experiment as permission to connect a payment API.
The paper’s tools were chosen because their outputs could be checked against a language-modeling loss. A side effect cannot.
Cite Schick and colleagues for the method, and cite your own forbidden-call tests for the deployment.
What to record before you rely on it
Record which tools exist in any system you compare to the paper. If your tool list is different, say the comparison is only about the pattern.
Keep one example where a tool call was unnecessary.
Keep the model id or the paper citation, the date you read it, and the inputs you used beside that note, so a later change can be compared with the same test.
When the result is a failure, store the artifact and change the design. Do not relax the check until the model happens to pass.
Name a person who owns that failure.
Common questions
Is a function-calling API Toolformer?
No. Function calling is an interface. Toolformer is a training procedure for inserting calls.
A model can have the interface without that training, and the calls can still be bad.
Will the model only call a tool when it is unsure?
Not reliably. The paper’s filter kept calls that helped prediction, which is not the same as a calibrated sense of uncertainty.
Measure the calls you get.
Sources & further reading
How this story was made
Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.
Our editorial standards





