TradingCalc
FROM THE BLOG · BENCHMARK · NOVEMBER 2026

We gave six AI models a calculator and asked 105 trading questions

In October I asked six models 105 trading-maths questions without any tools and 613 of 630 answers were right. The obvious next question is what changes when the model may call a calculator. So I ran the same 105 questions again, through the vendors' own APIs, twice: once with nothing, and once with the 91 TradingCalc MCP tools connected. Short answer: five of the six did not need them on this set, one small model needed them badly, and a tool is no help if the model reaches for the wrong one.

OPEN THE BENCHMARKRAW DATA
01THE RESULT

Right answers out of 105, without and with tools

Bar chart of right answers, out of 105 questions, for six models with tools, with the no-tools result beside each: gpt-6.1-sol 105 and 105, claude-opus-5-5 105 and 105, gemini-3.8-flash 105 and 105, gemini-4-argon 105 and 102, claude-haiku-5-5 102 and 104, gpt-5.4-mini 93 and 45.
ModelNo toolsWith toolsCost, no toolsCost, with tools
gpt-6.1-sol105 of 105105 of 105$0.539$1.19
claude-opus-5-5105 of 105105 of 105$1.79$7.32
gemini-4-argon102 of 105105 of 105$1.13$5.53
gemini-3.8-flash105 of 105105 of 105$0.556$5.79
claude-haiku-5-5104 of 105102 of 105$0.070$0.247
gpt-5.4-mini45 of 10593 of 105$0.106$0.926

Right means within 1% of the reference answer. Four of the six models were at or near the ceiling before the tools arrived, so for them there was little left to fix: claude-opus-5-5 and gpt-6.1-sol were at 105 of 105both ways. gemini-4-argon went from 102 to 105. claude-haiku-5-5 went the other way, from 104 to 102, and I come back to it below.

02THE ONE THAT NEEDED THEM

A small model, 45 right and then 93

Called directly through its API with default settings, gpt-5.4-mini got 45 of 105 right without tools. All 60 wrong answers came with a stated confidence of 80 or more, so asking the model how sure it is would not have caught them. With the tools it got 93, and it called a tool on every one of the 105 questions. Every question and every reply behind these figures is open: the raw data file lets you check any row.

One caution before you compare this with October. In October the same model scored 103 of 105 through a different channel, and here without tools it scored 45. I did not change the questions. What changed is how the model is called: this time I passed no reasoning setting, so it ran on the API default, and the first run went through another provider. I did not run the October setup again, so I cannot say how much of the gap is the channel and how much is anything else. The fair comparison is the one inside this run, same channel, same settings, tools off against tools on.

03A TOOL IN THE WRONG HAND

The 12 it still missed were not arithmetic

I read every one of gpt-5.4-mini's 12 misses with the tools in hand. It called a tool each time, and each time the tool answered a slightly different question from the one the task asked:

The numbers the tools returned were right for what they were asked. The model picked the wrong tool, or the right tool with a default it should have changed, and then quoted the result with confidence 99 or 100. That is a different failure from the hand arithmetic of the October run, and it is the one to watch for when you give an agent a toolbox: the tool does not check that the question matches it.

claude-haiku-5-5 shows a third pattern. It lost two answers with the tools on, and neither was a wrong number: it solved 16 of the 105 questions by hand without calling a tool, and on two it ended without the final answer line the format asks for, so there was nothing to score. One more of its misses was a hand calculation of a liquidation price. When a model may skip the tool, some of the time it will.

04COST

Tools made the run about five times dearer

At the vendors' prices the 630 answers without tools cost $4.20 and the 630 with tools cost $21.01. The reason is the prompt: every call carries the definitions of 91 tools. For claude-opus-5-5 that took $1.79 to $7.32 for the same 105 questions, with no change in the number it got right. If your agent only needs a few of the tools, connect only those.

These are the vendors' prices applied to the tokens of the measured replies, not what a billing page showed. The Google flagship and the Google small model were on introductory prices when I ran this. At the prices that follow, the same replies would cost $5.89 without tools and $32.33 with them. Replies that failed on the vendor side or hit a rate limit were run again and are not in the sums.

05LIMITS

What this run does not show

06CHECK IT YOURSELF

The data and the method are public

Every question, the expected answer, every full reply, the tool calls and the token counts of all 1260 measured answers are in the raw data file. The instructions given to the models in each mode and the prices used are in the same file. The October run, the method and the recompute are on the benchmark page, and the earlier article on how I checked the answer key is here. If a reference answer of mine disagrees with yours, I want to know about it.

To run the same questions against your own agent, connect the calculators it can call (the MCP tools) from the API and MCP guide and compare its answers with the expected ones in the data file.

OPEN THE BENCHMARKUSE THE TOOLS