Build note
Teaching it to call tools, and the form it fills wrongly
The model learned to emit a perfectly well-formed tool call, and then filled it with the wrong answer. The mechanism works. The model does not.
The comparison is filling in a form. Someone can now put a valid entry in every box, including the ones that need a date format or an account number. They still write the wrong number. Good handwriting is not the same as a correct answer.
What a tool call actually is
Three terms, because this phase is where they get confused.
A tool call is one turn in which the model emits a structured request, such as {"expr": "2 + 3"}, and the system runs it and feeds the result back. An agent loop is the multi-turn version: call, result, call again, then answer. A skill is a named tool with a description and a JSON schema for its arguments.
None of this requires a large model. It requires a format the model can emit reliably and a harness that can execute what it emits. That is what this phase builds.
The format and the constraint
The template is chatml-tools, a system prompt plus role tags plus a schema listing the available tools. The model is fine-tuned on a synthetic corpus of requests and calls, data/tools/train.txt, which started at 88 examples.
The interesting part is constrained decoding. --constrain maps to a required tool call, which means the model cannot answer in prose at all; it must produce a call. To make that possible, the sampler applies a JSON logit mask at each step, zeroing any token that would break the grammar, and forcing the opening <tool_call> token.
A logit mask is a simple idea with a large effect. Instead of hoping the model emits valid JSON, you compute which next tokens are legal given the grammar so far, and you remove every other token from consideration. The output is valid by construction.
The numbers
The warm start is the GPT-2-vocabulary base, 14.5 million parameters, fine-tuned for 1,500 steps at about 42 ms/step. Training loss reached 0.0404, and the in-corpus tail validation was 0.3208. Two 20-step runs were bit-identical under sha256sum, so the path is deterministic.
That low training loss is overfitting, not skill. 1,500 steps is about 17 passes over 88 examples. The held-out table is what tells the truth, over 18 tool tasks and 4 no-tool tasks:
| Arm | Tool parse | Tool result | No-tool parse | Answer |
|---|---|---|---|---|
| free | 4/18 | 0/18 | 3/4 | 0/22 |
| constrained | 17/18 | 0/18 | 0/4 | 0/22 |
Read the first column and last column together. The constrained arm parses 17 of 18 tool calls, against 4 of 18 unconstrained. That is the mechanism working: the mask makes the format reliable.
Then the results column says 0 of 18, and the answers say 0 of 22. The calls are well-formed and wrong.
Where it actually fails
The failure is specific and worth naming. The model memorises entities instead of copying them.
After all the capitals were added to the corpus, asking “What is the capital of China?” produced {"entity": "capital of China"} and resolved to Beijing. Asking “What is the capital of India?” produced the same argument, and so returned Beijing again. It had learned the answer to the question it saw, not the rule for extracting the entity.
There is a second, narrower failure. The final answer turn hallucinates arithmetic instead of using the tool result. Given a correct result of 5, it wrote “5 * 4 = 20.” as though it had been asked a different question.
One fix did work cleanly. Removing the literal schema examples, 2 + 3*4 and capital of France, fixed the common prompts: “What is 2 + 3?” then produced {"expr": "2 + 3"} and returned 5. The examples had been teaching the model to echo them.
A 2000-step retrain added web tools, and the same limit showed: asked about Peru, the model chose the facts tool and emitted a memorised entity rather than selecting web_search. The C harness routes it correctly anyway, falling through to a web search and returning Lima. Machinery complete; model selection is the limit.
The vLLM arm was even blunter. The export and the OpenAI-compatible client both work, but the smoke model never emitted a clean opener through vLLM’s stock hermes parser, so that arm scored 0 of 20. Same model limit, different frame, not a wiring problem.
What it taught me
The mechanism is not the model. A constrained decoder can make output parseable without making it correct, and those two properties are measured separately. Reporting only the parse rate would have looked like success.
The second lesson is that a small model’s failure mode is memorisation, and it looks like capability until you vary the input. China and India are the same question to a model that learned answers rather than a rule.
The third lesson is architectural. Because the machinery and the model quality are separable, the harness can carry reliability today. The C tool registry, the executor and the composer make the tools work with a weak model, and a better model would slot into the same format later.
How this was made
- Written by: Rupesh, from the
ghedi-crepository (src/tools.c,src/agent.c,src/chat_template.c,scripts/tools/eval-tools.py,docs/28-tool-calling.md,docs/34-harness.md,runs.md), with AI drafting and editing. - Numbers: the held-out table, the loss figures, the step count and the entity-memorisation example come from
runs.md“Phase 17” sections anddocs/28-tool-calling.md. - AI-assisted sections: structure, explanations and copy editing.