Case 07 · Tool use

The best thing my fine-tune learned was to wait

“It is a capital mistake to theorise before one has data.”

Arthur Conan Doyle · A Scandal in Bohemia, 1891 (Project Gutenberg)

A small assistant I fine-tuned answers one kind of factual question through a tool: which zodiac sign the Sun or Moon was in on a given date. Asked from memory, the 3B base model stated a sign on 3 of 5 date prompts and got 2 wrong. Another base run cited the lookup tool as its source without calling it. The fine-tune calls the tool with the right date and body, and says nothing about the sign until the tool answers. My first grader could not tell those apart. Now it can.

0:00 / 1:16

Status · Measured on the main setThe behavior is measured on 5 date prompts from the main set, one sample each. 9 held-out tool tests now exist; the served model has not been scored on them yet.

01 · The claim

The case, in four numbers

2 of 3

Signs the base model stated from memory that were wrong: Pisces for 21 Mar 2024 and Virgo for 23 Sep 2024.

5 of 5

Date prompts where the fine-tune called the tool with exactly the right date and body.

0 of 5

Prompts where the fine-tune named a sign before the tool returned.

4 of 5

Right arguments from an earlier adapter. It asked about the Sun on a Moon question, and my old grader passed it.

02 · Transcript

Transcript

0:01The question: which sign was the Sun in on a date. A tool holds the answer.

0:14The 3B base model answered from memory: Pisces for 21 Mar 2024.

0:20Another base run named the tool as its source without calling it.

0:30The fine-tune calls the tool with the exact date and body, 5 of 5.

0:37It names no sign until the tool answers.

0:44My first grader passed any tool call, even a Moon question asked about the Sun.

0:49Now the arguments must match, and no sign may come before the tool result.

0:55Nine held-out tool tests catch swapped dates and invented signs.

1:00On the main set the served model passes. One sample per prompt.

1:05Its held-out tool score is still open.

03 · Sources

Sources and honest limits

PRED Predictions of the base model and two adapters on the main set, 21 and 24 Sep 2026 (private lab)
EQX US Naval Observatory, Earth seasons 2024, equinox times in UTC: https://aa.usno.navy.mil/api/seasons?year=2024
GRD Held-out tool-intent fixtures and argument checks, 6 Oct 2026 (private lab)
DOY Arthur Conan Doyle, A Scandal in Bohemia, in The Adventures of Sherlock Holmes, Project Gutenberg ebook 1661: https://www.gutenberg.org/ebooks/1661

The Sun entered Aries at the equinox on 20 Mar 2024, 03:06 UTC, and Libra on 22 Sep 2024, 12:44 UTC, so both base answers named the sign the Sun had just left. One sample per prompt, and the base runs used two different hosts. The fine-tune does not compute positions; the tool does, and the model has to wait for it. The new checks are lexical: a sign named anywhere in an answer before the tool result fails it. The held-out date tests have no score for the served model yet. Product and client names are left out.

Clear solution.