Case 04 · Fine-tune evaluation

14 of my 22 tests were in training

“… ignorance more frequently begets confidence than does knowledge …”

Charles Darwin · The Descent of Man, 1871, Introduction (Darwin Online)

I fine-tuned a 3B model with QLoRA and scored it on 22 test prompts: 13 passed on the base model, 18 on the adapter. Then I checked each test prompt against the training data. 14 were in it word for word and 5 more were near copies, so part of that lift was recall. I wrote a held-out set of 21 tests with zero overlap, and a promote rule that needs a pass on both sets. The served model has not been scored on the held-out set yet.

0:00 / 1:16

Status · Audit done, held-out score pendingThe audit, the 21 held-out tests and the promote rule are built and checked on CPU. Scoring the served model on the held-out set needs one GPU inference run, which has not happened.

01 · The claim

The audit, in four numbers

14 of 22

Test prompts found word for word in the 298 training rows: all 4 pricing tests, all 7 refusal tests and 3 of 4 glossary tests.

19 of 22

Exact or near copies. The 5 near ones are date lookups built from the same templates with other dates.

0 of 21

Held-out tests that overlap training, at the 0.6 threshold and again at 0.45.

31 of 40

Validation rows flagged too, so the validation loss was optimistic as well.

02 · Transcript

Transcript

0:01A 3B model, fine-tuned. 22 tests. The base passed 13, the adapter 18.

0:14I checked every test prompt against the training data.

0:2014 were in it word for word. 5 more were near copies.

0:30So I wrote 21 new tests with zero overlap, checked at two thresholds.

0:36Ideal answers score 21 of 21. A set with planted mistakes scores 11.

0:41It catches swapped dates, the wrong body and answers given from memory.

0:46A candidate model now has to clear the gate on both sets.

0:52The gate itself is checked both ways: promote, and refuse to swap.

1:00The served model scores 19 of 22 on the main set.

1:05Its held-out score is still open. It needs one GPU run.

03 · Sources

Sources and honest limits

AUD Contamination audit of 22 fixtures against 298 training rows, 5 Oct 2026; rerun 8 Oct against 315 rows with the same fixture result (private lab)
HO Held-out fixtures and audit reports, 5, 6 and 8 Oct 2026 (private lab)
DAR Charles Darwin, The Descent of Man, 1871, vol. 1, Introduction, Darwin Online: https://darwin-online.org.uk/converted/published/1871_Descent_F937/1871_Descent_F937.1.html

Near means a character 4-gram similarity of 0.6 or more. My own audit note from 5 Oct says 15 exact matches; rerunning the audit script on 8 Oct lists 14 exact and 5 near, and this page uses the script output. The near matches on date lookups are templates with new dates, which is acceptable for testing whether the model calls the tool. The pricing tests still caught a weak behavior despite exact copies in training, so that failure was real. Each test is one sample. The held-out score for the served adapter is not measured. Product and client names are left out.

In solution.