A trained model is released the way a feature launches: against evidence named before the work began. This lesson follows one small language model to release under four rules, each written down before the work it judges: the gate it must pass, a test set used once, splits set apart by construction, and a base chosen by a declared rule.
The training itself, which needs a GPU, and the merge after it appear as sketches. Everything around them runs here, and every number in it is invented for the lesson.
The gate, written first
The question at the end is the one to write first: does the trained model get worse anywhere it is measured? The gate compares the trained model with the model in service, the one it would replace, at every point of the probe factorial from Measuring reasoning on the geometry, and a point where it may have got worse by more than a declared margin blocks the release.
Scores carry noise, so “worse” is defined before any score exists. Both models answer the same probe pairs, and the gate reads the change at each point pair by pair, with an error bar, as Evan Miller recommends for comparing two models.
A margin, declared with the gate, is the largest drop at one point the release accepts, and a point blocks when its error bar reaches past it: a drop that large can’t be ruled out.
# pipeline/release/gate.py
"""The gate, written before training: no point where a drop past a declared margin from the model in service can't be ruled out."""
from __future__ import annotations
from dataclasses import dataclass
from math import sqrt
from typing import Mapping, Sequence
Z = 1.96 # the error bar's reach in standard errors: its lower end is a one-sided 97.5% bound
@dataclass(frozen=True)
class Gate: # declared before training
points: tuple[str, ...] # every point of the probe factorial
pairs: int # probe pairs scored at each point, enough to resolve the margin
margin: float # the largest drop in accuracy at one point that the release accepts
@dataclass(frozen=True)
class Paired: # both models scored on the same probe pairs at one point
pairs: int
gains: int # pairs the trained model gets right and the model in service gets wrong
losses: int # pairs the model in service gets right and the trained model gets wrong
def change(p: Paired) -> float:
"""The change in accuracy, trained less in service, read pair by pair."""
return (p.gains - p.losses) / p.pairs
def lower_bound(p: Paired) -> float:
"""The change less its error bar: the largest drop these pairs can't rule out."""
variance = ((p.gains + p.losses) / p.pairs - change(p) ** 2) * p.pairs / (p.pairs - 1)
return change(p) - Z * sqrt(variance / p.pairs)
def gate_errors(scored: Mapping[str, Paired], gate: Gate) -> tuple[str, ...]:
"""Plain errors: every way the scores miss what the gate declared, or once they match,
one per point that blocks the release. None at all is a pass."""
faults = (
*(f"{point}: not scored" for point in gate.points if point not in scored),
*(f"{point}: not a point of the gate" for point in sorted(set(scored) - set(gate.points))),
*(
f"{point}: scored on {p.pairs} pairs, the gate declared {gate.pairs}"
for point, p in sorted(scored.items())
if p.pairs != gate.pairs
),
)
if faults:
return faults
return tuple(
f"blocks at {point}: changed by {change(scored[point]):+.3f}, error bar down to {bound:+.3f}"
for point in gate.points
if (bound := lower_bound(scored[point])) < -gate.margin
)
def stop_on(errors: Sequence[str]) -> None:
"""The one place a list of errors becomes an exception: the edge of the run."""
if errors:
raise SystemExit("not released:\n" + "\n".join(errors))from pipeline.release.gate import Gate, Paired, change, gate_errors, lower_bound
POINTS = ("000", "001", "010", "011", "100", "101", "110", "111")
gate = Gate(points=POINTS, pairs=2000, margin=0.05) # declared before training (invented for the lesson)
# Both models on the same 2,000 probe pairs at each point (invented): better by 0.05 at six points,
# unchanged at 101, and worse by 0.15 at 110.
scored = {point: Paired(2000, gains=240, losses=140) for point in POINTS}
scored |= {"101": Paired(2000, gains=200, losses=200), "110": Paired(2000, gains=100, losses=400)}
print(f"mean change {sum(map(change, scored.values())) / len(scored):+.3f}")
print(gate_errors(scored, gate))
print(f"101's error bar reaches {lower_bound(scored['101']):+.3f}; on 20 pairs it would reach {lower_bound(Paired(20, 2, 2)):+.3f}")
print(gate_errors(scored | {"111": Paired(500, gains=60, losses=35)}, gate))mean change +0.019
('blocks at 110: changed by -0.150, error bar down to -0.171',)
101's error bar reaches -0.020; on 20 pairs it would reach -0.201
('111: scored on 500 pairs, the gate declared 2000',)The trained model is better at six points and unchanged at 101, and its mean change is +0.019. At 110, four hops among three distractors in the generator’s own words, it falls by 0.15. A gate on the mean would ship it; the gate per point names 110 and stops it.
Point 101 passes without improving: its error bar reaches −0.020, inside the margin. On twenty pairs the same point’s bar would reach −0.201 and block, because twenty pairs can’t tell a point that held steady from a large drop.
So the number of pairs is declared with the gate, sized by the power analysis Miller describes: enough that a point training leaves alone clears the margin almost every time, since all eight must clear.
A point scored on any other number of pairs is refused, so no one adds pairs to a blocked point until it passes. A point not scored, or one the gate doesn’t name, is refused too.
Each check returns its errors as plain values, and only stop_on, at the edge of the run, turns them into an exception, the way the lecture Practical Applications of Functional Programming treats expected failure: as a value to report.
A refusal that ends a decision outright, such as a base rule with nothing over its floor further down, raises at once. The gate collects every error of a kind: first every way the scores miss what it declared, since no point is judged on pairs it didn’t declare, and once those are right, every point that blocks.
Pairing settles one more thing: both models answer the same pairs through one evaluation, in one run, so no score from another evaluation is ever compared. The probe pairs are a fresh batch for each decision, from seeds no earlier decision used, and the seeds are recorded with the decision.
A test set used once
A test set is good for one look. Look again after acting on what it showed, and the second look is no longer independent of the work. Dwork and colleagues name the risk, false discoveries, and a careful way to reuse a holdout; the simple defense is to use each test set once.
So each decision’s record keeps the places its test set asked about, and a test set that shares any place with the record is refused, with the decision named. Checking places, not whole sets, catches the same rows reordered, a few of them dropped, or one slipped in among fresh ones:
# pipeline/release/testset.py
"""A test set is good for one look: the record keeps the places each decision was tested on."""
from __future__ import annotations
from dataclasses import dataclass
from typing import Iterable, Mapping
@dataclass(frozen=True)
class Decision:
name: str # what was decided, such as the model that shipped
places: frozenset[str] # every place its test set asked about
def reused(rows: Iterable[Mapping[str, str]], record: Iterable[Decision]) -> tuple[str, ...]:
"""Every earlier decision these rows share a place with; none at all is a fresh test set."""
places = {row["place"] for row in rows}
return tuple(
f"{len(shared)} of these places already decided {decision.name}, {shared[0]} among them"
for decision in record
if (shared := sorted(places & decision.places))
)row, from the splits module in the next section, makes one generated question about an invented place, with its split, so a test set here is every test-split row of one batch:
from pipeline.release.splits import row
from pipeline.release.testset import Decision, reused
def test_set(name): # every test-split row the generator makes for one batch of invented places
return [r for i in range(200) if (r := row(i, name))["split"] == "test"]
record = [Decision("model-6", frozenset(r["place"] for r in test_set("Spur")))] # the record so far
print("model-6 was decided on", len(record[0].places), "places")
for label, rows in (
("the same rows, one dropped", test_set("Spur")[1:]),
("fresh places and one used one", test_set("Ridge") + test_set("Spur")[:1]),
("a fresh batch of places", test_set("Ridge")),
):
print(f"{label}:", *reused(rows, record) or ["fresh, so its places go in the record with the next decision"])model-6 was decided on 21 places
the same rows, one dropped: 20 of these places already decided model-6, Spur 118 trail among them
fresh places and one used one: 1 of these places already decided model-6, Spur 27 trail among them
a fresh batch of places: fresh, so its places go in the record with the next decisionSo a test set passes only when none of its places has been asked about before.
Splits by place, checked
The rows a release trains and tests on are generated the way Python, the team’s way generates its trail rows, turned by index, here over invented places, the Spur trails. A change to them is a change to the generator. The six rows that lesson keeps in data/train.jsonl are a small fixed set for its checks, each one marked train, and the rule below doesn’t govern them.
A row’s split comes from a stable hash of its place, so every row about one place lands in the same split, on every run and every machine, which Python’s own hash() doesn’t promise. Then a check proves what the construction promises: no place appears in two splits. Rows from serving’s log split the same way, by a stable hash of a key of their own.
# pipeline/release/splits.py
"""Splits by place: a stable hash of each row's place decides its split, and a check proves it."""
from __future__ import annotations
import hashlib
from typing import Iterable, Mapping
SHARES = (("train", 80), ("validation", 10), ("test", 10)) # percent of places, invented for the lesson
QUESTIONS = ("Is the {place} open?", "Can I hike the {place} today?", "Any news on the {place}?")
def split_of(key: str) -> str:
"""The same key lands in the same split on every run and every machine, unlike Python's hash()."""
bucket = int(hashlib.sha256(key.encode()).hexdigest(), 16) % 100
for split, share in SHARES:
if bucket < share:
return split
bucket -= share
raise ValueError("the shares must add up to 100")
def place(index: int, name: str = "Spur") -> str:
return f"{name} {index} trail" # invented, so only the notice can say what it is
def row(index: int, name: str = "Spur") -> dict[str, str]:
"""A row's split comes from its place, so every row about one place lands in one split."""
where = place(index, name)
return {"split": split_of(where), "place": where, "question": QUESTIONS[index % len(QUESTIONS)].format(place=where)}
def shared_between_splits(rows: Iterable[Mapping[str, str]]) -> tuple[str, ...]:
"""Any place that appears in more than one split."""
splits: dict[str, set[str]] = {}
for r in rows:
splits.setdefault(r["place"], set()).add(r["split"])
return tuple(sorted(where for where, seen in splits.items() if len(seen) > 1))from pipeline.release.splits import row, shared_between_splits
for index in (412, 413, 402, 414):
print(row(index)["split"], "|", row(index)["question"])
rows = [row(i) for i in range(600)]
print({split: sum(r["split"] == split for r in rows) for split in ("train", "validation", "test")})
print("shared between splits:", shared_between_splits(rows))
rows.append({**rows[412], "split": "test"}) # a training row, relabeled
print("shared between splits:", shared_between_splits(rows))train | Can I hike the Spur 412 trail today?
train | Any news on the Spur 413 trail?
validation | Is the Spur 402 trail open?
test | Is the Spur 414 trail open?
{'train': 484, 'validation': 61, 'test': 55}
shared between splits: ()
shared between splits: ('Spur 412 trail',)A training row relabeled as test is caught by its place. The three question templates here turn up in every split, so this test set measures new places, not new phrasings; a set meant to test phrasing gives each split templates of its own.
A base chosen by a declared rule
Fine-tuning starts from a small open-weight model, and choosing one is an experiment like any other. Every base under consideration answers the same validation prompts through the same evaluation, and results from two evaluations are refused rather than ranked.
The rule is written down before any result is seen: a floor on the pass rate, and among the bases over it, the one that takes the least memory to serve. A base under the floor is never chosen, however light, and when none clears it, the answer is to train from none of them:
# pipeline/release/select.py
"""Choose a base by a rule written down before any result: a floor on quality, then the lightest to serve."""
from __future__ import annotations
from dataclasses import dataclass
from typing import Iterable
@dataclass(frozen=True)
class Result:
base: str
evaluation: str # the prompts, settings and scorer, named together
pass_rate: float # validation answers that pass their oracle
memory_gb: float # what serving it takes
@dataclass(frozen=True)
class Rule: # declared before any result is seen
floor: float # the pass rate a base must reach to be considered at all
def choose(results: Iterable[Result], rule: Rule) -> str:
results = tuple(results)
evaluations = sorted({r.evaluation for r in results})
if len(evaluations) != 1:
raise ValueError(f"results from different evaluations can't be compared: {evaluations}")
eligible = [r for r in results if r.pass_rate >= rule.floor]
if not eligible:
raise ValueError(f"no base reaches the floor of {rule.floor:.0%}: train from none of them")
return min(eligible, key=lambda r: (r.memory_gb, r.base)).basefrom pipeline.release.select import Result, Rule, choose
rule = Rule(floor=0.85) # written down before the comparison ran (invented)
results = (
Result("base-x", "eval-1", pass_rate=0.93, memory_gb=14.0),
Result("base-y", "eval-1", pass_rate=0.88, memory_gb=6.5),
Result("base-z", "eval-1", pass_rate=0.79, memory_gb=3.0),
)
print(choose(results, rule))
try:
choose((*results, Result("base-w", "eval-2", pass_rate=0.95, memory_gb=2.0)), rule)
except ValueError as error:
print(error)
try:
choose(results, Rule(floor=0.95))
except ValueError as error:
print(error)base-y
results from different evaluations can't be compared: ['eval-1', 'eval-2']
no base reaches the floor of 95%: train from none of themThe rule picks base-y, though base-x scores highest and base-z is lightest: base-y is over the floor, and the lightest of those. Fixing the rule first keeps a favorite from winning on whichever measure flatters it. A base is only a starting point, and whether anything ships is the gate’s question.
Supervised, then preference training
Training runs in two stages on one LoRA adapter: a small set of added weights trained over a base model that stays fixed, so only a small share of the model’s parameters train.
- Supervised fine-tuning (SFT) teaches the answers, with the loss counted on the answer and not on the prompt.
- Direct preference optimization (DPO) then trains the same adapter on the pairs. It makes the chosen answer more likely than the rejected one, measured against a reference copy that stays fixed, the adapter as SFT left it. One
DPOConfigsetting,beta, sets how closely that comparison holds the model to what SFT taught.
Supervised first, then preferences, is the public recipe: the DPO paper (Rafailov and colleagues, NeurIPS 2023) starts its reference model from a supervised one.
The rows come from Geometric reasoning in model training, in TRL’s formats. The libraries are transformers, datasets, peft and trl. Every tunable setting comes from the JSON config, read once as Python, the team’s way reads it, and the loss rule and the paths are fixed in code:
# A sketch of the two training stages, not run here: it needs a GPU and the
# model's weights. Every tunable setting comes from the JSON config, read once:
# the seed and the epochs from its own fields, the rest from the "load",
# "adapter", "sft", "dpo" and "serving" settings its record here also carries,
# beside a "supervised" data file of rows with a prompt and a completion.
# The loss rule and the output paths are fixed in code.
from pathlib import Path
from datasets import load_dataset
from peft import LoraConfig, PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOConfig, DPOTrainer, SFTConfig, SFTTrainer
from pipeline.config.read import read_config
config = read_config(Path("config/train.json"))
name, revision = config.model, config.revision
tokenizer = AutoTokenizer.from_pretrained(name, revision=revision)
def base():
return AutoModelForCausalLM.from_pretrained(name, revision=revision, **config.load)
def rows(path):
return load_dataset("json", data_files=path, split="train")
# 1. Supervised fine-tuning: a new LoRA adapter, loss on the answer only.
# completion_only_loss needs rows with a "prompt" and a "completion".
sft = SFTTrainer(
model=base(),
args=SFTConfig(
output_dir="runs/sft",
completion_only_loss=True,
seed=config.seed,
num_train_epochs=config.epochs,
**config.sft,
),
train_dataset=rows(config.supervised),
processing_class=tokenizer,
peft_config=LoraConfig(**config.adapter),
)
sft.train()
sft.save_model("runs/sft/adapter")
# 2. Preference training: the same adapter learns from chosen and rejected
# answers, measured against a frozen copy of where it started.
policy = PeftModel.from_pretrained(base(), "runs/sft/adapter", is_trainable=True)
dpo = DPOTrainer(
model=policy,
args=DPOConfig(output_dir="runs/dpo", seed=config.seed, num_train_epochs=config.epochs, **config.dpo),
train_dataset=rows(config.train),
eval_dataset=rows(config.validation),
processing_class=tokenizer,
)
dpo.train()
dpo.save_model("runs/dpo/adapter")That fixed copy is the trainer’s own doing. In TRL since 0.29, a model that already carries an adapter, as policy does, is trained against a copy of that adapter, frozen when the trainer is built (the DPO trainer’s source, TRL 1.14.1). Earlier releases measured it against the base model, with the adapter switched off, and 1.14.1 still does, with a warning, for an adapter that targets parameters rather than modules (target_parameters) under a PEFT older than 0.20.
TRL cuts any sequence longer than its max_length setting, which is on by default, and says nothing. In TRL 1.14.1, both trainers here also drop outright a row whose prompt alone fills that length, with only a progress bar’s label to show for it.
Cut short, a chosen and a rejected answer can end as the same text, so every row is counted against that length, through the model’s own tokenizer and chat template, before training starts.
No learned reward model sits in this loop, because the oracles judge the rows. Pushing hard against a learned reward finds its blind spots: Gao, Schulman and Hilton let a fixed gold-standard reward model stand in for people, and show its score falling while a learned proxy’s keeps rising.
Evaluated as served
The trained adapter is merged into the base and saved, with the base loaded without quantization so the adapter’s weights merge in full. The serving runtime’s own tools then convert the merged model to the format it loads, quantizing it where the runtime calls for that, and the quantization type is a setting like any other, read from the config:
# A sketch, not run here: merge the trained adapter into the base and save the result.
# The base loads without quantization, so the adapter's weights merge in full.
# The serving runtime's own tools convert it, with the type read from the config.
full = AutoModelForCausalLM.from_pretrained(name, revision=revision)
merged = PeftModel.from_pretrained(full, "runs/dpo/adapter").merge_and_unload()
merged.save_pretrained("runs/merged")
tokenizer.save_pretrained("runs/merged")Quantizing changes the weights, so it can change answers. The evaluation that decides the release runs on the converted model, through the runtime that will serve it, inside the harness from The tandem harness, with the composition, checks and fallbacks a real request meets.
First come named checks that conversion kept what training taught, before the gate from the first section runs on the same served model. The checks are a table of scenarios and one runner, and they ask about places from training, so they use no test set:
# pipeline/release/served.py
"""Checks on the converted model as served: named scenarios as data, and one runner."""
from __future__ import annotations
from typing import Any, Callable, Mapping
Reply = Mapping[str, Any] # {"text": ...}
# Places from training: these checks score no answer's quality, so they use no test set.
SCENARIOS: tuple[tuple[str, dict[str, str], Callable[[Reply], bool]], ...] = (
("a closed trail is still closed after conversion",
{"place": "Spur 101 trail", "notice": "closed until Friday"},
lambda r: "closed" in r["text"]),
("a trail the notice doesn't name is still unknown",
{"place": "Spur 102 trail", "notice": ""},
lambda r: "not on the notice" in r["text"]),
("the answer names the trail it was asked about",
{"place": "Spur 103 trail", "notice": "open"},
lambda r: "Spur 103" in r["text"]),
)
def served_errors(serve: Callable[[dict[str, str]], Reply]) -> tuple[str, ...]:
"""The name of every scenario the served model fails."""
return tuple(name for name, request, holds in SCENARIOS if not holds(serve(request)))from pipeline.release.served import served_errors
def served(request, damaged=False): # a stand-in for the converted model behind the harness
if not request["notice"]:
return {"text": f"The {request['place']} is not on the notice."}
state = "open" if damaged else request["notice"]
return {"text": f"The {request['place']} is {state}."}
print("as served:", served_errors(served))
print("damaged:", served_errors(lambda request: served(request, damaged=True)))as served: ()
damaged: ('a closed trail is still closed after conversion',)Here a stand-in plays the converted model, one that has lost a closure, so the runner names the scenario it fails. These checks are one rung of three, and each rung catches what the one below it can’t:
Evidence that names the inputs
Before a trained adapter goes any further, the run shows what it was trained on. Its record names the inputs exactly: the config, the model’s pinned revision, the package versions, and each data file by its path and a SHA-256 digest of its contents. Each file among them, the config and the data, is checked again before release:
# pipeline/release/inputs.py
"""Evidence that names the exact inputs a run used, checked again before release."""
from __future__ import annotations
import hashlib
from pathlib import Path
def fingerprint(path: Path) -> dict[str, str]:
return {"path": path.as_posix(), "sha256": hashlib.sha256(path.read_bytes()).hexdigest()}
def changed_inputs(recorded: list[dict[str, str]]) -> tuple[str, ...]:
"""Every recorded input that is gone, or whose bytes are no longer the bytes the run used."""
present = [entry for entry in recorded if Path(entry["path"]).is_file()]
return (
*(f"{entry['path']} is missing" for entry in recorded if entry not in present),
*(
f"{entry['path']} changed since the run: recorded {entry['sha256'][:12]}, found {now['sha256'][:12]}"
for entry in present
if (now := fingerprint(Path(entry["path"])))["sha256"] != entry["sha256"]
),
)Here a stand-in for the training rows, data/sample.jsonl, holds three lines, {"id": "row-000"} through {"id": "row-002"}. The snippet’s last step rewrites that file, so run it on a fresh copy of data/ each time:
from pathlib import Path
from pipeline.release.inputs import changed_inputs, fingerprint
recorded = [fingerprint(Path("data/sample.jsonl"))] # written when training starts
print(recorded)
print("changed:", changed_inputs(recorded))
rows = Path("data/sample.jsonl") # one id rewritten after training: same size, same lines
rows.write_text(rows.read_text().replace("row-001", "row-009"))
print(*changed_inputs(recorded))[{'path': 'data/sample.jsonl', 'sha256': '1f1ea15caf02b22c8fbda2bb33f632e9ef7d231dc416c281fd870b88d9925ab8'}]
changed: ()
data/sample.jsonl changed since the run: recorded 1f1ea15caf02, found 474f637b110dA changed input is named with both digests, and a missing one by its path. Here the rewritten row keeps the file’s size and its line count, and only the digest catches it, so the record can’t quietly describe a different run from the one that made the model.
The model file gets the same treatment: fingerprint records it like any input, and changed_inputs checks it again before release. Every document that ships with the model, the model card and the serving config among them, names that digest, so each names the very file serving loads.
That is Module 6 in one model:
- The AI protocol map put a contract at every join, and Geometric reasoning as data declared behaviors as points with a distance.
- The tandem harness let the model propose and typed code decide, Server and client in tandem carried a booking across a stateless wire, and Extending the harness grew both at documented points.
- Python, the team’s way kept the core pure, Measuring reasoning on the geometry wrote the measures before training, Geometric reasoning in model training built rows from what serving logged, and this lesson released a model only on rules written before it ran.
That is the course, from a user story to a released model. Webmasters at any stage of the craft grow into this practice under the Sean Dinwiddie’s Webmastery name, on client work with a written scope and a review against it before every launch, where what one job teaches becomes the team’s, and keep 70% of the fee on their work. Joining the team sets out the terms; to join, they send a link to work they’ve built and the kind of job they want.