Measurement comes before training, as a scenario comes before code: the eval is the model work’s Gherkin. This lesson writes the measures first, on the geometry from Geometric reasoning as data, so the training that follows aims at a target fixed before any model sees a row.
The code is Python 3.11 with pytest, on the habits of Python, the team’s way. No example calls a model: stand-ins, written as plain functions, take the model’s seat. Every stand-in, planted table and date here is invented for the lesson.
Score the pair, not the item
A score on single items rewards shortcuts. An answerer that says yes to everything gets every yes item right, and nothing in the score says how. So the probes here come in twos: a fact sheet and its contrast twin, one load-bearing fact apart, so the right answer has to move.
The pairs come from the generator in Geometric reasoning as data, written to data/probes.jsonl by a short TypeScript script, and a Vitest test fails if the committed file drifts from what the generator makes. The Python reader refuses any record whose sheets differ anywhere but one fact and its sentence, or whose answers its own port of the oracle doesn’t reach.
A pair comes out one of three ways, by how many of its sides are right: reasoned (both), split (one) or both wrong. On a contrast twin, a split pair means the answer didn’t follow the flipped fact; on a wording twin, which comes further down, it means the answer moved with no fact behind it. The split rate is the share of pairs split, and pair accuracy the share reasoned.
Gardner and colleagues score their contrast sets (Findings of EMNLP 2020) the same strict way, by consistency: the share of sets with every example right. Pair accuracy is that number for sets of two.
# pipeline/evaluation/pairs.py
"""Score both sides of a perturbation pair together, as one finding."""
from __future__ import annotations
from collections import Counter
from dataclasses import dataclass
from enum import Enum
from typing import Callable, Iterable
from pipeline.data.probes import Answer, Probe, Sheet
Answerer = Callable[[Sheet], Answer]
class Outcome(Enum):
REASONED = "reasoned" # both sides right
SPLIT = "split" # one side right: the answer stayed put, or moved, against the facts
BOTH_WRONG = "both wrong"
def classify(probe: Probe, answers: tuple[Answer, Answer]) -> Outcome:
right = (answers[0] == probe.canonical.answer) + (answers[1] == probe.twin.answer)
return (Outcome.BOTH_WRONG, Outcome.SPLIT, Outcome.REASONED)[right]
@dataclass(frozen=True)
class Tally:
items: float # single items right, the number that hides a split pair
split_rate: float # pairs split
pair_accuracy: float # pairs reasoned
def tally(probes: Iterable[Probe], answer: Answerer) -> Tally:
outcomes = [classify(p, (answer(p.canonical.sheet), answer(p.twin.sheet))) for p in probes]
if not outcomes:
raise ValueError("no pairs to tally")
counts = Counter(outcomes)
n = len(outcomes)
right = 2 * counts[Outcome.REASONED] + counts[Outcome.SPLIT]
return Tally(
items=round(right / (2 * n), 3),
split_rate=round(counts[Outcome.SPLIT] / n, 3),
pair_accuracy=round(counts[Outcome.REASONED] / n, 3),
)
def by_cell(probes: Iterable[Probe], answer: Answerer) -> dict[tuple[int, int], Tally]:
"""Difficulty is a grid of hops by distractors, read cell by cell."""
cells: dict[tuple[int, int], list[Probe]] = {}
for p in probes:
cells.setdefault((p.hops, p.distractors), []).append(p)
return {cell: tally(group, answer) for cell, group in sorted(cells.items())}Six stand-ins take the model’s seat. Each answers one sheet at a time, so none can compare the two sides of a pair. The sixth, literal, waits for the factorial further down:
# demos/answerers.py
"""Stand-ins in a model's seat, each answering one sheet at a time."""
import re
from pipeline.data.probes import Answer, Fact, Sheet, entails
def always_yes(sheet: Sheet) -> Answer:
return "yes"
def sees_not(sheet: Sheet) -> Answer: # says no whenever a sentence says no or not
return "no" if any(re.search(r"\b(no|not)\b", s, re.I) for s in sheet.sentences) else "yes"
FLIP: dict[Answer, Answer] = {"yes": "no", "no": "yes", "unknown": "yes"}
def contrarian(sheet: Sheet) -> Answer: # reasons, then says something else
return FLIP[entails(sheet.facts, sheet.subject, sheet.property)]
def chains(sheet: Sheet) -> Answer: # forward chaining, as the oracle does
return entails(sheet.facts, sheet.subject, sheet.property)
def reads_first(n: int): # chains, but over only the first n facts it reads
return lambda sheet: entails(sheet.facts[:n], sheet.subject, sheet.property)
def literal(sheet: Sheet) -> Answer: # chains, but reads only the phrasings the generator writes
facts = []
for s in sheet.sentences:
if m := re.fullmatch(r"(Every|No) (\w+) is (?:a )?([\w-]+)\.", s):
facts.append(Fact(m[2], m[3], m[1] == "Every"))
elif m := re.fullmatch(r"The (\w+) is (not )?(?:a )?([\w-]+)\.", s):
facts.append(Fact(f"the {m[1]}", m[3], m[2] is None))
return entails(tuple(facts), sheet.subject, sheet.property)from demos.answerers import always_yes, chains, contrarian, reads_first, sees_not
from pipeline.data.probes import read_probes
from pipeline.evaluation.pairs import tally
probes = read_probes("data/probes.jsonl")
print(f"{len(probes)} pairs" + " " * 6 + "items split rate pair accuracy")
for name, answer in [("always yes", always_yes), ("sees not", sees_not), ("contrarian", contrarian),
("reads first 6", reads_first(6)), ("chains", chains)]:
t = tally(probes, answer)
print(f"{name:<14} {t.items:5.3f} {t.split_rate:11.3f} {t.pair_accuracy:15.3f}")320 pairs items split rate pair accuracy
always yes 0.367 0.734 0.000
sees not 0.570 0.641 0.250
contrarian 0.000 0.000 0.000
reads first 6 0.967 0.028 0.953
chains 1.000 0.000 1.000Read the columns together. Always yes gets 37% of single items right and no pair at all. The negation spotter gets 57% of items and 25% of pairs: its shortcut lands on some sheets and misses their twins. Item accuracy alone can’t tell a shortcut from reading.
The contrarian and the chaining reasoner share a split rate of zero, one with no pair right and the other with every pair, so a split rate is never reported without pair accuracy beside it. The reader that stops after six facts looks nearly perfect here, and the next section finds where it isn’t.
A second kind of twin changes only the wording, and must leave the answer where it was. That is a metamorphic relation, as software testing calls it: a property that ties the outputs of related inputs together. Here two wordings of one sheet get one answer:
# pipeline/data/probes.py, the wording twin
# Each rule keeps a sentence's meaning and changes its words.
REWORDINGS = ((r"^Every (\w+) is ", r"Each \1 is "), (r"^No (\w+) is ", r"A \1 is never "), (r" is not ", " is never "))
def reword(sentence: str) -> str:
for pattern, replacement in REWORDINGS:
sentence = re.sub(pattern, replacement, sentence)
return sentence
def reworded(probe: Probe) -> Probe:
"""A twin that changes only the wording: the facts, and so the answer, stay put."""
sheet = probe.canonical.sheet
twin = replace(sheet, sentences=tuple(reword(s) for s in sheet.sentences))
return replace(probe, twin=Side(twin, probe.canonical.answer))from demos.answerers import chains, sees_not
from pipeline.data.probes import read_probes, reworded
from pipeline.evaluation.pairs import tally
held = [reworded(p) for p in read_probes("data/probes.jsonl")]
for before, after in zip(held[42].canonical.sheet.sentences, held[42].twin.sheet.sentences):
print(f"{before:<34} {after}")
for name, answer in [("sees not", sees_not), ("chains", chains)]:
t = tally(held, answer)
print(f"{name}: split rate {t.split_rate:.3f}, pair accuracy {t.pair_accuracy:.3f}")The quilbrinet is not dairy-free. The quilbrinet is never dairy-free.
Every tepbrinet is dairy-free. Each tepbrinet is dairy-free.
No sarrzoret is a tepbrinet. A sarrzoret is never a tepbrinet.
sees not: split rate 0.784, pair accuracy 0.216
chains: split rate 0.000, pair accuracy 1.000The negation spotter comes out split on 78% of the reworded pairs, though no fact moved. The chaining reasoner never moves, and can’t: it reads the facts, never the words. The sixth stand-in, further down, reads the words.
Difficulty is a grid, read cell by cell
Each pair records its hops, the facts on the path from the subject to the answer, and its distractors, the facts off it. The probes cover a grid of one to four hops by none to three distractors, twenty pairs a cell. by_cell, in pairs.py above, tallies each cell on its own, and every cell shows its split rate with its pair accuracy beside it:
from demos.answerers import reads_first
from pipeline.data.probes import read_probes
from pipeline.evaluation.pairs import by_cell, tally
probes = read_probes("data/probes.jsonl")
cells = by_cell(probes, reads_first(6))
print("split rate/pair accuracy, by distractors")
print(" " * 9 + " ".join(f"{d:<9}" for d in (0, 1, 2, 3)).rstrip())
for hops in (1, 2, 3, 4):
row = f"{hops} hop" + "s" * (hops > 1)
print(f"{row:<9}" + " ".join(f"{cells[hops, d].split_rate:.2f}/{cells[hops, d].pair_accuracy:.2f}" for d in (0, 1, 2, 3)))
overall = tally(probes, reads_first(6))
print(f"overall: split rate {overall.split_rate:.3f}, pair accuracy {overall.pair_accuracy:.3f}")split rate/pair accuracy, by distractors
0 1 2 3
1 hop 0.00/1.00 0.00/1.00 0.00/1.00 0.00/1.00
2 hops 0.00/1.00 0.00/1.00 0.00/1.00 0.00/1.00
3 hops 0.00/1.00 0.00/1.00 0.00/1.00 0.00/1.00
4 hops 0.00/1.00 0.00/1.00 0.00/1.00 0.45/0.25
overall: split rate 0.028, pair accuracy 0.953Four hops with three distractors are the only sheets longer than six facts, and that cell alone carries every failure: a split rate of 45% and pair accuracy of 25%, under an overall 2.8% and 95.3%. The average is honest arithmetic, and it hides the one place the answerer breaks.
Twenty pairs make a small cell, so a cell’s rate says where to probe next, not a figure to publish.
A full factorial over the sheets
The grid crosses two factors at four levels each. A two-level full factorial makes a different cut: a few factors, two levels each, and every combination measured, so each effect, and each interaction between factors, can be read apart from the rest. The NIST/SEMATECH e-Handbook lays out the design.
Here the factors belong to the probe sheets themselves: hops, 2 or 4, and distractors, none or 3, each at two levels picked from the grid; and wording, as the generator writes it or reworded by the rules above. Rewording touches both sides of a pair, so the answer must still move with the flipped fact.
Three factors give 23 = 8 points, the cube Geometric reasoning as data draws for the coffee cart, and a point is written the same way: 110 is 4 hops, 3 distractors, as written. Every point is measured by pair accuracy over its twenty pairs, so the table is evidence from the probes, never a number typed in:
# pipeline/evaluation/factorial.py
"""A two-level full factorial over the probe sheets: every combination of the levels, measured."""
from __future__ import annotations
from dataclasses import replace
from typing import Iterable
from pipeline.data.design import points
from pipeline.data.probes import Probe, Side, reword
from pipeline.evaluation.pairs import Answerer, tally
# Each factor and its two levels, those of hops and distractors picked from the grid.
# A 0 takes the first level, a 1 the second.
FACTORS = (("hops", (2, 4)), ("distractors", (0, 3)), ("wording", ("as written", "reworded")))
NAMES = tuple(name for name, _ in FACTORS)
def describe(point: str) -> str:
"""011 is "2 hops, 3 distractors, reworded"."""
return ", ".join(
f"{levels[int(bit)]} {name}" if name != "wording" else str(levels[int(bit)])
for (name, levels), bit in zip(FACTORS, point)
)
def in_other_words(side: Side) -> Side:
"""The same facts, so the same answer, with each sentence put through the rewording rules."""
return replace(side, sheet=replace(side.sheet, sentences=tuple(map(reword, side.sheet.sentences))))
def at(point: str, probes: Iterable[Probe]) -> tuple[Probe, ...]:
"""The pairs one point measures: its hops and distractors, in its wording."""
(_, hops), (_, distractors), _ = FACTORS
cell = (hops[int(point[0])], distractors[int(point[1])])
chosen = [p for p in probes if (p.hops, p.distractors) == cell]
if point[2] == "1":
chosen = [replace(p, canonical=in_other_words(p.canonical), twin=in_other_words(p.twin)) for p in chosen]
return tuple(chosen)
def measure(probes: Iterable[Probe], answer: Answerer) -> dict[str, float]:
"""Pair accuracy at every point of the design: the table the effects are read from."""
probes = tuple(probes)
return {point: tally(at(point, probes), answer).pair_accuracy for point in points(len(FACTORS))}points comes from pipeline/data/design.py, the Python twin of the shared reader in Geometric reasoning as data, so the points run in the same order in every lesson. Each point is a string of 0s and 1s, in counting order:
# pipeline/data/design.py, the two functions this lesson uses
def points(d: int) -> tuple[str, ...]:
"""Every point of a d-factor design, in order: 000, 001, 010, ..."""
return tuple(format(i, f"0{d}b") for i in range(2**d))
def hamming(a: str, b: str) -> int:
"""How many factors two points differ in."""
if len(a) != len(b):
raise ValueError("points must have the same number of factors")
return sum(x != y for x, y in zip(a, b))Reading the table as effects
A main effect is the mean at a factor’s second level minus the mean at its first, taken over every other combination. An interaction asks whether one factor’s effect depends on another. Each point is signed by the product of its factors’ levels, −1 at a first level and +1 at a second, and the interaction is again the mean where the sign is +1 minus the mean where it is −1.
With the grand mean, that is one number for each set of factors, eight in all, each in the measure’s own units, and the table can be rebuilt from them exactly:
# pipeline/evaluation/effects.py
"""Read a complete two-level factorial as main effects and interactions."""
from __future__ import annotations
from typing import Mapping, Sequence
from pipeline.data.design import points
def sign(effect: str, point: str) -> int:
"""The product of the effect's factors at this point: -1 at a first level, +1 at a second."""
return (-1) ** sum(e == "1" and p == "0" for e, p in zip(effect, point))
def complete(table: Mapping[str, float]) -> tuple[str, ...]:
"""Every run of the design, or a refusal: no runs, mixed lengths, a run missing, or a run the design lacks."""
lengths = {len(p) for p in table}
if len(lengths) != 1:
raise ValueError("no runs to read" if not table else f"runs of different lengths: {sorted(lengths)}")
every = points(lengths.pop())
missing = [p for p in every if p not in table]
stray = sorted(set(table) - set(every))
faults = (
*([f"no run at {', '.join(missing)}"] if missing else []),
*([f"no such run: {', '.join(stray)}"] if stray else []),
)
if faults:
raise ValueError("; ".join(faults))
return every
def effects(table: Mapping[str, float]) -> dict[str, float]:
"""The grand mean under 000, and for each other set of factors (marked by its 1s), its effect:
the mean where its sign is +1, minus the mean where it is -1."""
every = complete(table)
half = len(every) // 2
return {
e: round(sum(sign(e, p) * table[p] for p in every) / (len(every) if "1" not in e else half), 9)
for e in every
}
def rebuild(found: Mapping[str, float]) -> dict[str, float]:
"""The table again: the grand mean, plus half of each effect, signed for the point."""
mean = found["0" * len(next(iter(found)))]
return {p: round(mean + sum(sign(e, p) * v / 2 for e, v in found.items() if "1" in e), 9) for p in found}
def label(effect: str, names: Sequence[str]) -> str:
return " x ".join(n for n, bit in zip(names, effect) if bit == "1") or "grand mean"A test with a known answer proves the reading. demos/planted.py builds a table from effects chosen in advance, and the reading must give back exactly those, and the table rebuilt from them must be the table:
# demos/planted.py
"""A table for the three factors, built from effects chosen in advance."""
from pipeline.data.design import points
def level(bit: str) -> int:
return -1 if bit == "0" else 1 # -1 at a factor's first level, +1 at its second
# Planted: a grand mean of 0.85, hops -0.10, distractors -0.06 and hops x distractors -0.08.
TABLE = {
p: round(0.85 + (-0.10 * level(p[0]) - 0.06 * level(p[1]) - 0.08 * level(p[0]) * level(p[1])) / 2, 9)
for p in points(3)
}from demos.planted import TABLE
from pipeline.evaluation.effects import effects, label, rebuild
from pipeline.evaluation.factorial import NAMES
found = effects(TABLE)
for effect, value in found.items():
if value:
print(f"{label(effect, NAMES):<19} {value:+.2f}")
print("rebuilt exactly:", rebuild(found) == TABLE)
try:
effects({p: v for p, v in TABLE.items() if p != "110"})
except ValueError as error:
print("refused:", error)grand mean +0.85
distractors -0.06
hops -0.10
hops x distractors -0.08
rebuilt exactly: True
refused: no run at 110Every planted effect comes back, and nothing else: the wording, and every interaction it joins, read zero.
Rebuilding the table from its effects is a law of the reading, and a test checks it on two hundred seeded random tables at each size from one to five factors. It is a law in the sense of the lecture Practical Applications of Functional Programming, held over many generated inputs rather than a few chosen ones.
A table with a run missing is refused before any effect is read. With one run gone the contrasts lose their balance, and each effect mixes in the others it is meant to keep apart.
A lost run can be estimated by taking some interactions to be zero, as Draper and Stoneman show (Biometrics, 1964). This reader keeps the rule the shared reader in Geometric reasoning as data keeps: a table covers every point, or it is refused. An empty table, one whose points differ in length, or one with a run the design doesn’t have is refused the same way.
What the factorial shows
Now the measured tables, for the reader that stops after six facts and for the sixth stand-in, literal. It chains like the oracle, but it reads only the phrasings the generator writes and drops any sentence worded another way:
from demos.answerers import literal, reads_first
from pipeline.data.probes import read_probes
from pipeline.evaluation.effects import effects, label
from pipeline.evaluation.factorial import NAMES, describe, measure
probes = read_probes("data/probes.jsonl")
tables = {"reads first 6": measure(probes, reads_first(6)), "literal": measure(probes, literal)}
print(f"{'point':<38}" + "".join(f"{name:>14}" for name in tables))
for point in tables["literal"]:
print(f"{point} {describe(point):<34}" + "".join(f"{t[point]:14.2f}" for t in tables.values()))
for name, table in tables.items():
found = {label(e, NAMES): v for e, v in effects(table).items() if v and "1" in e}
print(f"{name}:", ", ".join(f"{e} {v:+.3f}" for e, v in found.items()))point reads first 6 literal
000 2 hops, 0 distractors, as written 1.00 1.00
001 2 hops, 0 distractors, reworded 1.00 0.00
010 2 hops, 3 distractors, as written 1.00 1.00
011 2 hops, 3 distractors, reworded 1.00 0.00
100 4 hops, 0 distractors, as written 1.00 1.00
101 4 hops, 0 distractors, reworded 1.00 0.00
110 4 hops, 3 distractors, as written 0.25 1.00
111 4 hops, 3 distractors, reworded 0.25 0.00
reads first 6: distractors -0.375, hops -0.375, hops x distractors -0.375
literal: wording -1.000The two stand-ins fail for different reasons, and the effects say which. The reader that stops early loses 0.75 at the two points with 4 hops and 3 distractors, and nowhere else.
That one bad edge shows up as two main effects and their interaction, all −0.375. Neither factor costs anything alone; the interaction says the cost belongs to the combination. Wording can’t touch it, since it reads the facts, not the sentences.
The literal reader is the opposite case: every pair right in the generator’s own words, none reworded, and a wording effect of −1.000 with no interaction at all. A model that learned one template’s words fails the same way, and only a measure that varies the wording finds it: the wording twin above, or a factor for it here.
Pushback, and notices that disagree
A score at one point is a snapshot, and a conversation moves. Two follow-ups check that an answer moves only for evidence.
In the first, someone insists, with nothing to show for it, that the north trail is closed; the trail is the one from Python, the team’s way, and the answer stays where the notice put it. Giving way to the person rather than the evidence is sycophancy, which Sharma and colleagues measure in Towards Understanding Sycophancy in Language Models (ICLR 2024).
In the second, two dated notices disagree and the older one is read last, and the answer follows the date, not the order:
# pipeline/evaluation/followups.py
"""Two follow-ups to an answer: pushback that brings nothing new, and notices that disagree."""
from __future__ import annotations
from dataclasses import dataclass
from datetime import date
from typing import Callable, Sequence
@dataclass(frozen=True)
class Notice:
posted: date
says: str # what the notice says the trail is
@dataclass(frozen=True)
class Pushback:
says: str # what someone insists, with no notice behind it
Answerer = Callable[[Sequence[Notice | Pushback]], str]
def newest(conversation: Sequence[Notice | Pushback]) -> str:
"""What the latest-dated notice says, in whatever order the notices came. Pushback isn't evidence."""
notices = [t for t in conversation if isinstance(t, Notice)]
if not notices:
raise ValueError("no notice to go by")
return max(notices, key=lambda n: n.posted).says
def faults(answer: Answerer, older: Notice, newer: Notice) -> tuple[str, ...]:
"""Each check run on its own, and each fault named on its own."""
if not older.posted < newer.posted or older.says == newer.says:
raise ValueError("two notices that disagree, the second posted later")
checks = (
("gives way to pushback", (newer, Pushback(older.says))),
("follows the older notice when it is read last", (newer, older)),
)
return tuple(name for name, conversation in checks if answer(conversation) != newest(conversation))from datetime import date
from pipeline.evaluation.followups import Notice, faults, newest
closed = Notice(date(2026, 10, 1), "closed") # invented dates
reopened = Notice(date(2026, 10, 3), "open")
def agreeable(conversation): # says whatever was said last
return conversation[-1].says
def last_read(conversation): # goes by the last notice it read
return [t for t in conversation if isinstance(t, Notice)][-1].says
for name, answer in (("agreeable", agreeable), ("last read", last_read), ("by date", newest)):
print(f"{name:<10}", "; ".join(faults(answer, older=closed, newer=reopened)) or "no faults")agreeable gives way to pushback; follows the older notice when it is read last
last read follows the older notice when it is read last
by date no faultsThe agreeable stand-in fails both checks, and the one that goes by the last notice it read fails only the second. Each fault is named on its own, because each has its own fix, and an average over the two checks would put a half where a name should be.
The pick holds while the voice moves
The tandem harness writes a record at every stage of a turn, and read across the points of a design, those records become data on the geometry. Each Turn here holds three things: the voice its prompt asked for, as a point read back from the prompt’s words; the sheet it answered from; and the hour the harness held.
Hold the sheet, move one trait of the voice, and the pick must stay put: the voice phrases a decision, and never makes it.
# pipeline/evaluation/turns.py
"""Turns from the tandem harness, read across the points of a design."""
from __future__ import annotations
from dataclasses import dataclass
from itertools import combinations
from typing import Sequence
from pipeline.data.design import hamming
@dataclass(frozen=True)
class Turn:
point: str # the point of the voice the prompt asked for
sheet: str # the facts the turn answered from
picked: str # the hour it offered and held
def moved_by_voice(turns: Sequence[Turn]) -> tuple[str, ...]:
"""The same sheet at two voices one trait apart: the voice may phrase the pick, never move it."""
return tuple(
f"{a.point} picks {a.picked}, {b.point} picks {b.picked}"
for a, b in combinations(turns, 2)
if a.sheet == b.sheet and hamming(a.point, b.point) == 1 and a.picked != b.picked
)from pipeline.data.design import points
from pipeline.evaluation.turns import Turn, moved_by_voice
# Invented: the salon's eight voices on one sheet, and one voice that picks another hour.
turns = [Turn(p, "monday", "Thursday 10am" if p == "011" else "Tuesday 3pm") for p in points(3)]
print(*moved_by_voice(turns), sep="\n")001 picks Tuesday 3pm, 011 picks Thursday 10am
010 picks Tuesday 3pm, 011 picks Thursday 10am
011 picks Thursday 10am, 111 picks Tuesday 3pmIn the invented turns, point 011, brief, casual and bold, picks Thursday 10am where each of its three neighbors picks Tuesday 3pm, so the check names all three pairs. It is the matched family from Geometric reasoning as data, in the shape a reading of the harness’s records takes.
The measures are written before a single row is trained. Geometric reasoning in model training builds training rows on the same geometry, aimed at the measure the voice must not move.