An average score can rise while a model gets worse at something that matters, and the cure starts in the data. Measuring reasoning on the geometry wrote the measures first. This lesson builds training rows on the same geometry, aimed at the measure the voice must not move: the facts hold at every point, and only the voice changes.
The trail rows come from a teaching catalog with a renderer of its own, and rows for anything served come from the prompts serving really sent. Every answer is judged field by field, and voice pairs move one trait at a time.
The trail answers reply to the invented notices in the CATALOG of Python, the team’s way; the first is the north trail, closed until Friday because of a washed-out bridge.
The answers vary on the three voice traits the coffee cart and the salon use: brief or detailed, formal or casual, cautious or bold, declared once in data/trail.design.json. The traits give the eight points Geometric reasoning as data draws, and a point is three bits in that order, 0 for a trait’s first level and 1 for its second, so 000 is brief, formal and cautious.
The code is Python 3.11 with pytest, on the practices of Python, the team’s way, and every row and figure is invented for the lesson.
Rows as contracts, in TRL’s format
A training row is a contract. A supervised row holds the prompt as messages, the answer to learn, a few stable fields (its family and its point) and the oracle that judges the answer. A row taken from serving’s log is a type of its own and carries no oracle, because serving’s own check already passed its answer, so corpus_errors, which judges every answer by its oracle, takes only supervised rows. The loss falls on the answer alone.
A preference row holds the same prompt, a chosen answer and a rejected one that fails for one named reason. That is the shape direct preference optimization learns from, and TRL’s dataset formats give it public columns: a prompt and a completion, or a prompt, a chosen and a rejected answer. In their conversational form, each column is a list of messages with a role and content.
# pipeline/data/rows.py
"""Training rows as contracts: the prompt, the answer to learn, and what judges it."""
from __future__ import annotations
from dataclasses import dataclass
from typing import Sequence
from pipeline.data.oracle import Oracle, score
Message = tuple[str, str] # (role, content)
@dataclass(frozen=True)
class SupervisedRow:
id: str
prompt: tuple[Message, ...] # exactly what the model is shown
completion: tuple[Message, ...] # the loss falls here, and only here
oracle: Oracle # what judges the answer
family: str
point: str
@dataclass(frozen=True)
class ServedRow:
"""An answer serving's own check passed. Its ruling is in the log, so it carries no oracle."""
id: str
prompt: tuple[Message, ...] # exactly what the model was sent
completion: tuple[Message, ...]
family: str
point: str
@dataclass(frozen=True)
class PreferenceRow:
id: str
prompt: tuple[Message, ...]
chosen: tuple[Message, ...]
rejected: tuple[Message, ...] # fails as the failure it teaches
teaches: str
oracle: Oracle
family: str
point: str
def as_trl(row: SupervisedRow | ServedRow | PreferenceRow) -> dict[str, list[dict[str, str]]]:
"""Only the columns TRL reads. The oracle and the metadata stay with the corpus."""
sides = ("prompt", "chosen", "rejected") if isinstance(row, PreferenceRow) else ("prompt", "completion")
return {side: [{"role": r, "content": c} for r, c in getattr(row, side)] for side in sides}
def said(side: tuple[Message, ...]) -> str:
return " ".join(content for role, content in side if role == "assistant")
def corpus_errors(rows: Sequence[SupervisedRow]) -> tuple[str, ...]:
"""Every answer, judged by its oracle."""
return tuple(
f"{row.id}: {miss.value} {detail!r}"
for row in rows
for miss, detail in score(said(row.completion), row.oracle).misses
)as_trl hands a trainer only the columns it reads, in the conversational form. For rows with a prompt and a completion, TRL’s supervised trainer counts the loss on the completion alone unless told otherwise (its completion_only_loss setting). The oracle and the metadata stay with the corpus, where the graders read them.
Prompts from what serving logged
The prompt is the easiest part to get wrong, because two programs could write it: serving, which renders it for the model, and training, which needs the same text to learn from. Martin Zinkevich’s Rules of Machine Learning names a gap between how training and serving handle data as one cause of training-serving skew.
Its Rule 32 says to reuse code between the two pipelines whenever possible, and warns that two languages make that nearly impossible. Rule 29 gives the remedy this course takes: save what serving used, and train on that log.
The tandem harness records every prompt exactly as sent, and its servedLog turns those records into JSON, one value per call, which the committed log holds a line each. served_rows turns each answer that passed serving’s own check into a row, and never renders a prompt again:
# pipeline/data/served.py
"""Rows from what serving logged: the prompt exactly as sent, never rendered a second time."""
from __future__ import annotations
import json
import re
from pathlib import Path
from pipeline.data.design import Design, describe, points
from pipeline.data.rows import ServedRow
LOG = Path("../hs/the-tandem-harness/served.jsonl") # serving's own log, never a copy
def point_of(prompt: str, design: Design) -> str:
"""The design's point a prompt asked for, read back from its voice in words."""
voice = re.search(r"Answer in a (.+?) voice", prompt)
found = [p for p in points(len(design.factors)) if voice and describe(design, p) == voice[1]]
if len(found) != 1:
raise ValueError("the prompt names no voice of this design")
return found[0]
def served_rows(lines: list[str], design: Design) -> list[ServedRow]:
"""A row for every answer serving's check passed. A rejected answer never becomes a row."""
rows = []
for n, line in enumerate(lines):
call = json.loads(line)
if call["ruling"] == "passed":
rows.append(ServedRow(
id=f"served-{n}",
prompt=(("user", call["prompt"]),),
completion=(("assistant", call["output"]),),
family="salon",
point=point_of(call["prompt"], design),
))
return rowsThe course keeps its Haskell project in hs/, beside the Python one, so ../hs/ reaches serving’s own file rather than a copy.
A row’s point is read back from the voice in words, through the salon’s design file and the shared reader, so the row is filed at its point although the prompt never names one:
import json
from pipeline.data.design import load_design
from pipeline.data.rows import as_trl
from pipeline.data.served import LOG, served_rows
lines = LOG.read_text(encoding="utf-8").splitlines()
rows = served_rows(lines, load_design("../hs/the-tandem-harness/salon.design.json"))
print(len(lines), "call logged, and", len(rows), "row from the answer that passed")
trl = as_trl(rows[0])
print(list(trl), rows[0].family, rows[0].point)
print(json.dumps(trl["completion"]))
print("the prompt is the logged prompt, byte for byte:", trl["prompt"][0]["content"].encode() == json.loads(lines[0])["prompt"].encode())1 call logged, and 1 row from the answer that passed
['prompt', 'completion'] salon 010
[{"role": "assistant", "content": "{\"choice\":\"Tuesday 3pm\",\"reply\":\"Monday's full, sorry. Tuesday 3pm is open. Want it? The price isn't on my sheet yet.\"}"}]
the prompt is the logged prompt, byte for byte: TrueThe chat template is part of the prompt too. A trainer renders these messages through the tokenizer’s chat template, so training has to use the template, the roles and the system message serving used, or the tokens still differ.
The log keeps the prompt’s text, not the role it traveled in, so the row’s one user message has to match how serving’s adapter sent it. Logging the messages as sent, roles and any system message included, closes that gap too.
The completion is the answer exactly as the model sent it, the choice and the reply together, as the schema shapes them, so a model trained on it learns the shape serving asks for. A rejected answer stays in the log, where a measure can read it, and never becomes a row.
The trail notices below are a teaching catalog that nothing serves, so they have a renderer of their own, trail_prompt. A row for anything served comes from the log.
$ python -m pytest -v tests/test_served_rows.py | sed -n 's/.*:://p'
test_a_row_keeps_the_logged_prompt_and_answer_exactly PASSED
test_a_rejected_answer_never_becomes_a_row PASSED
test_the_rows_point_is_read_back_from_the_voice_in_words PASSED
test_a_prompt_in_no_voice_of_the_design_is_refused PASSEDStructured answers, judged field by field
Overlap with a reference answer measures wording, and an oracle measures meaning. Meaning is easiest to judge when an answer carries it in fields, so each trail answer is a small JSON object: the state, the reason, what the question asked that the notice doesn’t say, and the sentence the reader sees.
parse_answer reads the object into an Answer or says why it isn’t one, the rule Alexis King calls Parse, don’t validate. The oracle compares the fields exactly and reports every miss at once. The sentence is free to move with the voice, so the oracle asks one thing of it: that it never states the state the notice rules out.
# pipeline/data/oracle.py
"""A structured answer, judged field by field: the state, the reason and what isn't known."""
from __future__ import annotations
import json
from dataclasses import dataclass
from enum import Enum
from pipeline.data.negation import stated
FIELDS = ("state", "reason", "unknown", "say")
@dataclass(frozen=True)
class Answer:
state: str
reason: str
unknown: tuple[str, ...] # what was asked and never supplied
say: str # the sentence the customer reads
def parse_answer(raw: str) -> Answer | str:
"""The answer in its most precise form, or why it isn't one."""
try:
data = json.loads(raw)
except json.JSONDecodeError:
return "not JSON"
if not isinstance(data, dict) or sorted(data) != sorted(FIELDS):
return f"fields must be exactly {', '.join(FIELDS)}"
if not all(isinstance(data[f], str) for f in ("state", "reason", "say")):
return "state, reason and say are text"
if not isinstance(data["unknown"], list) or not all(isinstance(u, str) for u in data["unknown"]):
return "unknown is a list of text"
return Answer(data["state"], data["reason"], tuple(data["unknown"]), data["say"])
class Miss(Enum):
SHAPE = "not an answer"
STATE = "wrong state"
REASON = "wrong reason"
UNKNOWN = "unknown differs"
RULED_OUT = "states what the notice rules out"
@dataclass(frozen=True)
class Oracle:
state: str
reasons: tuple[str, ...] # every phrasing of the reason it accepts
ruled_out: str # the state the notice rules out: the sentence must never state it
unknown: tuple[str, ...] = () # asked about, and never supplied
@dataclass(frozen=True)
class Score:
misses: tuple[tuple[Miss, str], ...]
@property
def passed(self) -> bool:
return not self.misses
def score(raw: str, oracle: Oracle) -> Score:
"""Every miss at once. The fields are compared exactly; the sentence only for what the notice rules out."""
answer = parse_answer(raw)
if isinstance(answer, str):
return Score(((Miss.SHAPE, answer),))
found = []
if answer.state != oracle.state:
found.append((Miss.STATE, answer.state))
if answer.reason not in oracle.reasons:
found.append((Miss.REASON, answer.reason))
if sorted(answer.unknown) != sorted(oracle.unknown):
found.append((Miss.UNKNOWN, ", ".join(answer.unknown) or "nothing"))
if stated(answer.say, oracle.ruled_out):
found.append((Miss.RULED_OUT, oracle.ruled_out))
return Score(tuple(found))The fields report every miss together, the way the lecture Practical Applications of Functional Programming collects every failure instead of stopping at the first (Accumulate every error with validation).
The sentence check is negation-aware, in the manner of NegEx, Chapman and colleagues’ 2001 algorithm for negated findings in discharge summaries: a trigger such as “not” negates a phrase that follows it within a window of a few words. Here the window is three words, a figure invented for the lesson:
# pipeline/data/negation.py
"""Is a phrase stated, or negated? A fixed window after a trigger, in the manner of NegEx."""
from __future__ import annotations
import re
TRIGGERS = {"not", "no", "never"} # and every word ending in n't
WINDOW = 3 # words after a trigger that it negates (invented for the lesson)
def words(text: str) -> list[str]:
return re.findall(r"[a-z0-9]+(?:['-][a-z0-9]+)*", text.lower().replace("’", "'"))
def is_trigger(word: str) -> bool:
return word in TRIGGERS or word.endswith("n't")
def found(text: str, phrase: str) -> list[int]:
"""Where a phrase starts, matched as whole words: "closed" never hides in "enclosed"."""
ws, target = words(text), words(phrase)
return [i for i in range(len(ws) - len(target) + 1) if ws[i : i + len(target)] == target]
def stated(text: str, phrase: str) -> bool:
"""The phrase appears with no trigger in the window of words before it."""
ws = words(text)
return any(not any(map(is_trigger, ws[max(0, i - WINDOW) : i])) for i in found(text, phrase))from pipeline.data.negation import found, stated
for sentence in (
"The north trail is open.",
"The north trail is not open.",
"It isn't open until Friday.",
"No, the north trail is open.",
"The trail is not, as far as the notice says, open.",
"The picnic shelter reopened.",
):
print(f"{'stated' if stated(sentence, 'open') else 'negated' if found(sentence, 'open') else 'absent':<8}{sentence}")stated The north trail is open.
negated The north trail is not open.
negated It isn't open until Friday.
stated No, the north trail is open.
stated The trail is not, as far as the notice says, open.
absent The picnic shelter reopened.The fourth line is the window doing its job: the “No” answers something else, and sits too far back to count. The fifth is its blind spot. Six words separate its “not” from “open”, which puts the “not” outside the window, so a negated phrase reads as stated.
Beside the oracle, for contrast, sits f1_overlap: token F1 as SQuAD’s scoring script computes it, after lowercasing and dropping punctuation and articles.
import json
from pipeline.data.contexts import CATALOG
from pipeline.data.design import load_design
from pipeline.data.family import build_family
from pipeline.data.oracle import score
from pipeline.data.overlap import f1_overlap
from pipeline.data.rows import said
row = build_family(CATALOG[0], load_design("data/trail.design.json"), "Is the north trail open?")["001"]
reference = json.loads(said(row.completion))
for name, say in (
("reference", reference["say"]),
("says open", reference["say"].replace("closed", "open")),
("reworded", "The north trail stays shut until Friday, since the bridge washed out."),
):
result = score(json.dumps({**reference, "say": say}), row.oracle)
print(f"{name:<10} token F1 {f1_overlap(say, reference['say']):.2f}, passes: {result.passed}",
[f"{m.value} {d!r}" for m, d in result.misses])reference token F1 1.00, passes: True []
says open token F1 0.90, passes: False ["states what the notice rules out 'open'"]
reworded token F1 0.50, passes: True []The sentence that calls the trail open scores 0.90 on token F1 and fails on meaning. The reworded one scores 0.50 and passes: its fields are right, and it never says the trail is open.
The graders are tested before any model sees them. Six tests check that every reference answer passes its own oracle, each field is compared on its own, a sentence that states what the notice rules out fails, a reworded sentence passes, anything but the four fields is refused, and token F1 rates the opposite answer highly.
The first and the fifth are boundary tests in the sense of the lecture Practical Applications of Functional Programming: both branches of the parser, the answer and the refusal:
$ python -m pytest -v tests/test_graders.py | sed -n 's/.*:://p'
test_every_reference_answer_passes_its_own_oracle PASSED
test_each_field_is_compared_on_its_own_and_every_miss_is_reported PASSED
test_a_sentence_that_states_what_the_notice_rules_out_fails_though_every_field_is_right PASSED
test_a_reworded_sentence_passes_when_the_fields_are_right PASSED
test_anything_but_the_four_fields_is_not_an_answer PASSED
test_f1_overlap_is_high_for_an_answer_that_says_the_opposite PASSEDFamilies at every point, and pairs one trait apart
Each notice is answered at all eight points, a family. The fields are the same at every point, and only the sentence moves with the voice. Each trait leaves one mark on it: a detailed answer adds a practical second sentence, a casual one uses contractions, and a cautious one credits the notice.
# pipeline/data/voice.py
"""The notice's voice on three traits, each leaving its own mark, and a style check that reads them back."""
from __future__ import annotations
import re
from pipeline.data.contexts import Context
def say_at(notice: Context, point: str, unknown: tuple[str, ...] = ()) -> str:
"""The sentence at one point: length, register, certainty. Each trait leaves one mark, and none touches a fact."""
detailed, casual, bold = (bit == "1" for bit in point)
verb = "'s" if casual else " is"
text = f"{notice.place}{verb} {notice.state}, because of {notice.reason}."
text = text[0].upper() + text[1:] if bold else f"The notice says {text}" # certainty: cautious credits the notice
for topic in unknown:
text += f" The notice {'doesn’t' if casual else 'does not'} mention a {topic}."
return text + (" Check the posted notice before you set out." if detailed else "") # length
def voice_of(say: str) -> str:
"""The point a sentence's marks put it at, read from the sentence alone."""
detailed = "Check the posted notice" in say # a second, practical sentence
casual = re.search(r"\w['’](s|t)\b", say) is not None # a contraction
cautious = say.startswith("The notice says") # the notice credited
return "".join("1" if mark else "0" for mark in (detailed, casual, not cautious))voice_of reads those marks back from the sentence alone. It is a style check, kept apart from the oracle: the oracle judges what an answer says, and the style check how it says it.
# pipeline/data/family.py
"""One notice answered at every point of its voice, and voice pairs only between neighbors."""
from __future__ import annotations
import json
from pipeline.data.contexts import Context
from pipeline.data.design import Design, describe, hamming, points
from pipeline.data.oracle import Oracle
from pipeline.data.rows import PreferenceRow, SupervisedRow
from pipeline.data.voice import say_at
def trail_prompt(notice: Context, design: Design, point: str, question: str) -> str:
"""The catalog's own renderer: nothing serves these notices, so no serving log holds them."""
return (
f"Answer in a {describe(design, point)} voice, in JSON with the fields state, reason, unknown and say. "
f"The posted notice says {notice.place} is {notice.state}, because of {notice.reason}. {question}"
)
def oracle(notice: Context, unknown: tuple[str, ...] = ()) -> Oracle:
ruled_out = "open" if notice.state.startswith("closed") else "closed"
return Oracle(state=notice.state, reasons=(notice.reason,), ruled_out=ruled_out, unknown=unknown)
def answer(notice: Context, point: str, unknown: tuple[str, ...] = ()) -> str:
"""The same fields at every point; only the sentence moves with the voice."""
fields = {"state": notice.state, "reason": notice.reason, "unknown": list(unknown), "say": say_at(notice, point, unknown)}
return json.dumps(fields, ensure_ascii=False)
def build_family(notice: Context, design: Design, question: str, unknown: tuple[str, ...] = ()) -> dict[str, SupervisedRow]:
name = notice.place.removeprefix("the ").replace(" ", "-")
return {
point: SupervisedRow(
id=f"{name}-{point}",
prompt=(("user", trail_prompt(notice, design, point, question)),),
completion=(("assistant", answer(notice, point, unknown)),),
oracle=oracle(notice, unknown),
family=name,
point=point,
)
for point in points(len(design.factors))
}
def voice_pairs(family: dict[str, SupervisedRow]) -> list[PreferenceRow]:
"""For every point, a pair with each neighbor one trait away: the right voice, then the other."""
return [
PreferenceRow(
id=f"{row.id}-vs-{other}",
prompt=row.prompt,
chosen=row.completion,
rejected=family[other].completion,
teaches="wrong-voice",
oracle=row.oracle,
family=row.family,
point=point,
)
for point, row in family.items()
for other in family
if hamming(point, other) == 1
]import json
from collections import Counter
from pipeline.data.contexts import CATALOG
from pipeline.data.design import load_design
from pipeline.data.family import build_family, voice_pairs
from pipeline.data.negatives import pair_errors
from pipeline.data.rows import said
from pipeline.data.voice import voice_of
family = build_family(CATALOG[0], load_design("data/trail.design.json"), "Is the north trail open?")
for point in ("000", "001", "011", "111"):
say = json.loads(said(family[point].completion))["say"]
print(point, "read back as", voice_of(say), "|", say)
pairs = voice_pairs(family)
print("pairs from each point:", sorted(set(Counter(p.point for p in pairs).values())), "| problems:", [e for p in pairs for e in pair_errors(p)])000 read back as 000 | The notice says the north trail is closed until Friday, because of a washed-out bridge.
001 read back as 001 | The north trail is closed until Friday, because of a washed-out bridge.
011 read back as 011 | The north trail's closed until Friday, because of a washed-out bridge.
111 read back as 111 | The north trail's closed until Friday, because of a washed-out bridge. Check the posted notice before you set out.
pairs from each point: [3] | problems: []A voice pair holds one notice and question, the answer written for the point asked for, and the answer written for a point one trait away. Both pass the oracle, so the pair says nothing about the facts. It parts on style alone, the one thing it means to teach.
Pairs run only between neighbors, three from every point. A pair of neighbors shows which trait owns which mark; a pair two traits apart would move two marks at once.
$ python -m pytest -v tests/test_families.py | sed -n 's/.*:://p'
test_a_family_answers_its_notice_once_at_each_of_the_eight_points PASSED
test_the_style_check_reads_every_point_back_from_its_sentence PASSED
test_every_answer_in_a_family_gives_the_same_fields PASSED
test_voice_pairs_run_only_between_neighbors_three_from_every_point PASSED
test_every_voice_pair_passes_the_oracle_on_both_sides_and_parts_on_style PASSEDNegatives named by the failure they teach
A preference pair teaches whatever separates its two sides, so every rejected answer carries the name of the failure it teaches, and the name is checked. The trail declares three: a wrong state, a made-up detail that fills in what the notice never said, and a wrong voice. A failure nobody declared is an error.
The practice is a published one. Kaushik, Hovy and Lipton had people revise a text just enough to flip its label, so the two versions differ only in what decides it (counterfactually augmented data, ICLR 2020), and Gardner and colleagues test models on the same kind of small, deliberate change (contrast sets, Findings of EMNLP 2020).
# pipeline/data/negatives.py
"""Every rejected answer is named by the failure it teaches, and has to fail for that reason."""
from __future__ import annotations
import json
from pipeline.data.design import hamming
from pipeline.data.oracle import Miss, Oracle, score
from pipeline.data.rows import PreferenceRow, said
from pipeline.data.voice import voice_of
FAILURES = { # the miss each failure must raise; a wrong voice is judged by the style check instead
"wrong-state": Miss.STATE,
"made-up-detail": Miss.UNKNOWN,
"wrong-voice": None,
}
def negative_errors(failure: str, rejected: str, oracle: Oracle, point: str) -> tuple[str, ...]:
"""Empty when the rejected answer fails the way the named failure says it should."""
if failure not in FAILURES:
raise ValueError(f"no failure declared as {failure!r}")
raised = {miss for miss, _ in score(rejected, oracle).misses}
expected = FAILURES[failure]
if expected is None: # a wrong voice
if raised:
return (f"{failure}: the rejected answer gets a fact wrong too, so the pair teaches two things",)
heard = voice_of(json.loads(rejected)["say"])
return () if hamming(heard, point) == 1 else (f"{failure}: {heard} is not one trait from {point}",)
if not raised:
return (f"{failure}: the rejected answer passes, so the pair prefers one right answer to another",)
return () if expected in raised else (f"{failure}: it fails, but never as {expected.value}",)
def pair_errors(row: PreferenceRow) -> tuple[str, ...]:
chosen = score(said(row.chosen), row.oracle).misses
return (*(f"chosen: {m.value} {d!r}" for m, d in chosen), *negative_errors(row.teaches, said(row.rejected), row.oracle, row.point))import json
from pipeline.data.contexts import CATALOG
from pipeline.data.family import answer, oracle
from pipeline.data.negatives import negative_errors
from pipeline.data.oracle import score
from pipeline.data.voice import voice_of
trail, point = CATALOG[0], "001"
right = json.loads(answer(trail, point))
NEGATIVES = (
("wrong-state", oracle(trail), {**right, "state": "open", "say": "The north trail is open."}),
("made-up-detail", oracle(trail, ("detour",)), {**right, "say": right["say"] + " Take the river detour."}), # "Is there a detour?"
("wrong-voice", oracle(trail), json.loads(answer(trail, "011"))),
)
for failure, judge, rejected in NEGATIVES:
raw = json.dumps(rejected)
raised = ", ".join(m.value for m, _ in score(raw, judge).misses) or f"read back as {voice_of(rejected['say'])}"
print(f"{'ok' if not negative_errors(failure, raw, judge, point) else '??'} {failure:<15} {raised}")
reordered = json.dumps({**right, "say": "Because of a washed-out bridge, the north trail is closed until Friday."})
print(*negative_errors("wrong-state", reordered, oracle(trail), point))
try:
negative_errors("rude", reordered, oracle(trail), point)
except ValueError as error:
print(error)ok wrong-state wrong state, states what the notice rules out
ok made-up-detail unknown differs
ok wrong-voice read back as 011
wrong-state: the rejected answer passes, so the pair prefers one right answer to another
no failure declared as 'rude'A negative that fails for no reason at all, its facts merely reordered, is refused: it would teach the model to prefer one right answer over another. A wrong voice must pass the oracle, so its pair teaches the voice and nothing else, and it must sit one trait away.
Find where the fact went missing
When an answer leaves out a fact, trace it the way any pipeline bug is traced: follow the failing request through its steps (the input, the prompt built from it, the answer), and the first step whose text no longer holds the fact, read as whole words, is where the fix starts.
If the prompt never carried the reason, because the code that builds it trims a long notice, the fix goes in that code. A fact the prompt carried and the answer dropped is the model’s to learn.
When the scorer is what’s wrong, rejecting a valid phrasing of a reason, the oracle widens to accept it and its revision changes. Scores under two revisions are never compared, since they come from two different rulers. The revision covers the negation check’s triggers and window too, and RULES, a number bumped by hand whenever the checking code changes, since a hash of the settings can’t see a change to the code that reads them.
EleutherAI’s lm-evaluation-harness asks the same of its tasks: a task’s version goes up with every breaking change, so users can tell its versions apart, and the task guide’s own example is a fix to answer extraction that had underestimated performance.
# pipeline/data/diagnose.py
"""Find where a fact went missing, and version the ruler that judges answers."""
from __future__ import annotations
import hashlib
from dataclasses import replace
from typing import Mapping
from pipeline.data.negation import TRIGGERS, WINDOW, found
from pipeline.data.oracle import Oracle
STEPS = ("input", "prompt", "answer")
RULES = 1 # bumped by hand with every change to the checking code
def went_missing(trace: Mapping[str, str], fact: str) -> str | None:
"""The first step whose text no longer holds the fact, as whole words, or None when the answer kept it."""
skipped = [step for step in STEPS if step not in trace]
if skipped:
raise ValueError(f"trace every step: {', '.join(skipped)} not traced")
return next((step for step in STEPS if not found(trace[step], fact)), None)
def revision(oracle: Oracle) -> str:
"""A short hash of the ruler: the oracle's fields, the negation check's triggers and window,
and RULES, the number that stands for the checking code."""
return hashlib.sha256(repr((oracle, sorted(TRIGGERS), WINDOW, RULES)).encode()).hexdigest()[:8]
def widen(oracle: Oracle, phrasing: str) -> Oracle:
"""Accept one more phrasing of the reason, which makes a new revision of the oracle."""
if phrasing in oracle.reasons:
raise ValueError(f"{phrasing!r} is already accepted")
return replace(oracle, reasons=(*oracle.reasons, phrasing))import json
from pipeline.data.contexts import CATALOG
from pipeline.data.diagnose import revision, went_missing, widen
from pipeline.data.family import answer, oracle
from pipeline.data.oracle import score
notice = "The north trail is closed until Friday, because of a washed-out bridge."
cut = {"input": notice, "prompt": notice[:39], "answer": "The north trail is closed until Friday."} # a prompt builder that trims (invented)
print("prompt cut short:", went_missing(cut, "washed-out bridge"))
print("prompt whole: ", went_missing(cut | {"prompt": notice}, "washed-out bridge"))
print("closed in enclosed:", went_missing({"input": notice, "prompt": notice, "answer": "The lot is enclosed."}, "closed"))
judge = oracle(CATALOG[0])
reworded = json.dumps({**json.loads(answer(CATALOG[0], "001")), "reason": "the bridge washed out"})
for o in (judge, widen(judge, "the bridge washed out")):
print(revision(o), [f"{m.value} {d!r}" for m, d in score(reworded, o).misses])prompt cut short: prompt
prompt whole: answer
closed in enclosed: answer
58154e40 ["wrong reason 'the bridge washed out'"]
1293086d []The tandem harness treats its checker the same way, reading “Tuesday at 3 pm” as 3pm rather than refusing it.
From fine-tune to release follows a model through the whole pipeline around this geometry, from choosing its base to the gate that decides whether it ships.