Part 2 · Lesson 12 of 12 · about 15 minutes

Fair baselines for evals

A score says nothing until you know what it is being compared with.

A score comparison is a diff with one number

An eval is a script that runs your code on fixed inputs and prints a score. When an agent's branch claims "the eval went up," that claim has two ends, like a diff: the score on the branch, and the baseline you subtract from it. The branch end is easy. The mistake is at the other end.

The tempting baseline is whatever main scores today. But if main has moved since the branch split, that is the two-dot comparison from Lesson 10 with numbers in place of lines. GitHub's docs say of the two-dot diff that it "changes when the base branch is updated, even if you haven't made any changes to the topic branch."4 A score difference taken that way does the same. It mixes the agent's change with everything else that landed on main.

The last two rows are both fair, and they answer different questions. This lesson's exercise gets a number for each row.

A small eval for penguins-lab

The code under test is a cleaning pipeline built on the drop_missing function you wrote in Lesson 2. Two of its steps don't do anything yet:

import pandas as pd
from clean import drop_missing


def fix_species(species: pd.Series) -> pd.Series:
    """Return species names in one spelling, such as 'Adelie'."""
    return species


def fix_sex(sex: pd.Series) -> pd.Series:
    """Return sex labels as 'male' or 'female'."""
    return sex


def clean_penguins(df: pd.DataFrame) -> pd.DataFrame:
    """Drop unusable rows, then tidy the labels."""
    df = drop_missing(df, ["bill_length_mm"])
    df["species"] = fix_species(df["species"])
    df["sex"] = fix_sex(df["sex"])
    return df

The eval feeds the pipeline ten messy rows and compares what comes out with a table that was checked by hand. A row counts as right if it matches the expected row exactly, or if it was meant to be dropped and is gone.

"""Score clean_penguins() against a hand-checked expected output."""
import pandas as pd
from pipeline import clean_penguins

COLUMNS = ["id", "species", "sex", "bill_length_mm", "body_mass_g"]

MESSY = [
    (1, "Adelie", "male", 39.1, 3750),
    (2, "Adelie", "female", 39.5, 3800),
    (3, "adelie ", "male", 40.3, 3250),
    (4, "GENTOO", "female", 46.1, 4500),
    (5, "Gentoo", "MALE", 50.0, 5700),
    (6, "Gentoo", "F", 45.5, 4650),
    (7, "Chinstrap", "M", 51.3, 3650),
    (8, "Chinstrap", "female", None, 3700),
    (9, "Chinstrap", "female", 46.5, 3500),
    (10, "Adelie", "male", 38.9, 3600),
]

# Checked by hand: what each row should look like after cleaning.
# Row 8 has no bill length, so it should be gone.
EXPECTED = {
    1: ("Adelie", "male", 39.1, 3750),
    2: ("Adelie", "female", 39.5, 3800),
    3: ("Adelie", "male", 40.3, 3250),
    4: ("Gentoo", "female", 46.1, 4500),
    5: ("Gentoo", "male", 50.0, 5700),
    6: ("Gentoo", "female", 45.5, 4650),
    7: ("Chinstrap", "male", 51.3, 3650),
    9: ("Chinstrap", "female", 46.5, 3500),
    10: ("Adelie", "male", 38.9, 3600),
}

cleaned = clean_penguins(pd.DataFrame(MESSY, columns=COLUMNS))
rows = cleaned.itertuples(index=False, name=None)
got = {row[0]: row[1:] for row in rows}
ids = [row[0] for row in MESSY]
wrong = [i for i in ids if got.get(i) != EXPECTED.get(i)]

print("wrong rows:", *wrong or ["none"])
print(f"score: {len(ids) - len(wrong)}/{len(ids)}")

Two properties make this eval useful for comparing branches:

• It needs only pandas, which you installed for Lesson 2.
• It is deterministic. There is no randomness and no network call, so the same commit prints the same score every time. Any difference between two runs comes from the code.

It also prints which rows are wrong, so you can see what a change fixed, not just how many.

Do it for real: four runs

You'll build a short history, then run the eval four times, in this order: the tip of main, the merge base, the branch tip, and the branch after updating. The scores printed below came from a rehearsal of these exact steps.7 Yours should match.

1. Put the eval on main. Create the two files above on a branch and land them.

   $ git switch main
   $ git pull
   $ git switch -c add-eval
   #   ...create pipeline.py and eval_clean.py...
   $ git add pipeline.py eval_clean.py
   $ git commit -m "Add cleaning pipeline and its eval"

   Use the Lesson 4 loop: git push -u origin add-eval, gh pr create --fill, gh pr checks --watch, gh pr merge --squash --delete-branch. Then git switch main and git pull.

2. Play the agent. So that every student gets the same numbers, you type the agent's change yourself. In pipeline.py, change the last line of fix_species to:

       return species.str.strip().str.title()

   $ git switch -c normalize-species
   #   ...edit pipeline.py...
   $ git commit -am "Normalize species names (eval: 7/10)"

   The commit message carries the score the agent reports for its own branch: 7 out of 10.

3. Meanwhile, main moves. A second session fixes the sex labels and its pull request merges first. In pipeline.py, change the last line of fix_sex to:

       return sex.str.lower().str[0].map({"m": "male", "f": "female"})

   $ git switch main
   $ git switch -c normalize-sex
   #   ...edit pipeline.py...
   $ git commit -am "Normalize sex labels"

   Land it with the Lesson 4 loop, then git switch main and git pull. normalize-species is now a stale branch.

4. Run 1: the tip of main. This is the baseline most people reach for.

   $ python eval_clean.py
   wrong rows: 3 4
   score: 8/10

   The agent reported 7/10. main scores 8/10. Subtract, and the branch looks like a loss of one row. Write down that verdict: −1. It is wrong, and the next two runs show why.

5. Run 2: the merge base. Check out the commit where the branch split, which git merge-base finds for you,1 in a second worktree, and run the same eval there.

   $ git worktree add ../penguins-lab-baseline -b baseline $(git merge-base main normalize-species)
   Preparing worktree (new branch 'baseline')
   $ cd ../penguins-lab-baseline
   $ python eval_clean.py
   wrong rows: 3 4 5 6 7
   score: 5/10
   $ cd ../penguins-lab

   $(…) runs the inner command first and puts its output, the merge base's hash, into the outer command. So this is Lesson 7's git worktree add <path> -b <branch> with a starting commit on the end: a new branch named baseline "starting at" that commit.2 Stay in the same terminal. An activated Python environment belongs to your shell, so the same pandas runs in the new folder.5

6. Run 3: the branch tip. Don't take the agent's number on trust. Reproduce it.

   $ git switch normalize-species
   $ python eval_clean.py
   wrong rows: 5 6 7
   score: 7/10

   The report was accurate. Compare it with run 2: the branch started at 5/10 and reached 7/10. Rows 3 and 4 left the wrong list. That is +2, and it belongs to this branch alone. It is the score version of a three-dot diff, which compares the merge base with the branch tip.3

7. Run 4: the branch after updating. Bring in main as you did in Lesson 11, then run the eval and the tests.

   $ git fetch
   $ git merge origin/main
   Auto-merging pipeline.py
   Merge made by the 'ort' strategy.
   $ python eval_clean.py
   wrong rows: none
   score: 10/10
   $ python -m pytest -q

   This is what main will score after the merge. Compare it with run 1: 8/10 becomes 10/10, another +2.

8. Remove the baseline. You have the number, so the worktree and its branch can go.

   $ git worktree remove ../penguins-lab-baseline
   $ git branch -d baseline
   Deleted branch baseline (was c3b53d0).

   Lowercase -d is enough. The baseline commit is part of your branch's history, so nothing is lost.6

9. Decide, then ship. The branch improves the score on its own and on top of today's main, and the tests pass. It is safe to merge.

   $ git push -u origin normalize-species
   $ gh pr create --fill
   $ gh pr checks --watch
   $ gh pr merge --squash --delete-branch
   $ git switch main
   $ git pull

Reading the four numbers

• Run 3 against run 1: −1. The unfair comparison. main had gained the sex-label fix, worth three rows, and the stale branch didn't have it. The agent's +2 was buried under a −3 that had nothing to do with the agent. A gain was reported as a loss.
• Run 3 against run 2: +2. Attribution. Both ends lack the sex-label fix, so the only difference between them is the branch's own change.
• Run 4 against run 1: +2. Merge safety. Both ends have the sex-label fix, so the difference is what merging adds to main today.

Here the two fair answers are the same number, because the two changes fix different rows. They don't have to match. If one row needed both fixes, it would stay wrong in runs 1, 2 and 3 and come right only in run 4. Attribution would still say +2 and merge safety would say +3. So choose the comparison by the question you are asking: what did this branch do, or what will merging it do.

The same baseline rule applies when the score is a model metric or an LLM eval, and a later course on evals starts from there.

Check yourself

Command drills