Model-guided experiment loops for screening campaigns

A closed-loop replay of published cell-culture media campaigns: a Bayesian surrogate refit every five experiments reached a top-1% medium in 45–65 experiments where random selection needed 95–105. Pre-registered, with a CLI a client can run on their own CSV.

Domain

R&D screening campaigns, with cell-culture media formulation as the worked case. The code does not know its inputs are amino acids

Decision

Which formulations to run next; how many designs to seed a new campaign with and how often to refit. Scored in experiments saved

Data

Kim & Audet 2019 (Communications Biology 2:48, CC BY 4.0): TF-1 progenitor cells, 2,340 formulations of 15 components across three independent campaigns; primary T cells, 2,124 formulations of 14 components. Narayanan et al. 2025 (Nature Communications 16:6055, CC BY 4.0; data MIT): K. phaffii, three proteins. A fourth arm was run on a CC BY-NC-ND dataset and is withheld from commercial material. All are adaptive-search data: the pools are enriched for good media by the authors’ own search

Methods

Closed-loop replay on a finite pool (seed, batch, reveal, refit); policies: random, greedy and Thompson sampling on a Bayesian linear surrogate, GP-BO with expected improvement; pre-registered comparisons of partial pooling and curvature; CRPS, coverage and PIT scoring; a mechanical sampler health rule on every fit

Tools

Python 3.12, PyMC, scikit-learn. CLI mediamix replay. 51 tests in the public snapshot; 43 dated decisions in the log

Status

Five pre-registration tags; eleven hypotheses with verdicts. Delivered 18 September 2026

Repo

github.com/ellenmarsh1612/mediamix-replay. make install && make test in under a minute; make replay-example fetches a public campaign and writes a replay report in about two minutes

A model-guided experiment loop reached a top-1% medium in 45–65 experiments where random selection needed 95–105, on three published cell-culture campaigns replayed on their own measurements. The pools were enriched by the authors’ own search, and every gain is measured on that basis. The tool that does it takes one CSV and returns a one-page report with that caveat printed inside the headline sentence.

45–65 vs 95–105

experiments to a top-1% medium, a Bayesian linear surrogate refit every five experiments against random selection, three TF-1 campaigns. Campaigns 2 and 3 pre-registered; campaign 1 post hoc.

2 of 3

campaigns on which GP-BO found the campaign’s single best medium in every seed set within 200 experiments. On the third it did so in 6 of 10.

10–20 seeds

Seed a campaign with 10–20 designs and refit every 5, and the model helps from its first batch, at a third to a half of random’s experiments. Seed with 80 and the saving shrinks to a quarter. Exploratory, one campaign.

Problem and decision

A team screening a growth medium with fifteen or more components runs campaigns of a few hundred formulations in batches, choosing at each batch what to run next. The question I set out to answer is the one they ask before committing to a model: would it have chosen better than picking sensibly? The way to answer it without running new experiments is to replay a finished campaign on its own measurements. A policy sees a small seed of designs, picks a batch, is shown the real outcomes, refits and repeats, and the count of experiments it needs to reach a top design is compared with random selection on the same pool.

The second decision is how to design the next campaign: how many designs to seed the model with, how often to refit, and what to run in every batch so that batch-to-batch drift can be removed.

Data and its mess

• Adaptive-search pools. Every dataset was collected by the authors’ own search, so later batches concentrate where their search went, and the pools are enriched for good media. A replay on such a pool measures the saving on the campaign’s own media, and the report says so in its headline sentence.
• Batch offsets. Each generation is normalised to one positive-control value, so every forward fold carries a shared offset that no composition model can forecast. T-cell generation 4 sat 0.16 above the linear model’s forecast, with 46% coverage of its 90% intervals.
• Replicate variance thrown away upstream. The public benchmark’s loader sets each TF-1 row’s replicate variance to zero; the source carries four replicate counts per row. I carry it.
• Total culture failures. Twenty-six TF-1 formulations and three T-cell formulations killed the cells. I keep them at zero; a medium that kills the cells is information about toxicity.
• Coded concentrations. The Kim & Audet campaigns record design codes rather than real units, so saturation cannot be identified from them. The arm with real units is the withheld one.
• A withheld arm. A fourth dataset (Ozawa, Hashizume and Ying 2025, Biochemical Engineering Journal 214:109591) is licensed CC BY-NC-ND 4.0 with no licence on its repository. Its results stay in the private audit record. One observation from it is used in words, cited to the authors: selecting media on a cheap early readout weakens that readout’s relation to the endpoint.

All 43 decisions, with the alternative rejected, are in docs/decision-log.md.

What was tested

Eleven hypotheses across five pre-registration tags, each frozen as a git tag before the fit it governs; the confirmatory model classes refuse to fit until their tag exists. Every fit ran under a sampler health rule (R-hat at most 1.01, zero divergent transitions, bulk ESS at least 400), with an escalation ladder and exclusion, in an environment pinned by lockfile.

Tag	Hypothesis	Observed	Verdict
prereg-v1	H1: partial pooling across campaigns beats complete pooling on cold start by 2% CRPS	1.007 against a bar of 0.893, on the one campaign of three the health rule allowed to score	Fail
	H2: cold-start 90% coverage within 85–95%	99.2%, too wide	Fail
	H3: experiments to 95% of best below 8.0, regret within 1% of pooled linear’s	Both held on the scorable campaign (2.0; 0.031); two campaigns excluded by the health rule, and the rule fixed in advance has no path from incomplete to pass	Not supported
prereg-v2	H4, H5: curvature and saturation on the withheld arm	Withheld	Fail
prereg-v3	H6: curvature ranks the next T-cell generation better in 4 of 4 folds	2 of 4; mean gain +0.008 (90% interval −0.015 to +0.032)	Fail
	H7: curvature CRPS at most 0.95 of linear’s	1.007	Fail
prereg-v4	H9: greedy-linear halves random’s experiments to top 1% on campaigns 2 and 3	Campaign 2: 45 against a bar of 50. Campaign 3: 50 against 47.5, on three surviving seed sets	Fail
	H10: GP-BO at the campaign’s best within 120 experiments in 80% of seed sets, both campaigns	Campaign 2: 87.5%. Campaign 3: 0 of 3	Fail
prereg-v5	H11: H9’s rule on campaign 3 with ten guaranteed seed sets	65 against 95: a ratio of 0.68, bar 0.50	Fail
	H12: H10’s rule on campaign 3 with ten seed sets	1 of 10 by 120; 6 of 10 by 200	Fail

H8 was registered as “reported, no rule” and has no verdict by design. The v5 re-test settled whether campaign 3’s misses were a protocol defect: they were not. It is a harder surface, with its seven top designs in its last two generations.

Two of the bars were missed narrowly on one campaign and clearly on the other; the rest were missed outright. I set the bars at the effect sizes a client would need to justify the method, and the loop’s saving holds at a lower level than the one I asked for.

Results

TF-1 campaign 2, pre-registered: seed 40 designs, batch 5, 200 experiments, top 1% target. Lines are medians over seed sets, bands the interquartile range. Source: results/replay/ path files, listed in site/site_numbers.json.

Campaign	Policy	Seed sets	Median experiments to top 1% (IQR)	At the campaign’s best by 120 / by 200
1 (post hoc)	Random	134	105 (70–150)	14% / 36%
	Greedy, Bayesian linear	8	45 (45–50)	0% / 50%
	Thompson, Bayesian linear	8	50 (49–51)	25% / 63%
	GP-BO, expected improvement	8	60 (49–68)	100% / 100%
2 (pre-registered)	Random	118	100 (75–159)	18% / 48%
	Greedy, Bayesian linear	8	45 (45–51)	100% / 100%
	GP-BO	8	50 (45–53)	88% / 100%
3 (pre-registered, ten seed sets)	Random	128	95 (70–151)	12% / 19%
	Greedy, Bayesian linear	10	65 (51–86)	20% / 40%
	GP-BO	10	78 (58–85)	10% / 60%

Kim & Audet TF-1. Campaign 3’s figures are from the pre-registered re-test with ten guaranteed seed sets, which supersedes the three-seed-set run. Source: site/site_numbers.json, entries replay_campaign1 to replay_campaign3.

First good medium. The linear surrogate refit after every batch reached the top 1% in 45, 45 and 65 experiments across the three campaigns, against 105, 100 and 95 for random selection: ratios of 0.43, 0.45 and 0.68. Random selection failed to find a top-1% medium at all in 9–11% of its seed sets within 200 experiments; the model policies never did.

Best medium. GP-BO reached the campaign’s single best design in every seed set by 200 experiments on campaigns 1 and 2, and in 6 of 10 on campaign 3, where random managed 19%. Two questions, two winners: the linear surrogate for the first good medium, GP-BO for the best one.

Designing the next campaign. Replaying campaign 2 under different seed sizes and refit intervals gives a planning table. With a seed of 10 and a refit every 5 experiments, the model reached the top 1% in 20 experiments against random’s 60; with a seed of 20, 35 against 77.5; with a seed of 80, 85 against 125. Each extra seed well is budget the model would have spent better, and refitting every 5 beats every 10 or 20 at every seed size. Exploratory, one campaign.

Seed	Refit every	Random, median	Model, median	Ratio
10	5	60	20	0.33
20	5	77.5	35	0.45
40	5	100	50	0.50
80	5	125	85	0.68

Run a reference medium in every batch. Every dataset showed batch-to-batch shifts no recipe model can forecast. With a reference well in each batch the shift is a one-line addition to any of these models; without it, it cannot be removed.

What a client sees. mediamix replay --csv campaign.csv --components "A,B,C" --response yield --batch-col plate returns a one-page report: the pool’s shape, the median experiments to a top design per policy with its spread, the share of runs reaching the pool’s best, a regret figure, and a headline sentence that carries the enrichment caveat. On campaign 2 exported as a client file with a seed of 20, the report gives 35 experiments for the linear loop and 40 for GP-BO against random’s 77.5.

What the tests ruled out

• Partial pooling. On cold start, the hierarchical model scored worse than complete pooling on the one campaign the sampler allowed to score and on the two it excluded (descriptive, +0.07 CRPS on each). Within a campaign the pooled and unpooled linear models are within 0.002 CRPS of each other. Three campaigns cannot estimate a between-campaign variance, and a model built to use one cannot show its value without it.
• Curvature. Squared component terms cut cold-start CRPS from 0.911 to 0.770 in exploration and then failed twice under pre-registration: on the withheld arm and on the T-cell campaign, where they ranked the next generation no better than linear. The likeliest account is selection among exploratory comparisons. The surrogate in the loop is linear.
• GP-BO on the first hit. Slower than the linear surrogate on every campaign, 60 against 45 on the first. Exploration has a price at the start.
• GP-BO at the optimum on campaign 3. 0 of 3 seed sets, then 1 of 10 by 120 experiments under the re-test; regret flat from round 8. Its record on the other two campaigns did not replicate there.
• Greedy selection after the first batch. Regret flattens from round 4: the policy that wins the first round does not win the campaign.
• Calibration. Coverage looks right and the PIT test fails for every model; the cause is the fold-wide offsets above, which motivate the reference-medium rule.
• The bars. Every pre-registered threshold was missed. The savings reported here are the ones observed, at the level observed.

Where else this goes

The harness is agnostic to what the design columns are. The same replay, with the same policies and the same report, fits any screen where designs are chosen in batches and outcomes are measured: formulation, fermentation conditions, catalyst screens, assay development, and campaigns outside biology. I ran it on a pool of 504 trading-rule parameter sets scored on US Treasury yields to check that the loop’s behaviour is not a property of cell-culture data; the model policies reached the top 1% in about 30 backtests against random’s 75, and the pool’s own selection proxy failed in the same way the early readout fails in cell culture. That study is semi-synthetic and exploratory, and is written up in the writing section rather than here.

Runs without me

• make install && make test runs in under a minute; make data fetches the Kim & Audet campaigns at a pinned commit and the Treasury series; make replay-example writes a replay report. The demo notebook runs in about two and a half minutes.
• Data are fetched by script and never committed, although the source licence permits redistribution.
• Sampler health is a gate in CI: fits with divergences or R-hat above 1.01 fail the run.
• The client intake guide (docs/replay-intake.md) says what to send and how to read the result.

Limitations

• Every pool was enriched by the campaign’s own search; the saving is measured on the campaign’s own media. A campaign run from a design-of-experiments seed would give different numbers.
• Three campaigns from one laboratory; the T-cell campaign is a near-replication of TF-1. Campaign 1’s protocol was fixed after a pre-fixed probe had stopped the session, so its figures are post hoc.
• Every pre-registered bar was missed. The observed savings are 1.5 to 2.3 times faster to the first good medium, and the reliability at the optimum did not replicate on the third campaign.
• Replay refits run without the sampler health rule and record no diagnostics.
• No component prices exist in any of these datasets, so the decision is scored in experiments, not pounds.
• The withheld arm cannot be used commercially, and the licence on Kim & Audet’s data files, as distinct from the article, has not been checked against the article’s data-availability statement.

Artefacts

• Repository: github.com/ellenmarsh1612/mediamix-replay, a clean public snapshot; the private repository is the audit record.
• One-page memo and technical write-up: docs/memo.md, docs/technical-writeup.md.
• Client intake guide: docs/replay-intake.md.
• Pre-registrations and verdicts: docs/PREREGISTRATION*.md; the decision log, docs/decision-log.md.
• Every number on this page, with its source path and label: site/site_numbers.json.
• Datasets catalogue: Public media-optimisation datasets, one schema.

Positioned against

• Narayanan et al. 2025, Nature Communications 16:6055, report Bayesian optimisation reaching better media with 3 to 30 times fewer experiments than design-of-experiments estimates. This project’s comparator is random selection on an enriched pool, and its saving is 1.5 to 2.3 times to the first top-1% medium; the two numbers answer different questions. Their K. phaffii pools, about 100 designs each, were too small for this replay to discriminate between policies.
• Cosenza et al. 2023, Engineering in Life Sciences 23:e2300005, report 23% more growth at 62.5% of medium cost. No prices exist here, so the cost comparison was not attempted.
• The Amii applied-AI cell-agriculture benchmark hosts the Kim & Audet data and compares optimisers on semi-synthetic surrogate landscapes. This project replays real outcomes only, and departs from the benchmark’s loader on replicate variance and on the T-cell response definition, with the reasons in the decision log.