Shadow Compiles: How We Test the Future Without Breaking the Present

Every product that replaces a core workflow faces the same problem: how do you test the new workflow without disrupting the old one? The answer at Olmex is the shadow compile — a parallel synthesis pipeline that runs the same inputs through a new model, produces the same output format, and compares — without ever delivering the shadow output to a user.

The Two-Picture Architecture

Every week, the compile pipeline builds two Operating Pictures:

Production: The existing raw-signal prompt. Unchanged, undelivered, the baseline that customers trust.

Shadow: The new world-model-first prompt. Per-subject current belief + trend + this week's delta + evidence refs, then the week's signals. Persisted as digests.kind='shadow', run through the same grounding gate, never delivered, excluded from every user-facing query path.

Both pictures are grounded. Both are cited. Both pass the evidence gate. But they are built from different substrates: one from raw signals, one from accumulated beliefs.

The Diff and Review

Per cycle, the system logs: - Shadow vs. production grounding rejections: which claims did each pipeline block? - Brief-set overlap: which items appear in both, which only in shadow, which only in production? - Prompt tokens, cost, duration: is the shadow pipeline cheaper or more expensive? - Judgment precision: if feedback exists, which pipeline's items are rated more useful?

The founder reviews these diffs side by side. Not as a product feature --- as a validation gate. The shadow pipeline does not ship until three conditions hold for two consecutive cycles: 1. Shadow grounding-rejections + corrections ≤ production baseline 2. Judgment precision ≥ production baseline 3. Founder prefers shadow output

The Budget Discipline

Shadow compiles run in the existing batch pipeline (Message Batches API, 49% discount) with their own token budget and a per-org cap. If the budget is hit, shadow is skipped and logged. The production compile is never delayed or starved by shadow.

This is not A/B testing. It is not a canary. It is a parallel validation pipeline with strict resource isolation and explicit cutover criteria.

Proving a new pipeline is better before it ships is how a persistent model — requirement four of The Olmex Standard — gets replaced without breaking the trust it took months to build.

The Honest Position

Olmex does not ship the shadow compile because it is newer. We ship it when it is better — measured, gated, and preferred. The raw-signal pipeline continues to run until the shadow pipeline proves itself. There is no hard switch, no flag day, no "trust us, the new model is better."

The shadow compile is how we test the future without breaking the present. It is slower than shipping and iterating. It is also the only way to replace a trust product without losing trust.