The analysis starts at exposure, the randomisation stays at login

The rewritten support prompt has been re-run properly this time. Assignment happens at login, hashed on user_id with an experiment salt, and it holds for the full two weeks. There are 181,400 users in the test, 90,600 in control and 90,800 in treatment, and the balance check passes on every covariate anyone thought to look at — the pre-existing attributes such as plan tier, tenure and locale, which a fair split should leave evenly distributed. The result is nothing at all: task completion is 61.9% in control and 62.1% in treatment, a lift of 0.2 percentage points with a confidence interval of [-0.7, 1.1] and a p-value of 0.66.

The team's reading is that the new prompt does not work. The number that decides whether that reading is right is not on the experiment dashboard: 54,300 of the 181,400 assigned users — 29.9% — ever opened the assistant during the two weeks. The other 127,100 logged in, did whatever they came to do, and never touched the feature the experiment was about. They are all in the denominator, and every one of them contributes an outcome that is identical in expectation across the two arms, because nothing different happened to them.

What the 70% do to the estimate#

Dilution has a closed form when the unexposed cannot possibly be affected. The measured difference over everyone assigned is the exposure rate times the effect among the exposed, plus the exposure rate difference times whatever the unexposed do, and when the second term is zero the arithmetic is one multiplication.

exposure_rate = 0.299
effect_among_exposed = 0.030      # +3.0 pp, the quantity of interest
diluted = exposure_rate * effect_among_exposed
print(f"{diluted:.4f}")           # 0.0090 → +0.9 pp over all assigned

A real 3-point improvement for the people who use the feature shows up as 0.9 points over everyone. That is still positive, so dilution alone does not explain a reading of 0.2, but it explains why the reading is indistinguishable from zero: the interval is [-0.7, 1.1], which is 1.8 points wide, and a 0.9-point signal sits comfortably inside it. The test has enough traffic to detect a 3-point effect and nowhere near enough to detect a 0.9-point one. Detecting a third of the effect needs roughly nine times the sample, because required sample size scales with the inverse square of the effect size, and nine times two weeks is four and a half months.

The variance side makes it worse than the point estimate suggests. Those 127,100 unexposed users are not neutral filler — they have their own outcome distribution, and on this product they complete tasks at a different base rate from assistant users, so they add variance to both arms while adding signal to neither.

The three rows are the same fourteen days of traffic, analysed over three populations. The middle row is the correct one and it recovers the effect. The bottom row has the treatment losing by nearly three points, and it is the row a team produces when it does the natural thing.

Why the natural exposure rule inverts the sign#

The natural thing is to log exposure where the feature visibly does something. The assistant renders a suggestion when retrieval returns at least one passage above a relevance threshold, so the instrumentation gets attached at the render call, and a user counts as exposed once they have seen a suggestion. Both arms are logged by the same code path, which feels like it settles the fairness question.

It does not, because the treatment prompt changed the retrieval call. The rewrite added a query-expansion step, which pushed the share of turns clearing the threshold from 71% in control to 84% in treatment. The 13 points of difference are not random users. They are the turns where the original query was vague, underspecified, or phrased in a way that matched nothing — the hard ones. In control those users never render a suggestion and are excluded from the analysis. In treatment they render a suggestion, get counted, and mostly fail to complete their task anyway, because a vague question is hard regardless of which prompt is answering it.

Control's exposed population has therefore been filtered down to its easy cases while treatment's has not, which is why control reads 64.8% and appears to win. The filtering happened after assignment and was caused by the treatment, which is the definition of post-treatment selection, and it reintroduces exactly the confounding that randomisation existed to remove. Randomisation guarantees comparable groups at the moment of assignment and guarantees nothing about groups defined by anything that happens afterwards.

The test for whether an exposure rule is safe is whether the condition could have been evaluated identically in both arms using only information that exists before the branch. "The user opened the assistant" passes: opening it is a user action taken before either configuration produces anything. "A suggestion was rendered" fails, because rendering depends on the retrieval behaviour that the treatment changed. "The user clicked the suggestion" fails harder, since clicking is a response to the output and is exactly the kind of behaviour the experiment is trying to measure.

Logging exposure at the branch#

The implementation that survives review puts the exposure event immediately before the code diverges, in a path both arms execute, with a predicate that reads only pre-branch state.

def handle_turn(user, conversation, message):
    arm = variant_for(user.id, "support-sysprompt-v4")

    # Exposure is logged here: after we know the user reached the assistant,
    # before either configuration has run. Both arms hit this line.
    exposure.log(
        experiment="support-sysprompt-v4",
        unit_id=conversation.id,
        arm=arm,
        at=now(),
    )

    prompt = TREATMENT_PROMPT if arm == "treatment" else CONTROL_PROMPT
    return generate(prompt, conversation, message)

The awkward case is a treatment that adds something control does not have at all — a new "explain this answer" panel, say, that appears only when the model emits a structured rationale. There is no branch-free point at which control can log the same event, because control never computes the rationale. The fix is counterfactual logging: control runs enough of the treatment's trigger logic to know whether it would have fired, logs exposure on that basis, and then discards the result without serving it. The cost is real, since control now pays for computation nobody sees, and on this feature that was an extra 180 ms and about $0.0004 per turn on 90,600 users' traffic. The alternative is an analysis whose two populations were selected by different rules, which is not a cheaper experiment but a worthless one.

Validating the rule while the test is running#

An exposure rule can be correct in the design document and broken in the code, and the diagnostic for that is the exposure rate itself. Under a valid rule the share of assigned units that become exposed should be equal in the two arms, because exposure is caused by things that happened before the branch and the branch was randomised. A gap between them is either a bug or evidence that the rule reads post-branch state.

On this test the exposure rates were 29.8% in control and 30.0% in treatment on 90,600 and 90,800 assigned users, a difference of 0.2 points with a chi-square p-value of 0.34, which is the expected result. The broken rule from the previous section produces 21.4% against 25.9%, a gap of 4.5 points on those sample sizes with a p-value below 1e-40. That check costs one query, runs daily, and is the single most reliable indicator that an exposure rule has drifted from what was declared — more reliable than reading the code, because the code path that logs exposure is frequently three services away from the code path that branches.

Two smaller mechanics matter once the rule is right. A user can be exposed many times over fourteen days, and the analysis needs a single first-exposure timestamp per unit, both because the metric is aggregated per unit and because the novelty diagnostics in a later chapter are indexed on time since first exposure rather than on the calendar. Recording every exposure event and taking the minimum at analysis time is more useful than recording only the first, since the full series also gives the exposure frequency distribution, which is worth having when the treatment might change how often people come back.

The second is the boundary. A user assigned on day 1 who first opens the assistant on day 14 has a fortnight of assignment and one session of exposure, and whether they belong in the analysis depends on whether their outcome can mature inside the observation window. Declaring the exposure window explicitly — exposures counted for the first thirteen days, outcomes collected through day fifteen — avoids an argument at analysis time about a population that is defined partly by when the query happened to be run.

The population that is not wrong, only different#

Analysing everyone assigned is a legitimate design with a name — intention to treat — and it answers a question that sometimes matters more than the one about exposed users. If the decision on the table is whether to ship a change to the whole product and the change might alter how many people adopt the feature, the intention-to-treat estimate is the honest measure of what the company gets, because it includes the adoption effect. A new prompt that is 3 points better for existing users and drives 5% more of the base into the assistant has a larger total effect than the exposed-only analysis will report.

The two are answering different questions and neither is a corrected version of the other. Exposed-only tells you whether the configuration is better for the people who meet it. Intention-to-treat tells you what happens to the population when you ship it. The failure in the opening result was not choosing intention to treat; it was choosing intention to treat by accident, without the sample size that estimate requires, and then reading its null as evidence about the configuration.

There is also a case where exposure filtering is the wrong instinct. If the treatment is expected to change adoption itself — a change to the entry point, an onboarding nudge, anything that alters who arrives — then restricting to the exposed throws away the entire effect and, worse, compares two differently-selected groups again. The rule holds in both directions: filter on the exposure event only when the exposure event is unaffected by the arm.

A practical consequence is that the exposure rule belongs in the same predeclaration document as the metric and the unit, written at the level of the event name and the predicate, before launch. Once results exist, the choice between "all assigned", "opened the assistant" and "saw a suggestion" is a choice between +0.2, +3.1 and −2.9 points on identical data, and no amount of good faith makes that choice afterwards in a way anyone should trust.

With the unit fixed and the population fixed, the middle row of that table is a defensible +3.1 points on the primary metric. The next thing to establish is what else moved while it did, because the same fourteen days also contain the spend per conversation, the tail latency and the safety classifier rate, and none of those were in the predeclaration yet.