Power gives you one day; the week and the novelty give you the rest

Someone puts the numbers into a sample-size calculator before the launch review: baseline task completion 62%, minimum detectable effect 2 percentage points, alpha 0.05, power 0.80. Alpha is the false-positive rate the test is willing to run at, power is the probability of detecting an effect of the stated size when one is genuinely there, and the minimum detectable effect is the size in question. It returns 9,145 per arm. The feature serves about 20,000 sessions a day split evenly, so that is 10,000 per arm per day, and the calculator's answer is that the test needs 22 hours. The proposal on the table is to run it Tuesday to Wednesday and decide Thursday morning.

from math import sqrt
from scipy.stats import norm

def n_per_arm(p1, mde, alpha=0.05, power=0.80):
    p2 = p1 + mde
    z_a, z_b = norm.ppf(1 - alpha / 2), norm.ppf(power)
    return (z_a + z_b) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / mde ** 2

print(round(n_per_arm(0.62, 0.02)))   # 9145

The arithmetic is right and the plan is wrong, because the calculator was asked one question — how many observations are needed to distinguish 2 points from 0 — and the horizon depends on three more that it was never given.

Two days of traffic is not two days of the product#

Weekday and weekend traffic on this assistant are different populations doing different things. Weekend sessions are 22% of volume, they skew toward consumer account holders rather than business users, they are shorter, and they complete at 66.1% against 60.9% on weekdays. A test that runs Tuesday to Wednesday contains no weekend at all, and its estimate is an estimate of the weekday effect.

That would be tolerable if the effect were constant across the week. It usually is not, because the effect is an interaction between the change and the kind of task the user brought. The rewritten prompt in this test helps most on underspecified queries, which are more common at the weekend, so the weekday-only estimate systematically understates the shipped effect. A different change — one that improves handling of long technical threads — would be flattered by the same window.

Running whole weekly cycles is the standard correction and it is nearly free once the horizon is longer than a week anyway. Seven days, or fourteen, starting and ending on the same weekday. The pathological version is a 10-day test, which contains one and a half weekends and weights Saturday and Sunday at 1.5× their true share, producing a mix that matches neither the weekday population nor the real one.

The first three days lie in a predictable direction#

The larger correction is behavioural. When a visible change lands, the users who notice it react to the fact that it is new, and that reaction decays. On this test the lift by exposure window looks like this.

A test stopped at day three reports +5.1 points. The same change, measured after the novelty has worn off, is worth +2.2. Both numbers are correct measurements; only one of them is a measurement of what the product will be like in a month. Shipping on +5.1 and then explaining a quarter later why the metric only moved 2 points is a recurring and entirely avoidable conversation.

The effect runs both ways. Primacy is the mirror image, where established users are slowed down by a change to something they had learned, so the early estimate is worse than the truth and recovers over one to two weeks. Enterprise tools see this more often than consumer ones. Either way the first days are the unreliable ones, and the direction of unreliability depends on whether the change is a novelty to be explored or a habit to be relearned.

The measurement detail that matters is the column header. Slicing lift by calendar day mixes cohorts, because on day 10 of the test some users are on their first exposure and others are on their fourteenth, and the mixture changes as the test runs. Slicing by days since that user's first exposure separates the two, and a treatment effect that is flat on that axis while varying on the calendar axis is telling you about a launch or an outage, not about novelty.

-- lift by tenure in the experiment, not by calendar date
SELECT
  DATE_DIFF(e.exposed_at, f.first_exposed_at, DAY) AS days_since_first,
  e.arm,
  AVG(CAST(o.completed AS INT64)) AS completion
FROM exposures e
JOIN (SELECT unit_id, MIN(exposed_at) AS first_exposed_at
      FROM exposures GROUP BY unit_id) f USING (unit_id)
JOIN outcomes o USING (exposure_id)
GROUP BY days_since_first, e.arm
ORDER BY days_since_first;

If the curve has not flattened by the end of the window, the honest report is that the long-run effect is unknown and the test was too short, which is a more useful sentence than an average over a decaying series.

The outcome has not happened yet#

The primary metric is defined as a resolved status set within 24 hours of the last user turn, which means a conversation that starts at 22:00 on the final day of the window has its outcome determined the next afternoon. Cutting the data at the window boundary censors those conversations as failures and drags the last day's completion rate down in both arms — usually symmetrically, so the effect estimate survives, but not always, because if the treatment resolves faster it has fewer conversations pending at the cut.

The fix is to separate the exposure window from the observation window: exposures are counted for fourteen days, outcomes are collected for fifteen, and the analysis runs on day sixteen. The gap is the length of the metric's own lag. For a completion metric that is a day. For seven-day retention it is seven days, and a two-week test becomes a three-week test, which is a fact worth surfacing before anyone commits to a date.

Randomising at conversation or user level rather than per request also costs power, through the design effect established in chapter 1. Here, with user-level assignment, 20,000 daily sessions come from about 12,500 users, and the intra-user correlation across sessions of 0.12 at 1.6 sessions per user gives a design effect of 1.07. The required 9,145 becomes 9,800 users per arm, and at 6,250 users per arm per day that is 1.6 days rather than 0.9.

Assembling the actual number#

The horizon is fourteen days of exposure, starting on a Monday, with outcomes collected through day fifteen and the analysis run on day sixteen. Power contributed the smallest constraint in the whole table, which is the general case on any feature with meaningful traffic and the reason a sample-size calculator makes such a poor planning tool on its own. On a low-traffic feature the ordering flips and power dominates everything, and then the honest options are a larger MDE, a more sensitive metric, or not running the test.

Going faster without cheating#

Since the horizon is set by the slowest constraint, shortening it means attacking that constraint rather than the sample size. The lever that works on power is variance reduction, and the standard technique is CUPED — controlled experiments using pre-experiment data — introduced by Deng and colleagues at Microsoft in 2013. It adjusts each unit's outcome by its own pre-experiment behaviour on a correlated covariate, which removes the between-user variation that has nothing to do with the treatment. On metrics with a strong pre-period correlation the variance falls 30–50%, and since required sample scales linearly with variance, a test that needed 9,800 users per arm needs 5,400–6,900. That does not shorten a horizon that is set by the weekly cycle, and it very much shortens one set by power.

Nothing similar exists for the novelty constraint, which is a property of user behaviour rather than of the estimator. The closest thing is a long-run holdback: ship the change to 95% and keep 5% on the old configuration for a quarter, which measures decay directly at the cost of maintaining two configurations. Teams that run holdbacks routinely report that a meaningful share of shipped wins are substantially smaller after three months, and the only way to know which ones is to keep measuring.

The other lever is the metric's own variance, which is easy to forget because the calculator asks for a baseline rate and hides the variance inside it. For a binary metric the variance is fixed by the base rate, so there is nothing to choose. For a continuous or ratio metric — spend per conversation, turns to resolution, time to first useful answer — the variance is a property of the distribution, and a long-tailed one destroys power. Spend per conversation on this feature has a mean of $0.021 and a standard deviation of $0.038, driven by a small number of conversations that make forty tool calls, and a test powered on the raw mean needs several times the sample that the same test needs on a winsorised version capped at the 99th percentile. Capping changes the estimand slightly — the estimand being the quantity the test is actually estimating, which after capping is mean spend among the 99% rather than mean spend outright — and it should be declared in advance, and it routinely halves a horizon.

Running longer is not a free hedge either. Over a long window users churn in and out of the assignment pool, other teams ship changes into the same surface, and any cookie-based unit gradually leaks as devices are cleared, all of which pushes the estimate toward zero and can produce a sample ratio mismatch — the arms arriving in the analysis at a split materially different from the one that was configured — which invalidates the test outright. A horizon has an upper end as well as a lower one, and four weeks on a two-week question buys contamination rather than confidence.

When the horizon comes back implausible#

The table above assumes 20,000 sessions a day. A feature serving 600 sessions a day, which is an ordinary situation for a newly launched assistant inside a larger product, needs 9,800 users per arm at 300 per arm per day, which is 33 days before any of the other constraints apply, and closer to seven weeks once whole weeks and novelty are respected. Seven weeks is longer than the change will stay relevant, and pretending otherwise produces a test that gets abandoned in week three and quoted anyway.

The options at that point are all trades and all better than the pretence. Raising the MDE to 5 points cuts the requirement by a factor of six and gives a test that can only see large effects, which is fine if the change is expected to be large and useless if it is expected to be incremental. Moving to a more sensitive primary metric — one that is continuous rather than binary, or measured per turn rather than per session, or with a higher base rate — can buy a factor of two or three, provided the substitute is genuinely a proxy for the outcome and not merely a number that moves easily. Pooling several related changes into one arm answers a coarser question about a bundle, which is often the question the business actually has. Running the test as a Bayesian analysis with an informative prior from previous similar changes shortens the horizon at the cost of making the conclusion depend on a prior somebody has to defend. And deciding without a test is a legitimate option that becomes more legitimate the smaller the traffic: at 600 sessions a day, shadowing the candidate and having three experienced people read 200 outputs may simply be a better use of three weeks than an underpowered experiment that will conclude nothing.

The minimum detectable effect deserves one clarification, since it is routinely misread as a prediction. The 2 points in the calculator is the smallest effect the test can reliably detect, not the effect anyone expects, and it should be set to the smallest effect that would change the decision. If a 2-point improvement would not justify the extra 35% spend from chapter 3, the test should be powered on whatever effect would, and the answer is often that the required horizon is implausible — which is itself a result, arrived at before spending two weeks.

All of this assumes the comparison is between two arms serving different users. There is a way to compare two configurations inside a single user's request, which is dramatically more sensitive and answers a narrower question.