Where the step list cannot be written before the run

Diagnosing a failing test in a repository nobody on the team can still explain is work whose steps cannot be listed before it starts. That is a claim about the work rather than about the person doing it, and drawing that line honestly is what this chapter is for. On one side sits everything a designer can draw as a graph: a fixed set of nodes joined by edges chosen when the file was saved, which is what the earlier chapters of this topic called a workflow. On the other sits work where the argument to the next step is manufactured by the step before it, out of a space that has no list behind it.

Two words carry the weight and both need pinning down first. A step here is whatever unit a design would turn into a node — a command, a function call, one model call — and the granularity is chosen by the designer rather than given by the world. Enumerable means the steps and their order can be written down in advance for nearly every input, which is not the same as their being identical for every input; a rule that demanded sameness would forbid any branch at all. The reason to care is that a criterion is only worth having if it can be refused. A rule that classifies every task as enumerable has stopped measuring anything and become a house style, and the designs it produces fail quietly rather than loudly.

Six commands, and where each argument came from#

test_reconciliation_window fails on CI. The repository is 340,000 lines of Python across nine packages, you joined the team on Monday, and the engineer who wrote the ledger module left in March. Here is what the first four minutes actually consist of, transcribed from a terminal:

$ pytest tests/ledger/test_reconciliation.py::test_reconciliation_window -x
E   AssertionError: expected 3 settled rows, got 0

$ sed -n '108,141p' tests/ledger/test_reconciliation.py
    rows = seeded_ledger(count=3, age_days=2)
    settled = window.settle(rows, as_of=NOW)

$ grep -rn "def seeded_ledger" tests/
tests/conftest.py:88:def seeded_ledger(count, age_days):

$ sed -n '88,104p' tests/conftest.py
    created = datetime.utcnow() - timedelta(days=age_days)

$ git log -S "utcnow" --oneline -- src/ledger/window.py
8ac31f2 replace naive clock with aware clock in settlement window

$ git show 8ac31f2 --stat
 src/ledger/window.py | 6 +++---
 (tests/conftest.py untouched)

Six commands, and the fault is now visible: a commit three days ago made the production clock timezone-aware and left the fixture naive, a fixture being the helper that builds a test's data before the test body runs, and an aware timestamp carrying a timezone where a naive one carries none, so every comparison in settle silently excludes rows whose created is naive.

What matters for this chapter is the shape of that sequence rather than the bug, and the shape is easiest to see by asking, of each command, where its arguments came from. The first command's argument is the test identifier, which was in the failure report. The second command's arguments are a file and a line range, both read off the traceback the first command printed. The third command searched for the string seeded_ledger, and that string did not exist anywhere in the problem statement, in the codebase's documentation, or in anyone's head before the second command printed it.

That third command is where enumeration breaks, and the break is worth taking one link at a time. A designer writing this procedure in advance would have to write the literal text grep -rn "def seeded_ledger" into it. To write that text they would need the identifier. The identifier is a Python function name, so it is drawn from the set of strings a programmer may legally choose, which has no upper bound and no listing. Therefore the argument cannot be present in the design, and therefore the step containing it cannot be present either. Nothing about the fourth, fifth and sixth commands rescues the situation: each one's arguments — tests/conftest.py, then utcnow, then a commit hash — were also produced by the command immediately above it, from the same unbounded space. Nobody designing this procedure in advance holds any of them.

Why bisection is code#

The obvious formulation of all this — later steps depend on earlier results — is too weak, and git bisect, the binary search over commit history that locates the commit at which a passing test started failing, is the counterexample that shows why. Bisection's next probe depends entirely on the outcome of the last one, the sequence of commits it visits is different for every bug, and its length is not known in advance. It is also a fixed algorithm that fits in fifteen lines:

def bisect(commits: list[str], test) -> str:
    lo, hi = 0, len(commits) - 1          # lo is known good, hi is known bad
    while hi - lo > 1:
        mid = (lo + hi) // 2
        checkout(commits[mid])
        if test():                        # the only input from the world
            lo = mid
        else:
            hi = mid
    return commits[hi]

Every branch here is written down. The result of the previous step feeds the next one, and the mapping from that result to the next action is a rule the designer chose: pass means move the lower bound, fail means move the upper bound, and there is no third outcome. The argument to the next step is an index into a list that existed before the run, which is the property the debugging transcript lacked.

The run length is unknown per bug and bounded all the same, and the bound is worth deriving rather than quoting, because the derivation is what lets a designer reuse it. Each probe replaces the interval between the known-good and known-bad ends with one of its two halves, so after one probe the interval is half its original width, after two probes a quarter, and after each further probe half of whatever it was. The search stops when the interval has been narrowed to a single commit. So the number of probes is the number of halvings that take the starting width down to one, and the number of halvings that take a quantity down to one is what a base-two logarithm reports. That is why the bound is log₂ n, which is 18 for a 250,000-commit history.

Written that way the relation answers questions nobody has to run a bisection to settle. Doubling the length of the history adds exactly one probe, because one extra halving is all a doubled interval costs. A history a thousand times longer adds about ten probes, for the same reason applied ten times over. A designer can therefore state the worst-case step count of a bisection over any repository, at design time, from a number they can look up — and that is precisely the property the debugging transcript could not offer, where the worst-case step count is not merely unknown but not a function of anything the designer holds. Bisection is a workflow with a loop in it, and the loop is controlled by code.

The two halves of the criterion#

Putting the transcript and the algorithm side by side gives the criterion two parts rather than one, and it fails a task if either part fails.

The first part is the one the earlier chapters stated: the steps and their order have to be listable in advance. The second part is the one bisection forces into view: the rule that selects the next step from the last result has to be writable. Bisection fails the first and passes the second, which is why it is code — a designer who cannot list the probes can still write the rule that produces them, and the rule is the program. Diagnosing an unfamiliar failure fails both.

The property that defeats enumeration is therefore narrower than dependence. What the debugging transcript has and bisection does not is that the space of possible observations is open, and the response to each observation is a judgment rather than a lookup. The assertion message could have been a missing import, a flaky sleep, a fixture ordering problem, a version skew in a transitive dependency, or a genuine regression in the code under test, and each of those leads to a different next command with different arguments. You cannot pre-compute the table because you cannot pre-compute its keys.

The stumble here is a belief that a defence works, and it is worth naming in its strongest form because it is tempting for a good reason. The belief is that a task whose next step depends on the last result has, by that fact, escaped enumeration. Notice what stays constant across bisection and the debugging session: in both, the run length is unknown before the run, and in both, every step is chosen from what the previous step returned. Those two properties — the ones a reviewer looks at — do not separate the cases at all. What varies is the property nobody looks at: whether the mapping from a result to the next action is a rule the designer chose from a closed set of outcomes, or a judgment over a space whose members appear during the run. A design review that checks the constant properties and not the varying one will classify bisection and unfamiliar-repo debugging the same way, and will be wrong about one of them whichever way it goes.

Four tasks against the same test#

The middle column is where the test is actually applied, and it rewards being literal. For the invoice job, the argument to the extraction step is the name of a template, drawn from a set of three that exists in a dictionary. For the migration, the argument is a path, drawn from a list produced by git ls-files before anything ran. For the email reply, the arguments are the email body and a customer id, both present at the start. For the debugging session, the argument to command three was def seeded_ledger, drawn from the set of all strings, and it came into existence during the run. Writing the arguments out this way removes most of the ambiguity from a judgment that otherwise gets made on impressions.

Reading the table across rather than down is what makes it teach. The verdict column is not the finding; the gap between the second column and the third is. Three of the four rows name an argument source that a designer holds before the run, and those three rows have a rule in the third column that is a lookup, a constant or nothing at all. The fourth row names an argument source that does not exist yet, and its third column is the only one that says judgment. The two columns move together, which is what it looks like when a criterion has one underlying cause rather than two independent ones.

Three of those four tasks are frequently described as open-ended in a design document, and the third one especially. Drafting a reply feels open-ended because the output is free text with unbounded variety, and output variety has nothing to do with the question. The control flow is: read the email, retrieve the customer's last three tickets, generate a draft, check it against the tone policy, route to a human if the policy check fails. Five nodes, one path, enormous variety in what comes out of node three. The reason the signal misleads is that variety is the property a reader can see without opening the code, while the argument sources are the property that decides the question.

The migration is the other common confusion. Four hundred files is a lot of steps, and long is not the same as unlistable; the list is for f in files: rewrite(f); run_tests(f), and its length is 400 because someone counted. Difficulty is a third false signal. A task can be extremely hard and completely fixed — a numerical solver runs the same steps on every input and the hardness is in the mathematics, not the control flow. All three false signals share a root: marketing language tracks output impressiveness and effort, and the criterion tracks whether the designer can write the graph.

Confident output from an incomplete search#

It is worth building the workflow version of the debugging task in your head, because its failure is instructive and it is not a crash. A plausible five-node design exists and somebody has shipped it:

def diagnose(test_id: str) -> Diagnosis:
    failure = run_test(test_id)                     # 1. get the traceback
    frames  = parse_traceback(failure)              # 2. extract file:line frames
    sources = [read_file(f.path) for f in frames]   # 3. read every frame's file
    history = git_log_for(frames[0].path, days=14)  # 4. recent changes to the top frame
    return model_diagnose(failure, sources, history)  # 5. one call, all context

Run that on the reconciliation bug and it fails in a specific way, and the failure is a chain in which every link is checkable. A fixture runs before the test body and returns a value; by the time the assertion fires, the fixture has already returned, so its frame is gone from the stack. Therefore tests/conftest.py appears in no frame of the traceback, even though the bad value came from it. Therefore step two, which extracts frames, produces test_reconciliation.py and src/ledger/window.py and nothing else. Therefore step three reads those two files and not the file that holds the fault. Therefore step four looks at history for window.py — and here the chain turns, because it does find commit 8ac31f2, which really did change the clock. Therefore step five receives a genuinely suggestive input: a failing comparison in a settlement window, and a recent commit to the settlement window. It produces a fluent, confident, wrong diagnosis: that the settlement window logic introduced in 8ac31f2 has an off-by-one on the age comparison.

Nothing errors. The output has the right shape, cites a real commit, and would be accepted by a schema validator and by a tired engineer at 17:40. That is the characteristic failure of forcing enumeration onto non-enumerable work: not an exception, but confident output from an incomplete search. The property that makes it dangerous is that the coverage gap — the fact that the fault can live in a file no frame mentions — is invisible from inside the run. The graph did exactly what it was told, every node returned, and there is no signal anywhere in the execution that the search was incomplete. So the rate at which it happens is not measured, because measuring it would require knowing the right answer, and every run already looks like a success.

Testing the criterion on fifty real cases#

Where the boundary moves#

Recognising that a task is not enumerable establishes exactly one thing: a fixed workflow will not be sufficient for it. It does not follow that an unbounded loop is a good solution, and it does not follow that the whole system should become one.

The insufficiency is worth stating precisely, because it is easy to overclaim. A fixed workflow applied to unfamiliar-repo debugging would not fail loudly; it would produce a plausible answer from whichever three files it was told to read, and be wrong on the cases where the fault was somewhere else. That is the failure mode of forcing enumeration onto work that resists it — silent coverage gaps rather than errors — and it is why the fallback instrumentation from chapter 3 — the counter that records every input a fixed graph could not handle — is not a universal answer. It only works where the graph knows it has failed. When the fallback rate is 60%, the counter has stopped being a measurement and started being a description of the design. That figure is a rule of thumb rather than a derived threshold, and what makes it usable is the direction rather than the number: a counter that fires on most inputs is describing the shape of the work, not measuring an exception to it.

The boundary also moves, in both directions, and the direction it moves more often is toward enumerability. The first three months of any debugging-assistant project look non-enumerable because nobody has looked at 200 real cases yet; by the time somebody has, it usually turns out that 70% of them are one of five patterns — a dependency version mismatch, a fixture leak, a timezone or encoding assumption, a test-order dependency, a genuine regression — and five patterns is a router. The remaining 30% stays genuinely open. That split is a general expectation drawn from how such projects have tended to go, and not a measurement anyone took on this task; treat it as a reason to run the fifty-case exercise rather than as a substitute for running it. That shape, a large enumerable head and an irreducible tail, is what most real domains look like once measured, and it is the reason the choice is rarely whole-system.

It moves the other way too. A pipeline that was correctly enumerable for two years becomes non-enumerable when the business changes underneath it, and the signal for that is the fallback rate climbing without the reason codes clustering. Nothing about the design decayed; the domain did.

What remains genuinely unsolved is the middle of the distribution. For the clearly enumerable head, fixed control flow is straightforwardly correct, and for the clearly open tail, model-controlled iteration is the only thing that works at all. In between sits a class of tasks where the step list exists but is large and thinly populated — perhaps forty distinct patterns, each occurring a few times a week — and neither approach fits well. A forty-node graph is unmaintainable, and an unbounded loop over the same domain wastes most of its iterations rediscovering things the graph already knew. The current state of the art is to enumerate what you can measure and leave the remainder to iteration, which is a division rather than a solution, and it is the design most production systems converge on for reasons that are practical rather than principled.

The two halves of the criterion are what a design review can actually check. Ask where the argument to each step comes from, and ask whether the rule that picks the next step could be typed out. A task fails the test only when both answers come back empty, which is why so much work that feels open-ended — long, hard, or various in its output — turns out to be a graph somebody had not yet drawn. When both answers do come back empty, the honest conclusion is narrow: this part of the work cannot be scheduled in advance, and a fixed graph laid over it will return confident answers from an incomplete search rather than telling you it failed.

That division is the actual move. Having found the part of the work that cannot be scheduled in advance, the task is to contain it, and containment has a specific shape: one node of an otherwise fixed graph, with its own tool set and its own budgets, while everything around it keeps the properties that made it testable.