Fighting an abstraction is a cost, not a differentiator

The pull request was 340 lines and its description opened with an apology. It added a module called run_bridge whose job was to take a document-extraction run that had paused for human approval — sometimes for three days, because the approver is a paralegal with a queue — and resume it inside an agent framework that assumes a run lives and dies inside one process. The bridge serialised the framework's internal step list, wrote it to Postgres, and on resume reconstructed the objects field by field, including two attributes with leading underscores. A comment near the bottom read # this breaks on 0.4.x, check before upgrading.

The engineer who wrote it proposed, reasonably, that this was the fourth workaround of its kind and that the team should build its own orchestration layer. The framework did not fit how they thought about the problem, and every release forced another bridge. Leadership was sympathetic and asked a broader question: five layers were in play for this twelve-engineer document-processing product — the gateway, the vector store, the evaluation platform, observability, and the agent framework — and they were willing to fund building exactly one of them in-house this year. Which one.

The friction is real and it is not evidence#

Working around an abstraction is what using an abstraction feels like. Every team that has used an ORM — an object-relational mapper, the layer that lets you address database rows as objects instead of writing SQL — has written raw SQL for the one query the ORM generates badly; every team that has used a web framework has fought its request lifecycle for the one endpoint that streams. The presence of workarounds tells you that a general tool is being applied to a specific problem, which is the definition of using a general tool. It carries no information about whether the specific problem is one your customers pay for.

That matters because the friction is genuinely expensive and belongs in the analysis — just in the right column. In chapter 1's terms, four bridge modules and an upgrade that is blocked on private attributes are integration and maintenance costs on the buy side, and they should be written into the table as engineer-weeks per year. A framework that costs you three weeks a year in workarounds is a £30,000-a-year line item, and that is an argument for pricing the alternative properly. It is not an argument that orchestration is where this company wins.

Two questions do the discriminating work. The first is whether the constraint is genuinely yours: is there a workload characteristic or a control requirement that no product serves, and that follows from what the product does rather than from how the team prefers to write code? The second is about the third year: when the person who designed it has moved to another team, who extends it, how fast is the surrounding ecosystem moving underneath it, and what does a new engineer have to read to be useful.

Applied to the durable-pause problem, both questions have unwelcome answers. Runs that suspend for days waiting on a human are not an unserved requirement; they are the ordinary case for durable execution engines — the workflow runtimes an earlier topic covered, which persist a run's progress step by step so that it survives a deploy, a crash or a week of waiting — and those have existed for a decade and are bought, not built. The correct move is not to build orchestration but to invert the ownership so the framework stops being the thing that holds the run.

# The workaround: the framework owns the run, so pausing means smuggling its guts through a database
state = pickle.dumps({
    "steps": agent._steps,          # private
    "cursor": agent._cursor,        # private
    "scratch": agent.scratchpad,
})
db.execute("update runs set framework_state=%s, status='awaiting_approval' where id=%s",
           (state, run_id))

# The inversion: a durable workflow owns the run; the model call is one activity among several
@workflow.defn
class ExtractContract:
    @workflow.run
    async def run(self, doc_id: str) -> Extraction:
        pages = await workflow.execute_activity(ocr, doc_id, start_to_close_timeout=MIN_5)
        draft = await workflow.execute_activity(extract_fields, pages, start_to_close_timeout=MIN_5)
        if draft.confidence < 0.85:
            await workflow.wait_condition(lambda: self.approved, timeout=DAYS_7)
        return await workflow.execute_activity(finalise, draft, start_to_close_timeout=MIN_5)

    @workflow.signal
    def approve(self) -> None:
        self.approved = True

The second version is shorter than the bridge it replaces, the three-day pause is a first-class concept rather than a serialisation trick, and the model call is an activity that can be retried, timed out and versioned like any other. The team's requirement was real; the conclusion that it implied building was not, because the market had already solved it under a name nobody had thought to search for.

Scoring the five layers against the same questions#

The exception is the evaluation layer, and the reason is specific rather than sentimental. This product's quality is a claim about documents: whether the counterparty on an amendment resolves to the same legal entity as on its parent contract, whether an effective date expressed as "the first business day following execution" is computed correctly, whether a clause classified as a limitation of liability really is one. A vendor platform can score helpfulness, groundedness, toxicity and instruction-following on any application, and none of those is the question this business is judged on. The labelled corpus of 2,400 documents with adjudicated field-level answers is an asset the company built and no one can sell them.

What gets built is narrower than "an evaluation platform", and the distinction is what keeps this from contradicting chapter 1's arithmetic. That analysis compared two things at parity, and it explicitly did not apply when the two columns are not the same product. Here they are not. The build is a harness: roughly 1,200 lines that load the golden corpus, run the domain scorers, write per-field results to Postgres, and fail a release when field-level accuracy on any document type drops more than 1.5 points. It has no UI, no seat model, no trace ingestion, and no ambition to acquire them. Buying a platform for trace capture and generic scoring while building the domain scorers on top is not a compromise between the two options; it is the shape most teams end up with, and choosing it deliberately at the start saves the year spent arriving at it.

The best case for the worst candidate#

The agent framework is the weakest candidate on that table, so it deserves the strongest argument anyone can make for it. Orchestration is where the product's behaviour actually lives: the sequence of extraction steps, when to escalate to a human, how tool failures are handled, what gets retried and what gets abandoned. A framework imposes its control flow on all of that, and the four bridges are evidence of a mismatch that will keep costing. Owning it would give deterministic replay of a run from its event log, typed tool contracts checked at build time rather than discovered at runtime, and pause semantics designed for this workload rather than smuggled through pickled private attributes. It is also the layer where the team has the most expertise, which means the build estimate is more likely to hold than for any other row.

That case is coherent and it still loses. Orchestration is the fastest-moving part of this stack — the shape of tool calling, structured output, streaming semantics and multi-agent handoff has changed materially every few months, and an in-house framework means absorbing each of those changes with your own engineers instead of upgrading a dependency. For twelve engineers, that is a permanent obligation in a domain where the state of the art is being rewritten by teams much larger than yours. The control it buys is control over machinery that no customer is paying for; the buyer of this product cares whether the effective date is right, not whether the run loop is yours. And the specific constraint that motivated it — the three-day pause — is served by a durable execution engine at a fraction of the cost, which means the requirement was never the unserved kind.

What has to be true a year from now#

A build decision that cannot be falsified is a preference. Four conditions were written down alongside this one, chosen because each can be checked without a debate.

The harness has a named owner who is still on the team, and at least one other engineer has changed it unaided. If the only person who can extend it is the person who wrote it, the third-year question has already been answered badly, and a year is long enough to find that out while the cost of stopping is still one sprint.

It is still small. Under 2,000 lines, no user interface, no seat management, no trace ingestion of its own. Scope growth here is the specific failure mode, because every request that arrives — a web view for the product manager, a comparison across runs, a permissions model — is individually reasonable and collectively reconstructs the vendor product that was deliberately not built.

The golden corpus has grown, from 2,400 adjudicated documents to something above 4,000, and covers the two document types added since. A build justified on a proprietary asset that nobody invests in was justified on nothing, and corpus growth is the cheapest observable proxy for whether the organisation actually believes the argument it made.

It has blocked or reversed at least one shipping decision. A quality gate that has never failed is either measuring nothing or measuring something that never regresses, and both of those mean the harness could be deleted with no effect.

The reverse conditions were written too, and the sharpest is the vendor one: if a mainstream evaluation product ships field-level rubric scoring against a customer-supplied labelled corpus, at a price under £30,000 a year, the differentiating argument evaporates and the correct response is to migrate to it rather than defend the thing already built. Naming that in advance is what makes it possible to act on later without anyone having to admit an error.

What this test does not license#

It is not "build the thing your team is best at". Skill concentrated in a layer makes the build cheaper without making it worth doing, and a team of strong distributed-systems engineers can build an excellent gateway that their customers will never notice.

The differentiator question also has a blunt probe that settles most arguments quickly: would a customer, told plainly what you built and why, pay more for it, or choose you over a competitor because of it? A legal operations director choosing between two contract-extraction products will care that field-level accuracy on effective dates is measured against 2,400 adjudicated documents and reported per release, because that is a claim about whether the product works. She will not care which orchestration library sequences the extraction steps, and if she did care it would be a signal that something is wrong. The probe is not infallible — infrastructure that lowers unit cost can differentiate on price without a customer ever hearing about it — but it separates "this would be better engineering" from "this would be a better business" more reliably than any amount of internal enthusiasm.

It is also not a permission slip for a second layer. The scoring above produced exactly one build, and the discipline is that the number stays one — each additional in-house component adds a maintenance obligation that outlives whoever argued for it, and the failure mode of small teams is not choosing wrong once but choosing to build three times over four years and staffing none of them properly.

And it does not settle the question forever. What it settles is this year, under these conditions, with these vendors, at this team size. That is worth stating precisely, because a decision without its conditions written down cannot be checked later, and the conditions here are unusually easy to name: twelve engineers, a domain-specific definition of correctness, a durable execution engine that solved the pause, and vendor evaluation products that score generic dimensions. If a vendor ships field-level rubric scoring against a customer-supplied corpus next spring, the build stops being justified, and someone should notice.

Nobody notices by accident. The next chapter is about the record that makes noticing possible, and about why the one currently sitting in the repository for the retrieval layer cannot support any review at all.