Alternatives to a credential store

The previous chapter left an argument unfinished. A server that stores tenants' refresh tokens and can decrypt any of them on demand yields all of them to one remote code execution, and none of the application-layer controls changes that. The standing counter-argument is that the alternative does not scale, and it is a fair one, because an objection that produces no design is indistinguishable from a delay.

All three shapes below answer the same question — how does the process that runs tool code stop being the process that holds every tenant's credentials — and they differ in what they give up to do it. The numbers attached to them are the point of the exercise: a costed alternative can be compared to the design it criticises, and an uncosted one cannot.

Three ways to stop holding the credentials#

A cell per tenant keeps the design exactly as it is and changes the deployment unit. Each tenant gets its own process, or its own small group of processes, holding only that tenant's grants, with routing by tenant at the edge. Attacker code inside a cell reaches one tenant's credentials because there is nothing else in the address space or in the pod's IAM role. Nothing about the application code changes, which is its main attraction: the adversarial suite, the per-request identity and the scoped credential store all stay, and their remaining job is to prevent bugs rather than to be the only thing standing between one tenant and forty.

Credentials presented at request time removes the store instead of partitioning it. The tenant's own client holds the downstream credential and passes it with the call, so the server has it for the duration of one request and never writes it anywhere. This is the strongest of the three against a process compromise, and it is the most restrictive: nothing can run when the user is not there, which rules out the nightly sync, the webhook reconciliation and the scheduled digest.

A broker splits the difference by separating the two jobs the current process does. One small service holds the refresh tokens and does nothing else — no tool code, no model-influenced input, no third-party parsers in its dependency tree. It exposes a single operation: present a validated identity assertion, receive a short-lived access token scoped to that tenant and that provider. The tool process holds nothing reusable, because everything it obtains expires in minutes and is confined to one tenant.

The cell numbers assume a 0.25 vCPU / 512 MB pod per tenant on shared nodes at roughly $9 per month each, which is where the "does not scale" objection usually points. It is a weak objection at forty tenants and a strong one at four thousand, and the shape that answers both is a pool of cells rather than one per tenant: route tenants by hash into, say, sixteen cells, and the blast radius of one compromise becomes the tenants in that cell instead of all of them. Placing the largest or most regulated tenants in cells of their own and pooling the rest is the arrangement most teams converge on, and it makes blast radius a dial rather than a binary.

The compute figure is one multiplication, and doing it in the open is what makes the objection answerable in either direction. Forty tenants at roughly nine dollars a pod each is about three hundred and sixty dollars a month, which is the lower end of the band in the table; the upper end is the same multiplication with headroom for the tenants whose traffic will not fit inside a quarter of a core. Set against a single engineer-day, that is not a number anyone declines. The same multiplication is what makes the objection real further out, because the cost is linear in tenants: nine dollars against four thousand tenants is a monthly figure two orders of magnitude larger, and at that point nobody is being rhetorical.

Pooling turns that linear cost back into a fixed one, and what it does to blast radius is a division. Hash tenants into a fixed number of cells and each cell holds the tenant count divided by the number of cells, so sixteen cells over forty tenants holds two or three tenants apiece, and one compromise reaches those rather than all forty. Add tenants after that and the number of deployments does not move while the count inside each cell does, so the operational cost stops growing and the blast radius resumes growing. That trade is what the dial is actually setting, and giving the largest or most regulated tenants a cell each is the same dial turned all the way down for the tenants where the divisor matters most.

The scaling cost that actually hurts in the cell design is not compute. It is that forty deployments drift, that a migration now runs forty times with forty chances to fail halfway, and that a per-tenant configuration error is invisible until that tenant complains. Teams that run this well run it with no per-tenant configuration at all — identical images, tenant identity injected at start, one control plane, the service that creates, upgrades and destroys cells as distinct from the cells that serve traffic — and getting there is a quarter of platform work, not a sprint.

The "no" in the unattended-work row of the middle column is not a policy choice, and it is worth deriving because teams reliably try to argue with it. If the only copy of a credential arrives with a request and is discarded when the response is sent, then at any instant when no request is in flight the process holds nothing that could act for anybody. A nightly sync runs when no request is in flight. The restriction is therefore not imposed on top of the design; it is the design read from the other side, and it is the same property that makes the row above it say the yield is the credentials of users active during the compromise. Those two cells are one fact stated twice. Any feature that must run while the user is absent needs something stored, which is what moves it back into one of the other two columns.

What the broker actually does#

The exchange is a standard one. RFC 8693 defines token exchange for exactly this: present a token proving who is asking, receive a different token scoped for a downstream use.

POST /oauth2/token HTTP/1.1
Host: broker.internal
Content-Type: application/x-www-form-urlencoded

grant_type=urn:ietf:params:oauth:grant-type:token-exchange
&subject_token=eyJhbGciOiJFZERTQSJ9...        # the caller's validated MCP token
&subject_token_type=urn:ietf:params:oauth:token-type:access_token
&audience=https://acme.my.salesforce.com
&scope=sobjects.read

{
  "access_token": "00D5f000...",
  "issued_token_type": "urn:ietf:params:oauth:token-type:access_token",
  "token_type": "Bearer",
  "expires_in": 600,
  "scope": "sobjects.read"
}

The broker derives the tenant from subject_token rather than accepting it as a parameter, which is the same rule that governs the tool server and matters more here, since this service is the one place that can produce a credential for anybody. It refuses to issue anything with a lifetime longer than ten minutes, it will not return a refresh token to the caller under any circumstances, and it rate-limits by tenant so that a caller asking for forty tenants inside a minute is stopped by an ordinary quota rather than by anyone noticing.

Those three refusals do different jobs, and separating them matters because a broker with two of the three buys much less than one with all three. Deriving the tenant from the presented token rather than from a parameter is the get_helpdesk(identity) rule from the chapter on credentials as arguments, moved across a network boundary: the set of credentials reachable from one call is one, whatever the caller writes into the request body. The short lifetime — expires_in is counted in seconds, and six hundred seconds is ten minutes — means anything stolen decays without anyone having to notice it was stolen. And refusing to hand back a refresh token is what makes the second refusal binding rather than advisory, because a caller holding a refresh token can mint its own access tokens for as long as it likes and the ten-minute ceiling stops applying to it. Drop that third refusal and the other two are decoration.

What an RCE in the tool process is worth under this arrangement is different in three ways that all matter. The attacker can mint tokens only for tenants whose users are currently presenting valid MCP tokens, so the yield is the active set rather than the whole book. Each token expires in ten minutes, so continued access requires continued presence inside the compromised process, which is a much noisier thing to maintain than a stolen refresh token used from elsewhere. And revocation works: disabling the broker or rotating one tenant's grant ends access within the TTL, whereas revoking eighty refresh tokens scattered across forty third-party systems is forty support conversations.

The first of those three deserves a shape, because "the active set rather than the whole book" sounds vaguer than it is. Under the credential store the yield is every tenant that has ever connected, and it does not depend at all on when the compromise happens — the store holds the same eighty tokens at three in the morning as at midday. Under the broker the yield is whatever fraction of tenants presents a valid token while the attacker is inside, so the answer is the store's number multiplied by that fraction, and for a product used during working hours by a minority of its customers at once the fraction is small. What has changed is that time is now a term in the answer, where before it was absent. That is precisely why the patient harvest below is the right objection to raise, and why it is an objection about duration rather than about whether the arrangement works.

The honest limitations are two. The broker is itself a target, and the argument for it rests entirely on it being a small, boring service — if someone adds a tool to it, or a template renderer, or a dependency that parses untrusted input, the concentration has simply moved. And an attacker who is patient can harvest slowly: sitting inside the tool process for a week, minting for each tenant as they arrive, reconstructs a large fraction of the coverage that the bulk theft achieved in two seconds. The difference is that the slow version has to survive a deploy, leaves a week of broker logs, and trips the rate limits if it hurries. That is a real improvement and it is not immunity, which is the right way to describe it in a document someone will sign.

When a refusal is the right answer, and what can wait#

A refusal is warranted when the stored grants carry write access to systems where a write is irreversible — moving money, sending mail to a tenant's customers, changing records that a regulator audits — or when tenants are individually large enough that one of them being breached ends the company. There the condition is cells for those tenants, with the pooled arrangement for the rest, and the argument is not that shared processes are unacceptable but that the concentration in this particular process is worth more than the controls on it can be.

What can reasonably be accepted as follow-up work is the structural half: per-tenant search indexes rather than filtered ones, moving enforcement into a gateway, splitting the pooled cells further as tenant count grows. Those are quarters of work, they improve the picture, and none of them is a precondition for a second customer in the way that removing the reusable credential is.

The reason all three shapes stay available is a property of the code rather than of the architecture. A server built with credentials as arguments, from a store keyed on identity, can move behind a broker by changing one function's implementation; a server that reaches for a module-level client cannot move anywhere at all until that is fixed first. The design work from the chapter on credentials as arguments is what keeps these options open, which is the practical argument for doing it before there is an incident to justify it.

Almost nobody gets to design this from scratch, so the useful first move on an existing server is small. Write the blast-radius sentence for it as it stands today, without changing anything, and see whether you can. If the sentence needs a qualifier — "assuming the gateway does not pool connections", "assuming no path uses the startup client", "assuming the cache keys include the tenant" — each qualifier names a chapter of this topic and a place to look. Run the audit hook over one tool call and reconcile the outbound destinations against the query log. Add the positive control to the isolation suite and break a tenant predicate on purpose to see whether the suite notices.

That sequence takes about a week on a server of moderate size, and it ends with a written statement of what one compromise costs, which is the artefact every later decision here gets made against — including the decision about somebody else's server, which is a different problem with the trust boundary drawn on the other side.