Secret managers and per-execution credentials

A secret manager is a service that stores credentials outside your images and code and hands them to workloads that can prove who they are — AWS Secrets Manager, Vault, Google Secret Manager, whichever the platform already runs. Reaching for one is the correct first move on any credential you find in a container image, and it is worth being exact about which property it fixes, because a team that stops there has spent a sprint on one of three.

The properties to keep separate are extraction, lifetime and scope: who can obtain the credential, how long it stays valid, and what it permits. A secret manager addresses the first and leaves the other two exactly where they were, which for a sandbox running model-generated code is where the interesting failure lives.

What moving it to a secret manager actually fixes#

Start with the worst version, which is common enough to be worth showing. Running docker history against an analysis sandbox image prints the layers that built it:

IMAGE          CREATED       CREATED BY                                        SIZE
a91f3c2e...    3 weeks ago   ENV WAREHOUSE_USER=svc_analysis                     0B
<missing>      3 weeks ago   ENV WAREHOUSE_PASSWORD=Qx7!vNr2p...                 0B
<missing>      3 weeks ago   COPY profiles.yml /home/agent/.dbt/profiles.yml    1.2kB
<missing>      6 weeks ago   RUN pip install pandas pyarrow duckdb            412MB

That credential belonged to a service account with SELECT on every schema in an analytics warehouse, which at that point was 340 datasets covering every customer the product had. It was three weeks old, had never been rotated, and was readable by anyone who could pull the image — which included every engineer, every CI runner, every node that had ever scheduled the workload and kept a layer cached, and the image registry's own backups. Extracting it required no exploit and no access to a running container, since docker save and tar -x are sufficient.

Put the warehouse password in a secret manager and the container fetches it at startup using its workload identity, meaning the identity the platform itself vouches for, such as a Kubernetes service account or an instance role, so that no credential is needed to obtain the credential. The credential leaves the image and the registry, which closes the extraction path entirely: pulling the image now yields a pip install layer and nothing else. Rotation becomes possible, because there is one place to change the value rather than a rebuild and a redeploy. Access to the secret becomes auditable, since the secret manager logs every read with the identity that made it.

Now list what did not change. The credential is fetched once, at container start, into an environment variable in a worker process that serves many executions. It is the same credential for every execution on that worker. It carries the same scope, which is SELECT on all 340 datasets. It stays valid for as long as the pod lives.

The blast-radius arithmetic falls straight out of those facts. A worker pod that lives 6 hours and handles four executions a minute serves about 1,440 executions, and if the tenant mix is 180 customers, that credential is present in the same process memory as code submitted on behalf of most of them. One execution that reads os.environ["WAREHOUSE_PASSWORD"] — which requires no privilege at all, since the process was handed the variable — holds a key that works for every other execution on that worker and for every customer's data, for the rest of the pod's life.

The multiplication is worth doing in the open, since it is the number the argument turns on. Four executions a minute is two hundred and forty an hour, and six hours of pod life makes one thousand four hundred and forty executions passing through one process that holds one key. Against a tenant mix of a hundred and eighty customers, that is eight executions per customer on average, so a given pod has almost certainly run code on behalf of every customer you have. Neither the four a minute nor the six hours is a security setting. They are a throughput figure and a deployment cadence, chosen by people who were not thinking about credentials, and the blast radius is their product.

There is a lifetime figure in the table further down that looks like an improvement and needs reading carefully, because two different clocks are wearing the same word. Moving the secret into a manager takes the rotation interval from months to whatever the pod's lifetime is, which is a real gain for the credential considered as a stored value. It does nothing to the interval that matters here, which is how long a single execution's neighbour holds a working key: that was the pod's lifetime before and is the pod's lifetime after, because the fetch happens once at startup and the value sits in the process from then on. When this chapter says a secret manager leaves lifetime where it was, it means the second clock. An execution that lasts a few hundred milliseconds is sharing a process with a credential good for hours, and the ratio between those two intervals is the thing the redesign attacks.

A natural objection at this point is that chapter 3, on default-deny egress, closed the outbound path, so a compromised run cannot send the credential anywhere. That is right, and it does not help much, because the run does not need to send it anywhere. The credential's value is realised in-band: the run queries a different customer's dataset with it and returns those rows as its own output, which travels back through the tool result, into the model's context, and into the answer shown to whoever triggered the execution. Default-deny egress governs sockets, and this path never opens one. The database connection it does open goes to the warehouse, which is on the allowlist because reading the warehouse is the job.

Put that as a question about what the egress policy is able to distinguish and the limit becomes structural rather than unlucky. A query for tenant 4471's orders and a query for tenant 5002's orders are the same protocol, to the same host, on the same port, from the same pod, differing only in bytes inside an encrypted session. The policy's verdict is identical on both, and it has to be, because refusing the second would mean refusing the first. A control that cannot tell two cases apart cannot be the control that separates them, however correctly it is configured — the same objection this topic has raised against a docstring, a self-supplied tenant identifier and a rubber-stamped approval. The property that distinguishes the two queries lives in the credential, since only the credential can be made to work for one tenant and not the other, which is why the fix is in the credential path and not in the network path.

Injecting per execution, scoped to the run#

The redesign moves the credential from the container's lifetime to the execution's, and from the service's authority to the request's. Two pieces do the work: an identity broker, a small service that holds the authority to mint credentials and will do so only when the harness asks on a run's behalf, and the warehouse's security token service — an STS, the component that issues short-lived credentials against a longer-lived role. The scope is derived outside the sandbox, from the request context that already identifies the tenant, and it is never derived from anything the model produced.

Two arrows carry most of the design. The mint call takes the tenant from the harness's own record of who initiated the request, so an injected instruction telling the agent to "use tenant 5002" changes nothing, because the model is not a party to that call. The revoke call at the end means the credential's validity is bounded by the run rather than by its TTL — the time to live stamped into the token, after which it stops being accepted — and the TTL is only the backstop for a run that crashes.

The session policy, which is a permission document attached to a token that can only narrow what the underlying role was already allowed to do, is where "scoped" becomes a checkable statement instead of an adjective.

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow",
      "Action": ["warehouse:Query"],
      "Resource": "arn:wh:analytics:::db/prod/schema/t4471/table/orders",
      "Condition": { "StringEquals": { "wh:RunId": "8f21ac" },
                     "DateLessThan": { "aws:CurrentTime": "2026-08-08T14:35:00Z" } } },
    { "Effect": "Deny", "Action": ["warehouse:Unload", "warehouse:CreateStage",
                                   "warehouse:Write"], "Resource": "*" }
  ]
}

With that token, an execution can read one customer's orders table and nothing else. It cannot read a second table in the same schema, cannot read another tenant, cannot write anywhere, and cannot use the warehouse's own bulk-export mechanism. The explicit deny on Unload and CreateStage is worth its own sentence: warehouse products let a query write its results to an external object store, and that transfer is performed by the warehouse's network rather than the sandbox's, so it walks straight past the egress policy from chapter 3. A credential scoped only by table, with export left permitted, is an outbound path with a SQL interface.

That deny is the one line in the policy that closes something no other chapter in this topic can reach, and the reason is a chain worth having explicitly. The sandbox's egress policy governs sockets the sandbox opens. An UNLOAD opens no socket in the sandbox; it is an instruction to the warehouse, and the warehouse then makes its own connection to an object store from its own network, on its own credentials. Every control between the untrusted code and the internet is therefore behind the request rather than in front of it. Which is why an allow-list that names a table is incomplete on its own: it bounds what can be read and says nothing about where the result may be sent, and those turn out to be separate permissions in every warehouse product that has an export feature. Read the shape once and it generalises — any dependency that will act on your behalf against a third system is an egress path that your egress policy is not in a position to see.

Getting the token into the sandbox without leaving copies is a small piece of plumbing with a real consequence. Writing it to a tmpfs file that the run's process can read, tmpfs being a filesystem that exists only in memory and vanishes when the instance does, with the tmpfs discarded at teardown, keeps it out of /proc/<pid>/environ — which chapter 2, on shared namespaces, showed is readable by anything running beside the process — and out of any core dump, the image of a crashed process's memory that the kernel writes to disk. Passing it over a unix socket from a broker process is stronger still, since the sandbox then never holds a durable copy and the broker can refuse to hand out a second one after the run is marked finished.

The broker itself becomes the most valuable thing in the system, since it holds the authority to mint tokens for any tenant, and it has to sit where the sandbox cannot reach it. In practice that means the broker listens on a socket the harness holds a file descriptor to, or on an address that the egress policy denies alongside everything else, so that the only way to obtain a token is for the harness to ask on the run's behalf. A broker exposed on the sandbox's network — even behind an API key — hands a compromised run the ability to request its own scope, which converts the whole design back into the shared-credential case with a nicer audit trail.

The design is also worth asserting in the same CI suite as the mount table from chapter 4, on filesystem scoping, with two checks that take a second to run: that no environment variable, file or process argument visible from inside the sandbox matches a long-lived credential pattern, and that a token captured from a completed run fails when replayed sixty seconds later. The second check is the one that catches the failure mode where revocation is wired up, logs a success, and is quietly a no-op because the warehouse's revoke endpoint only invalidates future sessions.

The reason that particular failure needs its own test is that every signal available without one points the wrong way. The harness calls revoke, the endpoint returns success, the audit line reads revoked, and the dashboard is green — and all of that is equally true whether the token stopped working or not, because the response is describing the request being accepted rather than the token being dead. The replay test is the only instrument that varies with the thing you care about: it takes the actual token from a finished run, waits, uses it, and reads the answer from the warehouse rather than from your own logs. That is the same discipline as the egress canary in chapter 3, on default-deny egress, where the verdict comes from outside the sandbox, and it is the same reason: a component's report about its own success is evidence about the report.

Three designs, side by side#

The middle column is worth reading as a set of differences rather than as a description. Against the first column it improves extraction, rotation and the number of executions sharing a value, and it moves the rotation clock. Against the same column, one row is word-for-word unchanged — every dataset, every tenant — and one is explicitly written as same. Those two are scope and audit granularity, and they are the two that decide how large an incident is and how quickly you can describe it. So the sprint that moved the password bought four rows of a seven-row table and left the two that answer "how much was reached" and "reached by which run" exactly where they were. That is the same shape the allowlist table in topic 7.5, on tool execution safety, had: a migration everybody agrees with, improving every property except the one that bounds the damage, with the remaining cell only flipping in the third column.

The latency row is the only cost with teeth, and it is manageable rather than free. An AssumeRole-style call typically returns in 40–80 ms, which is a real bite out of a 300 ms end-to-end budget if it happens serially at the front of the request. Minting in parallel with sandbox assignment hides most of it, since assignment from a warm pool takes a few milliseconds and the sandbox has nothing to do until the workload arrives anyway. Caching by scope is the tempting optimisation and it should be resisted for anything but identical (tenant, table, ttl-remaining) triples, because a cache keyed loosely enough to get a good hit rate is a reinvention of the per-container credential with extra steps.

The reason parallelism is a requirement rather than a nicety falls out of putting the mint against the budget. Forty to eighty milliseconds is between about an eighth and a quarter of a three-hundred-millisecond end-to-end target, spent before any customer code has started, and no amount of tuning gets an AssumeRole-style call materially below that, because it is a signed request to another service. Run it serially and the credential design has eaten a quarter of the budget; run it alongside sandbox acquisition, which takes milliseconds and produces an instance with nothing to do until work arrives, and it costs whatever the mint exceeds the acquisition by. The two operations are independent — neither needs the other's output — which is what makes the overlap available, and it is the reason the budget table in chapter 7, on signing off a warm pool, can list the mint and then not subtract it.

Within-scope reads, retroactive effect, and the tenant argument#

The scoping bounds cross-tenant reach and it leaves within-scope behaviour completely open. A run legitimately holding a read token for tenant 4471's orders table can select all 8 million rows of it and return them, and every control in this chapter is satisfied while that happens. If the thing that triggered the run is itself attacker-influenced — a support ticket, an inbound email, a document the customer uploaded — then the correct scope is still the scope of a run doing something you did not want. Bounding blast radius to one tenant is a large improvement over 340 and it is not the same as preventing misuse, and the controls for the second problem are the tool-level ones from topic 7.5, on tool execution safety, rather than anything in the credential path.

It also does nothing retroactively. A run that already read a row and returned it has done so, and shortening the credential's life from six hours to five minutes changes what happens next rather than what happened. If you are asked which of two changes would have prevented a specific past exfiltration, the honest answer depends on what the run held: a per-execution token scoped to the tenant would have prevented a cross-tenant read, and would not have prevented a run from returning data it was entitled to read.

The third limit is procedural and it is the one that erodes designs over time. The mint call's inputs must come from a context the model cannot influence, and the pressure to relax that is constant, because the moment someone wants an agent that can "look up the parent account's data too", the easy implementation is to let the agent name the tenant. The design holds only while the tenant is a fact about the request rather than an argument in a tool call.

Credential lifetime, then, converts directly into incident size: months and every dataset, a pod's six hours and every dataset, or five minutes and one table. Each step down that list costs a piece of engineering — a secret manager, then a broker, then a revoke path somebody tests — and buys a smaller answer to what one bad run reached.

A run that finished cleanly with a revoked token has still left the disk in whatever state it left it in: packages installed, files written into the workspace overlay, a cached copy of the query results sitting in /tmp. The next chapter deals with that residue — what snapshotting before a run and rolling back afterwards actually restores, and which of the effects that run produced are simply standing.