Where the credential lives decides more than where the code lives

The requirement arrived as a paragraph in a planning doc: the agent should be able to answer analytical questions about product usage by querying the production read replica directly, instead of waiting on the weekly extract. Three proposals came back within a day of each other. The first was a skill containing the query conventions — which of the four events tables is the canonical one, that account_id in usage_daily is the billing account and not the workspace, that anything scanning more than 90 days needs a partition predicate or it will take the replica down. The second was a tool, run_query(sql), running inside the agent process against a role with SELECT on three schemas and nothing else. The third was an MCP server operated by the data platform team, who already ran a query gateway for the BI stack and were willing to put a tool interface on it.

The review spent forty minutes arguing about which was safest before anyone noticed that two of the three were answers to the same question and the third was an answer to a different one. Sorting that out is easier if you stop comparing them as options and trace one thing through each of them: the secret that ultimately gets presented to the replica.

The skill holds nothing, because it is text#

The skill is a file that gets loaded into the model's context when a query task comes up. It contains no SELECT that anyone runs, no connection string, no service account, and no code path. If it is wrong — if it names a table that was dropped last quarter — the failure is that the model writes a query against a table that does not exist, and something else has to return the error. Nothing in the skill can reach the database, which is why the security conversation about it is short and why it is not in competition with the other two.

That is worth stating plainly because the rest of this chapter is about a decision between two things, and the skill is not one of the two. A capability that produces an effect needs something that can produce effects, and the skill is the part that decides which query is the right one to run. The team ended up shipping the skill regardless of how the execution question resolved, and that is the normal outcome rather than a compromise.

Two shapes for the executable half#

The tool version is the smaller diff by a wide margin. The agent service already has a database connection pool, a secrets mount, and a deployment; the capability is a role, a connection, and a call.

# analytics.py — inside the agent service
_pool = psycopg.pool.ConnectionPool(
    conninfo=os.environ["ANALYTICS_RO_DSN"],   # role: agent_analytics_ro
    min_size=1, max_size=4,
)

MAX_ROWS = 500

def run_query(sql: str) -> dict:
    with _pool.connection() as conn:
        conn.execute("SET statement_timeout = '20s'")
        conn.execute("SET default_transaction_read_only = on")
        rows = conn.execute(sql).fetchmany(MAX_ROWS + 1)
    return {
        "rows": [dict(r) for r in rows[:MAX_ROWS]],
        "truncated": len(rows) > MAX_ROWS,
    }

The DSN in that environment variable — the connection string carrying host, port, database, role and password in one line — is the whole story. It is mounted into the agent's pod from the cluster's secret store, it is readable by any code running in that process, and it is presented to the replica on every connection. The role behind it is genuinely constrained — SELECT on three schemas, statement_timeout enforced server-side as well as here, no write grants anywhere — and the constraint belongs to the role rather than to this function. Anything else in the agent process that opens a connection with the same DSN gets the same reach.

The server version moves that secret across a process boundary and puts an identity in its place.

What the agent holds afterwards is a workload token that says who it is, meaning a short-lived credential issued to a named service by the platform's identity provider, which proves that this pod is agent-support-prod and grants nothing beyond the right to be recognised as it. The database role lives in the data platform team's secret store, is rotated on their schedule, and is never present in the agent's memory at any point. If somebody finds a way to make the agent emit its own environment into a response, the tool version leaks a database credential and the server version leaks a token that is only good for calling one endpoint that the platform team is already rate-limiting and logging.

Five things to follow, not one#

Credentials are the axis that decides this case and they are not the only axis a reviewer needs traced. The useful discipline is to write down five facts for each shape and see which of them actually differ, because most arguments in this space are conducted with three of the five left implicit.

Rotation is where the second and third rows stop being abstract, and it is the cheapest way to feel the difference before you have committed to either shape. The replica's roles are rotated every 90 days under the same control that governs the BI stack. In the tool version, rotation means a new DSN lands in the cluster secret store, and the agent's pods pick it up whenever their projected secret volume refreshes, which on this cluster is up to 60 seconds behind the write. The connection pool, however, is holding four open connections opened with the old password, and those keep working until the old credential is revoked, at which point every checkout from the pool fails until the process notices and rebuilds. The team found this the honest way, during a rotation that produced 11 minutes of password authentication failed in the agent's traces before a pod restart cleared it. The fix is a few lines — catch the auth error, drop the pool, re-read the DSN — and the point is that those lines are yours to write, and nobody writes them until the first rotation.

In the server version, that entire sequence happens inside the data platform team's service and you do not observe it at all. What the agent holds is a workload token with a 15-minute time-to-live, minted by the cluster's identity provider and refreshed by a sidecar, and the failure mode you own is a token that failed to refresh, which surfaces as 401 from the server rather than as a database error. That is a smaller and better-shaped failure, and it comes with a dependency you did not have before: if the platform team's rotation goes wrong, your agent is down and the fix is not in your repository. Neither arrangement removes the failure; they relocate it to a different team's pager and change what your traces are able to tell you at three in the morning.

The row that surprises teams is the last one. Moving the capability behind an operated server does not consolidate the audit trail, it splits it. Their logs know that a caller identified as agent-support-prod ran a query at 14:02 and got 84 rows. Your traces know which customer conversation prompted it and what the model did with the answer. Neither side has both halves, and reconstructing an incident means someone filing a request with the other team, which took the data platform team in this example about a day and a half in practice because their query logs were sampled at 10% and the retention was 14 days. That is not an argument against the server. It is a line item that belongs in the write-up, and it is invisible if you only trace credentials.

The single fact that decides it#

The shallow version of this decision is that production data should never be reachable from the agent process, so the server wins by default and the other two proposals both hand production access to an agent. It sounds like the security-conscious position, and it misreads what the boundary is doing. A read-only role scoped to a replica, with a statement timeout and no write grants, is a real constraint on what any caller can do — and it is exactly as real whether your process holds it or the data platform team's process holds it. The replica does not care which pod the connection came from. Relocating the same role behind a network hop removes no access that the caller previously had, which is the test 6.16 set out for whether a boundary takes something away.

What the server actually buys is that a different team holds the credential, and that is worth something precisely when your team is not permitted to hold it. So the deciding fact is a fact about your environment rather than about the architecture: is your agent's workload allowed to be granted a role on the production replica at all? In some organisations that is a form and two days. In others the replica sits inside a data classification tier where grants are only issued to services owned by the data platform team, reviewed under a control that an auditor tests annually, and no amount of good intentions on your side changes the answer. When the grant is available to you, the tool is the smaller, faster, more observable implementation of the same constraint. When it is not, the server is not a preference, it is the only shape that can exist, and arguing about latency is beside the point.

There are second-order versions of the same fact worth checking before you conclude. If your agent process also holds credentials that must never be combined with replica access in one blast radius — the set of systems an attacker reaches once they are inside a single process — the grant may be available in principle and unwise in practice. If the agent runs on infrastructure that is shared with a tenant you do not control, the process is not really yours to put a secret in. Each of these resolves the same way: they are statements about whether your process is an acceptable place for the secret to sit, and they are answered by your security model rather than by a general principle about agents.

What the trace leaves open#

Following the credential settles which of the two executable shapes fits, and it says nothing about whether the capability should have been converted at all. The data platform team's server existed already, which made it look free, and the same argument gets made routinely for capabilities where nobody has built anything yet and the server is a proposal rather than an endpoint. The proposal is usually framed as a conversion: we have this working as a tool, or as a skill plus a browser, and moving it behind a server will make it reusable and better governed.

Some of those conversions genuinely improve things and some of them quietly delete the part of the capability that was doing the work. The next chapter takes two conversions that look identical on a slide, runs both, and shows why one of them produces a regression that every dashboard in the building reports as healthy.