A vendor's changelog entry is a deploy into your agent

The pipeline review on a Thursday morning was where it surfaced. The sales team's inbound router — an agent that enriches every new signup against a company-data vendor and assigns it to a tier — had put 1,870 of the previous two days' 3,100 signups into the catch-all tier, against a historical rate of about 12%. No alert had fired. The agent's error rate for the window was 0.4%, which was normal. The vendor's status page was green, and it was green honestly, because nothing at the vendor was broken.

At 02:14 UTC on the Tuesday, the vendor had published a changelog entry. It said that lookup_company now returned employee_count in place of employees for consistency with their REST API, that the old field would be removed in the same release rather than deprecated because it had "no documented consumers", and that search_companies now required a dataset argument to disambiguate their new EU-only corpus. The entry was on a docs subdomain with an RSS feed nobody in the company was subscribed to. The team's own repository had not changed in nine days.

Two days of a system working exactly as designed#

The renamed field is the more instructive of the two changes, because nothing about it produces an error anywhere. Before Tuesday, a call came back looking like this:

{
  "domain": "aureliondata.io",
  "employees": 420,
  "employees_range": "201-500",
  "hq_country": "DE",
  "sic_primary": "7372",
  "confidence": 0.91
}

After Tuesday it came back looking like this:

{
  "domain": "aureliondata.io",
  "employee_count": 420,
  "employees_range": "201-500",
  "hq_country": "DE",
  "sic_primary": "7372",
  "confidence": 0.91
}

Both are valid JSON, both are perfectly reasonable objects, and the transport carried the second one exactly as happily as the first. The agent's routing skill said to place an account in the enterprise tier when headcount is 500 or more, mid-market between 50 and 499, and the catch-all tier when headcount is unknown. The model read the new object, found no employees key, treated headcount as unknown, and applied the rule it had been given, which is the correct behaviour for the information in front of it. In roughly a fifth of the traces it did something slightly worse: it inferred a number from employees_range and routed on the midpoint, which produced right answers often enough that the wrongness was not visible even in spot checks.

The new required argument failed more loudly and still not loudly enough. Calls to search_companies came back as tool errors — {"error": "missing required argument: dataset"} — and the model, holding a tool definition it had fetched at session start that did not mention dataset, could not fix them. In 41 of 63 affected conversations it retried once with the same arguments, then fell back to lookup_company on a guessed domain, then continued. The 0.4% error rate the dashboards showed was these calls, and 0.4% is well under any threshold anyone had set, because a small number of tool errors is the normal weather of an agent.

Why none of the usual defences were in position#

An engineer on that team had spent years being careful about dependencies. The service pinned its Python packages with hashes, its container base image by digest, and its internal service clients by generated-code version, and a change to any of those arrived as a pull request with a diff, a CI run, and a human approving it. None of that machinery was engaged here, because the vendor's server was not a dependency in any sense the tooling understood. It was a URL in a config map.

The distinction that matters is not internal versus external, since plenty of external code is managed perfectly well. It is whether a change on the far side can reach production without passing through something you control.

The third column is the one people underestimate when they say a vendor server is "just an API call". A versioned REST API and a vendor-operated MCP server are both someone else's code, and the version in the path is what makes the first one a decision you make on a Tuesday of your choosing. Without something playing that role, the vendor's release calendar is your release calendar. The API-vendor framing in Postman's Claude Skills vs MCP: What's the Difference and When to Use Each? is worth reading precisely on this point, because it is written from the side that publishes the contract rather than the side that consumes it, and it is candid that maintaining a stable interface is a commitment the operator either makes or does not.

The questions that decide this class of dependency#

Whether the capability works today is nearly uninformative, because a vendor demo works today by construction. The useful evaluation is about the interface's lifecycle, and four questions carry most of it.

Whether the server is versioned at all is the first, and it is worth being specific about what counts. A version string in the server's metadata that never changes is decoration. What counts is whether two versions can be live at once, so that you can move between them on your schedule — a /v2 path, a version header the server honours, a per-tenant pin in their console. When the answer is that there is one version and it is whatever is deployed, everything downstream of that question has a worse answer than it looks.

The deprecation policy is the second, and it should be a document rather than an intention. The vendor in this story had one for their REST API — 180 days' notice, email to the account's technical contact — and had not extended it to their MCP server, which was eight months old and treated as a preview surface. That gap is common right now and it is knowable in advance by asking.

The third is how a change is announced and whether the channel reaches a human on your side. An RSS feed is a real channel if someone has wired it into a room where engineers are, and is otherwise a way for the vendor to have been technically correct. The team in this story fixed this part first, in about an hour, by pointing an RSS-to-Slack integration at the docs feed.

The fourth is whether you can exercise a change before it reaches you, which usually means asking for a staging endpoint or an early-access flag. Many vendors have one and do not advertise it.

What to build when the answers are unsatisfying#

You will often consume a server whose answers to those four questions are poor, because the capability is worth it and there is no alternative. The mitigation that pays for itself fastest is treating the tool definitions as a contract you have committed to your repository and diffing them on a schedule, which turns an invisible change into a failing job.

# tests/contract/test_vendor_tools.py — runs hourly in CI, not on PRs
import json, pathlib
from mcp_client import connect

SNAPSHOT = pathlib.Path("contracts/vendor_company_data.json")

def _shape(tools):
    return {
        t["name"]: {
            "required": sorted(t["inputSchema"].get("required", [])),
            "properties": sorted(t["inputSchema"]["properties"]),
        }
        for t in tools
    }

def test_tool_definitions_match_snapshot():
    with connect("https://mcp.vendor.example/sse") as session:
        live = _shape(session.list_tools())
    expected = json.loads(SNAPSHOT.read_text())
    assert live == expected, (
        "vendor tool contract changed; review before the agent sees it"
    )

That job would have caught the dataset argument within the hour, because a new entry in required is a diff. It would not have caught the renamed response field, since tools/list — the protocol call a client makes at session start to fetch the server's tool definitions — describes inputs and, for most servers, says nothing useful about the shape of what comes back. Response drift needs a second, cheaper check: call the tool with a known fixture and assert on the keys you actually consume.

def test_lookup_company_returns_fields_we_consume():
    with connect("https://mcp.vendor.example/sse") as session:
        result = session.call_tool("lookup_company",
                                   {"domain": "aureliondata.io"})
    payload = json.loads(result.content[0].text)
    for field in ("employees", "hq_country", "confidence"):
        assert field in payload, f"vendor dropped {field}"

Two tests, one fixture account, and a scheduled run. The cost is a few minutes of vendor quota an hour and the discipline of updating the snapshot deliberately when a change is real. The value is that the vendor's deploy now arrives as a red build with a name on it instead of as a tier distribution someone notices on a Thursday.

There is a limit to how far this goes, and it is worth being clear about it rather than selling the tests as a solution. Contract tests detect changes in structure. They do not detect changes in meaning, and a vendor that starts returning headcount for the legal entity rather than for the operating group will pass both tests above while making a large number of your routing decisions wrong. That failure is only visible in outcomes, which means the routing distribution itself needs to be a monitored number — the catch-all tier at 12% is a fact about the business that a dashboard can hold, and it was the fact that eventually told the story here.

Putting the dependency in the write-up#

When you take this class of dependency deliberately, the reviewer's question is not whether the vendor is reputable. It is what you are handing over and what you will do when it moves. The write-up that survives review names the change owner explicitly — the vendor's release process governs a behaviour of your agent — and then states the four lifecycle answers, the detection you have built, and the fallback when detection fires. The fallback is the part most often missing, and it does not have to be sophisticated: for the routing agent it was that unknown headcount stops meaning catch-all tier and starts meaning a queue a human looks at, so that a vendor change degrades into work rather than into wrong answers.

That accounting works for a single external dependency, where the question is who controls the interface. It stops being sufficient as soon as one operated server carries several unrelated systems at once, because then the interface is not one dependency but several bundled together, and the failures start to arrive in shapes that argue with each other. The next chapter picks apart a server that had grown that way.