Connection pooling and client lifetime

Connection reuse is the reason an interactive request path answers in 800 milliseconds. HTTP/1.1 made persistent connections the default in 1997 precisely because the handshake-per-request behaviour of HTTP/1.0 dominated page load times, and TLS made the same argument again with a bigger constant. A long-lived client with a warm pool amortises one handshake over thousands of requests.

A bulk job written as a fan-out over a list frequently gives that up without anyone deciding to, because the code that gives it up reads as tidy resource management:

async def summarise(attachment: Attachment) -> str:
    async with httpx.AsyncClient(timeout=30.0) as client:      # new client per item
        response = await client.post(PROVIDER, json=prompt_for(attachment))
    response.raise_for_status()
    return response.json()["content"][0]["text"]

Creating a client inside the per-item function means the pool is created empty, one connection is opened into it, one request travels over that connection, and the async with closes the pool and the socket underneath it. Thirty-six thousand items produce thirty-six thousand pools, thirty-six thousand TCP connections and thirty-six thousand TLS negotiations, and nothing is ever reused because nothing outlives the call that created it.

Where the extra 250 milliseconds goes#

Take a provider in a different region, with a round-trip time to its edge of about 62 ms. Everything a fresh connection has to do is measured in round trips.

Every row but three is a count of round trips, which is why the total is predictable rather than measured. A TCP handshake costs one round trip, so at a 62 ms round-trip time to the provider's edge it costs 62 ms. A TLS 1.2 handshake costs two, so it costs 124. DNS resolution, SSL context construction and certificate chain verification are local or near-local work rather than round trips, and they add 18, 12 and 34. Those five come to 250 ms, and 250 on top of the 800 ms the provider actually spends is the 1,050 ms the job observes.

TLS 1.3 cuts the handshake to a single round trip, which would take the overhead from 250 ms to about 190 ms and leave the shape of the problem unchanged. The one row worth singling out is the SSL context: httpx.AsyncClient() builds a default context, which reads and parses the certifi CA bundle — roughly 150 certificates, about 250 KB of PEM — using synchronous CPU on the event loop. A blocking call on an event loop stops every other coroutine in the process for its duration; here it is 12 ms of it per item, which across 36,000 items is seven minutes during which nothing else in the process runs, distributed in slices too small for any profiler pointed at the provider call to attribute.

"Blocking" is doing precise work in that sentence, and it is worth unpacking, because 12 ms sounds negligible beside an 800 ms call. An event loop runs one callback at a time, and a synchronous call inside a callback cannot be suspended, because it contains no await point for the loop to seize. During those 12 ms the process is therefore not slow, it is stopped: no other response is parsed, no other request is written, no timer fires. Twelve milliseconds repeated across 36,000 items is four hundred and thirty-two thousand milliseconds, a little over seven minutes of a process doing nothing at all, and it hides from profiling because it arrives as 36,000 separate slices rather than as one stall anybody would notice.

Where the connection pool belongs#

The client is a process-level resource with the same lifetime as the process, so it is constructed once at startup and closed once at shutdown. In a batch job that is the outermost async with; in a web service it belongs in the framework's lifespan hook, so that handlers receive it rather than build it.

CONCURRENCY = 8

LIMITS = httpx.Limits(
    max_connections=CONCURRENCY + 2,          # never smaller than the concurrency bound
    max_keepalive_connections=CONCURRENCY + 2,
    keepalive_expiry=60.0,                    # default 5 s drops idle connections
)


async def main(attachments: AsyncIterator[Attachment]) -> None:
    async with httpx.AsyncClient(limits=LIMITS, timeout=30.0, http2=True) as client:
        gate = asyncio.Semaphore(CONCURRENCY)
        async with asyncio.TaskGroup() as tg:
            async for attachment in attachments:
                tg.create_task(summarise(client, gate, attachment))


async def summarise(
    client: httpx.AsyncClient, gate: asyncio.Semaphore, attachment: Attachment
) -> str:
    async with gate:
        response = await client.post(PROVIDER, json=prompt_for(attachment))
    response.raise_for_status()
    return response.json()["content"][0]["text"]

Each of the three Limits values is a decision rather than a default worth accepting. max_connections is a hard ceiling on how many requests can be in flight through this client, so if it is smaller than the concurrency bound then it — and not the semaphore — is the real bound, and the semaphore's value becomes a comment. That is the failure the next chapter opens on.

max_keepalive_connections controls how many idle connections survive after their request completes, and httpx defaults it to 20 while defaulting max_connections to 100. A job running at 50 in flight under those defaults keeps 20 connections and closes 30 after every batch of responses, so 60% of calls handshake again — a partial version of the same defect, slower and much harder to spot, because the average latency lands between the pooled and unpooled numbers.

That 60% is one subtraction. With max_connections at 100 and max_keepalive_connections at 20, a job running 50 requests in flight opens 50 connections and may keep only 20 of them idle, so 30 are closed as their responses complete. Thirty out of 50 is the 60% of the next round that handshakes again. The version of the defect that lives in the defaults is harder to find than the per-item client for exactly that reason: three calls in five pay the handshake and two do not, so the mean latency sits between the two figures and matches neither, and nothing in a latency histogram looks like a handshake.

keepalive_expiry defaults to five seconds, which is fine for a saturated batch and wrong for a job that pauses to write a checkpoint, page a database cursor or wait on a rate-limit backoff. Any gap longer than the expiry retires the whole pool, and the next burst pays for a full set of handshakes.

Turning on HTTP/2 changes the accounting rather than tuning it. A single HTTP/2 connection multiplexes many concurrent streams, so 8 or 80 concurrent requests can share one socket, and max_connections stops being a concurrency bound at all. That is usually what you want against a provider that supports it, with the caveat that concurrency is then bounded only by your semaphore and by the server's SETTINGS_MAX_CONCURRENT_STREAMS, a number you do not control and are not told about. Provider SDKs are built on httpx or an equivalent and inherit all of this: a long-lived AsyncAnthropic() or AsyncOpenAI() client holds a pool the same way, accepts the same http_client if you want to configure the transport, and suffers the same problem when constructed inside a per-item function — which is the more common shape of this bug, because an SDK constructor looks cheaper than an HTTP client constructor.

The waiting that a pool introduces#

A single timeout=30.0 sets four different deadlines, and once a pool exists they stop being interchangeable. Writing httpx.Timeout(connect=5.0, read=60.0, write=10.0, pool=5.0) names them separately, and the one that only exists because of pooling is pool — the time a request will wait for a free connection before raising httpx.PoolTimeout. With max_connections set above the concurrency bound that exception should never fire, which makes it an excellent alarm: a PoolTimeout in the logs means requests are queueing for connections, and that is a bug in the sizing rather than a transient to be retried.

The read timeout deserves separate thought on model calls, because 30 seconds is generous for a classification and short for a long structured extraction, and a read timeout that fires mid-generation costs the full input tokens with nothing to show. Streaming changes the meaning again, since the read timeout applies between chunks rather than to the whole response, so a streamed call can legitimately run for minutes under a 30-second read timeout.

Name resolution is the other cost that hides behind pooling. getaddrinfo is a synchronous C call, and asyncio runs it in the default thread pool executor, sized at min(32, os.cpu_count() + 4) — on a two-core container, six threads. A client that resolves on every request funnels every call through those six threads, so on a slow resolver the resolution stage becomes an invisible concurrency limit of six, sitting underneath whatever number was configured. A warm pool sidesteps this entirely by resolving once, which is one more reason the difference between a shared client and a per-item client is larger than the handshake table suggests.

A batch that starts cold pays for its handshakes in the first seconds, which is negligible on a 40-minute job and visible on a 30-second one. Where that matters, sending a cheap request — a models list, or a one-token completion — for each connection you intend to keep, before the real work starts, fills the pool in parallel rather than serially and removes the ramp from the measurement. It is also the only way to make a short benchmark comparable to steady-state behaviour, since a 60-second benchmark on a cold pool spends its first 250 ms per call on setup and reports a latency nobody will ever observe in production.

The fix is invisible in application logs, so it is worth confirming directly. The transport layer underneath httpx logs every connection event, and enabling it for one run answers the question in the first ten lines of output:

logging.basicConfig(level=logging.DEBUG)
logging.getLogger("httpcore.connection").setLevel(logging.DEBUG)

A working pool logs connect_tcp.started and start_tls.complete a handful of times at the beginning of the run and then nothing, while every subsequent request produces only send_request_headers. A broken one logs a start_tls.complete for every item, which on a 36,000-item job is unmistakable. From outside the process, the socket table tells the same story without a code change: ss -tan state established '( dport = :443 )' | wc -l on the running container should settle at roughly the pool size and stay there. A count that climbs, or a large and growing number of sockets in TIME_WAIT — the state the kernel parks a closed socket in before it will reissue that port — means connections are being opened and closed per request. Sixty seconds of TIME_WAIT per connection at 500 connections per minute is 500 sockets held by the kernel at steady state, which is survivable; at 5,000 per minute it is a port exhaustion incident whose error message mentions neither TLS nor pooling.

Proving reuse from outside the process is the check to reach for when the logs belong to somebody else, and the socket arithmetic behind it is worth stating. A closed socket sits in TIME_WAIT for about a minute, so the number of sockets the kernel holds at steady state is the connection rate multiplied by that minute: 500 connections a minute leaves 500 parked, which nothing notices, and 5,000 a minute leaves 5,000, which starts competing for the machine's supply of ephemeral ports. What decides the outcome is the rate rather than the total, which is why a job that opens 36,000 connections over 40 minutes is fine and the same job compressed into four is not, and why the eventual failure is so often misread as the provider throttling you.

Latency drops and the bound moves with it#

Connection reuse removes a fixed cost per call and has no effect on any of the constraints from the first three chapters. The account still serves about 530 requests per minute, the token budget is unchanged, and removing latency raises the request rate a given concurrency bound presents — which is why this fix can arrive looking like a regression. The relationship is the one Little's law states: in-flight equals rate times latency, so holding in-flight constant while latency falls raises the rate, in exact proportion.

There are also cases where reuse is worth less than it looks. A client that spreads calls across many hosts pools per host, so ten providers at low volume each keep a mostly cold connection. A CLI that makes one call and exits cannot amortise anything. And a long-lived connection behind a load balancer is pinned to whichever backend accepted it, so if that backend degrades, every request on that connection degrades with it until the connection is retired — which is one reason to set keepalive_expiry to a minute rather than an hour, and why some providers close connections periodically on purpose.

Where the client lives is therefore a decision with two separate consequences. One is the 250 milliseconds per call, which is arithmetic anybody can check against a handshake table. The other is that the pool has a second identity: it caps how many requests can be outstanding, which makes it a concurrency limiter nobody declared, sitting underneath the one that was declared. When a job's observed throughput and its configured concurrency disagree, that gap is measurable in a single division, and the next chapter turns that division into the first thing to run.