Mid · 7 chapters
Concurrency and throughput
Async clients, pooling, semaphores, worker queues for bulk jobs
Unbounded concurrency turns provider limits into outages, while poorly managed bulk work makes latency and operational costs unpredictable.
Read these first
What you will know
Run many model calls concurrently without exhausting rate limits
A semaphore around the call site, sized to the provider's limit rather than to your ambition.
Bound in-flight requests and tune the bound against observed 429s
A zero-429 target can underuse quota, while burst windows, retries, and token limits can make request-count concurrency look healthy as throughput collapses.
Move bulk work to a queue and explain why it doesn't belong in a request handler
Bulk work in a request handler blocks a worker for minutes and ties throughput to HTTP timeouts.
Reuse connections instead of opening one per call
Connection reuse matters more than it looks when every call is TLS to the same host.
Diagnose whether a slow batch is limited by the provider or by your own code
Compare provider latency, throttling, token utilization, queue wait, CPU, and connection-pool metrics; account-level token limits alone miss client serialization and hidden queueing.
Chapters
- 1
Semaphores and rate limiters
A semaphore bounds requests in flight and a provider publishes requests per minute, and the two are different quantities related by latency: 500 slots at 800 ms each is roughly 37,000 requests a minute. This chapter derives the in-flight bound from the published rate through Little's law, shows why the answer is a single-digit number, and explains why a client needs a rate limiter alongside the semaphore rather than instead of it.
- 2
Bounding in-flight tokens
An account rated at 500 requests per minute is usually also rated at some number of tokens per minute, and the two are exhausted by different traffic. A bound derived only from the request rate holds until prompt sizes grow, and then the token budget trips while request concurrency still looks healthy. This chapter derives both bounds, shows which headers say which budget refused you, explains why max_tokens is a reservation charged up front, and builds the weighted gate that bounds tokens outstanding.
- 3
Tuning the concurrency bound
A derived in-flight bound is a starting point, not a measurement, and tuning it down until the rejection count reaches zero pins the configuration to the unluckiest second of a five-hour run. This chapter argues for items completed per minute as the variable to optimise, prices what a 429 actually costs in slot time and quota, walks a sweep whose completed-work curve goes flat well past the first rejection, and writes the stopping rule down for whoever turns the knob next.
- 4
Connection pooling and client lifetime
A batch that constructs a fresh HTTP client inside its per-item function gives every call its own connection pool, so nothing is ever reused and each item absorbs DNS, a TCP handshake, a TLS negotiation and a certificate verification. That is a few hundred milliseconds per call the provider never sees. This chapter prices each round trip, places the client at process scope, sizes the three pool limits against the concurrency bound, and shows how to prove connections are being reused.
- 5
Diagnosing effective concurrency
Two hundred tasks in flight against an 800 ms provider should complete around 250 items a second; observing twelve means the tasks are not concurrent at all. This chapter turns that division into a diagnostic, shows where the in-flight gauge and the service-time histogram must be placed for it to mean anything, tabulates six things that serialise a fan-out with the measurement that confirms each, and explains why adding a second pod makes a serialisation bug permanent.
- 6
Bulk work in a request handler
An endpoint that summarises every row of an uploaded CSV survives a twenty-row fixture and fails on eight hundred: the load balancer cuts the connection at sixty seconds while the handler keeps working and pays for output nobody receives. This chapter traces what breaks in the order it breaks — shared provider gates, occupied request slots, deploys — and builds the job-id, per-row-state and streamed-progress design that keeps the synchronous-feeling experience the product asked for.
- 7
Concurrency on a worker fleet
Moving bulk work onto workers makes the API responsive and often leaves throughput exactly where it was: if each worker processes one item at a time, worker count is now the concurrency bound. This chapter locates concurrency inside the job, compares job granularities, replaces the per-process gate with a bound that survives autoscaling, caps the autoscaler against a ceiling it cannot move, and settles what happens when bulk and interactive traffic draw on one account allowance.
External resources
- Mastering asyncio.gather and as_completed for LLM Processingpython.useinstructor.com
Instructor's own pattern for semaphore-bounded concurrent LLM calls.
- pkempire/llm_processorgithub.com
Working bulk-processing harness you can read and adapt.
- Limiting concurrency in Python asyncio: async imap_unordered()death.andgravity.com
Deep dive on why naive gather() blows up memory on large batches.
- Why I Chose arq and RQ Over Celery for LLM Workloadsdangquan1402.github.io
Picking a worker queue when calls take 2-30s and can't block your server.
- Concurrent LLM Calls - Implementing Asyncio for Parallel Agent Execution EngS3CloudHub
Working asyncio code for fanning out many LLM calls in parallel
- Arq in Python: Run Async LLM Job Queues Without CeleryProfessor Py: AI Engineering
Worker-queue pattern for bulk LLM jobs without dragging in Celery