Async and concurrency in Python

Read these first

• Python for AI work

What you will know

• Compare asyncio concurrency with thread or process parallelism by running the same I/O workload and recording throughput and CPU usage

  Asyncio overlaps waiting work but does not make CPU-bound Python execute in parallel on one event loop thread.

• Write an async function, await it, and run many of them concurrently rather than in sequence

  Ten sequential API calls take ten round trips; ten concurrent ones take roughly one. This is the single biggest speed-up in most AI code.

• Use a task group so one failing task cancels its siblings instead of leaving them orphaned

  A task group turns a fan-out into an all-or-nothing unit, so one failure does not leave siblings running and unwatched.

• Bound concurrency with a semaphore and say what number you picked and why

  Unbounded concurrency just moves the failure to the provider's rate limiter. Pick a number you can defend from measured 429s.

• Run an async workload containing a blocking call, then move it off the event loop and verify the scheduling impact with timing or loop-lag instrumentation

  A single synchronous library call can stall unrelated requests until it returns, even when the surrounding function is declared async.

Chapters

1. 1

   Sequential and concurrent awaits

   Ten awaits written one after another cost ten round trips; ten started before any of them is awaited cost roughly one. This chapter covers the mechanics — declaring a coroutine function, what await actually hands back to the event loop, why a for loop with an await in its body is strictly sequential however much async syntax surrounds it — and the before-and-after wall-clock measurement that proves a rewrite worked rather than assuming it.

   Read chapter
2. 2

   Event loops, threads and processes

   Run one workload three ways — on an event loop, on a thread pool and on a process pool — and record throughput alongside CPU. Two facts account for everything you measure: an event loop is a single thread that only overlaps waiting, and the global interpreter lock means one CPython process executes one bytecode stream at a time, so pure-Python CPU work stays pinned near a single core however many threads you add.

   Read chapter
3. 3

   Failure semantics of a fan-out

   With default arguments, asyncio.gather hands the first exception to your caller and leaves every other call in the fan-out running with nobody watching its result or its error. A TaskGroup cancels the siblings instead and surfaces an ExceptionGroup once they have all finished unwinding. This chapter covers which contract a fan-out over documents actually wants, how cancellation is delivered, and the narrower case where gather is still the right call.

   Read chapter
4. 4

   Concurrency limits and queue time

   A semaphore is a queue with N servers, so the number in Semaphore(20) sets a service rate, and once arrivals exceed it the excess shows up as p99 latency that has nothing to do with the provider. This chapter covers how to split semaphore wait from provider latency in one metric, why retrying with backoff while still holding a slot removes servers rather than adding load, and how to derive the limit from a provider's own rate-limit headers.

   Read chapter
5. 5

   Bounded queues and worker pools

   Creating a task per item and letting them all queue on a semaphore is correct and produces the right answer, but every coroutine and its payload sits in memory from the first instant and nothing pushes back on whatever is producing the work. Feeding the same 10,000 calls through a bounded queue drained by 16 long-lived workers gives flat memory, real backpressure and results written as they land, at identical wall clock.

   Read chapter
6. 6

   Blocking calls on the event loop

   A synchronous 800ms PDF extraction inside an async handler freezes every other request sharing that worker, including ones that were a millisecond from returning, which is why the p99 damage lands hardest on endpoints that never touch a PDF. This chapter covers the one measurement that proves it — a no-op scheduled every 100ms, recorded by how late it fires — what asyncio.to_thread recovers, and where a thread stops being enough.

   Read chapter