Async Backpressure and Bounded Queues
How bounded queues, overload policy, cancellation, and admission control keep async services from failing slowly.
Async code does not remove overload. It makes it easier to accept more work than the system can finish. Backpressure is the mechanism that turns finite capacity into explicit behavior.
Bounded queues are policy
A bounded queue has capacity $C$. If arrival rate $\lambda$ exceeds service rate $\mu$ for long enough, queue length grows until it hits $C$:
\[\lambda > \mu \Rightarrow q(t) \rightarrow C\]After that, the system must choose: wait, shed, degrade, or fail fast. If the queue is unbounded, the choice is delayed until memory, latency, or the process fails.
Tokio’s bounded mpsc channel applies backpressure by making sends wait for capacity. Its unbounded channel can always accept sends, so memory becomes the hidden bound.
Backpressure surfaces
Backpressure can appear at several layers:
| Layer | Signal |
|---|---|
| listener | stop accepting or apply accept limits |
| request parser | stop reading from sockets |
| worker queue | bounded send waits or fails |
| dependency client | semaphore, rate limit, or retry budget |
| response writer | stop reading when writes back up |
A service needs a pressure path from the saturated resource back to the producer. Otherwise work keeps entering after the bottleneck is already overloaded.
Service readiness
Tower’s Service::poll_ready is an explicit backpressure boundary. Callers must wait for readiness before calling call. Buffer and concurrency-limit layers change where capacity is reserved, so layer order is part of the overload design.
Cancellation and shutdown
Cancellation must release capacity. A dropped request should release queue slots, permits, buffers, and child tasks. Clean shutdown should close producers first, drain accepted work according to policy, then stop consumers.
For Tokio mpsc, receiver close prevents new sends. The receiver can then drain buffered messages before dropping the channel.
Failure modes
Unbounded channels move overload from the scheduler into the allocator. Latency increases, memory grows, and the service can die far away from the callsite that accepted too much work.
Retry loops can bypass backpressure. A failed dependency call that immediately retries consumes capacity while adding no new user value.
Head-of-line blocking appears when one expensive request occupies a shared worker and cheap requests wait behind it. Class-based queues or cost-aware admission may be needed.
Practical checks
- Every queue has a capacity, owner, and overload behavior.
- Queue depth and time-in-queue are metrics.
- Dependency calls have concurrency limits and retry budgets.
- Cancellation tests verify permits and buffers are released.
- Load tests include traffic above capacity and confirm graceful rejection.
- Logs distinguish local shedding from dependency failure.
Design guidance
Prefer bounded queues with explicit overload semantics. If an unbounded queue is used, name the reason and prove another resource bounds it. A reliable async service should reject excess work before it turns into memory pressure and tail latency.