Epoll Readiness and Nonblocking IO

How readiness notification, nonblocking file descriptors, draining loops, and backpressure fit together.

Linux · Internals

epoll reports readiness, not completion. It tells a program that an operation might make progress without blocking. The program still has to call read, write, accept, or connect and handle short results, EAGAIN, and close events.

Readiness model

A nonblocking file descriptor returns immediately. If no data is available:

read(fd, buf) = -1, errno = EAGAIN

epoll_wait lets one thread wait for readiness on many descriptors. When a socket is readable, the event loop should drain until EAGAIN or until its fairness budget is exhausted.

Level and edge triggering

Level-triggered epoll repeats events while the condition remains true. If data remains unread, the next wait reports readiness again.

Edge-triggered epoll reports transitions. If the handler fails to drain the socket to EAGAIN, no new edge may arrive and the connection can stall.

Safe edge-triggered read shape:

for (;;) {
    n = read(fd, buf, sizeof buf);
    if (n > 0) { consume(buf, n); continue; }
    if (n == 0) { close(fd); break; }
    if (errno == EAGAIN || errno == EWOULDBLOCK) { break; }
    handle_error(errno);
    break;
}

Writes and backpressure

Writable readiness does not mean the whole response fits. write can accept a prefix. The rest needs to stay in an output buffer and the event loop should watch for writable readiness only while there is pending output.

If output buffers grow without bound, a slow peer can consume memory. Apply limits:

maximum queued bytes per connection,
global queued-byte limit,
priority or traffic class,
close or shed policy for slow consumers.

Failure modes

Treating readiness as completion and assuming full reads or writes.
Edge-triggered handlers that do not drain to EAGAIN.
Always subscribing to writable events and spinning because most sockets are usually writable.
Accept loops that accept one connection per event under high load.
Forgetting that regular files are usually always ready.
Closing a file descriptor while another subsystem still has it registered or reused.

Practical checks

Test partial reads and writes.
Test peer close, half-close, and reset.
Enforce output buffer limits.
Drain accept loops until EAGAIN under load.
Track event-loop iteration time and ready-list size.
Keep protocol parsing independent from the event loop so parser bugs are testable.

Design guidance