Aulon: NATS-Core Broker on io_uring

Aulon is a NATS-core-compatible message broker built thread-per-core on tokio-uring. It uses fixed buffers registered with the kernel, an allocation-free protocol codec, sharded subscription routing, and a benchmark harness that compares publish-to-deliver latency against nats-server.

The design target is narrow on purpose: a single-node NATS-core broker with a small, explicit compatibility surface. JetStream, clustering, gateways, leafnodes, TLS, and authentication are out of scope for v1.

Goal

Build the NATS hot path as a systems project rather than a framework exercise:

• parse wire frames over borrowed bytes,
• avoid allocation on the publish path,
• keep routing local to cache domains where possible,
• use kernel-registered fixed buffers for TCP I/O,
• validate NATS CLI compatibility on the supported verb subset,
• keep benchmark caveats attached to every number.

Protocol scope

The wire protocol implements the verbs needed for the official nats CLI and nats bench to run unmodified on the supported surface:

Unsupported production NATS features return errors or stay out of scope: TLS, auth, JetStream, clustering, gateways, leafnodes, WebSocket transport, MQTT bridging, and multi-node operations.

Workspace

aulon-proto   allocation-free wire codec, no_std-clean, fuzzed and proptested
aulon-core    fixed-buffer pool, wildcard trie, topology, cross-shard inbox
aulon-server  broker binary, config, admin listener, graceful shutdown
aulon-bench   HDR-histogram benchmark client

aulon-proto owns the parser and emitter. Frame<'a> holds borrowed bytes, so payloads do not need to be copied into protocol objects. ParseOutcome::NeedMore supports streaming TCP input without treating partial frames as errors.

Runtime and buffers

Aulon uses tokio-uring because its public API exposes IORING_REGISTER_BUFFERS, FixedBufPool, and fixed-buffer read/write operations. Each worker owns its buffer pool. There is no global pool and no cross-core buffer migration.

The TCP path uses registered 4 KiB buffers and read_fixed / write_fixed_all. Fixed buffers are not magic. The project docs record that fixed-buffer opcodes were slower than the earlier Monoio baseline in the first single-connection VM test. Aulon kept tokio-uring because the registered-buffer API is the design surface the project wanted to explore, and because the meaningful validation point is higher-concurrency fanout, not one small echo connection in a VM.

Routing and fanout

Routing moved from a flat exact-match table to a per-token wildcard trie. The trie supports exact subjects, *, terminal >, and queue groups. Match emits through a callback rather than collecting matches into a Vec, which keeps the publish path allocation-free.

Each connection uses separate reader and writer tasks. Outbound delivery uses a pre-allocated per-connection byte ring instead of an mpsc channel. A release review caught an early bug where that buffer behaved like a linear bump allocator under steady near-keepup. The fix changed it to a true wrap-around ring with monotonic counters and added regression tests for wrap and eviction behavior.

Topology

The worker topology follows cache domains. Aulon discovers topology from sysfs, creates one worker per L3 cache domain, and uses SO_REUSEPORT so accepted connections are sharded. Single-shard publish stays on the allocation-free path. Cross-shard publish uses a bounded inbox and an eventfd wakeup, with one shared published frame object for cross-shard fanout.

This is a clear tradeoff: local fast path first, bounded cross-shard coordination only when subscribers live elsewhere.

Operator surface

The v0.1 operator surface includes:

• graceful SIGTERM drain,
• admin listener with /healthz, /readyz, /metrics, and /version,
• JSON logs behind a flag,
• CLI and TOML config with typo guard,
• --version with git SHA and rustc,
• Dockerfile,
• systemd unit,
• reproducible release script.

Shutdown uses a ShutdownToken plus one EventfdWaker per shard. The server integration test spawns the binary, opens TCP connections, sends SIGTERM, and asserts each connection receives -ERR server shutting down followed by FIN before the child exits successfully.

Benchmark evidence

The headline v0.1 chart is explicitly labeled as an in-VM, single-process bench client result. Bare-metal and multi-process benchmarking are deferred to v0.2.

Setup from the project performance log:

Result from the README headline table:

The comparison is useful but not oversold. Each backend runs at the smallest pace its measured behavior permits in the current single-process client. The project records this as a harness limitation, not a broker law.

Verification surface

The v0.1 release review records:

• cargo fmt --check,
• cargo clippy --all-targets --all-features -- -D warnings,
• cargo test --all-features,
• cargo doc --no-deps --all-features with warnings denied,
• cargo deny --all-features check,
• cargo audit,
• public API snapshot check,
• publish dry runs for aulon-proto and aulon-core,
• one-minute fuzz smoke against parse_frame and parse_pub.

The protocol crate is fuzzed and proptested. The core crate includes loom tests for shutdown ordering and cross-shard inbox behavior.

Limits

Aulon is Linux-only because io_uring is Linux-only. Daily development can happen from a macOS host through an Ubuntu VM, but the broker itself has no cross-platform fallback.

Aulon is not a replacement for production NATS operations. It is a low-level broker implementation focused on NATS-core framing, routing, fixed-buffer I/O, shutdown behavior, and measured hot paths.

Related notes

io_uring Fixed Buffers and Parser Fuzzing, GRE, FOU, and NAT Traversal, and Design Verification and Test.

View on GitHub