Zephyr Devlog #3

This week started with a pretty simple plan: add streaming static files, implement HTTP range requests, and optimise the metrics system I talked about in the last devlog.

All three landed, but while working on streaming I ended up finding one of the most interesting bugs in the project so far.

Streaming static files

Until now, Zephyr always loaded an entire file into memory before sending it. That is fine for small assets, but not for large downloads.

Static files now stream directly from disk, while cached files still come straight from memory. The cache behaviour hasn’t changed, but anything outside the cache no longer has to allocate the whole file first.

I also added support for HTTP range requests (206 Partial Content), including If-Range, so downloads can resume correctly instead of starting over.

The bug

Everything passed its tests, so I benchmarked it with a 256 MB file.

Memory usage was still growing almost linearly with the file size.

After a lot of testing I tracked it down to a single Hyper configuration option I enabled back in week one. It disabled the backpressure that streaming depends on, so the server buffered data as fast as the disk could read it instead of as fast as the client could receive it.

Removing one line changed peak memory usage from this:

256 MB file Peak RSS Old implementation 402 MB “Streaming” 240 MB Fixed 7.4 MB

The throughput difference on normal workloads was only around 1 to 4%, which is close to benchmark noise, but the memory improvement is massive. More importantly, it removes an easy denial of service case where slow clients could force the server to hold onto huge amounts of memory.

It was a good reminder that passing tests only proves the output is correct. It does not prove the implementation is efficient.

Faster metrics

The other big change was the metrics system.

Every request updates several counters. They were already sharded per thread, but I was still doing seven atomic fetch_add operations on every request.

Owned shards now use simple load/store operations instead, while shared shards still use atomics when they have to. That reduced the cost of recording metrics from about 26 ns to 3.5 ns, roughly a 7 to 8x improvement.

End to end, that translated into about a 10% throughput increase in the plaintext and cached static benchmarks.

Also changed

• Target::File now streams instead of reading the whole file into memory.
• Cached files now reuse pre-built ETag headers instead of recreating them for every request.
• Added 13 new tests covering streaming, range requests, 416, If-Range, and the metrics changes.
• Removed every source code comment from the project.

Next

• Finally get the benchmark suite running against PostgreSQL on Linux.
• Add large file workloads to the benchmark suite.
• Look into reusable buffers for the streaming path.
• Continue work on the async plugin ABI.

The biggest lesson this week was that benchmarks need to measure more than correctness. The streaming implementation passed every test, but it was not actually streaming.