Zephyr Devlog #3

This week started with a pretty simple plan: add streaming static files, implement HTTP range requests, and optimise the metrics system I talked about in the last devlog.

All three landed, but while working on streaming I ended up finding one of the most interesting bugs in the project so far.

Streaming static files

Until now, Zephyr always loaded an entire file into memory before sending it. That is fine for small assets, but not for large downloads.

Static files now stream directly from disk, while cached files still come straight from memory. The cache behaviour hasn’t changed, but anything outside the cache no longer has to allocate the whole file first.

I also added support for HTTP range requests (206 Partial Content), including If-Range, so downloads can resume correctly instead of starting over.

The bug

Everything passed its tests, so I benchmarked it with a 256 MB file.

Memory usage was still growing almost linearly with the file size.

After a lot of testing I tracked it down to a single Hyper configuration option I enabled back in week one. It disabled the backpressure that streaming depends on, so the server buffered data as fast as the disk could read it instead of as fast as the client could receive it.

Removing one line changed peak memory usage from this:

256 MB file Peak RSS Old implementation 402 MB “Streaming” 240 MB Fixed 7.4 MB

The throughput difference on normal workloads was only around 1 to 4%, which is close to benchmark noise, but the memory improvement is massive. More importantly, it removes an easy denial of service case where slow clients could force the server to hold onto huge amounts of memory.

It was a good reminder that passing tests only proves the output is correct. It does not prove the implementation is efficient.

Faster metrics

The other big change was the metrics system.

Every request updates several counters. They were already sharded per thread, but I was still doing seven atomic fetch_add operations on every request.

Owned shards now use simple load/store operations instead, while shared shards still use atomics when they have to. That reduced the cost of recording metrics from about 26 ns to 3.5 ns, roughly a 7 to 8x improvement.

End to end, that translated into about a 10% throughput increase in the plaintext and cached static benchmarks.

Also changed

• Target::File now streams instead of reading the whole file into memory.
• Cached files now reuse pre-built ETag headers instead of recreating them for every request.
• Added 13 new tests covering streaming, range requests, 416, If-Range, and the metrics changes.
• Removed every source code comment from the project.

Next

• Finally get the benchmark suite running against PostgreSQL on Linux.
• Add large file workloads to the benchmark suite.
• Look into reusable buffers for the streaming path.
• Continue work on the async plugin ABI.

The biggest lesson this week was that benchmarks need to measure more than correctness. The streaming implementation passed every test, but it was not actually streaming.

Zephyr Devlog #2

Yesterday I published my first Zephyr devlog with a list of things I wanted to tackle next: better WebSocket support, more realistic benchmarks, and measuring RAM. I expected it to be a few small improvements, but it ended up becoming a much bigger project than I thought.

A lot of this update isn’t just adding features. It involved redesigning parts of the benchmark suite, reworking parts of the server to support more realistic workloads, fixing bugs I only discovered while building those workloads, and making a number of performance improvements along the way. There were a lot of small technical decisions and trade-offs behind the scenes that took far longer than I expected.

The biggest change was throwing away the old benchmark suite entirely. It did its job, but it wasn’t measuring the kinds of workloads that real applications actually run. If I wanted Zephyr’s benchmarks to mean something, I needed to build a new suite from the ground up that tested real-world scenarios instead of just the HTTP layer. Anyways, the benchmarks are below, you can go check them out, where Zephyr is consistently top 3 in all benchmarks.

Zephyr Devlog #1

This week I focused on getting the core of Zephyr working and making it as fast as possible.

What I built

• Core HTTP server using Hyper
• Router with support for static and dynamic routes
• Static file serving
• Plugin system
• Metrics for tracking requests
• Benchmark suite to compare Zephyr with other frameworks

Bugs I fixed

I found a few bugs while writing integration tests:

• Body size limits weren’t always enforced.
• If-None-Match headers weren’t working correctly.
• The rate limiter rewrote some valid configs.

These were all fixed after writing more tests.

Performance

After benchmarking, I realised the server wasn’t using all CPU cores efficiently. I rewrote the threading model so each worker had its own runtime, which increased throughput by about 45%.

Current plaintext benchmark:

| Framework | Requests/sec |
| Zephyr | 321k |
| actix-web | 314k |
| Node.js (cluster) | 266k |

I’m happy that Zephyr is now competitive with one of the fastest Rust web frameworks.

Next

• Better WebSocket support
• Streaming static files
• More realistic benchmarks (JSON, databases, uploads)
• More plugin improvements
• Benchmarking things like RAM usage too
  Overall I’m really happy with how the project is progressing. The biggest lesson this week was to benchmark before assuming something is the bottleneck.