Devlog #2

Last time I said the plan is to store meaning, not the waveform, the way face recognition stores an embedding instead of the photo. That left a question I skipped over: how do you actually train a compressor to do that?

The naive version is to run the EEG through an encoder, get an embedding, and store just that. But it has two problems. First, I’d be throwing the raw signal away for good, and a brain recording is expensive and reusable, you might want to ask it a different question later. Second, and bigger: what target do I train the embedding toward? An autoencoder left to itself just learns whatever compresses easiest, which lands me right back at fidelity.

The insight I landed on: I don’t have to invent the meaning space. It already exists. Image and text models like CLIP have already learned a space where a picture of an accordion and the word “accordion” sit in the same spot. So instead of hoping my compressor discovers meaning on its own, I borrow a model that already has it and use it as a judge.

Here’s the shape. The codec compresses and rebuilds the EEG like a normal codec. Then I take the rebuilt signal, push it through a frozen model that maps it into that shared image-text space, and score how close it lands to the embedding of the image the person actually saw. If the reconstruction drifts away from the right meaning, that score punishes it. The judge never changes, it only grades.

The important part is that the gradient has to flow through the frozen judge back into the compressor. The judge is frozen but not detached: it’s a fixed grader whose opinion shapes what the codec learns to keep. That one detail is easy to get wrong, and it’s the whole trick.

So the loss becomes three things added together: spend few bits, rebuild the waveform roughly, and stay decodable in the judge’s space. That third term is the entire idea. It’s what makes the codec hold onto the accordion instead of the loudest wiggle in the signal.

And because the judge lives in a text-image space, search comes for free. Type a word, it becomes a point in the same space, and I rank the stored recordings by how close they land. I never trained on text.

The Problem and My Idea

The problem. Brain-to-image datasets are huge and growing, and the only way we compress them today is generic lossy compression that optimizes how the waveform looks. That throws away the part that actually matters: what the person saw. And there’s no way to search the data. A lab with decades of recordings can’t ask “find the trials where the subject saw a face.”

My idea: embeddings. Instead of storing the raw signal, store an embedding: a short vector that captures the meaning of the data, not its pixels.

The clearest example is faces. Face recognition already works this way. A face gets turned into a vector (a few hundred numbers), and identity lives in the geometry of that space: the same person’s faces land close together, different people land far apart. You never store the photo. You store the embedding, and that tiny vector is enough to recognize, match, or search the face.

Where I want to take it. My idea is that the same trick should work for the “what you saw” signal in a brain recording. If I can map an EEG epoch into a latent space that captures meaning (the kind of space face and image models already live in), then I could:

• Store it tiny. Keep the compact embedding instead of the full signal.
• Search it by meaning. Type a word, find the matching recordings, because text and these embeddings can share a space.
• Compress against meaning, so what survives is what the data is for, not the loudest part of the waveform.

This is the direction I want to take it: treat embeddings as the storage format, the way we already do for faces, and apply it to the brain.