About project

Zephyr is a magic mod, with spell casting powered by speech live translated into International Phonetic Alphabet, to detecting custom-made spell words in it.

Adding Voice Activity Detection - part 2

After the refactor I was able to finally add the VAD. But after few tests I noticed that the VAD was making things worse, not better!
That was as a part of the previous devlog (which ran out of character limit).
I added a pipeline part that writes every chunk of audio that passes through it to a file, so it can be inspected.
With it I found the main problem: It took few hundred milliseconds before the VAD detected speech, so a small chunk was always cut off from the start. To fix this I added context keeping: A bunch of previous chunks are kept, and when speech is detected they are also taken into consideration.
I also made the sliding window chunking aware of VAD - the VAD detector sends a signal when it detects an end of speech. The chunker then flushes its buffer into the pipeline, so that it doesn’t get stuck until next speech, and replaces it with precise amount of silenece. This way the start of next audio doesn’t get cut off later due to the sliding window mechanism.
The author of the bunsen crate (providing the VAD model implementation) also released version 0.29.0, which fixed all the issues I encountered in the last devlog, and updated its API slighly, which meant that after adjusting my code I could stop using my own fork of it.

Improving word detection from IPA

My friend found a crate named phonetics-rs and suggested using it to improve the algorithm. This crate calculates the difference between two IPA strings, taking into account how similar they sound. This is much more accurate than just comparing strings on grapheme level. I modified my algorithm to use it, and it turned out to be a great idea, that significantly improved the detection rate of magic words.
I also changed the dictionary to be sorted by longest words first, so that they take priority over short ones.

So far, testing and improving the word detection algorithm has been the most fun out of the entire project. Looking at IPA strings, and figuring out why some speech has or hasn’t been classified as some word is quite fascinating. For example, I encountered a string pɾilifɛra which the model output after me saying prizim and fɛra, the two magic words that I made up for the spell system. Surprisingly, my algorithm said the string contained just prizim despite there being a literal fɛra in it.

As I found out, pɾilif is closer then pɾili to prizim, and it fell under the threshold, so the algorithm took that part and left out ɛra in the buffer which was too far from fɛra. After seeing this, I made the algorithm scan the string for exact matches first, and then do fuzzy phonetic matching, also ensuring the order is preserved. I also tuned the thresholds, adding more and more tests (there is now 8 of them) to ensure the algorithm behaves well.
Finally, I noticed the IPA output from the model changes based on audio’s offset in time. So I wrote a script that lets me test this, and see the effect of different offsets.

What’s next

There is a saying in data science: garbage in - garbage out. This is also true for my project. The detection works decently, but it also quite often outputs false positives during normal speech. I believe I did I all could to compensate for the unsatisfying quality of the speech to IPA translation model.
Now I will be shifting to the Minecraft part of the project - creating a mod, integrating the detection system into it and creating spells. In paralell, I will be exploring the possibility of finetuning the model, to improve the accuracy of it and hopefully get better results.

Watch the video with audio, that’s the part that matters right? :)

Finding words in long IPA strings

For the magic system I need to find individual words in the ever accumulating string of IPA characters. To do that I hold a list of valid words in memory. When there is some text, I iterate over the dictionary, calculating a levenshtein distance between each word and start of the string. Then the word with the lowest score is taken and if the score is under a set threshold, the word is considered a match, and the matching part is remved from the buffer. To cover more cases different lengths of the string are checked. If no match is found, one character is removed from the string, and the process repeats, until all words are found.

Fun fact: When comparing IPA and other non-ascii strings you need to make sure you are considering graphemes and not characters, because in UTF-8 what we perceive as one symbol, can be encoded as many characters in memory.

So far this algorithm works quite well, though in the future it might need some performance optimization, or tuning, if it starts yelding too many false positives.

Other misc changes:

• Migrated main fn to use clap for parsing arguments
• Added flake.nix with dev env so I can run this code on my NixOS laptop
• Added time information to the pipeline - Together with the audio from the mic, a timestamp is captured, and passed through entire pipeline up to the end, where it is used to discard text older then 200ms from the buffer, so that stale data doesn’t mess up new words.

Adding VAD

I noticed that the speech to IPA model likes to output random data when no one is talking, so I decided to add VAD - Voice Activity Detection - a system that detects if someone is talking, and only then passes the audio on. I estimated this to take me around 2 hours, and oh, how wrong I was?
It took 10x that, which is why this devlog documents 3x the normal amount of time. And it still isn’t fully ready! So let me tell you what I did and why it took so long.
In the Burn (ML/AI lib for Rust) discord I noticed a solution just dropped out of the sky: Merely few days before a Silero VAD model implementation was added to a library named bunsen. ML models take input as Tensors - multidimensional arrays, which do not say in what form or shape the data is expected. So I had to figure that out myself. Fortunetely it was just 512 elements long chunks of 16khz audio with no additional pre-processing. Knowing that I fed the microphone input to the model and began to watch it output complete nonsense unrelated to whether I was speaking or not. So I started debugging - checked my code, wrote an equivalent in python using the official silero implementation to compare results - completely different. So I cloned the bunsen library, wrote an unit test there and ran off a Opencode agent. I also contacted the library author. He found one major issue, and the agent identified a second problem and wrote a solutuion - Silero VAD expects 64 samples of the previous chunk to be passed along the 512 to work correctly. I cleaned up the code, which was quite a pain because rust-analyzer was super slow on that repository, and made a PR, but it wasn’t merged because the author went with his own way of implementing it.

The big refactor

Coming back to my project I realized that my code strucutre makes it very hard to integrate VAD into it. So I spent the next 6 hours of so refactoring almost every file in the codebase into a new pipeline system. It lets you compose a pipeline out of a producer, many processors, and a consumer. It makes use of async, which saves on the amount of threads used. Static type checking ensures you can’t compose parts that wouldn’t work together, and all elements can also define contracts on the size of data chunks they produce and consume.

I ran out of the character limit, so the rest of VAD story will be in next devlog.

Live speech to IPA transcription in Rust using ML

I finished my rough implementation of a live speech transcription pipeline. It captures my speech from the mic, and processes it with the Multipa model using the Burn framework.
To make it work live I implemented a sliding window that chunks the audio into 2 second fragments, each of which is processed separately, and then assembled into a single string. The chunks overlap, and are then sliced, so that audio that was on the borders of a cut (which will reduce the quality) is thrown away, and the text for it is taken from the next chunk, for which it is at the center.

The inference is running on cpu for simplicity and compatiblity, so by sacraficing a bit of pain I added threading (where each chunk is processed on a separate thread) to make sure the pipeline can keep up. I’m slightly worried about how the perfromance will be when this is going to run along Minecraft, but hopefully I will be able to make it as lightwieght and unintruisive as possible.

Finally I reworked how the text is accumulated, switching from a vector of logits to a string. Since IPA has no concept of words, just phonemes, everything I say is appened to one long string.
So I added a very simple silence detection, where if a captured chunk returns no text a space is added to the string.

For easier teasting I also improved the main function, so that the program can:

• take a file and process it all at once
• take a file and process it using the pipeline
• run live mic capture on the pipeline

What’s next - improving the detection

In theory this is all I need, and I can start working on the Minecraft mod. However I’m not satisfied with the quality of the transcription, and at the level it currently is most spell detection will probably fail.
The quality of the model itself is pretty bad. If the description on github of the Multipa model is correct, the model was trained on merely 9 hours of speech data, which probably explains it. The model is also sensitive to noise, and distractions like other people talking, and sometimes outputs garbage.
So the next devlog I will be experimenting with various techinques of filtering and cleaning up the audio the model is receiving, and potentially even training a new model myself on a bigger dataset.

Attached video

Working demo of live capture and transcription of me talking.

Sephyr - project overview

Sephyr is an unique magic mod for Minecraft. It aims to differentiate itself by:

• Introducing a voice powered spell casting system, that’s both fun to use and intricate. You will be reading spells out loud into your mic in a magic language, and what will you say will affect the result of the spell.
• Having stunning, detailed visuals unique to each spell. This is going to be achieved by spending way too much effort on rendering. I want spell casting to be something satisfying, and the look is a key part of it.

The project consits of two parts:

• Sephyr - The minecraft mod itself containing the spells, mechanics and visuals, written in kotlin.
• Sepple - Speech processing engine, responsible for transforming your babble into something Sephyr can work with and figure out what have you just said.

Sepple overview

Sepple is a foundation of the voice powered spell casting. It is an audio processing library written in rust, that Sephyr will depend on and embed. It captures your mic live and pre-processes the speech. Then it uses the Burn framework to locally and quickly run a speech recognition model called Multipa. The model transcribes speech into the International Phonetic Alphabet (IPA), a textual notation of various sounds that make up the spoken language. Finally it tries to match the result against a custom dictionary of magic words, which will form sentences - the spells you will be casting.

This devlog

Before Stardance launched I have written a rough skeleton for Sepple. With some trouble I managed to import the Multipa model to Burn via ONNX export from Pytorch, and implement the required post-processing to decode model’s output into IPA. I also started working on a pipeline for mic capture and sliding window processing, so that the audio can be transcribed live.
After Stardance started I noticed that the same input audio file gives quite different result in Pytorch then in Rust. I spent few hours debugging why that is, and found a missing piece in my implemenation. For the Multipa model to work correctly the samples need to be normalized using a so-called z-score normalization. That means calculating the mean of the input and its standard deviation (a statistics measure), and applying a formula that rescales the data, so that it has a mean of 0 and standard deviation of 1. After implemenating that the model started to work correctly and give the same output for the same input on both sides.

Picture: “Hello Stardance. Sepple is working now” translated to IPA using pytorch and the fixed Rust implementation. It is not veeery accurate, but it is good enough.