stardance has been extended another month! the new deadline is october 31 :)

You are browsing as a guest. Sign up (or log in) to start making projects!

Sephyr Magic mod for Minecraft

  • 13 Devlogs
  • 138 Total hours

Voice powered spell casting mod for Minecraft, something you have never seen before.

Open comments for this post

10h 8m 4s logged

Debugging gui is (almost) done

What is going on, why already 10 hours passed?
This was menat to be a quick sidequest, where I built this gui tool for debugging how well detecting magic keywords is working
BUUT
given it is the third devlog about it
That means it took over 30 hours!
Well, happens.
So, all of this devlog was spent building all the plumbing needed to capture all the data from the detection pipeline and sending it in realtime to the app. All of it is using done using channels. They are injected into the detection pipeline either as Tap<T>‘s - a new pipeline element that clones data it sees, sends to channel, and passes it on, or using wrapper types around existing processors. The app then receives the data and displays in on the plot. I also encountered and fixed few bugs, that I didn’t expect.
For example originally the zoom went to crazy values, because a timestamp used to determine where the data ends wasn’t converted to be relative to the timeline start.
I also found an quite an issue introduced 3 months ago, where the timestamps attached to audio weren’t calculated properly, making entire audio chunks have identical stamps. It was minor enough that I never noticed till now, when the data landed on the plot and resulted in this funny thing you can see on the attached screenshot.
Finally I added a simple scrolling animation, to amortize the jitter caused by bigger chunks of audio coming, so the plot doesn’t jump that much when going live, but it’s still not as smooth as I would like it to be.
As mentioned before I was using AI assistance in strictly controlled manner (I carefully planned, revieved, cleaned up and adjusted to my liking all the code generated by it) to build this debugging tool faster, and by now I’m very tired of it. At the end of the work it started doing stupid things (possibly because some context was lost?), which made me question if it was making me faster or slower. Luckily the work is mostly over, and I can finally get to other, more interesting stuff, that will be fun to do, with no AI. Next I want to use the tool to improve the detection a bit, and spend more time working on the actual mod.

0
0
12
Open comments for this post

10h 15m 35s logged

Keyword detection UI is done
Look as this! It’s so pretty! Seeing all the data visualised is quite mesmerizing. I know the image will probably attract more views, so lemme explain what you are looking at.
I’m making a minecraft mod where you cast spells by saying custom made up words, and this is a tool for debugging the keyword detection system. It’s meant for me, to help me make the detection more accurate, and reliable. I’ve been building it for the last ~20 hours, and in the last two I built the rendering of all the 8 strips. Check previous devlog if you want to know what exactly they display.
It was pretty smooth sailing. As said before, since this is just a debugging tool for me, that I want to have finished quickly for it I’m using AI assistence, by letting it do stuff in small, pre-planned steps, and unsloppifying all the code between them. It’s working well. What helps, is that entire app is built on the egui/eframe crates, which I used before, and are somewhat familiar with. I gotta give credit, where is due, their api is really intuitive for the most part, and simple to use.
The entire chart is also zoomable, and scrollable, and making this work was a bit of pain. I wanted to limit the accesible range to only around the data (so you can’t scroll away to far from it), and egui_plot doesn’t really provide a way to do it, so some manual calculations had to be done. I wanted to have some breathing space by allowing 0.25x of the zoom after last audio sample to be visible. This was the worst part, as it’s kind of a recursive problem - the zoom value affects how much zoom is allowed. But after around two hours I manged to figure it out, but keeping the original zoom value independantly of the limit calculated from it.

Worth noting - what you see is just a mockup data, and the gui isn’t actually hooked up to the detection system, as that’s what I’m going to do next. After that I can hopefully improve the detection, and the finally go back to minecraft coding.

0
0
18
Open comments for this post

11h 32m 13s logged

Visualizing the keyword detection pipeline

I’m not happy with the accuracy of the magic words detection system. It work half of the time, sometimes better, sometimes worse depending on the word. This make casting longer spells quite troublesome.
Debugging why a specific word wasn’t detected is hard, debug prints can only get you so far.
So I decided to go on a crazy sidequest and build a gui tool, that will help me debug this. I spent over 2 hours designing the concept, and over 10 implementing it, and it’s around 40% ready.
The tool will display 8 strips / plots, and each plot will display some data, all on a shared timeline on the bottom, that will update live as the audio is processed.

  • Strips 1, 2, and 4 will display waveforms of the audio at different processing stages.
  • Strip 3 will show Voice Activity Detection score (which is used to filter out non-speech).
  • Strips 5, 6 and 7 will display detected IPA tokens, accumulated text buffer, and detected words respectively.
  • Strip 8 will display latency stats, which will help in optimizing it.

This way I will be able to easily see, “oh, this phoneme wasn’t detected” or “oh, VAD cut out this word” or “oh, some mess accumulated in the buffer” and tweak my algorithms.

AI - it’s bad, or is it?

Despite looking quite simple on the surface, the app is still quite an undertaking. And it’s just a debugging tool! A thing to help me get the main goal done! I don’t want to spend too much time on it.
So I considered using AI as a tool (NOT vibecoding). I hate vibecod(ing, ers), and I’ve always steered off AI, because very often I wasn’t satisfied with the output quality. I take pride in writing good code. But recently I’ve been trying out OpenCode here and there (on free models), and was quite surprised with it’s capabilities. Sure, it’s still often slop, but a one you can quite quickly correct, and sometimes be more effective if used correctly.

AI - is it helpful?

So I decided to do an experiment - can carefully planned AI usage help me write the debugging tool quickly?
I started off by designing the functionality of the tool - I described the idea in detail to MiMo-V2.6-Flash, and asked it to interrogate me about as many details as possible and needed. I had to answer 35 questions, some of which were quite observant, picking me on nooks and crannies. After that a markdown file describing the design was written, and then went through few corrections.
Next up, in a new session I asked MiMo to create a bullet point outline of steps needed to be done to implement the tool specified in the design file. Then one by one I went over each bullet point, asking the just released LongCat 2.5 (which has a bigger ctx window) to ask me questions, and formulate an implementation plan of it, present it to me, I gave corrections, and the point in file was replaced. This way I ended up with a second markdown file - an implementation plan.
This took way too long, and was really boring. So, finally coding time. I do it like this:

  • Ask opencode to implement a step of the plan
  • Review the diff in vscode
  • Remove fluff and slop manually, make sure all new code is good quality.
  • Give feedback if I something doesn’t work like expected, or also fix it manually.
  • After it’s good - commit
  • Repeat for the next step

It’s still not super fun, but at least I’m touching the code now, so much better already. And it is efficient! I believe it definitely would take me longer, if I wasn’t using AI. So the morale is: If you want fun, or doing something important - do it manually. If you want to do something quickly and get it out of the way less important.

Here is a screen of how it is looking now: the general shape is in place, but there is no plots yet.

0
0
29
Open comments for this post

6h 21m 32s logged

The Stack Overflow

For some reason my mod broke and started crashing during startup, and it did so in the worst possible way - without any error message! All I got in the Intellij Idea, was a message that the process exited with the signal code 11. WHAT?
That signal corresponds to SIGSEGV signal on Linux, which generally means you did something wrong with the memory. But this is Rust! There shouldn’t be any memory issues!
I started searching, and I learned there is a tool called coredumpctl, which lets me access a special dump file generated in such cases, and analyze it in a debugger like GDB. So I did that, and what I found was even more bizzare! The debugger was pointing to a start of a function, before the first line. I added some debug prints, and to my disbelief, the print before calling that function ran, and the the second one, which was the first line inside the function didn’t!
Fast forward a bit I managed to figure out the problem, with some help from a friend. It turned out I encountered an stack overflow! When the function was called, Rust tried to expand the stack, and crashed because the stack ran out!
Why?
Because the code was loaded in Java, and Java by default allocates less memory for the stack!
The solution was manually requesting a bigger stack size during thread creation in Java.
I also discovered, why the problem appeared - in the Rust file download code I had allocated an array buffer of 1MB. It landed on the stack, and that was too much! The solution was using a Vec, which lives on the heap instead.
Stupid problem, but took 3 hours of my time 😩

Mod init progress reporting

Sephyr requires the IPA speech recognition model to work, which needs to be downloaded on the first run, and subsequently loaded from the disk. The logic was overhauled providing more corectness, and hooked up to a series of toasts. File download progress bar is shown, and the intialization state (or failure) is coummunicated to the user!
See it in the video

0
0
17
Open comments for this post

9h 45m 55s logged

Wind spells have sound now

Using LMMS and ZynAddSubFx I made two wind sound effects, which now play when casting a wind spell. You can hear them in the attached video.

Cross-Compiling is fun

The speech recognition system is written in Rust, which compiles down to native code. This means that it needs to be compiled separately for each operating system, and processor architecture.
Rust is more friendly to it, then C or C++ for example, but as always, compilation finds it own weird ways to break.
I took the setup used by Sable as a base, with two Docker containers, one for Linux and Mac, and one for Windows. BUT:

  • The Rust version installed in the docker image was not new enough for one of my dependencies - I had to update the rust toolchain in the container image.
  • Compiling for linux required ALSA headers for audio access - I had to figure out, how to install them in a way, that works for both x86_64 and arm64, without breaking each other.
  • The latest version of the base image has Mac cross compilation broken - I had to use a previous one.
  • The image ships a MacOS SDK from 5 years ago - too old, doesn’t work, so it also had to be replaced. - the container is kinda becoming a Ship of Theseus xD
  • My code was pulling openssl as a dependency, which was failing to compile - I had to spent over an hour, figuring out how to get rid of it from the dependency tree, discovering I can do so by updating a dependency to an unreleased version.
  • Gradle obviously required some tweaks, who doesn’t love Gradle right?

In the end I managed to get it working, which brings the mod closer to being usable by the public, outside dev environment.

Downloading the model

The mod works by running an IPA speech recognition model locally. The model weights around 1.2G, making it too large to be embedded in the mod jar.
So I added automatic download support - the model can now be downloaded automatically on the first time an user runs the mod. There is even progress tracking, though it’s not hooked up to Minecraft yet.
A Toast is also shown when the loading fails, making the mod more user friendly.

0
0
10
Open comments for this post

10h 13m 51s logged

SPELLCASTING WORKS!

The main mechanic of the mod is working!

YOU CAN CAST SPELLS WITH VOICE!

Finally! It took soo long…
After over two months of development, I finally have a proof, that my crazy idea actually works, and was worth trying out.
As you can see (and hear) in the video I implemented a simple wind leap spell, as the first one in the mod.
Note: There are no visuals or sounds yet, but they will be added, and this is just a proof of concept.
The spell comes in three variants

  • ʔalɪˈvɑn vɛɾtɛɾ ɛvi ˈonːdˌoː - wind launches you forward
  • ʔalɪˈvɑn vɛɾtɛɾ ɛvi ãntikɔ - wind launches you backward
  • ʔalɪˈvɑn vɛɾtɛɾ ɛvi fɒksɑm - wind launches you up (no matter where you are looking), not shown in the video

Additionaly, it can be cast as just ʔalɪˈvɑn vɛɾtɛɾ ɛvi, which is a shortcut for the first variant.
This can seen in the third attempt, where the system didin’t catch onːdˌoː, but the spell worked anyway.
This flexibility is easily possible, thanks to Mojang’s brigadier library. Brigadier powers the / commands in vanilla, and now the spell syntax and execution in my mod. I settled on it after trying two alternative approaches and failing to make them work. After all, my spells are just sentences, with the only difference, is that you speak them instead of writing.

0
0
47
Open comments for this post

10h 37m 24s logged

Note: This one is kinda boring, mostly plumbing stuff to get things to work, feel free to skip it.

Sephyr (the mod)

I’m working on setting up the the logic for registering and casting spells. This is quite hard, as the design of the mod is far from complete, which means I don’t even know what exactly do I need. I created a registry of elements, which are a source of a color, and are going to have other effects in the future. The first word of a spell is its element. For example in the spell ʔalɪˈvɑn vɛɾtɛɾ ɛvi the word ʔalɪˈvɑn means wind. I also made a basic system for figuring out the spell the player is casting. For now its very simple and can’t handle more complex spells, for example, with optional words.
The entirety is a big mess as I’m figuring things out, and the fact I’m making this mod for both Neo and Fabric doesn’t make it any easier.
I also updated the mod to Minecraft 26.1.2, which means there is going to be no 1.21.1 release.

Sepple (word detection system)

I keep working on collecting data to finetine the speech to ipa ml model. After discovering a better way of getting the audio files while writing last devlog I scraped the download script, and instead replaced it with a new one to extract needed files from the 20GB Wiktinary audio dump tar.
I also cleaned up the entire code, turning it back to human standards, after a bit of Opencode usage. (Since those are one-time scripts to extract data I decided using a bit of AI in this area won’t hurt. I don’t use AI for the rest of the project.).
I also prepared a fourth script, that will use the existing multipa model, to generate IPA transcription of the files. Then files with generated IPA being too different from original IPA will be discarded, as a way to detect files, where the IPA doesn’t match the audio (because the audio contains more text for example).

0
0
33
Open comments for this post

10h 14m 8s logged

Sephyr (the mod)

  • A proper lifecycle of the detection system was added. The IPA model is loaded in background during game start, but the audio capture and processing start waits until you join a world, and stops once you leave it.
  • The Rust code now has a logging hook, so that it can do civilized logging using the Java logger, instead of blatantly writing to stdout.

Sepple (the voice casting detection system)

  • I made the pipeline system cancellable, so that the mod can cleanly stop the detection, which was not possible before.
  • Support for running the speech detection inference on GPU was added as a feature flag. For now it is not used by default, as it showed no measurable improvements over running on CPU on my PC, but it might be useful on other, weaker devices.

Collecting data for training

As I said too many times, the speech to IPA model used for magic speech detection is not quite the quality I would like it to be. I’m hoping to improve it by fine-tuning it on its weak areas, but for that I need some data.
My idea is to use wiktionary as a data source. It is a free online dictionary run by the same people who run Wikipedia. Most words on it have the pronunciation attached both as an IPA and an audio file, which makes it ideal for the task.
I downloaded a preprocessed jsonl dump file from kaikki and wrote a script to extract just words and ipa from it. Unfortunetely it was missing urls of the audio files. So I had to download a raw xml dump of the entire english wiktionary, and spend few hours writing a script to extract them. Then I used deepseek a generate a third script, to download all the audio files from a list. There is going to be almost 20k of them, and the script is running right now as I’m writing this, and it is going to take multiple hours as I can only download around 1 file per second to respect rate limits and not overburden wikimedia. And just now while writing I realized I’m an idiot, because kaikki provides a 20GB tar file with all the audio, downloading which would be more efficient.

0
0
11
Open comments for this post

9h 36m 38s logged

Integrating Rust IPA word detection with Minecraft

With the IPA based magic keyword detection (which is written in Rust) working I spent all of this devlog integrating it into Minecraft. I created a new repo for the mod, using the Jared’s template. I decided to go with supporting both Fabric and Neoforge, and picked the 1.21.1 Minecraft version, though in the future I’m planning to move to 26.2 or later. I set up an additional gradle subproject and a rust package, that both together act as a Java to Rust bridge, using the j4rs crate.
Using it I managed to initialize Sepple (the detection system), and get it running. The tricky part was going the other direction, to pass detected words from Rust to Java. For some reason Rust wasn’t detecting the JVM and was trying to spawn a new one!
After a bit of experimentation I concluded, that I can only call Java from Rust only during the period when Java calls Rust, and only on the same thread. In the end I managed to get it working, by spawning a new thread on the Java side, and calling a blocking Rust function, that kept calling a Java callback passed to it. Originally I tried to use a static function as a callback, but for some reason (probably some kind of classloading isolation) the system was failing with some crazy null errors.

I also had to adjust Sepple a bit, to not require a bunch of random json files at specific paths, and to provide an mpsc chanell directly outside its pipeline, for less boilerplate.

With that I have a proof of concept working! Minecraft is receiving the detected words and printing them to chat.

Note that those are all custom words, created by me specifically for this mod. This is why I had to create an entire new system for detecting them, and why they are written in the International Phonetic Language, which is also why they look a little weird in the minecraft’s font.

The Java<->Rust setup is also not very good yet. For example the packaging of compiled Rust is very basic, and you can’t control the word detection from Java - no way to stop it or restart. There is also no good error handling, which means if a file is missing it will just crash everything.
But it works, and I can continue working on the mod, and making it better!

0
0
29
Open comments for this post

10h 10m 19s logged

About project

Sephyr is a magic mod, with spell casting powered by speech live translated into International Phonetic Alphabet, to detecting custom-made spell words in it.

Adding Voice Activity Detection - part 2

After the refactor I was able to finally add the VAD. But after few tests I noticed that the VAD was making things worse, not better!
That was as a part of the previous devlog (which ran out of character limit).
I added a pipeline part that writes every chunk of audio that passes through it to a file, so it can be inspected.
With it I found the main problem: It took few hundred milliseconds before the VAD detected speech, so a small chunk was always cut off from the start. To fix this I added context keeping: A bunch of previous chunks are kept, and when speech is detected they are also taken into consideration.
I also made the sliding window chunking aware of VAD - the VAD detector sends a signal when it detects an end of speech. The chunker then flushes its buffer into the pipeline, so that it doesn’t get stuck until next speech, and replaces it with precise amount of silenece. This way the start of next audio doesn’t get cut off later due to the sliding window mechanism.
The author of the bunsen crate (providing the VAD model implementation) also released version 0.29.0, which fixed all the issues I encountered in the last devlog, and updated its API slighly, which meant that after adjusting my code I could stop using my own fork of it.


Improving word detection from IPA

My friend found a crate named phonetics-rs and suggested using it to improve the algorithm. This crate calculates the difference between two IPA strings, taking into account how similar they sound. This is much more accurate than just comparing strings on grapheme level. I modified my algorithm to use it, and it turned out to be a great idea, that significantly improved the detection rate of magic words.
I also changed the dictionary to be sorted by longest words first, so that they take priority over short ones.

So far, testing and improving the word detection algorithm has been the most fun out of the entire project. Looking at IPA strings, and figuring out why some speech has or hasn’t been classified as some word is quite fascinating. For example, I encountered a string pɾilifɛra which the model output after me saying prizim and fɛra, the two magic words that I made up for the spell system. Surprisingly, my algorithm said the string contained just prizim despite there being a literal fɛra in it.

As I found out, pɾilif is closer then pɾili to prizim, and it fell under the threshold, so the algorithm took that part and left out ɛra in the buffer which was too far from fɛra. After seeing this, I made the algorithm scan the string for exact matches first, and then do fuzzy phonetic matching, also ensuring the order is preserved. I also tuned the thresholds, adding more and more tests (there is now 8 of them) to ensure the algorithm behaves well.
Finally, I noticed the IPA output from the model changes based on audio’s offset in time. So I wrote a script that lets me test this, and see the effect of different offsets.


What’s next

There is a saying in data science: garbage in - garbage out. This is also true for my project. The detection works decently, but it also quite often outputs false positives during normal speech. I believe I did I all could to compensate for the unsatisfying quality of the speech to IPA translation model.
Now I will be shifting to the Minecraft part of the project - creating a mod, integrating the detection system into it and creating spells. In paralell, I will be exploring the possibility of finetuning the model, to improve the accuracy of it and hopefully get better results.

Watch the video with audio, that’s the part that matters right? :)

0
0
6
Open comments for this post

30h 24m 35s logged

Finding words in long IPA strings

For the magic system I need to find individual words in the ever accumulating string of IPA characters. To do that I hold a list of valid words in memory. When there is some text, I iterate over the dictionary, calculating a levenshtein distance between each word and start of the string. Then the word with the lowest score is taken and if the score is under a set threshold, the word is considered a match, and the matching part is remved from the buffer. To cover more cases different lengths of the string are checked. If no match is found, one character is removed from the string, and the process repeats, until all words are found.

Fun fact: When comparing IPA and other non-ascii strings you need to make sure you are considering graphemes and not characters, because in UTF-8 what we perceive as one symbol, can be encoded as many characters in memory.

So far this algorithm works quite well, though in the future it might need some performance optimization, or tuning, if it starts yelding too many false positives.


Other misc changes:

  • Migrated main fn to use clap for parsing arguments
  • Added flake.nix with dev env so I can run this code on my NixOS laptop
  • Added time information to the pipeline - Together with the audio from the mic, a timestamp is captured, and passed through entire pipeline up to the end, where it is used to discard text older then 200ms from the buffer, so that stale data doesn’t mess up new words.

Adding VAD

I noticed that the speech to IPA model likes to output random data when no one is talking, so I decided to add VAD - Voice Activity Detection - a system that detects if someone is talking, and only then passes the audio on. I estimated this to take me around 2 hours, and oh, how wrong I was?
It took 10x that, which is why this devlog documents 3x the normal amount of time. And it still isn’t fully ready! So let me tell you what I did and why it took so long.
In the Burn (ML/AI lib for Rust) discord I noticed a solution just dropped out of the sky: Merely few days before a Silero VAD model implementation was added to a library named bunsen. ML models take input as Tensors - multidimensional arrays, which do not say in what form or shape the data is expected. So I had to figure that out myself. Fortunetely it was just 512 elements long chunks of 16khz audio with no additional pre-processing. Knowing that I fed the microphone input to the model and began to watch it output complete nonsense unrelated to whether I was speaking or not. So I started debugging - checked my code, wrote an equivalent in python using the official silero implementation to compare results - completely different. So I cloned the bunsen library, wrote an unit test there and ran off a Opencode agent. I also contacted the library author. He found one major issue, and the agent identified a second problem and wrote a solutuion - Silero VAD expects 64 samples of the previous chunk to be passed along the 512 to work correctly. I cleaned up the code, which was quite a pain because rust-analyzer was super slow on that repository, and made a PR, but it wasn’t merged because the author went with his own way of implementing it.


The big refactor

Coming back to my project I realized that my code strucutre makes it very hard to integrate VAD into it. So I spent the next 6 hours of so refactoring almost every file in the codebase into a new pipeline system. It lets you compose a pipeline out of a producer, many processors, and a consumer. It makes use of async, which saves on the amount of threads used. Static type checking ensures you can’t compose parts that wouldn’t work together, and all elements can also define contracts on the size of data chunks they produce and consume.

I ran out of the character limit, so the rest of VAD story will be in next devlog.

0
0
6
Open comments for this post

4h 52m 31s logged

Live speech to IPA transcription in Rust using ML

I finished my rough implementation of a live speech transcription pipeline. It captures my speech from the mic, and processes it with the Multipa model using the Burn framework.
To make it work live I implemented a sliding window that chunks the audio into 2 second fragments, each of which is processed separately, and then assembled into a single string. The chunks overlap, and are then sliced, so that audio that was on the borders of a cut (which will reduce the quality) is thrown away, and the text for it is taken from the next chunk, for which it is at the center.

The inference is running on cpu for simplicity and compatiblity, so by sacraficing a bit of pain I added threading (where each chunk is processed on a separate thread) to make sure the pipeline can keep up. I’m slightly worried about how the perfromance will be when this is going to run along Minecraft, but hopefully I will be able to make it as lightwieght and unintruisive as possible.

Finally I reworked how the text is accumulated, switching from a vector of logits to a string. Since IPA has no concept of words, just phonemes, everything I say is appened to one long string.
So I added a very simple silence detection, where if a captured chunk returns no text a space is added to the string.

For easier teasting I also improved the main function, so that the program can:

  • take a file and process it all at once
  • take a file and process it using the pipeline
  • run live mic capture on the pipeline

What’s next - improving the detection

In theory this is all I need, and I can start working on the Minecraft mod. However I’m not satisfied with the quality of the transcription, and at the level it currently is most spell detection will probably fail.
The quality of the model itself is pretty bad. If the description on github of the Multipa model is correct, the model was trained on merely 9 hours of speech data, which probably explains it. The model is also sensitive to noise, and distractions like other people talking, and sometimes outputs garbage.
So the next devlog I will be experimenting with various techinques of filtering and cleaning up the audio the model is receiving, and potentially even training a new model myself on a bigger dataset.

Attached video

Working demo of live capture and transcription of me talking.

0
0
3
Open comments for this post

3h 45m 57s logged

Sephyr - project overview

Sephyr is an unique magic mod for Minecraft. It aims to differentiate itself by:

  • Introducing a voice powered spell casting system, that’s both fun to use and intricate. You will be reading spells out loud into your mic in a magic language, and what will you say will affect the result of the spell.
  • Having stunning, detailed visuals unique to each spell. This is going to be achieved by spending way too much effort on rendering. I want spell casting to be something satisfying, and the look is a key part of it.

The project consits of two parts:

  • Sephyr - The minecraft mod itself containing the spells, mechanics and visuals, written in kotlin.
  • Sepple - Speech processing engine, responsible for transforming your babble into something Sephyr can work with and figure out what have you just said.

Sepple overview

Sepple is a foundation of the voice powered spell casting. It is an audio processing library written in rust, that Sephyr will depend on and embed. It captures your mic live and pre-processes the speech. Then it uses the Burn framework to locally and quickly run a speech recognition model called Multipa. The model transcribes speech into the International Phonetic Alphabet (IPA), a textual notation of various sounds that make up the spoken language. Finally it tries to match the result against a custom dictionary of magic words, which will form sentences - the spells you will be casting.

This devlog

Before Stardance launched I have written a rough skeleton for Sepple. With some trouble I managed to import the Multipa model to Burn via ONNX export from Pytorch, and implement the required post-processing to decode model’s output into IPA. I also started working on a pipeline for mic capture and sliding window processing, so that the audio can be transcribed live.
After Stardance started I noticed that the same input audio file gives quite different result in Pytorch then in Rust. I spent few hours debugging why that is, and found a missing piece in my implemenation. For the Multipa model to work correctly the samples need to be normalized using a so-called z-score normalization. That means calculating the mean of the input and its standard deviation (a statistics measure), and applying a formula that rescales the data, so that it has a mean of 0 and standard deviation of 1. After implemenating that the model started to work correctly and give the same output for the same input on both sides.

Picture: “Hello Stardance. Sepple is working now” translated to IPA using pytorch and the fixed Rust implementation. It is not veeery accurate, but it is good enough.

0
0
2

Delete project?

Are you sure you want to permanently delete this project? This action cannot be undone.

All devlogs, followers, and associated data will be removed.

Followers

Loading…