Packet Types

After a (way too long) pause, I’m finally working on Simple Chat again. I’ve added a packet enum, which wraps all possible kinds of data communication.

What this means

Basically, now everything that’s being sent has the same interface: serialize and deserialize. You don’t need to worry about what is in this packet, just serialize it, encrypt it and it works.

This means handshakes aren’t loose bytes being sent between client and server and held together by pure hopes and dream (im praying somebody gets the reference), but instead a proper packet, which can also now be encrypted.

Heartbeats

Previously, there was also a big design flaw. If no message was being sent, you couldn’t tell if the others were just not sending anything or had already (uncleanly) disconnected. The TCP stream looked the same in both scenarios. Because of this, I’ve added a Packet::Heartbeat type. It contains no data, and if it gets received by any party, (almost) nothing happens. It’s sole purpose is to keep the stream alive.

Now with this, lets say we’re the server: a client doesn’t send a heartbeat twice. We now know that they’ve likely crashed or not properly disconnected in some other way. Because of this, we can now kick them instead of keeping a connection live.

Server Rewrite!

I’ve rewritten the server using Tokio instead of Threads for improved performance and less server-side overhead.

What Tokio is

It utilizes Rust’s async implementation with futures to create a much more lightweight alternative to threads. Basically, you can summon multiple tasks and instead of them being full threads with seperate 8MB stacks and so on, they have way less context switching overhead and a much smaller stack per instance.

Other Server-Side Changes

In the initial handshake, the client provides its name to the server. If the server receives a message with the wrong name from that client, it will replace the name with the originally given one.

Encryption

I’m still working on finishing up encryption for all messages and the initial handshake.

im almost done already what

So… it kinda works????
The client AND server, the TUI, the new message structs and all of that work now.

Changelog

• Reworked Message Struct(s) and multiple message types (Chat, System)
• Proper connect handshake
• Rework server to use SystemMessages to broadcast client connect/disconnect events

What’s planned

• A config file to store your name (~/.lan-chat)
• Encryption (between clients)

TUI Time! :)

The TUI works now, I only need to polish it (stuff like scrolling when message box is full)

Additonally, I’m planning to add a special kind of message that the server sends on client join/leave events.

New Project!!

This is LAN Chat, a TUI chat messenger to use in the same LAN as others.

Build using Rust with ratatui (a TUI library)

Currently, I have a working client/server system, which sends serialized message structs. Next up, I’m gonna work on getting the TUI running.

DebugWrite syscall!

The userspace process is now able to ask the kernel to write log messages! This means that finally, you can actually properly do something from the userspace program.

Next, I’ll be implementing userspace page fault handling

Syscall Dispatching, Syscall returning and Exit Syscall

Finally, after all this time: User programs can now interact with the kernel!! :)

Changelog

• Added Syscall enum which represents all possible syscalls and their respective IDs
• Added SyscallArgs whichs contains syscall number and args in an architecture-independent format
• Added SyscallResult and SyscallError
• Added syscall dispatching logic
• Fixed kernel data leaking into userspace via initial_userspace_jump by clearing CPU registers before the jump

the end of stardance is approaching… i wish all of you a lot of fun for the last 9 days.

Syscalls!

Changes so far:

• Added gs_info struct, which saves the user stack and holds a pointer to the kernel stack
• Added syscall::init which enables the syscall/sysret instructions and sets up the STAR, LSTAR and SFMASK registers
• Added syscall_entry which builds a UserFrame (a snapshot of the CPU registers on syscall entry) and calls syscall_dispatch
• Added syscall_dispatch, which currently logs the received syscall number and halts (the system call return path is not implemented YET)
• Added UserFrame which holds a CPU register snapshot which gets taken upon entering the kernel

ELF Loading Refactor and improved build system

After a week of basically no progress, I’ve finally finished moving the ELF init sequence from main.rs into the proper methods under the sched subsystem.

I’ve also simplified and improved the build system (build.py/docs.py). They now use a shared lib.py to avoid code duplication. Also, build.py got a few changes:

• Removed build cache (cargo already does it)
• Swapped console restore for proper serial control char output filtering

Bug Fixes

The docker permissions have been fixed, as under Linux, root would own parts of the build files which is of course annoying as you can’t easily interact with those files then.

The github-hosted rustdoc was failing to build as it could not find USER-BINARY, so I just faked user-binary via touch user-binary.

time to go to bed yippeeeee :D

Finally… after all that work, we’re back!!

After almost 30 hours of work over the last week, I’ve finally finished the big refactor.

Changelog of the refactor

• Rewritten kprint backend
• Restructured KSTATE
• Hit my head on the wall A LOT
• Added a few new datatypes and macros
• Optimizations all over the place
• Added quite a few TODOs (more work 🥲)
• Restructured the source tree
• Changed the entrypoint (kernel now starts in an arch-specific kernel_entry, which builds a BootInfo and hands to kernel_main)
• Moved the userspace/ELF stuff from main.rs into the sched subsystem
• Rewrote the UART driver

… and so much more.

and of course i organized it all into one slightly large 144 file commit :)

Big changes are still happening

oh boy this is taking way too long

Rewritten kprint

kprint, the kernel print macro has been fully rewritten. The public macros have not changed, but internally, the macros now:

• Construct a kernel writer
• Call core::fmt::write on the kernel writer

The kernel writer collects the text of all calls made to it using core::fmt::write in an internal buffer, and on drop, it pushes a new entry into the kernel log ringbuffer in a format that is (currently only partially) syslog-compatible. Then, the kernel writer dispatches a log header and the full log text to all active log recievers (like UART or the basic framebuffer).

New Types

• PackedU8: A type which wraps all the bitwise logic for packing multiple pieces of data which are less than one byte into a byte
• UncheckedCell: A datatype with interior mutability, which can be uninitialized via MaybeUninit. When the cargo feature debug-checks is enabled, this type also checks for double initialization and uninitialized access
• NicheCell: A datatype with interior mutability, which can be uninitialized via Option. This datatype is designed to be used with values that can take advantage of niche optimization (where the datatype T has a value which is guaranteed to be invalid, like a NonZeroU64 being zero)

and oh boy, the big refactor: redid kernel source tree, ... commit is gonna be one hell of a mess…

Big changes upcoming…

I’ve decided to change what the kernel aims to be and how it will work. I’ve dropped full linux drop-in compatibility in favour of creating a clean, modern and optimized kernel which tries to fix design mistakes other kernels have already made.

New Design

The kernel will be POSIX-compatible, feature a namespaced VFS to seperate virtual files or devices like /dev/sda, /sys/class/hwmon, /proc/self/ and more from actual files. It will have namespaces like fs:/ for the actual file system, dev:/ for device files (like dev:/sda), vdev:/ for virtual devices (like vdev:/urandom) and more, including hw:/ for a modern hardware info interface, log:/ for a standardized logging interface and reg:/ for a global registry-like configuration system. Internally, the kernel will also feature KSTATE, a central static struct for storing everything the kernel uses. Also, ther kernel will move from strictly limine to a dynamic bootloader format, where each architecture has its own kernel_entry, which prepares the CPU and kernel into a common state (long mode, paging, …) and builds a standardized BootInfo struct, which then gets handed to kernel_main when it is called.

New Source Tree

I’m also currently working on massively reorganizing and cleaning up the kernel source, by for example standardizing where arch-specific code lands, categorizing and organizing the code better and more. Also, the kernel print system (kprint) is currently being rewritten to improve performance and integrate it into KSTATE.

Refactoring hell is here again!

Before I start on syscalls for the userspace binary, I’ve decided to first get most TODOs done, like updating the safety comments and rustdoc, refactoring the ELF loading stuff to properly utilize KSTATE::procs, proper error delegation and more.

I’ve finally finished refactoring the TSS, GDT and IDT properly by removing the struct wrapper for GDT and IDT, getting rid of all the Onces and more. Additionally, I refactored KSTATE::cpu to be an arch-independent container of arch-specific structs.

holy crap ive got so much left to refactor… and the new bodycam update just came out i wanna play but am stuck doing kernel stuff and some other stuff :’)

@everyone… WE HAVE USERSPACE!!! 🎉 :D

This is something I’ve been waiting for sooo long… a binary is actually executing in USERSPACE now (ring 3 and stuff)!!!

How it works

Currently, the code is extremely hacked together, or “gepfuscht” as we like to call it in Austria ;). Basically, what should be cleanly integrated with KSTATE::procs, should use a proper userspace and kernel stack and so on, doesn’t. All of this of course won’t stay like that, but it currently is like that to just get something in userspace running. The one thing that is actually done cleanly is the ELF validation and definitions, which live under src/elf. There is a struct representing the ELF header, an ELF PHDR and also an enum for ELF validation errors. All other logic was just thrown into main for now, like creating the processes address space, mapping its memory, copying the binary into the mapped memor, setting up its stack and actually jumping to it.

The current testing binary

Written in good old assembly of course, the test binary currently has the job to cause some sort of CPU exception, as syscalls or any other way to interact with the kernel or output something doesn’t even exist yet.

The current program manages that via just running the hlt instruction, which, when run from userspace (Ring 3) causes a general protection fault (#GP). This is because a userspace program shouldn’t just be able to halt the CPU or disable interrupts (via the cli instruction). That would defeat the whole purpose of rings and kernel/userspace.

The userspace binary currently lies under src/user-binary.asm, next to src/kernel/. It is build via running docker exec ferrite_os /ferrite_os/build-user-binary.sh. NASM was also added to the Dockerfile to make building the assembly file possible.

Virtual Memory Management is complete! (for now)

I’ve added methods for mapping pages and VMA together. I’ve also improved the kernel VMA granularity and added a kernel paging remap step to mm_init() which applies the kernel VMA permissions to the kernel pages. This is only intermediate until I switch to the linux boot protocol.

Next up, I’ll be adding a (very basic) ELF loader!

Lots of stuff

 
Changelog for the last two days

• Fixed the clean option in build.py
• Add kernel VMA setup method
• Update rustdoc in several places
• Add panic types to kernel_panic output
• Move SIMPLE_STATE out of main
• Improve per-arch code layout

… and an INSANE amount of optimizations, including:

• Marking certain code paths as cold (which signals the compiler to expect that the code path is unlikely to occur and optimize accordingly)
• Remove a lot of overhead introduced by Once and similar by swapping them for UnsafeCell<MaybeUninit<T>> and others and taking advantage of things like niche optimization
• Using bit-packing instead of seperate booleans for state tracking

Next up, I’ll be doing some more work on the VMM and finally add the full mapping methods, which map pages and VMAs together.

AddressSpace and VMA data structs done!

I’ve introduced a struct called AddressSpace which contains the virtual address of the root page table (the PML4 on x86_64) and a BTreeSet of VMAs (virtual memory areas).

A BTreeSet is basically a type of set which organizes data in a tree-like structure (in a B-Tree structure to be exact) with a time complexity of O(log n) for inserting, removing and searching for elements.

A VMA (virtual memory area) is an area in memory with specific flags, such as READ, WRITE, EXEC or USER. One of their main uses is determining what to do on a #PF (page fault): should we allocate a page here (lazy-mapping)? Or was this truly a faulty operation like trying to execute READ-only memory?

Also, I briefly attempted to switch over to the linux boot protocol, but I decided to do that sometime later, because it is slightly complicated and definitely did not break everything ;)

Finally finished fixing the PMM free method!

After needing to quickly relearn the logic of the free method because I somehow forgot it, it finally works again!

Additionally, I’ve added a little safeguard to the bootloader memory reclamation. Next, I’ll be continuing on the VMM and refactoring it to support userland page tables too.

Also, I’ve added a few core::fmt traits to PhysAddr and VirtAddr and also added a constructor which creates a new address type (PhysAddr or VirtAddr) with an address of zero. In addition, PhysAddr got a few new getters like as_ptr or as_mut_ptr.

Memory Stuff and General Fixes

Currently working on the PMM and VMM a little more, also cleaned up the main for better readability.
I also moved the PMM and VMM into the mm subsystem of the KState struct.

Additionally, I switched from rust 2021 to rust 2024, which also needed a little fixing to get working again. Why? Well to use let chains in if statements of course cause its like a little bit nicer to write! :)

Also, I partially upgraded scripts/x86_64/build.py to log to a log file and restore your terminal config on termination.

Currently working on getting a bug in the PMM fixed, getting bootloader memory reclaim to work and then I’ll do the VMM’s VMAs.