I JUST DESERIALIZED AND RESERIALIZED A UPK. OMAIGOD IM SO HAPPY. my brain is so tired. Alright so the payload still doesn’t get transferred but that’s pretty simple, it’s just Vec::extend.
What a challenge! Deserialization actually ended up being the easy part, which is opposite of what I expected from starting the project. It went (relatively) fast since there are a few existing implementations which I could kinda just “borrow” structure parsing from.
But, mine is unique in that it almost completely reserializes the package itself. I might’ve been able to “borrow” reserialization, but the impl I’ve been looking at is STRAIGHT ai slop and it’s more annoying to read that than to just do it myself. It was useful though, so I have to respect it.
The first problem: basically, the first byte of an FString determines how many parts it has, NOT how many bytes it has. Which wasn’t a problem for parsing, since I had some word-based helpers which worked so I didn’t think much about it. But, when reserializing, I made two mistakes: 1. forgot to add the null terminator (which shouldn’t even be necessary) and 2. I accidentally put the wrong offset.
I have no clue why there’s a null terminator for FStrings. The format is [n chars][the char bytes], meaning it should already be known how long the string is. Pretty stupid, but whatever. Anyways, when reserializing you’re supposed to include the null terminator in the string length marker. So, my logic was that I should offset it by two since unicode characters are two bytes. Bleh! I should’ve only incremented by one, since it’s in terms of utf16 chars rather than bytes.
Yay, that worked! I don’t know how to summarize the next problem, so I’ll explain the reasoning. The package header is file summary + tables + something else (we’ll get to that later). Anyways, there’s some mysterious padding in between the summary and tables. Parsing, it’s easy to skip over because the summary includes the tables start offset (name_offset, the name table is the first one). But when reserializing, you gotta keep those offsets intact. Well, I’m not totally sure if you actually have to keep them, but just for the sake of simplicity I tried to (which ended up adding more complexity, which I’ll see if I can remove now). So, I serialize/push the summary to the big Vec. Then, I make a new Vec and serialize the name table at (name_offset - summary_size), since the new Vec has to be encrypted and then appended to the big Vec. I do the same thing for imports/exports (placing the table at import/export _table_offset - summary_size).
Oh no! Since name_table_offset isn’t equal to summary_size, there’s some padding in between, which is expected. HOWEVER! The previous logic would skip over it in the table Vec creation, meaning the table Vec would have some 0-padding at the start. This padding needs to be in the SUMMARY part instead, and the name_offset should basically be the ZERO of the tables Vec!
I doubt I would’ve been able to solve this part without the beautiful tool HxD. It’s really easy to use and is always helpful when I’m figgling around with binary.
So, it does some more math, calculates the spacing, adds it to the big vec, and removes it from the tables Vec. Then, it re-encrypts the tables Vec and pushes it onto the big Vec! Okay, now that’s done. NOPE! There’s a mysterious structure for some reason at the end of the real table part? Yep, it’s that something else I said we’ll get to. I actually still have no idea what it is. From some research, I gathered it must be the dependencies table, but it’s really poorly documented. So, I ended up just cutting the bytes out from after the export table to the end of the tables part, and “pasting” it back before re-encryption.
I do hope this devlog was readable. Now I’ll get to including the actual package payload and hopefully simplifying reserialization offsets. See you soon!