I made a couple improvements to the current system that caches site categorisation locally. Previously, it would simply keep categorisation valid for 30 minutes, then invalidate it and require another request to the server. Now, it instead runs on a loop that, every 10 minutes, sends all its cached site IDs (which is just an internal incremental ID for each hostname listing) to the server and gets back a list of versions, which it then checks against its own versions and updates only the ones it needs. Currently rewriters are not implemented, but they will be bundled in this system.
I wrote a simple test in Ruby to test this functionality and modified the extension code to act differently when in dev mode vs a production build, which helped a lot in verifying that it was working correctly and finding bugs. It was pretty annoying to make (I probably should’ve just used Python or JS, not sure why I didn’t) and is pretty buggy itself (it doesn’t properly stop the backend).
I also did a whole lot of planning. I made a flowchart attempting to detail how the whole system will work and wrote a MD file to plan out how the actual database will be stored. One thing I still need to decide is whether to use MariaDB or PostgreSQL.