Finished proxy - but I can’t use it

I finished writing the proxy. I had to:

1. Retrieve the provided containers internal IP
2. Generate a random IPv6 address within the given prefix and add it using netlink
3. Start a proxy, listening at the generated IPv6 for each port, and proxying it to the containers IP.

This works, but unfortunately I can’t use it!

My server provider, datalix, gives a /64 subnet for each VPS, but what they don’t tell you is that you can only have 10 routed IPv6 addresses! This is not enough, and I have to manually add the addresses anyways. This means that I can’t listen on the generated IP address, and also that I can’t give each container an IP. This is a big problem, because some applications (like a minecraft server) require a plain TCP connection, which does not work behind a reverse proxy since there is no hostname passed.

My only option is to rent a dedicated server, which I can’t afford right now.

So now I will continue work on other parts of the project, like making it viable and easy to deploy different services, offer one click apps/database and potentially work on a way provision servers from the cloud.

Commits:

1. Retrieve container IPv4 address on expose command
2. Finish proxy

Finished proxy communication

Wrote a library to make receiving and sending messages really simple. Uses unix sockets so they are pretty fast.

The server is started in the background, then different commands are received through channels.

On the client, the connection is started in the background but c.Start() waits for the connection to change to connected.

Then messages can be sent, but the connection has to be sustained using c.Wait() which waits until the server disconnects.

Now I can start work on querying the docker container for its internal ip address and proxying it.

Commit

Started work on the proxy API

Not much to show, but I started writing code to create a unix socket to listen for messages. I’ll need to pass a container name, and array of ports to publish. I’ll also have to figure out how to pick new ipv6 addresses

long time no see

I kinda hit a roadblock and waiting till I got past it until posting a devlog, but that never happened.

That roadblock was trying to give each container a public ipv6 address. I was trying to get docker to expose ports on a specific ipv6 address on each container, but no matter what I did there still ended up being port conflicts.

I found a better alternative now though, which is writing a proxy service that binds to an ipv6 address, and forwards the traffic to the containers internal ipv4 address. This works, but now I need to write an API so my virtual server can specify a container name and ports to forward to a public address, and dynamically create these addresses

I still worked on other stuff though, so here’s what I have changed: (Sorry for massive devlog)

1. The docker virtual server now syncs products automatically to the platform. Before it had to be done through a cURL command.
2. Add a default postgres url to configuration
3. Add a Key-Value input (UI library)
4. Add support support for Key Value inputs to platform
5. Allow users to configure volumes
6. Simplify virtual server configuration by removing suboptions
7. Allow regex validation for text inputs
8. Fixes for KV Input: 1, 2
9. Add a README
10. Disable email verification by default
11. Style scrollbar on chrome
12. Make error message more detailed
13. Allow users to specify port protocol with regex validation
14. Add a database utility to makefile

Lotta Features!: Toasts, Container Recreation, and Ability to reconfigure services

1. Added toast messages - needed a better way to show error messages so I implemented toasts. These were implemented using a MutationObserver, which watches an element for changes, in this case for new nodes being added to it. This lets me automatically remove toasts after a certain time and fade them out. Using a MutationObserver allows toasts to be added using HATEOS (HTMX) instead of only being able to create toasts from JavaScript.
2. Added the ability to reconfigure services. Before only the value was shown, but now you can update configuration and save it. Added an API endpoint so virtual servers could request new service configuration. Also reworked code to use toasts, and generally improved code quality by moving code into separate functions
3. Added the option to recreate containers. This is needed to update the container image, and/or refresh configuration, for example new ports or even a different image. This depended on the changes in 2.

1. Toasts
2. Reconfigure services
3. Recreate containers

Feature: Service Actions

Virtual servers can now list actions for the user.

This is fully programmatic, so when the virtual server sends statistics and links, it also responds with a list of actions like:

{
  "label": "Stop",
  "code":  "stop_container",
  "type": ui.ButtonTypeDanger,
  "confirm": {
    "title": "Are you sure you want to stop this container?",
    "description": "Application data may be lost.",
    "button_label": "Confirm"
  }
}

Each action has an optional and configurable confirmation dialog. Doing it like this allows the virtual server to have different actions based on the status of the container for example, like start if its stopped and stop/restart if its running.

I also added a loading indicator to all buttons, which shows whenever a HTMX request is being made.

1. Sort links by name (to prevent them shuffling)
2. Add service actions

Feature: Service overview page

Shows the status and utilisation of the container, and also the links.

The links are found by parsing the containers labels and extracting the traefik rules.

Fixed a few issues with the docker manager:

1. Only one manager was started but the channel width was 5, meaning only one service was handled at a time. Now I start 5 managers, so 5 containers are handled at once.
2. Made it so loading services at start runs in the background as well, so the API is available a lot quicker
3. Each port uses a different service name in traefik, not sure if this was an issue but could of potentially caused problems when a service has multiple ports

Also added loading indicator and status dot components.

Next I’ll implement manual start/stop of containers (general action buttons), then service deletion and then volumes

1. Service overview page
2. Loading indicator/status dot

Feature: Fully automatic reverse proxy

I added support for configuring ports, then implemented the Traefik reverse proxy giving each port a unique (non random - based on service id and port) subdomain. The service is given automatic TLS (error due to self signed cert - when I deploy I will have a domain and use letsencrypt)

Next I need to add a better display, so the link shows in app

Commit

Feature: Parse and store multi-input values

1. I allowed configuration of multi inputs, to set the min and max values/length of number/text inputs
2. I parsed and stored the values on the backend, so they are now available to the virtual server

I need to:

1. Allow updating configuration after creation
2. Figure out how to implement a reverse proxy with variable subdomains to expose ports

1. Configure multiinputs
2. Add parsing + storage

Feature: Multi-Inputs

I added an input to my ui library that lets you add infinite entries. Supports text/number types. Also fixed an issue where modals could overflow the screen.

Built this so I can have better configuration for my docker service, i.e to set ports and volumes.

Haven’t implemented the backend storage for these but the components work

Commit

Feature: Tabs

The service page looked pretty bad so I added tabs to my UI library.

Commit

Feature: View all your services

Doesn’t show much rn

Had a nightmare figuring out why the service list was overflowing the page. Solution was somehow to but min-width: 0 on the container 😭

Commit

Feature: Load existing on services on start + async container manager

I added an endpoint to the platform API (GET /api/services) that returns all of the clients services.

On start, the virtual server will request services to run, and start them one concurrently using golangs channel features (containers to run are added to a queue where up to five containers are handled at a time)

1. Add async container manager
2. Add API endpoint and start containers on virtual server start

Feature: Run docker images

I implemented the very basics of running a docker container, and added a logs endpoint

Definitely need to improve the UI for the service page, and make a tabs component. Also need to make stuff async so I don’t wait for the image to pull

Commit

Feature: Configuration groups

I added the ability to set configuration groups. These are purely visual. A virtual server may set whether the options are visible by default. For example to hide addons or advanced config options

Commit

Feature: Order services

I implemented the backend for ordering a service.

Diagram explaining flow
Text scored with red has not been implemented yet

It works by

1. Storing the service and configuration in a database
2. Sending a request to the virtual server to run the service (signing the message so the virtual server can trust it)

This setup is pretty complicated but it allows for virtual servers to run virtually anything and have configuration options for all of it.

I have yet to implement:

1. Per addon pricing (e.g +1 credit for +1 GB RAM)
2. Configuration groups, which would allow for sections of configuration (e.g simple, advanced, and addons)
3. Advanced options (required, selects etc)
4. A proper virtual server that actually runs stuff
5. A good service page

1. Implement service orders
2. Disable spinners on number inputs

WIP Feature: Get service modal

I implemented a modal that loads on click, and has all the addons/options of the service.

I need to style number inputs better, and make it so servers can set min/max values and lengths

Commit

Feature: Programmatic services

You can now add services through an API, using a generated client id and secret.

A server would call the API, passing a list of products to sync. You pass a name, description, unique code, a price and a pricing schedule (per hour, day, or month). You can also add addons with options, currently only text or numbers (e.g for additional RAM).

I also added a page to view all available services.

Next I’ll implement “getting” the service

1. Add services API + page
2. Show credit price

Feature: Disable email verification

Added an env variable (MAIL_ENABLE) that will disable any email sending and functions like verification. With this enabled, signing up will automatically mark the account as verified and sign them in, skipping the verify step.

Did this so I can develop entirely locally and without internet, I’m on holiday in the Outer Hebrides where internet isn’t great. Writing this on a 7 hour ferry 🫩

Also changed the flags field from bigint -> integer to save space, I won’t need 64 bits)

1. Disable Email Verification
2. Fix silliness

Feature: API client secret regeneration

You can now regenerate the client secret.

Mostly copied code from the confirm deletion, tweaked to show the new secret after.

Also improved error handling so now it should show an error message if something fails instead of sending the error into the void

Commit

Feature: Delete API clients + Danger button

I added a delete action to API clients, with a confirmation modal. This needed a new button type, danger.

Next I will add a regenerate action, and then I guess I will have to work on a products API

1. Danger button
2. API client deletion

Feature: API Client creation + Copy Button

I added a page to view, manage and create API clients. This is what the past couple of devlogs have been leading up to, there was a lot of interactivity required for this.

These will be used to programmatically set up “virtual servers”, which will be able to register “products” to the platform, and receive services to deploy, and they manage where to deploy them, for example on the current server, or provision a VPS, or to GKE, or a docker swarm. I’m designing it to be as flexible as possible.

I also added a copy button component

Need to implement actions next, probably just delete

1. Copy button
2. Fix issue where close modal buttons would submit form
3. Add copy button to kitchen sink (UI demo)
4. Add API clients

Feature: Tables + Modals

I added tables and modals to my UI library. I needed these for my next platform update.

Tables were more annoying to implement than you’d expect, the browser makes them pretty hard to style.

Modals on the other hand were pretty quick to implement, I just used my existing card component.

My UI library is coming together pretty quickly, I’ve now implemented most of the components you’d need for a interactive website!

1. Tables
2. Modals

Feature: Sign Out + Alerts

I added the ability to sign out, added a favicon, created a standard layout component.

I also added four types of alerts to my UI library, and the associated colours. I’ve designed everything to be themeable.

I needed this for my next update

1. Add Sign Out
2. Disable logging of read config
3. Add layout component and favicon
4. Added alerts and colours
5. Disable text wrapping in sidebar

Feature: Account page

I added the account page that shows after sign in/up.

Sign out is not implemented yet.

I also updated the sidebar styling, adding a gap between links and differentiating between active and inactive links.

Commit

Feature: Sign In

I added a sign in page, implemented session tokens, and fixed a few bugs

I also made it so you are automatically signed in after verifying your account. This is secure because a verification session is only made after signing in or creating an account.

Honestly I thought this would take longer to implement, but I have done this a million times before.

Next I’ll implement the account page.

1. Implement sign in and sessions
2. Make the code input required
3. Fix error message when entering wrong code
4. Make verification sign you in
5. Cleanup/remove verification session after use

Feature: Verification emails

When you create an account, an email with the verification code gets sent.

You are redirected to a verification page, where you can enter the code, submit and your account gets marked as verified.

The page after verification is not implemented.

To do this, I:

1. Added a hidden input to the code input that contains the actual value
2. Set up mail sending using AWS SES (over SMTP)
3. Added a flags column to users, and created a flagset with a verified and admin flag
4. Created verification sessions, which store the code, an expiry date and a unique identifier

Next, I’ll implement signing in (with sessions)

• Commit 1 - hidden input
• Commit 2 - verification sessions, emails and flags

Feature: Code input + Verify Page

I built a verification page (server side stuff not implemented yet) and to do this I build a custom code input.

It automatically focusses the next input when typing, or the previous when backspacing.

I also added support for pasting codes. In the video demo, after pasting the focus doesn’t go to the last input but I have since fixed this.

Next up, I’ll set up SMTP with AWS SES and get verification working on the server side.

• Commit 1 - Code input and verify page
• Commit 2 - Handle backspace and focus to next input when pasting

Feature: Sign up!

There a fair bit of set up involved with this.

I had to:

• Add config with environment variables
• Set up postgres database (automatic schema migration)
• Restructure again

So now you can create an account. You can’t do anything yet, and I need to implement the next step of the flow, which is to verify your email.

• Commit 1 - Setup config, database and pretty logging
• Commit 2 - Slight restructure
• Commit 3 - Implement sign up (with HTMX)

Feature: Ghost buttons

Also fixed an issue where link buttons got displayed differently due to left over styles from the browser on buttons.

Commit

Inputs

I added inputs to my UI library and added fields to my sign up form

Probably won’t keep the background but its fun.

Second attachment is one of my mockups from a long time ago that I might try make the UI look like.

Commit

Start work on platform

I’m starting work on the Runik platform interface.

To do this I restructured the code, added a “Full Page Card” component and created a basic sign up page

Minimise sidebar

I implemented the minimise button on the sidebar!

It backs to local storage so works through refreshes and different pages.

Also provides a way to display difference things based on if the sidebar is maximised or not

Commit

Sidebar and icons support!

Currently the minimise sidebar button doesn’t work.

I also need to add a section for bottom links.

Forgot to mention in my previous devlog, some time wasn’t tracked because I was experimenting with the helix editor.

Also, this project is only using CSS, not tailwindcss. Although tailwindcss is nice, I miss CSS selectors and I find it a bit silly to have a build step for CSS.

Commit

UI Library

I started working on a UI library. For my other projects I wrote the UI from scratch, where I was repeating a lot of code, or used Daisyui, which I never really liked the look of.

So instead, I’m writing my own UI library for templ (a templating language for golang) so I can get it right this time.

So far I’ve added buttons, button rows and cards, and a “kitchen sink” demo.

There are three types of buttons, primary, secondary and accent, with nice animations.

Button rows can also be aligned left, right, or center.