mesh importing and refraction

ima start off by saying that this isn’t true refraction, since that requires path tracing, which is really computationally intensive
 

instead, i went with screen space refraction offset approximation, which is a good, efficient refraction approximator that i still think looks pretty cool

material data for refraction

 
the value that determines the transparency of an object is called transmission, and the value for refraction is index of refraction (or IOR)
 

the current filesystem for materials that i have is .mtl files, because thats the standard filetype for exporting materials with objs from programs like blender. however, .mtl is a filesystem from the 1990s that was originally created for phong systems, meaning it doesnt really contain transmission values

 
so, i created a json sidecar system, that generates a better, more robust, human readable material file for every model. so now, when you import an obj/mtl pair:
 

• it checks for a json sidecar file
• if there is one already, just load from that and finish!
• if there isn’t one already, now read the .mtl, extract as much info as you can from that, and write it to a new json file

multi pass rendering

in order to correctly render transparent meshes, you need at least 2 render passes (render the opaque ones first, then the transparent ones)
 
also, for the refraction to work, it needs an image of everything behind it, so in between the opaque and transparent pass, it quickly copies everything on the screen to an offscreen framebuffer

how the refraction works

1. get the view vector (v):

this one’s simple, just evaluate the view direction from the fragment position to the camera position
 
also, adjust the normal (n) if rendering backface
 

2. refraction vector (snell’s law):

the refraction vector is actually easy too, glsl has a built in refract intrinsic so you just drive that with a clamped ior value
 

3. deviation mapping:

to determine how much to bend the background scene texture, just calculate the deviation vector between the refracted ray and the view ray:
 

vec3 rayDeviation = refractDir - (-V);
vec2 distortion = (push.viewProjMatrix * vec4(rayDeviation, 0.0)).xy * 0.05 * transmission;

4. uv sampling + composite blending

offset the screen coordinates, but in order to prevent the edges from exploding, it needs to be clamped
 
also, sample the background buffer to blend lighting and make a fresnel rim

other stuff i did:

 
i also made a full custom persistent ui system (this took a while), and a command system, because hardcoding the ui to do stuff is bad
 
so instead of the ui code directly running logic, clicking a button just sends a command to the command system
 
this meant creating another render pass, meaning i’m up to 3 render passes per frame now
 

the only command so far is file.import_mesh, which is also hooked into the ui (you can see in the video below)

sketching

all of this was just spent making some concept art (tbh i kinda just wanted an excuse to do some art and log it lol)

my art process is:

• blender concept model with matcap
• sketch in krita
• shade the sketch
• coloring
• shade the coloring

i have not started doing any actual game stuff or coding, because slate engine isnt far enough in development yet lol

pbr materials

pbr math

reflectance:

this is the main equation that a cook-torrance pbr system, all the other equations lead back to this one at some point

𝐿𝑜⁡(𝑝,𝜔𝑜) =𝐿𝑒⁡(𝑝,𝜔𝑜) +∫Ω𝑓𝑟⁡(𝑝,𝜔𝑖,𝜔𝑜)⁢𝐿𝑖⁡(𝑝,𝜔𝑖)⁢(𝐧⋅𝜔𝑖)⁢𝑑⁢𝜔𝑖

this equation essentially just determines what light does after hitting a surface with specific parameters

we approximate the integral with two different light interactions:

1. diffuse (⁢𝑘𝐷): any light that penetrates the surface, scatters internally, and re-emerges
2. specular (𝑘𝑆⁢): light that reflects directly off the micro-surface

in the code itself, the total outgoing radiance ($L_o$) is calculated with:

vec3 Lo = (kD * albedo / PI + specular) * radiance * NdotL;

cook-torrance brdf:

the specular component is calculated using the cook-torrance microfacet brdf:
 
⁢## 𝑓cook-torrance =𝐷⋅𝐺⋅𝐹 / 4⁢(𝐍⋅𝐕)⁢(𝐍⋅𝐋)⁢
 
where 𝐷⁢, 𝐺, and ⁢𝐹 represent:

1. normal distribution (𝐷):

   the NDF statistical distribution calculates how many microfacets are aligned with the halfway vector ($⁢𝐇=𝐕+𝐋 / abs(𝐕+𝐋⁢))

𝐷GGX⁡(𝐍,𝐇,𝛼) = 𝛼^2 / 𝜋⁢((𝐍⋅𝐇)2⁢(𝛼2−1)+1)2⁢

```glsl
float DistributionGGX(vec3 N, vec3 H, float roughness) {
    float a = roughness * roughness;
    float a2 = a * a;
    float NdotH = max(dot(N, H), 0.0);
    float NdotH2 = NdotH * NdotH;

    float num = a2;
    float denom = (NdotH2 * (a2 - 1.0) + 1.0);
    denom = PI * denom * denom;

    return num / max(denom, 0.000001);
}
```

1. geometry (𝐺): smiths schlick-ggx

   microfacets can cast shadows and obscure light from the camera too, so the geometry function calculates the self shading factor using smiths method:
       

𝐺⁡(𝐍,𝐕,𝐋,𝑘) =𝐺1⁡(𝐍,𝐕,𝑘) ⋅𝐺1⁡(𝐍,𝐋,𝑘)

     
where 𝑘⁢ maps roughness for direct lighting using ⁢𝑘 = (roughness+1)^2 / 8:

```glsl
float GeometrySchlickGGX(float NdotV, float roughness) {
    float r = (roughness + 1.0);
    float k = (r * r) / 8.0;

    float num = NdotV;
    float denom = NdotV * (1.0 - k) + k;

    return num / denom;
}

float GeometrySmith(vec3 N, vec3 V, vec3 L, float roughness) {
    float NdotV = max(dot(N, V), 0.0);
    float NdotL = max(dot(N, L), 0.0);
    float ggx2 = GeometrySchlickGGX(NdotV, roughness);
    float ggx1 = GeometrySchlickGGX(NdotL, roughness);

    return ggx1 * ggx2;
}
```

1. fresnel equation (𝐹⁡): fresnel-schlick approximation

   the fresnel factor calculates the percentage of light reflected versus absorbed based on the viewing angle (⁢𝜃):
      

𝐹⁡(𝜃,𝐹0) =𝐹0 +(1−𝐹0)⁢(1−cos⁡𝜃)^5

     
nonmetals reflect around 4% of light (𝐹0 =0.04), while metals (conductors) use their tinted base albedo color as 𝐹0:

```glsl
vec3 fresnelSchlick(float cosTheta, vec3 F0) {
    return F0 + (1.0 - F0) * pow(clamp(1.0 - cosTheta, 0.0, 1.0), 5.0);
}

// lower, inside main():
vec3 F0 = mix(vec3(0.04), albedo, metallic);
vec3 F  = fresnelSchlick(max(dot(H, V), 0.0), F0);
```

other stuff:

a surface cant reflect more light than it recieves, so a simple equation is used to make sure this doesnt happen:
       
⁢## 𝐤𝐃 =(1−𝐤𝐒) ⋅(1−metallic)
     
right now, all the material data is pulled from a .mtl file

i also added bindless ssbo storage so it doesnt explode when you have a lot of models

next, i might keep working on the custom persistent ui system

custom obj file loading

all of the obj loading is done using tinyobjloader, which is a popular header file that im using to get the raw vertex coords of any .obj file

however, all tinyobj does is pass the raw vertex data to my code (technically it can do some basic triangulation, but thats not its main purpose so its not perfect)

this means that any model with n-gons (complex polygons) will not work and will look super weird

technically you can tell blender to explicitly triangulate a file when you export it, but i think that it should be able to take any obj that you can throw at it without needing external setup

so i also added a popular header file called earcut, which at a basic level, just cuts any complex polygons into 3-vertex triangles (which my index pass and renderer accepts).

this now means that it can load any custom obj file

i want to add auto color loading (from the obj data) or texture loading next, or maybe work on the ui or add ambient occlusion

entered the 3rd dimension

wasd movement

made a whole separate Camera class (look at me keepin things organized )

• this means that now the vulkan_renderer class has model, projection, view, and transform matrices:
  • the model matrix tracks the transformations
  • the view matrix essentially tracks where the camera is
  • the projection matrix sets a perspective frustum in order to make things actually have perspective and not just be flat
• the lerping is also based on deltaTime, which means smooth movement no matter ur fps
• the mouse tracking is locked via SDL_SetWindowRelativeMouseMode, and can be unlocked with the escape key

vertices and index buffers

• right now, the vertices are being set manually in my code… however, importing obj (and maybe gltf and stl) files is next on the feature list, i’m planning on using tinyobj
• all the geometry is indexed, which basically means that instead of wasting a ton of memory repeating data for 36 raw vertices, you can just define the 8 corner vertices, then use an index buffer pass to tell the gpu how to connect those 8 vertices into 12 triangles

window resizing

• this was easy, basically just had to tell it to make new image views and framebuffers based on the current aspect ratio (w/h) when the size of the window changes

so now, its a spinning cube in a resizeable window that you can move around with wasd

the window opens (!!)

so, why did it take me 6 hours to get a window to open?
 
it kinda just comes down to the fact that vulkan is vulkan
 
on other apis such as opengl, the graphics driver just does stuff for you. you can say sumn like “open a window and make it pink” and it just does the memory, thread syncing, and goofy platform stuff for you
 
vulkan on the other hand, wants none of that. if you dont specifically ask for something, it wont happen. this does make it significantly harder (and much easier to create a gpu-ending vram overflow) to code with, but it does have the benifit of being much more powerful, having no overhead, and 100% control. its not gonna do anything that you dont tell it to

so what do you have to do to open a beautiful grey window with vulkan?

• os bridge (sdl3): instead of writing a whole abstraction for wayland/x11, win32 and metal, you just have to tell sdl3 that you’re making a vulkan window, and it does some magic stuff

• vulkan instance (VkInstance): this is how you get into the vulkan sdk. you kinda just have to tell vulkan everything it needs to know and ask sdl what extensions you need

• window surface (VkSurfaceKHR): vulkan is kinda dumb and doesnt know what a window actually is, so this is where sdl tells vulkan where it can actually write pixels

• pick a physical device: scan the current hardware, loop through every available device, and then just pick the best one (eventually ill probably add a way to select a custom one)

• logical device (VkDevice): the physical device is the actual doohickey in your computer that runs stuff, the logical device is the software’s interface to it.

• swapchain (VkSwapchainKHR): vulkan can’t actually write images directly to the screen (and that would look bad anyway), so you make a swapchain. my code does double buffering (basically vsync), which means that the screen displays image a, while the gpu is rendering image b.

• render pass and framebuffers: basically just telling vulkan what to do (in this case, clear the screen, then draw the color). the framebuffer just binds the swapchain image views to the render pass

• synchronization and command subsystem: believe it or not, your gpu’s clock speed is usually around (or more than) half of your cpu’s clock speed, so you have to make sure that your code waits for the gpu is ready before running the next c++ line. if you don’t, your gpu ram and cache will overflow.

 
overall, this took a total of 764 lines of code just to get a grey window
 
the nice part is that the progress that i’ll make is exponential, since the very beginning stuff takes the longest

i also made a logo (the current project page banner)

extra ui tweaks and readme updates

added some images of 3d fractals to the readme, did some general ui tweaks, and some idiot proofing

now its time to ship!!!

probably the final commit :o

most of this was spent idiot proofing, going through all my code and making sure that everything works well and actually makes sense
 
i also updated the tutorial to be a bit better

a major amount of time was spent figuring out how to get windows to actually compile
 
i swear windows is the bane of my entire existence

its not reflected in the devlog hours, but i did also make a demo video to show off some of the features
 
mystery link

other than that, it was mostly ui and readme updates :)

i set up a full tutorial system

the hard part was doing the whole thing where it darkens everywhere except the thing the tutorial is yapping about…
 
also, right now, the tutorial is really really easy to break, so i might have to implement some idiot proofing or something
 
to be honest, adding the whole darkening thing was really simple, but actually getting it to work was really annoying for some reason.
 
it essentially just gets the xy coord for each corner of the specified element, and then makes a dark block everywhere except inside those 4 corners.
 
 
also, i made it so the tutorial saves to the same json file as the preset manager, so that you only have to do the tutorial one time and it doesnt just get in your way every single time you open it.

aside from that, most of this time was spent trying to figure out how to get it to automatically build on github when i push a release.
 
i did this successfully for my other project, but the problem so far has been linking the qml for the binaries on the different platforms.
 
specifically, windows just really really doesnt want to behave, its starting to feel like getting a discord mod to eat anything other than gamersupps and pizza rolls

overall, maybe getting close to shipping? it honestly just depends on how much more features and idiot proofing i want to add
 
if this sounds interesting, check out (and star ) the repo here

3d wasd controls (wowie zowie)

the math for this was super hard, especially because i want it to work alongside with the other classic orbit style controls lol
 
i ended up adding a toggle in the sidebar, which felt like the cleanest way to do it, but did mean doing more qml

 
also, i pretty much had to guess when setting up all my vectors so i
spent like 15 presidential terms messing around with positives and
negatives and 1.0f / 0.0f everywhere until doing stuff in the right
directions actually did the right thing, which was hard
 
also had to implement auto horizon correction so things d

also added some live 3d settings, which took a while, but now theres a
live slider for both the mandelbulb power (exponent), and the brightness
(which was hard)

and, theres now a math section in the readme, which i spent a lot of
time pondering while making, but if you’re at all, check out the “mathy
information” section of the readme
 
https://github.com/swedishsplidney/fraktalized
 
(also star the repo )

exploring the mandelbulb in wasd mode genuinely feels like exploring an alien planet in a spaceship, lowk super fun

3d is finally working!!!!!

i have been having an epic samurai battle against opengl and 3d fractal math for what feels like forever, but it finally works, and I have a working 3d mandelbulb explorer!!!

i have actually been working on getting 3d working for the past multiple devlogs, it just took me sooo incredibly long to actually figure it out that i never really mentioned it.

the longest step was figuring out the controls. when i first started working on the 3d mode, then it was literally just a cube, and i had to figure out a good way to be able to rotate it
 
the hard part wasn’t even trying to figure out how to do the math to rotate the vertices, the hard part was trying to figure out how the thing should actually rotate
 
what i figured out how to do is having an orbit style camera, where dragging side to side rotates the model along its own local yaw, and then dragging up and down changes the camera’s global pitch. this way the model’s roll never changes (because that makes the model look tilted or skewed).
 
i had to spend an embarassingly long amount of time trying to figure the logic of an orbital camera…
 
the camera is also slerped (like lerping but 3d) so it feels smooth

color mappping on the 3d was actually surprisingly easy, i just did it in a very similar way to the 2d escape time ones, except with an extra dimension (its not as hard as it sounds)

one other major change was when i added the ability to add and remove gradient stops. this way, instead of always having 4 gradient stops, you can have anywhere from 2 to 16 stops, for more customizeability.

also, the readme is updated, go check it out! (and star the repo )
https://github.com/swedishsplidney/fraktalized

below: video demonstrating the 3d and camera system

added iterative function system fractals
 
basically, iterative function fractals are fractals that start at a set point, then repeatedly apply the same affine transformations (scaling, rotating, shifting), making an infinitely repeating, self-same fractal.
 
as far as actually getting this setup and working, i had to make a whole new compute shader that involves a lot of math and for loops to iterate forever
 
getting ifs fractals to render was a particular challenge, and i almost exploded my gpu and accidentally caused a vram cache overflow causing my display output to break. in the end, it turns out the trick was to not send 4,000,000 worker threads to the gpu all at once, and to do them 3-4 at a time

also finally got working on the whole 3d system, which will likely take forever, and completely does not work right now, but hopefully i will have figured out how in the world to get working 3d stuff by the next devlog 🙏
 
check out the github! -> https://github.com/swedishsplidney/fraktalized
 
image: barnsley fern, rendered in 4k

added support for trajectory based fractals
 
most people know the normal fractals (the ones ive covered so far), such as the mandelbrot and julia sets, that test wether or not a point escapes and maps how long it takes to escape
 
trajectory fractals look really cool, and the difference is that they map the trajectory that 𝑧 takes on its way out, while incrementing a counter for every pixel it steps on.

getting this to work was a lot of work, and required to write a whole new shader file from scratch, except instead of a .frag/.vert combo, it was a compute shader (.glsl).
 
instead of simply running a loop and writing color data to each independent pixel like with the escape-time fractals do,
 
the trajectory shader has to pick a random (x, y) starting coord, and then track its orbit across the complex plane, writing data to random pixels as it goes.
 
this required me to use atomics, because if 2 orbits cross, instead of one overwriting each other, you have to use imageAtomicAdd() to make sure that the brightness stacks up.

one other issue was that at first it was super grainy, which was fixed by assigning each gpu thread 50 random sampled to increase detail substantially.

i also nearly exploded my gpu because i accidentally had the render system run 50+ compute sequences at the same time, so i now have each compute wait for the one before it to finish

in other happenings, there is now a preset feature that writes your current color setup set, zoom, center, etc. to a JSON file to be loaded later
 
also added the newton set, but its a less interesting escape-time fractal, so not covered in as much detail
 
image: buddhabrot set, rendered at 4k

added the burning ship fractal, anti aliasing, and completely overhauled the color system to use a gradient input instead of one color.

burning ship fractal:
 
so the burning ship fractal is made when you take the mandelbrot equation, but you take the absolute value of the real and imaginary parts before squaring, basically destroying the symmetry:
 
𝑧ₙ₊₁ = (|𝑅𝑒(𝑍ₙ)| + 𝑖|𝐼𝑚(𝑍ₙ)|)² + 𝐶
 
this makes it look super duper cool (as seen in the image below)

gradient-based color mapping
 
so before, the color was just calculated using one color tint and boosting brightness quadratically near the edges.
 
now, it takes in a gradient (calculated between 4 color inputs), and does the same color mapping but essentially moves what color it uses as the iterations increase.
 
the hard part is figuring out how to get a good input system in qml with this, but I was able to do it using some colordialogues and a lot of trial and error.
 
the new color input system is kinda like creating a gradient in photoshop or illustrator, with 4 little handles that you can drag across and change their colors.

anti aliasing
 
also added anti aliasing, or more specifically, ssaa (super sampling anti aliasing), with a selector to choose between off, 2x, and 3x
 
so basically it renders the entire fractal at a higher resolution (2x or 3x) and then averages the color of every pixel block (2x2 or 3x3 respectively) and then displays that. this results in a much, much smoother image at the cost of much higher resource usage.
 
while i was screwing around with dynamic resolutions, i also added a resolution slider to change the rendered resolution to anywhere from 25 to 150% of the window resolution. turning it up past 100% basically just gives the ssaa architecture more information to work with, making it even smoother
 
my graphics card isnt very powerful and it basically stays pinned at 99% usage and 100% clock when i have anti aliasing turned on and the iterations above like 200.
 
image: burning ship fractal, rendered at 4k with 150% resolution scale and 3x ssaa

now has a render to file feature
 
basically you just import a custom resolution and it takes your current center and renders at that resolution with your current parameters. the hard part is that it uses a completely different framebuffer object from the regular viewport.
 
while testing i did almost explode my gpu when pushing the resolution to almost 32k, which is kinda unreasonable, but this was a good opportunity to add a tiled render system
 
basically it just splits up the image into 2000x2000 pixel sections and renders them, and then it does some math to position it
 
the weird thing is that opengl is kinda goofy and likes to mirror stuff for some reason (literally no idea why) so theres some math to render it in reverse (which means that its actually correct)
 
also increased the rendering to 64 bit, meaning it is now 536,870,912 times more accurate compared to the 32 bit one, and also means you can zoom in more than twice as far with better quality, but it is much more gpu intensive…
 
this image is one of the cooler ones i rendered out at 4k, it kinda looks like colliding galaxies or something

is now fully interactive and honestly feels super great, especially compared to my last devlog where it was static and unchanging entirely

added working real-time render parameters such as panning and zooming with mouse, and custom iteration setup and color setup on the sidebar.
 
lots and lots and lots of quality of life features, such as:

• the reset button for if you get lost
• adjustable iteration slider min/max
• direct text enter for iterations
• hideable iteration min/max menu
• camera smoothing
• color clamping so it doesnt look deep fried

also added the julia set (which is what is in the image)
 
so basically, both the mandelbrot and julia sets are the same quadratic recurrence relation on the complex plane:

• 𝑧ₙ₊₁ = (𝑧ₙ)² + 𝑐

the main difference is that you start at the origin (𝑧₀ = 0) and change the constant 𝑐 for every single pixel on the screen based on its coordinates, but with the julia set the constant 𝑐 is the same everywhere, and the starting point 𝑧₀ changes based on the pixel’s coordinate, so you are testing how different starting points react to the same gravitational pull of 𝑐.

the nice thing is that you can change the constant 𝑐 in the julia set, and completely change how the fractal looks, so i added that as an input field too

looks really really cool lowk

working mandelbrot fractal rendering through opengl!!!!!

using qt/qml and raw opengl shaders together was a mistake
 
they really dont like working together and i had to spend 3 straight hours (its 5:42am at the time of this devlog) debugging an eternal tug of war between qt and opengl because qt kept overwriting what i wanted to happen
 
the worst part is that an opengl fragment shader cant really debug to the log so if something is on fire over there i just kinda have to guess to fix it

eventually i just gave up and switched to framebuffer objects, which i lowk should have done from the start
 
the funnest part was actually writing the equations for the mandelbrot set lol (it also looks super pretty):
 
so basically the Mandelbrot set is the set of all complex parameters 𝑐 ∈ ℂ for which the orbit of 𝑧₀ = 0 under quadratic recurrence remains bounded. in the shader, i just separate its multiplication into real and imaginary components:

• Re(𝑧ₙ₊₁) = 𝑥² + 𝑦² + Re(𝑐)
• Im(𝑧ₙ₊₁) = 2𝑥𝑦 + Im(𝑐)

also infinity is impossible so the iterations are capped at 100
 
sorry if this was too nerdy or long lmao

just spent an hour debugging just to get a working build for microslop windows…

finally got it working successfully though, kinda goofy on occasion but does work, thankfully