mesh importing and refraction

ima start off by saying that this isn’t true refraction, since that requires path tracing, which is really computationally intensive
 

instead, i went with screen space refraction offset approximation, which is a good, efficient refraction approximator that i still think looks pretty cool

material data for refraction

 
the value that determines the transparency of an object is called transmission, and the value for refraction is index of refraction (or IOR)
 

the current filesystem for materials that i have is .mtl files, because thats the standard filetype for exporting materials with objs from programs like blender. however, .mtl is a filesystem from the 1990s that was originally created for phong systems, meaning it doesnt really contain transmission values

 
so, i created a json sidecar system, that generates a better, more robust, human readable material file for every model. so now, when you import an obj/mtl pair:
 

• it checks for a json sidecar file
• if there is one already, just load from that and finish!
• if there isn’t one already, now read the .mtl, extract as much info as you can from that, and write it to a new json file

multi pass rendering

in order to correctly render transparent meshes, you need at least 2 render passes (render the opaque ones first, then the transparent ones)
 
also, for the refraction to work, it needs an image of everything behind it, so in between the opaque and transparent pass, it quickly copies everything on the screen to an offscreen framebuffer

how the refraction works

1. get the view vector (v):

this one’s simple, just evaluate the view direction from the fragment position to the camera position
 
also, adjust the normal (n) if rendering backface
 

2. refraction vector (snell’s law):

the refraction vector is actually easy too, glsl has a built in refract intrinsic so you just drive that with a clamped ior value
 

3. deviation mapping:

to determine how much to bend the background scene texture, just calculate the deviation vector between the refracted ray and the view ray:
 

vec3 rayDeviation = refractDir - (-V);
vec2 distortion = (push.viewProjMatrix * vec4(rayDeviation, 0.0)).xy * 0.05 * transmission;

4. uv sampling + composite blending

offset the screen coordinates, but in order to prevent the edges from exploding, it needs to be clamped
 
also, sample the background buffer to blend lighting and make a fresnel rim

other stuff i did:

 
i also made a full custom persistent ui system (this took a while), and a command system, because hardcoding the ui to do stuff is bad
 
so instead of the ui code directly running logic, clicking a button just sends a command to the command system
 
this meant creating another render pass, meaning i’m up to 3 render passes per frame now
 

the only command so far is file.import_mesh, which is also hooked into the ui (you can see in the video below)

pbr materials

pbr math

reflectance:

this is the main equation that a cook-torrance pbr system, all the other equations lead back to this one at some point

𝐿𝑜⁡(𝑝,𝜔𝑜) =𝐿𝑒⁡(𝑝,𝜔𝑜) +∫Ω𝑓𝑟⁡(𝑝,𝜔𝑖,𝜔𝑜)⁢𝐿𝑖⁡(𝑝,𝜔𝑖)⁢(𝐧⋅𝜔𝑖)⁢𝑑⁢𝜔𝑖

this equation essentially just determines what light does after hitting a surface with specific parameters

we approximate the integral with two different light interactions:

1. diffuse (⁢𝑘𝐷): any light that penetrates the surface, scatters internally, and re-emerges
2. specular (𝑘𝑆⁢): light that reflects directly off the micro-surface

in the code itself, the total outgoing radiance ($L_o$) is calculated with:

vec3 Lo = (kD * albedo / PI + specular) * radiance * NdotL;

cook-torrance brdf:

the specular component is calculated using the cook-torrance microfacet brdf:
 
⁢## 𝑓cook-torrance =𝐷⋅𝐺⋅𝐹 / 4⁢(𝐍⋅𝐕)⁢(𝐍⋅𝐋)⁢
 
where 𝐷⁢, 𝐺, and ⁢𝐹 represent:

1. normal distribution (𝐷):

   the NDF statistical distribution calculates how many microfacets are aligned with the halfway vector ($⁢𝐇=𝐕+𝐋 / abs(𝐕+𝐋⁢))

𝐷GGX⁡(𝐍,𝐇,𝛼) = 𝛼^2 / 𝜋⁢((𝐍⋅𝐇)2⁢(𝛼2−1)+1)2⁢

```glsl
float DistributionGGX(vec3 N, vec3 H, float roughness) {
    float a = roughness * roughness;
    float a2 = a * a;
    float NdotH = max(dot(N, H), 0.0);
    float NdotH2 = NdotH * NdotH;

    float num = a2;
    float denom = (NdotH2 * (a2 - 1.0) + 1.0);
    denom = PI * denom * denom;

    return num / max(denom, 0.000001);
}
```

1. geometry (𝐺): smiths schlick-ggx

   microfacets can cast shadows and obscure light from the camera too, so the geometry function calculates the self shading factor using smiths method:
       

𝐺⁡(𝐍,𝐕,𝐋,𝑘) =𝐺1⁡(𝐍,𝐕,𝑘) ⋅𝐺1⁡(𝐍,𝐋,𝑘)

     
where 𝑘⁢ maps roughness for direct lighting using ⁢𝑘 = (roughness+1)^2 / 8:

```glsl
float GeometrySchlickGGX(float NdotV, float roughness) {
    float r = (roughness + 1.0);
    float k = (r * r) / 8.0;

    float num = NdotV;
    float denom = NdotV * (1.0 - k) + k;

    return num / denom;
}

float GeometrySmith(vec3 N, vec3 V, vec3 L, float roughness) {
    float NdotV = max(dot(N, V), 0.0);
    float NdotL = max(dot(N, L), 0.0);
    float ggx2 = GeometrySchlickGGX(NdotV, roughness);
    float ggx1 = GeometrySchlickGGX(NdotL, roughness);

    return ggx1 * ggx2;
}
```

1. fresnel equation (𝐹⁡): fresnel-schlick approximation

   the fresnel factor calculates the percentage of light reflected versus absorbed based on the viewing angle (⁢𝜃):
      

𝐹⁡(𝜃,𝐹0) =𝐹0 +(1−𝐹0)⁢(1−cos⁡𝜃)^5

     
nonmetals reflect around 4% of light (𝐹0 =0.04), while metals (conductors) use their tinted base albedo color as 𝐹0:

```glsl
vec3 fresnelSchlick(float cosTheta, vec3 F0) {
    return F0 + (1.0 - F0) * pow(clamp(1.0 - cosTheta, 0.0, 1.0), 5.0);
}

// lower, inside main():
vec3 F0 = mix(vec3(0.04), albedo, metallic);
vec3 F  = fresnelSchlick(max(dot(H, V), 0.0), F0);
```

other stuff:

a surface cant reflect more light than it recieves, so a simple equation is used to make sure this doesnt happen:
       
⁢## 𝐤𝐃 =(1−𝐤𝐒) ⋅(1−metallic)
     
right now, all the material data is pulled from a .mtl file

i also added bindless ssbo storage so it doesnt explode when you have a lot of models

next, i might keep working on the custom persistent ui system

custom obj file loading

all of the obj loading is done using tinyobjloader, which is a popular header file that im using to get the raw vertex coords of any .obj file

however, all tinyobj does is pass the raw vertex data to my code (technically it can do some basic triangulation, but thats not its main purpose so its not perfect)

this means that any model with n-gons (complex polygons) will not work and will look super weird

technically you can tell blender to explicitly triangulate a file when you export it, but i think that it should be able to take any obj that you can throw at it without needing external setup

so i also added a popular header file called earcut, which at a basic level, just cuts any complex polygons into 3-vertex triangles (which my index pass and renderer accepts).

this now means that it can load any custom obj file

i want to add auto color loading (from the obj data) or texture loading next, or maybe work on the ui or add ambient occlusion

entered the 3rd dimension

wasd movement

made a whole separate Camera class (look at me keepin things organized )

• this means that now the vulkan_renderer class has model, projection, view, and transform matrices:
  • the model matrix tracks the transformations
  • the view matrix essentially tracks where the camera is
  • the projection matrix sets a perspective frustum in order to make things actually have perspective and not just be flat
• the lerping is also based on deltaTime, which means smooth movement no matter ur fps
• the mouse tracking is locked via SDL_SetWindowRelativeMouseMode, and can be unlocked with the escape key

vertices and index buffers

• right now, the vertices are being set manually in my code… however, importing obj (and maybe gltf and stl) files is next on the feature list, i’m planning on using tinyobj
• all the geometry is indexed, which basically means that instead of wasting a ton of memory repeating data for 36 raw vertices, you can just define the 8 corner vertices, then use an index buffer pass to tell the gpu how to connect those 8 vertices into 12 triangles

window resizing

• this was easy, basically just had to tell it to make new image views and framebuffers based on the current aspect ratio (w/h) when the size of the window changes

so now, its a spinning cube in a resizeable window that you can move around with wasd

the window opens (!!)

so, why did it take me 6 hours to get a window to open?
 
it kinda just comes down to the fact that vulkan is vulkan
 
on other apis such as opengl, the graphics driver just does stuff for you. you can say sumn like “open a window and make it pink” and it just does the memory, thread syncing, and goofy platform stuff for you
 
vulkan on the other hand, wants none of that. if you dont specifically ask for something, it wont happen. this does make it significantly harder (and much easier to create a gpu-ending vram overflow) to code with, but it does have the benifit of being much more powerful, having no overhead, and 100% control. its not gonna do anything that you dont tell it to

so what do you have to do to open a beautiful grey window with vulkan?

• os bridge (sdl3): instead of writing a whole abstraction for wayland/x11, win32 and metal, you just have to tell sdl3 that you’re making a vulkan window, and it does some magic stuff

• vulkan instance (VkInstance): this is how you get into the vulkan sdk. you kinda just have to tell vulkan everything it needs to know and ask sdl what extensions you need

• window surface (VkSurfaceKHR): vulkan is kinda dumb and doesnt know what a window actually is, so this is where sdl tells vulkan where it can actually write pixels

• pick a physical device: scan the current hardware, loop through every available device, and then just pick the best one (eventually ill probably add a way to select a custom one)

• logical device (VkDevice): the physical device is the actual doohickey in your computer that runs stuff, the logical device is the software’s interface to it.

• swapchain (VkSwapchainKHR): vulkan can’t actually write images directly to the screen (and that would look bad anyway), so you make a swapchain. my code does double buffering (basically vsync), which means that the screen displays image a, while the gpu is rendering image b.

• render pass and framebuffers: basically just telling vulkan what to do (in this case, clear the screen, then draw the color). the framebuffer just binds the swapchain image views to the render pass

• synchronization and command subsystem: believe it or not, your gpu’s clock speed is usually around (or more than) half of your cpu’s clock speed, so you have to make sure that your code waits for the gpu is ready before running the next c++ line. if you don’t, your gpu ram and cache will overflow.

 
overall, this took a total of 764 lines of code just to get a grey window
 
the nice part is that the progress that i’ll make is exponential, since the very beginning stuff takes the longest

i also made a logo (the current project page banner)