I left this project for some time as I couldn’t wrap my head around hardware stuff. But guess what, I came back after a long time with a new layout. Now I am using oled + 2 encoders(without switches) + 6 switches (also 6 diodes).
I am starting to understand how these things work and feel confident that I will cook it.

I will definitely cook it this time.

This one feels a lot better and is possible with current pins.

Peace Out.