Devlog #5: Wrote memory system

TL;DR: Wrote a memory system that holds the city state and sends changes that occurred to frontend

Goal for this session:

Make city.py track which cells changed since the last read, so the server can send small updates instead of the full state 10 times per second.

What I did:

Made a Cell and City class that hold info about… city and cell. Could have guessed tbh. Decided that each cell will represent one ip, and ips used by the same process will spawn (if possible) neighboring each another so we can get distinct districts.

Problems I hit:

My first grid generating algorithm could not spawn cells in an even way around the center. I used ai to help me with the math needed for the new one. It now works really good.

What I learned

• Yes
• Math is hard

Next up:

• Establish the actual connection that will use this code

MY LONGEST SESSION EVER

Devlog #4: Wrote a web socket

TL;DR: Wrote a simple asynchronous web socket to connect backend with frontend.

Goal for this session:

Write a backend connection to upcoming frontend

What I did:

Wrote a script that turns snapshots from the aggregator into JSON and sends them to the browser window through a websocket. It worked pretty well (see the screenshots)

Problems I hit:

I spent a while searching for why is the port i am trying to use already taken (I could not open connection through it) and it turned out that MY OWN SERVER.PY WAS ALREADY RUNNING AND BLOCKING THE PORT. Nonetheless, lesson learned.
I also had problem with websocket.serve expecting only one argument while my handler needed two, but I solved it with lambda temporary function.

What I learned

• What a websocket is
• What a async function is
• That I can use lambda ws: handler(ws, agg) to pass more arguments into websocket.serve when I need to.

Next up:

• Create the 3D engine in JS

Devlog #3: Tested the aggregator

TL;DR: Built test_run.py to bridge the fake traffic generator and the aggregator in real-time using threads.

Goal for this session:

Connect the background event generator to the aggregator and test them both

What I did:

So I simply started the generator in a separate thread and wrote a loop that calculates the packets and bytes/s in/out in each tick (100ms) . It later prints it in terminal with info about packet and event count.

The output:

processes=4 packets=28 events=0 in : 133078 B/s, out: 20054 B/s
processes=5 packets=25 events=0 in : 114678 B/s, out: 21327 B/s
processes=5 packets=27 events=1 in : 86732 B/s, out: 29492 B/s
processes=4 packets=25 events=1 in : 67049 B/s, out: 7650 B/s
processes=4 packets=28 events=1 in : 124895 B/s, out: 15077 B/s
processes=6 packets=25 events=0 in : 83312 B/s, out: 30431 B/s
processes=6 packets=48 events=0 in : 241464 B/s, out: 60790 B/s
processes=5 packets=119 events=0 in : 751542 B/s, out: 74549 B/s
processes=5 packets=133 events=0 in : 790185 B/s, out: 63535 B/s

btw the sudden spike of packets we see at the end is a proof of the burst functionality working!

Problems I hit:

I accidentally overwrote the values of bytes per second so it only showed the values of the last packet, not the sum of them.

Next up:

• Write the web socket for frontend integration (this is fully new territory for me so I am excited)

P.S thanks to @ethmarks for devlog review

Devlog #2: Writing the aggregator

TL;DR: Wrote the Aggregator that empties the queue every tick and turns hundreds of single events into one snapshot. Tested it with asserts. Hit a few silent bugs on the way.

Goal for this session:

Write the part that actually reads the queue from the fake generator and summarizes it.

What I did:

• Wrote the aggregator with flush() function
• flush() takes everything out of the queue and returns one snapshot: bytes in, bytes out and packet count per process, plus a list of one-off events (DNS and connections)
• Traffic with no known process goes under "unknown"- Added a limit of 20000 events per tick, so a flood of events can’t make one tick last forever

This is how the snapshots looks like:

{  "t": 1791121104.34,
"processes": {    "chrome.exe":  {"bytes_in": 1000, "bytes_out": 200, "packets": 2},    
"spotify.exe": {"bytes_in": 500,  "bytes_out": 0,   "packets": 1},    
"unknown":     {"bytes_in": 0,    "bytes_out": 120, "packets": 1}  },  
"events": [    {"type": "dns", "process": "unknown", "request_name": "google.com"}  
]}

Problems I hit:

• I used =+ 1 instead of += 1 so the program just set the counter to 1 instead of adding 1

What I learned:

-That the bugs that do not crash the script are worse than the ones that do

#Next up:

• test_run.py: fake generator in a thread + aggregator every 100 ms
• Turn per-tick sums into bytes/s

P.S. : BUKU BUKU GO!

Devlog #1.1

Just a quick update: I managed to finish the fake traffic generator. If you want to read about it, check the last devlog.

Next up:

An aggregator that will collect and encapsulate all those random-timed events into organized packets

Devlog #1: Writing fake traffic generator

TL;DR: Wrote a fake event generator for normal events, still need to do it for DNS and connections. Had trouble with time.sleep()

Goal for this session

Write a bigger part of fake event generator

What I did

• Wrote helper functions for queueing, making states of processes and making packets from states
• Wrote a function that using them all adds packets to queue
• Implemented the burst functionality into it

Problems I hit

The output was not smooth. During 5 seconds it was sometimes generating much less packets than expected. It was because time.sleep(0.01) is not exactly 0.01s but at least 0.01s.

Fix:

measure the real time since the last iteration dt and scale by it, so the packet count follows real time, not loop speed. The fractional part is handled with a random coin flip, since I can’t send 0.033 of a packet:

expected = state["rate"] * mult * dt 
n = int(expected)
if random.random() < (expected - n):
    n = n + 1
for _ in range(n):
    put(q, make_packet(name, prof, state))

What I learned

• That time.sleep can’t be trusted
• What threading is
• What queues are

Next up

• Add DNS and connection events to this

Devlog #0: Project started!

TL;DR: Just some basic info about the project

What is this project?

Netcity is going to be an audiovisual representation of internet traffic being sent from and to your device. I want to visualize the flow of packet as city traffic, apps as building and apps categories as districts.

What is done?

I coded the classes. Event info will be held in NamedTuples, just like that

class PacketEvent(NamedTuple):
time_s: float    
process: str | None    
direction : str # in or out    
protocol : str # tcp or udp    
remote_ip: str    
remote_port : int    
size: int

Next up

• Test the data types
• Add fake readings generator