What changed

In short, I added some new code to scrap some other things like:

• missions on the project page, so if a project is attached to a mission or has shipped with one
• source urls on the project page
• the daily random number pages (including the past)
• banner urls on users and projects
• if a ship has been returned or is pending

I also added timeline tables of users, projects, devlogs, superstar and ship events. So that I now how projects have changed over the time.

Problem

I also run in a problem, while the program scraps the site, I cannot access it too, because I wrote it so that the requests are exactly calculated so that I can fully utilize the rate limit. I tried to avoid it by running the program on nest but the container just kills my program after 50,000 requests (I don’t know why, and the logs don’t help me and it wasn’t the OOM-Killer). So I am now running it on my Raspberry Pi 4b and because it is at my house, it uses the same IP)

Renaming

I am also going to rename this project from Stardance-Graph to Stardance-Scraper, because the program is now more then just a graph. I am still going to write the graph, but in a second project.

Scraping

Well I scraped Stardance 4 times, since the last devlog:

1. The first time was on the Raspberry PI for testing (and because I couldn’t run it on my pc), after around 40000 request it stopped due to a death lock (two threads waiting on each other, but on my case it were 4 threads), the chance that this happened was so low, but it did, I tried to fix it but I failed. So a just killed the program.
2. I tried to run it on nest but after 50,000 requests it killed it so not the right option
3. Then I rerun it on my raspberry pi (with the fix) and it worked pretty well
4. I added the timeline feature and since then it just does its work

In total I have done like 400,000 requests to Stardance.

Image

The image shows a line chart that shows the number of users who joined on each date. (Yes someone joined on 19.04.2026: @/cskartikey)

What changed

I have used HTTP/1.1 for my request to stardacen, then I found out that I can use HTTP/2. I tried it and it so much faster, like I could send 600 Requests under 1 minute.  
It was so fast that my Database Writer, couldn’t keep up and I needed to slow down, so that everything is stored in the DB. But then Stardance got (D)DOSed (which wasn’t me). So they added ratelimits. And now I can only do 120 Requests per minute.  
Which slows my down.
I also updated the DB so that now every request I do is stored in it, for better debugging and analytic.

Some cool statistics

Here are some statistics about my scraping and Stardance

Scraping

Yesterday I started scraping around 6 AM EST and it took 10 hours and 21 minutes.

• I send 63,364 requests.
    - 6,820 returned with 404
    - and 98 requests failed due to other reasons
• I received a total of 1.22 GB HTML
• To serve my requests the Stardance server executed 746,089 database queries

Stardance

And I found some really cool things
Like:

• 30,239 users are on Stardance
    - 113 users got achivments
    - And the most common one is „Presentable Hardware Project„
• 19,171 projects have been created
    - There are currently 17 project that have been marked with superstar
    - And 1,167 ship events
• 11,614 devlogs have been posted
    - With a total time of 26,182 hours
    - 837 devlogs have been reposted
    - And a total number of 10,981 likes have been given.
    - there are 741 comments.
    - I also run a (very) simple local AI detection tool on every devlog, which can check for some obviously Ai and it only found 35 devlogs.

Images

I haven’t worked on the graph, so I don’t have pictures of it. But I have some diagrams about the data I have scraped.

1. The first (left) image is about the sign in users over the time I scraped
   2  The second (right top) image shows the visitors over the time
2. And the last (right bottom) one shows the project id to the total hours in the project, and the interesting thing about it is that most of the hours came from the projects with small ids, where the users of those projects have joined near the start of the event. I wrote I small test program which takes all projects with more then zero hours, then take the first 355 users. Then request Slack for there start date. Of those 355 users, 165 started before the 2026-5-20, 107 after, and by 83 I couldn’t find the start date.

Please note that this data is from yesterday and can be outdated

I wrote a program that completely scraps Stardance. It starts with the projects (from 1 to 20000) that it can find on the page. It analyses them and stores the information in a database. It also searches for references to users, devlogs, followers, comments, etc. So for every project I get also the creator of it and then the creator gets also scraped. So in the end I have users, projects, devlogs, followers, following and comments.

But was do I want with does data

The name of this projects reveals it: I want to create a graph, of every possible connection between objects. I want to host it on a website, so you can search for users like @Fantamomo and see how I am connected with other projects and users, like I repost a devlog, if someone comments one one of my devlogs etc.

Image

The image show a prototype (coded by AI, because I have no clue how to write something like that). The problem is that the program constantly uses around 139% CPU and around 2 GB RAM. Even with that much CPU usage, it is still laggy and I can barely use it.  
So now I have to figure out how to make the graph faster so that I can host it as a website on nest.

Stats

Scrapping Stardance took 3 hours (I paused between request to avoid the non existing rate limits)
During the time:

• 23654 requests where made
• To serve my requests the Stardance server executed 388157 database queries (174135 queries had been cached)
• I received 0.6 GB HTML text as responses.

Fun fact

My first devlog was reposted by someone.  
At that time, the notification system did not exist yet, so I didn’t know who reposted it.
Through my scraping I found out: It was @neon (btw thanks)