Updates

Quite a bit this time that are important.

First up I added a link to the github in the header and fixed the bot to actually follow slack rate limtis.

Added a board for linked accounts page.
Big thing now.
Added consent gated slack scraping. Basically fd told me i need people to opt in.

And finally the last big thing
I added channel blacklisting as a shipwright and a goi linked through OAuth so I had to blacklist all sw channels and goi channel as they have sensitive messages in them

Update time

Been quite a bit since ship and i decided because of the stardance extension I’m going to do more devlogs.

Mainly i added an opt out feature and realized the new defaults for scraping speeds were shit so i undid them.
I then decided that because there were channels with fucking 3000 thread roots, i lowered the age limit of threads to 7 days instead of 30 days.

I nuked opt out/opt in and nuked the manual apps needed for scraping.
There is now an OAuth system with slack, you can link your account and it will scrape your messages in private channels you are in. works without this though.

Added a max scraping per minute variable so i dont strain my internet connection a bunch.
And finally I also added the [PRIVATE] and [ARCHIVED] badges to channels

Updates

Should be final update before I ship

• Added all users and all channels boards
• Renamed Leaderboards to just Boards
• Normalized colum widths for those boards and added pagination
• We were hammering hackatime, it now has a hard limit of 50k hackatime reqs/day and 1req/sec
• Massive update to stats page so it displays more info and is more relevant.
• Changed total time to total hours where the time was hours and added total time in format of years/months/ etc. as well
• We sped up docker builds a bunch
• We retry known private hackatime users ever 30 days now
• Fixed a shared message-scrape lock
• The response when you ask the bot for time now includes a link to your/person your checking time ofs stats page

I think thats about it

Updates

Quite a bunch of updates this time, a lot of work has been done, and i feel like I was nearly killed by fd.

Okay so Shiptalkers is now opt in.

Most of this devlogs time was spent:

• Adding a message locator system, basically maps out where every message is instead of storing content of it
• Process messages without storing message content
• Backfill message content for users who opt in
• Add a custom shiptalkers id assigned to messages so we dont store sender id of each message.
• To opt in you can send “Opt In” in the shiptalkers channel

Apart from that I migrated from in file SQL to a proper migrations system.

General updates

• Added mobile support

• Massive cleanup of AGENTS.md
  0 Full fetch and scrape all messages run in parrallel now.

• Archived channels card for stats page.

• Now persists thread rescans, added a thread timestamp index, persisted thread high-water marks.

• Reduced repeated history-wide scans

• Added durable consent backfill progress and fixed backfills blocking fresh message checks

• Fixed a per message update of the shared database counter row. It now updates per statement

• Score recomputation is now hourly instead of passly.

I think thats about it.
Main point of this is that we got scraping to be way faster, and we got opt in requirement.

Updates

Bunch of API related things this time

• Added api docs
• API key section in account tab and APIT key account endpoints
• Added CSRF support (lmao i dont know why i didnt have it before)
• Made the api key response be a popup in account page and centered it in a later commit
• Added Stats API endpoint
• Added Max Age to cookies
• Website and slack bot now share 1 source of truth for stats (stupid that i didnt have it like this already)
• Added granting perms to other api keys
• New user joins are now picked up in real time.

Thats about it

Updates

Really short devlog this time.

• Added Error logging for session_query
• Resumable channel sweeps
• Use a new action for build & deploys

Workflows are also greatly increased with speed

Finishing Touches

• So time to ship

Basically finished up by:

• Doing example env
• Using a proper deploy workflow
• Add env reading support via CLI args
• Nuked docker compose
  And ofc, a bunch of readme updates

Alot of updates this time

Updates

Brand new scraper crate separate from app, dk if I have mentioned it before in devlogs.
It does slack and hackatime scraping all in one so we can have the main app deploys not interrupt scrapes.

Updates^2

• Added “a” keyword - Alias for “all time”
• Updated settings.rs and example.env to fit new defaults
• Fixed refresh_daily_stats failing on a second
• Fixed a bunch of ::bigint needing to be everywhere a sum() was decoded
• switched scraper to mimalloc allocator
• Replaced ts with real message_tis / 1000000 (I dont know if i put enough/too many zeros here)
• Fixed slack time recomputing in new postgredb
• Precomputes page stats now
• Removed a useless half-hourly job
• ON CONFLICT now only fires when counts actually change
• Added DB available probe to tests
• Better logging (disabled timestamps in logger)
• Added a mark_seen dedupe
• Updated hackatime sync logging to be less taking up all logs
• Added zstd + drop duplicate tables
• Removed zstd as postgre18-alpine does not come with it and I am not rescraping slack
• Only the scraper can init tables now

I don’t think im missing anything else i did,
If I am though then sorry gois

Also see in one of the images below is total ram and cpu usage. somehow the postgre db is using over 4.5G of ram

Updates

Mighta lowkey forgot to be doing devlogs again…
Anyway

Updates^2

• Basically any keyword for time works now
• Cargo Chef is now used for docker
• Updated AGENTS.md
• No longer requires hackatime data to display slack time data.
• Ported everything from clickhouse to postgresql but havent deployed that yet
• You can now mention other people to get their stats

I think thats it
Might be wrong on that tho.

Anyway I got better from being sick last week and im sick again

Also used @mahad in screenshot below, def not an excuse just for a random pung

Updates

We still sick (not much better today) but i got some work done

Updates^2

• Split a bunch of existing code into separate files for maintainability
• added font caching
• parallel resyncs (hackatime)
• More memory/CPU optimizations in the code (I lowkey forgot what i changed for this

Thats it for now

A decent length devlog this time

So recently I have been sick so i got less time.
(Wish me to get better soon)
Anyway enough of that, on to the changes:

Updates

• Fixed authless hackatime syncing
• Using the spans api of hackatime so we get all data for time specific commands
• fix SQL and rust helper dividing by 1000000
• Wrapped the entire sum(if(…)) in toUInt64(), which fixed the slack bot reporting wrong time from public hackatime syncing change. It makes it so clickhouse sees a single output type regardless of what it infers inside.

Thats about it

Updates

• We swapped aws-lc-rs for ring
• futures swapped for futures-util
• nuked tower crate & lowered what is pulled for tokio
• Brought back dependabot auto merge
• Added cool charts
• Nuked cool charts
• Stats page changed again (wip)
• Broke hackatime time syncing when i tried to make it not need auth (tmrw me problem)
• Added page streaming and incremantal word totals

Updates

improving my devlog to hour ratio

Anyway..
New db scrape is completed!

Major changes

• Top words leaderboard
• Leaderboard search system
• Updated stats page (see below)
• Added a combined leaderboard (hackatime + slack time)
• Added backups in coolify
• Minor repo restructure (moved shiptalkers-app to its own shiptalkers-app/ dir)
• Added gzip support for slack api scrapers

im lowkey running out of things to add…
Drop any suggestions in the comments

Updates

• Tad bit of time that i forgot to log progressively.
  Anyway
• DB got corrupted
• We rescraping dis shi rn
• Added a whole new formula (multiple times actually)
• Added tests for formula
• Did a bunch of fixes to try and save corrupted db
• Afterwards i just nuked it and we started rescraping yesterday
• Made ini_tables probe slack_messages_by_user first
• Added default clickhouse values into the rust app
• Now recomputes stats page cache every 30s
• Fixed UNKNOWN_IDENTIFIER: ‘chars’ (Yeah i had claude do this one)
• Added a better system to add new slack keys to the env, instead of 1 big env var, its now, SLACK_BOT_TOKEN_X=xoxb….
  Where X is token number
• made it a shared work queue instead of static round robin sharding for scraper
• Switched to a mimalloc allocator from glibc
• It now does backfills for channels and users in parallel
• Added a system for incase hackatime goes down
• Did a commit for one new keyboard being “last 2 months”
• It also now scrapes reactions

Updates

• Added proper test system
• More keywords
• Heavily sped up per user pages & stats page
• Updates stats ui to mostly show 5 decs for db size in gib
• added a display load time in ms to footer
• Nuked the admin panel
• Dependabot & PR labeler system(s)
• We now use alpine base for docker
• We got a tests system now
• I think thats it, idk tho

Updates

• May have forgot to be doing devlogs
  Anyway
• Actual thread scanning and actually scraping its replies
• Recconect system for run_socket so the slack bot part stops dying
• Added an admin panel
• Is bot and Is Deleted acc tags to make leaderboard better
• Slack_app_creation script (to scrape with more tokens.. faster)
• Added error support from slack
• Proper slack stats time computation for in channel image
• Scan older threads support (up to 30 days)
• If slack stops sending periodic pings then slack bot auto reconnects
• Displays coding time as hours and minutes instead of just minutes
• DB schema updates

Oops, forgot to devlog for a bit

Anyway

Updates

• its online
• it works
• Round robin support for multiple tokens for > speed
• it replies in channel (ship-talkers channel)
• downgraded clickhouse to 26.4 for my older server
• added a proper health check endpoint
• Optimized slack messages daily
• Realized conversations.history supports 999 per request instead of just 100 msgs
• Auto deploy system
• Limited resources for clickhouse
• Consistant rescrape checks w/ round robin support ofc

Thats about it i think

Oh yeah it finished the full slack scrape

Next up

• Changing scrape system for privacy reasons (maybe)
• more scrape accs
• QOL features
• Idk gang

Updates

As usual we got a bunch of features

• Search system
  • Channel search
  • User Search
• Per user stats page
• Per channel stats page
• Hackatime/HCA login systems
• Header & Footer
• Added a cool banner at top of the page to show its scraping for initial scrape
• Fix docker failing due to forgeting to COPY templates for it
• Added scopes needed for everything to readme
• Switched out the purple for a soft pink
  Still not done scraping