@nosrep

oh no, 8 hours i need to devlog more holy moly
from last devlog, i kinda found a solution to the fail-soft problem?
Most engine’s LMR implementations will re-search twice if the previous search went over alpha (lower bound). The first search has reduced depth AND a null-window around alpha (search window of 1 centipawn; searches faster, but can only tell the direction of the true search result); if this is over alpha, it’ll re-search with full depth; if this is again over alpha, it’ll re-search with full depth and full bounds. My engine was skipping the middle step and jumping from reduced depth + null window -> full depth + full bounds. Apparently this is an accepted method and it was working fine compared to the other version, EXCEPT once I switched to fail-soft on beta.

in other news, I added:

• aspiration windows (75 elo): the initial search is normally unbounded, but aspiration windows try to clamp the search around a “guess” based on the previous depth of iterative deepening, and widen if it fails high or low. it did almost nothing before, and randomly became really good. don’t ask me why.
• razoring (22 elo): prunes a node if quiescence search says it’s too far below alpha. I already had it in a previous version and brought it over to the search rewrite. pretty simple, smallish elo boost.
• late move pruning (39 elo): prunes the latter part of move list; moves are sorted by how good they’re predicted to be, so latter moves are usually pretty garbage
• fixes (23 elo): small logic changes to LMR and LMP. don’t know how they didn’t lose more elo, the LMR bug was kinda bad.
• separate PV tracking: the PV (principal variation, the best moves for each player as calculated by the engine) was previously tracked by the transposition table, but that meant i didn’t have a lot of flexibility with the replacement strategy for the table, lest it wipe an important PV entry. tracking it separately lets me fiddle with the replacement strategy more, although I haven’t actually found one better than what I have now

here’s a graph generated from a google sheet with 100k rows (!?). it’s a histogram of every time a move was marked as “currently the best”, indexed by entry. ideally, it’d be all 0, meaning the best move was searched first every time, meaning there was basically no wasted effort.

6.5 hours? oh man
i was right, nmp only gained like 30 elo, gained much much more before. i tried to hunt down anything weird but i had to leave it at that. i also implemented pruning in quiescence search, so the engine only searches moves that have a positive static exchange evaluation (essentially, the result of all trades on a particular square).
i also accidentally found out my engine was doing fail-hard on beta cutoffs instead of fail-soft! in a fail-soft framework, the search is allowed to return outside of the bounds set by the alpha and beta parameters (it’s hard to explain succinctly here and i don’t want to keep doing it, search up alpha-beta search if you’re interested). thus, alpha and beta are soft bounds. this means each search returns more information about the move score.
i successfully did fail-soft on the alpha (lower bound) side, but not on the beta side due to what i can only assume is a coding mistake. the problem is, when switching to fail-soft on beta, the engine lost like 30 elo! i bisected the change to the LMR (late move reductions) patch, but still have no idea why LMR doesn’t play nice with fail-soft beta. i’m kinda at the end of my rope here, i might just have to eat the elo loss…

I finally got RFP to gain! but it might not be as good as it seems…

ive been struggling to get rfp, a pretty basic improvement, to actually do anything, and redid everything search step-by-step to make sure i didn’t do anything wrong. so far, i’ve added back SEE move ordering, butterfly heuristic, PVS, transposition table move ordering, and RFP. however, i don’t think this actually fixes my problem; the code is largely the same aside from some parameter changes, so my theory is that rfp is only working now because i added everything back in a different order, and now i might struggle with adding back different features that aren’t gaining anymore.

i’m currently testing null move pruning (based on the null move observation: doing something is basically always better than doing nothing), and it’s not doing as much as it used to… disturbing

ive nuked it….

i kept fighting rfp and talking with other people about this. apparently, rfp is one of the easiest gainers (something that gains elo/strength), and usually gains a lot, but again, nothing. i ran an overnight test of like 5000 games and the rfp version manages to LOSE 8 elo, where other people have gained 80.
i’m gonna try to restart search again, but basically fully. i don’t know if this will actually do anything, but i don’t see many other choices…

i’ve reentered the nightmare

when i last left off this project in ~september, I basically hit a wall with optimizations. I had a crapload of pruning and whatnot, but the elo of the engine didn’t really reflect it. I then did a search rewrite but got stuck on RFP (reverse futility pruning), which is supposed to gain a lot of strength but did basically nothing for me.

brief (??) aside: reverse futility pruning, in simple terms, checks if the static evaluation (evaluation without searching moves) is above beta (if a move scores above beta, it’s “too good,” and the opponent won’t allow that position to happen in the first place) by some margin. if it is, it’ll return without searching any moves, because it’s not likely that the player can lose that much material/positional advantage in whatever depth is left to search.

anyway, i’m still stuck on this. I’ve re-reimplemented half the search to recheck its correctness, because correctness issues can sometimes stuff new optimizations, but it seems to be all good. RFP still doesn’t do jack.

i don’t really know where to go from here. I could just ignore it, but the fact that RFP doesn’t do anything (for reference, another engine dev i talked to got like 80 elo out of RFP, which is insane) is a troubling sign and might mean something else is wrong.

hello! trout again. i’m gonna try to get this to #1 haskell engine, but it’s been such a pain…

after summer of making I did some more work on this, but hit a wall where all my changes either did nothing or gained like 10elo (which is not enough for the hundreds i need). I tried rewriting part of search but hit a wall again with the same problem, on features that should have made improvements!

i’ve been fighting with the search some more trying to get reverse futility pruning to do something meaningful. in someone else’s engine, it gained i think 80+ elo in total (!!!), which is bonkers compared to the 20 or something i get on a good day. it’s not a good day, so RFP actually manages to lose elo here.

i might have to rewrite search again. i don’t know how much of this is just wrong.

(here’s the commented out old search)

I made some duct modifications and now cooling seems to be much better. I think the new duct is choking the airflow a bit too much, but it’s more focused and can actually reach the spot right under the nozzle (which the old one seemed to struggle with?).

I also did a LOT of testing, didn’t track most of it. Probably could benefit from a little more tuning, but we can do the pin support sphere on fullcontrol.xyz now!

fixed some of the lingering issues, like the bltouch interfering with the duct and the bltouch interfering with the endstop (it’s always the bltouch…). we should be done now

it has been built! it is so loud!

I finished designing the ducting and finished printing out everything, immediately a couple problems: ducts bump into bltouch (this was also in the cad but i didn’t notice) and the ducts are really flimsy. The ducts themselves are fine but the front plate is too thin to keep it in place properly. They work but it’s kinda sketchy. To fix the bltouch thing I just melted the ducts with my soldering iron, already remodeled a fixed version but have yet to print.

The extruder seems to be not tuned very well. I set the manufacturer-recommended esteps, but it could be off. The cooling also seems to be sketchy? I can feel an insane amount of airflow but i don’t know if it’s going to the part yet. It aims a little low and I reprinted a bridging test I made earlier and the results seem to be worse.

I also need to do some fixes later if i want to use this long-term. I couldn’t make a new extruder cable because my crimping tool couldn’t handle jst-ph, so i’m gonna have to deal with the stock extruder cable which severely limits the amount of vertical travel I can handle.

i think i finally finished the ducts…
these were nightmarish to make. onshape conveniently doesn’t have 3d sketches, so I did a horribly janky system involving planes and a routing curve. The path the duct had to take ended up being kinda strange, so the loft would constantly crash out because the profiles turned too sharply. i did eventually get the loft to cooperate, and the duct looks okay? it’s still very lumpy, but it should transition relatively smoothly, which is what’s important for airflow.
i’ve also started printing the parts now. as i’m typing this, the main mount is finished and the fan mount is almost done. after that i should theoretically be good to assemble once i print the ducts

(this is the video i used for duct design principles: https://www.youtube.com/watch?v=YijwkQCOBEA)

I have no idea how this devlog ended up so long
Anyway, I cleaned up some potential crashes/undefined behavior, fixed up the README, and finally tagged a release + published to AUR!
Spent a lot of time wrestling with static linking. I remember using old versions of linux before and getting glibc problems so I wanted to link it statically. I first tried using musl-gcc, but I ended up needing too many static versions of libraries so I just made a Dockerfile with alpine (which has static versions of basically everything) and made the binary there.
Did you know PKGBUILDs are just bash scripts? I didn’t.

the clipboard-as-files thing is added!
I originally wanted to use FUSE/libfuse, but i realized i could just change the storage format to have the data be in a single file. now i’m just using symlinks instead, no dependencies
this was relatively easy, ran into a couple of bugs but nothing too crazy. Biggest thing I got hung up on was which directory to put it in ($XDG_STATE_HOME? $XDG_DATA_HOME? ~/clipboard?). I eventually decided on $XDG_RUNTIME_DIR which seems mostly appropriate - the only thing is that it’s kinda annoying to access, but you could just make your own symlink, change the path with $ZZZCLIP_SYMLINK_PATH, or add a bookmark like in the screenshot.

Another thing was to add extensions (e.g. .png) to the file, because apps like discord won’t recognize them if they don’t have extensions. I was initially gonna take and process one of the giant lists online, but then i found out most systems have a /etc/mime.types so i parsed that instead.

I added some more options to the CLI, which ended up being a lot more annoying than i expected. Here are the ones I added:

added to zzzclip list:

• -v: verbose mode. Verbose mode also lists the mimetype for a clipboard item.

added to zzzclip get:

• -f: runs in foreground. Here’s something you might not have known: wl-copy starts a background process when you run it. The clipboard contents aren’t owned by the compositor, so a process has to be running to supply the data. I made zzzclip get do the same thing, but like wl-copy I added an option to make it run in the foreground.
• -s: output summary. It’s basically zzzclip list -v but only for one item.
• -l: list stored mime types. It lists the mime types for an item.
• -m : directly prints the data for an item.

big ticket is overhauling the config format. Previously, the index looked like this:

4a93d8d7 6b5d69df
2143c202

Each random string of 8 characters was a label for data under a particular mime type, and each line was a selection (a selection can have multiple mimetypes). Each label had its own file to store the actual clipboard data, with the mimetype included in that file.

this worked fine, but I’ve been wanting to implement the feature where the current clipboard contents are accessible via file. If I moved the mimetype out of the data files, I could just use symlinks because each file has exactly the data I need; using the old format, i’d need to use libfuse to make “fake” files without the mimetypes instead.

this was a lot more annoying than i expected (thank god for address sanitizer, caught many mistakes), but I eventually got it working. I think i’m nearing the finish line

lots of minor-ish stuff for this batch of changes.
Biggest change is actually adding proper cli parsing and help messages! previously, the cli was extremely bare-bones and just good enough to actually work but didn’t have any decent error handling or help messages.

I’ve also ditched the option that runs the selector program (e.g. rofi, fuzzel). I didn’t know about he -nth those programs have, which let you hide data from the selector, so that functionality was kinda redundant.
(also the sizes are human-readable now, i.e. in kb/mb/gb instead of just raw bytes)

the tests were bad. I fixed them and now everything passes. hooray!

Anyway, welcome to macro hell!
i’ve been working through the todo list and one of the items I wanted to hit was to support the ext-data-control protocol on top of the wlr-data-control protocol. Wayland compositors work on “protocols,” and not every compositor supports every protocol; the *-data-control protocols are what allow clipboard managers to mess with the clipboard. wlr-data-control was first (i think) and is only supported by wlroots-based compositors, but ext-data-control is the standard now. (except the compositor I use, river, doesn’t support it yet…)

The problem with supporting both is that C is not a very expressive language. I had to generate wrapper functions for every function supplied by the protocol, which was bad enough, but I also had to write wrapper functions for my own functions that were used in listeners. This part could have been so much worse, but I was able to hide some function pointers in the user data and cut down even more boilerplate.

Not fun. 3/10.

there’s always the boring janitorial stuff (e.g. i renamed a few files, moved some stuff around, fixed a mem leak and a potential crash, etc.)
the cooler change is that I finally added testing! testing an app like this is infinitely more annoying than something like a chess engine (shameless plug) because it’s built around IO. I followed in fuzzel’s footsteps and did integration testing in a shell script, which isn’t something i realized is practical until seeing it used there. the very sparse tests actually already did find a bug, but i’ll fix it later……
the testing is to set up more disruptive features i’ll add later, like implementing the ext-data-control protocol on top of wlr-data-control, and I want to make sure they won’t break anything without having to manually test

two things:

1. finally got rid of libpcre2. For the people that didn’t read the previous devlog, I needed a regex implementation as part of the user-provided mimetype selection, but I didn’t realize POSIX had one in regex.h and pcre seemed like the standard. POSIX regexes should be more than enough, though, so that’s a dependency killed
2. added locking and index refreshing. nothing was locked previously which is Not Good™️, but more importantly, the daemon never refreshed the index of clipboard data, so if i deleted one, the index would get rewritten anyway with the deleted file and corrupt it. Solution was to stop persisting the index and only read it when it’s used.

minor changes for this devlog. spent way more time than is reasonable thinking of the best way to deal with allocation failures… it seems like everyone online has a different opinion on it. I tried doing it “properly” before seeing how many lines of checks that would require for a codepath that would realistically never get hit, so I gave up and just did xmalloc (basically, a wrapper around malloc that exits with an error message when it fails).
Also in the theme of error handling, I did a sweep of the code and wrapped (most) of the io functions I missed to check for failure. it seems like overkill… there has to be a better way but maybe this is how it is.
Finally, I switched from libpcre2 to posix regex.h. I didn’t realize POSIX had a regex.h and it’s way more than is required for this use-case (matching mimetypes).

I started this project much earlier than stardance (it’s years old by now), but it’s only recently become usable. think of this devlog as motivation/background
the main goal was to make a clipboard manager that could

1. persist clipboard selections after applications closed
2. have configurable mimetype selection

Here’s something interesting you might not have known about linux clipboards (both wayland and x11): the data doesn’t actually exist until a client requests it. This is how applications like firefox offer 20 morbillion image types; it’s converting the image on-the-fly. Given this, it’s impractical to request every offered type, so most clipboard managers I’ve found solve this by giving up and only doing text or text+hardcoded image mimetypes. zzzclip has a probably-overcomplicated system that lets you configure which mimetypes it’ll remember and priority between mimetypes (plus regexes).
Another feature I want to add eventually is a FUSE clipboard system where it can make a “fake” (meaning it’s backed by the program, not something on disk) file holding the current clipboard contents.

(as for what these 17 hours are actually for: it’s just adding config parsing and a load of cleanups.)