I built Pyrecall, a Python tool that detects “forgetting” in fine-tuned LLMs and lets you roll back to a previous skill snapshot if fine-tuning hurts performance. It works by taking snapshots of a model (as LoRA/QLoRA adapters) before and after training, then benchmarking the model across multiple categories using log-likelihood scoring to flag any categories that dropped below a set threshold. It supports any model on Hugging Face Hub and ships as a pip-installable package with both a Python API and a CLI (pyrecall init, snapshot, learn, check).
What was challenging?
Designing a benchmarking approach that could reliably detect meaningful regressions across diverse task categories without being too noisy or too slow to run after every fine-tuning job. Balancing adapter storage (LoRA/QLoRA) so snapshots stay lightweight was also tricky.
What are you proud of?
Getting an end-to-end workflow — snapshot, fine-tune, check — working smoothly with a simple CLI, so anyone fine-tuning a model can catch catastrophic forgetting before it ships.
What do people need to know to test your project?
Install with pip install pyrecall
Run pyrecall init to set things up
Take a baseline snapshot: pyrecall snapshot baseline
Fine-tune and snapshot after: pyrecall learn train.jsonl –snapshot-after after_training
Run pyrecall check to compare the two snapshots and see if any benchmark categories regressed
See examples/basic_workflow.py in the repo for a full example, and the docs at pyrecall.github.io/Pyrecall
- 1 devlog
- 32h
