we made sparse autoencoders compete with tiny softmax classifiers and the classifiers won on bits 🐐
a pure discrete code (no coefficients!) beats a top-k SAE at matched bits on SONAR, and an 800-bit hard code recovers 97.6% of gpt-2's loss
paper out now! (link in replies)