Rethinking Sparse Autoencoders: Select-and-Project for Fairness and Control from Encoder Features Alone
Lists
Only you see this
Abstract
Sparse Autoencoders (SAEs) have proven valuable due to their ability to provide interpretable and steerable representations. Current debiasing methods based on SAEs manipulate these sparse activations presuming that feature representations are housed within decoder weights. We challenge this fundamental assumption and introduce an encoder-focused alternative for representation debiasing, contributing three key findings: (i) we highlight an unconventional SAE feature selection strategy, (ii) we propose a novel SAE debiasing methodology that orthogonalizes input embeddings against encoder weights, and (iii) we establish a performance-preserving mechanism during debiasing through encoder weight interpolation. Our **S**election and **P**rojection framework, termed **S&P TopK**, surpasses conventional SAE usage in fairness metrics by a factor of up to $3.2$ and advances state-of-the-art test-time VLM debiasing results by a factor of up to $1.8$ while maintaining downstream performance.
BibTeX
@inproceedings{barbalau2025rethinking,
author = {Antonio Barbalau and Cristian Daniel Paduraru and Teodor Poncu and Alexandru Tifrea and Elena Burceanu},
title = {{Rethinking Sparse Autoencoders: Select-and-Project for Fairness and Control from Encoder Features Alone}},
booktitle = {Reliable ML @ NeurIPS},
year = {2025},
url = {https://openreview.net/forum?id=d3OYZ0XEc1},
}