Generative Latent Priors for Interpreting and Steering Protein Language Models
The method improves protein-model probes, but repeated sequence editing still compromises structural plausibility.
The researchers trained diffusion models on frozen ESM-2 activations at two model sizes. Individual GLP “meta-neurons” outperformed InterPLM’s released sparse-autoencoder features at detecting discrete Swiss-Prot concepts, largely by concentrating existing information. GLP correction also made steering three biochemical properties effective within the model’s representations. After repeated decoding into actual sequences, however, structural plausibility deteriorated—a failure the authors trace to rewriting coverage and the decoder rather than the prior.
ArXiv · AI/CL/LG's note
The researchers trained diffusion models on frozen ESM-2 activations at two model sizes. Individual GLP “meta-neurons” outperformed InterPLM’s released sparse-autoencoder features at detecting discrete Swiss-Prot concepts, largely by concentrating existing information. GLP correction also made steering three biochemical properties effective within the model’s representations. After repeated decoding into actual sequences, however, structural plausibility deteriorated—a failure the authors trace to rewriting coverage and the decoder rather than the prior.
ArXiv · AI/CL/LG's note
score 4