Anthropic and AE Studio on Wednesday published a method called Gradient-Routed Auxiliary Modules (GRAM) for isolating dangerous knowledge within AI models into removable modules. This technique allows for the separation of sensitive knowledge without affecting the model’s overall performance.
GRAM incorporates small auxiliary neural compartments into a language model, each focusing on specific sensitive categories such as virology, cybersecurity, or nuclear physics. When a module is deleted, the model functions as if it had never been trained on that data. Conversely, when a module is activated, the knowledge becomes fully available.
The method modifies the standard transformer architecture by extending the width of MLP layers with these auxiliary modules. During training, only the module corresponding to a dual-use category is active when the model encounters related data.
The researchers tested GRAM on models ranging from 50 million to 5 billion parameters. They trained an 800-million-parameter model on diverse text data alongside four dual-use domains. The dual-use data made up approximately 0.25% of the training data for each respective domain.
Results indicated that removing GRAM modules eliminated specific capabilities nearly as effectively as if the model had never been trained on that data, while general performance remained close to baseline levels. The GRAM approach proved robust against adversarial fine-tuning, unlike post-hoc unlearning methods that merely suppress knowledge instead of permanently removing it.
This research emerges amid recent challenges in AI governance, particularly concerning export controls on Anthropic’s models implemented by the Trump administration in June due to national security concerns. Those restrictions were lifted on June 30 after Anthropic worked with the Commerce Department to mitigate related risks.
GRAM presents a potential compromise in AI policy, enabling selective access control instead of broad model restrictions or behavioral guardrails. A vetted biosecurity lab could receive a model with intact virology knowledge, while a general deployment would exclude that module entirely.
However, the researchers noted that this work is preliminary and has not been applied to production models at Anthropic. Challenges remain regarding the scalability of the technique to larger models and difficulties in separating entangled capabilities, where general biology knowledge overlaps with dangerous virology knowledge. The work was led by AE Studio researchers in collaboration with Anthropic’s Cem Anil and Alex Cloud.
