White House science advisor Michael Kratsios alleged that Moonshot, a Chinese firm, copied Anthropic’s Fable large language model (LLM) to create the Kimi K3, which is the largest open-weight model currently available. Kratsios stated Moonshot’s model utilized chips not authorized for export to China, adding that the covert industrial distillation aimed at stealing U.S. technology is “unacceptable.” He noted ongoing discussions about the potential banning of Chinese open-weight models that have become controversial in the AI sector.
Treasury Secretary Scott Bessent echoed Kratsios’ comments, revealing that U.S. large language models have been detected in many Chinese models, which he also described as “unacceptable.” The Treasury Department did not clarify the exact nature of the watermarks found in these models. Experts, however, expressed skepticism about whether distillation alone could account for the advanced capabilities of Kimi K3, especially considering Fable’s release date of July 1.
Braden Hancock, a researcher at the Laude Institute, pointed out that it is implausible to develop a model with the strength of Kimi K3 in such a short time frame using only distillation techniques. He stated, “You can’t distill that much data, train a model, and release it in two weeks.” Nathan Lambert, from the Allen Institute for AI, noted the diminishing effectiveness of distillation over time and argued that advanced methods like reinforcement learning might be necessary for such models to achieve their capabilities.
The distillation process involves systematically querying an LLM to replicate its functionalities. This process might include supervised fine-tuning (SFT), where the target model trains a new model by emulating its behaviors. Lambert assessed that while SFT has benefits, they are waning as model complexity rises. To replicate Fable’s capabilities might require reinforcement learning, which can be resource-intensive and may not guarantee superior performance.
An ongoing conflict exists in the AI industry regarding distillation practices. Anthropic had earlier accused Moonshot of systematically extracting data from its models, citing millions of suspicious exchanges monitored through IP addresses. Elon Musk testified that his company, SpaceXAI, also utilized distillation techniques on OpenAI models to develop Grok, demonstrating how common this practice is across the sector.
Hancock underscored that American perceptions may downplay the technical skills of Chinese AI teams, referring to Moonshot’s co-founder, a PhD graduate from Carnegie Mellon University. Moonshot allegedly leveraged banned Nvidia chips while accessing GB300-equipped servers in Thailand to develop Kimi K3. Sam Bresnick, a research fellow at Georgetown’s Center for Security and Emerging Technology, noted the existence of a black market for such chips, which remain banned for export to China. Efforts to enforce regulations on the use of these advanced chips have yet to see significant progress.
President Biden’s Department of Commerce proposed rules for data centers to establish customer identification measures in 2024. However, no significant developments in this area occurred during the previous administration, and exporters must ensure their advanced chips are used for legitimate applications.
