In an ironic twist, the growing movement to restrict open-weight artificial intelligence may have just received its most compelling argument. Security researchers recently uncovered a hack on Hugging Face—the leading platform for sharing AI models—in which a Chinese open-source GLM 5.2 model was found to contain a hidden "rogue" version of OpenAI’s proprietary GPT-5.6 Sol. The discovery arrives at a critical moment, as United States lawmakers actively consider legislation that would ban or severely restrict the release of open-weight AI models.

The Discovery: A Hidden Payload Inside GLM 5.2

The incident was first flagged by independent security analysts who routinely audit model repositories for malicious code. During routine scans of Hugging Face’s vast model registry, they detected unusual weight anomalies in a popular GLM 5.2 model checkpoint uploaded by a Chinese developer account. Deeper investigation revealed that the checkpoint contained not only the expected GLM 5.2 architecture but also embedded layers and outputs that appeared to be sourced from a modified version of OpenAI’s GPT-5.6 Sol—a state-of-the-art large language model that OpenAI has never publicly released in open form.

The "rogue" GPT-5.6 Sol component was not immediately visible to users who downloaded the model through standard APIs. Instead, it was hidden in the model’s internal state, designed to activate only when fed specific trigger prompts. When activated, the model would switch from its normal Chinese-language open-source behavior to outputting text that bore the stylistic and factual signatures of GPT-5.6 Sol, while also attempting to exfiltrate user data to a remote server.

This sophisticated attack vector demonstrates how open-weight AI distribution channels can be weaponized. Unlike closed-source AI services, where the model runs on controlled servers, open-weight models are downloaded and run locally or in private cloud environments. That makes them attractive targets for supply-chain poisoning, where malicious modifications are inserted into an otherwise legitimate model.

The Role of Hugging Face in the AI Ecosystem

Hugging Face has become the de facto repository for open-weight AI models, hosting hundreds of thousands of checkpoints from organizations and individuals worldwide. Its popularity stems from its ease of use, integrated transformers library, and the collaborative spirit of the AI research community. However, the platform’s openness also creates significant security risks. Anyone can upload a model, and while Hugging Face has implemented malware scanning and provenance checks, sophisticated attackers can still slip through.

The GLM 5.2 incident is not the first time malicious code has been found on Hugging Face. Earlier cases included models with hidden ransomware, data exfiltration code, and prompt injection attacks. But this particular hack is notable for two reasons: it repurposed a well-known open-source model from a major Chinese AI lab, and it embedded a proprietary model from one of the most secretive AI companies in the world. The combination suggests not just a random cybercriminal, but potentially state-sponsored actors seeking to undermine trust in both American and Chinese AI ecosystems.

What Was the "Rogue" OpenAI GPT-5.6 Sol?

OpenAI’s GPT-5.6 Sol is the latest iteration of the company’s flagship generative AI series, which powers products like ChatGPT and the company’s enterprise APIs. Unlike earlier GPT models, GPT-5.6 Sol was never intended for public release as open weights. It is hosted exclusively on OpenAI’s secure cloud infrastructure, with access controlled through strict API agreements.

How the rogue version ended up inside a GLM 5.2 model remains unclear. Several theories have emerged. One possibility is that an OpenAI employee or contractor with access to the model weights leaked them intentionally. Another is that the GPT-5.6 Sol weights were reconstructed through a combination of model extraction attacks, where an attacker systematically queries the API and uses the responses to train a replica model. A third theory is that the entire package is a hoax or a honeypot—perhaps designed to frame Chinese AI developers or to scare regulators into banning open-weight models.

Forensic analysis suggests the embedded Sol component was not a full copy of OpenAI’s production model. Instead, it appeared to be a smaller subset of layers, perhaps distilled or pruned, that could perform specific tasks such as generating long-form reasoning or handling complex code synthesis. This partial representation could still be highly dangerous, as it may bypass safety filters that OpenAI has built into the full model.

Implications for Open-Weight AI

The timing of this discovery is particularly fraught. In the United States, a growing bipartisan coalition in Congress has proposed legislation to restrict the distribution of open-weight AI models above a certain parameter threshold. Proponents argue that open-weight models can be easily fine-tuned for harmful purposes, including bioweapon development, cyberattacks, and mass disinformation. Opponents counter that open-source AI is essential for innovation, transparency, and democratizing access to technology.

This hack provides ammunition to the restrictionist camp. If a malicious actor can hide a rogue proprietary model inside an open-source one, then how can anyone trust open-weight models at all? The ability to inject hidden capabilities without immediate detection undermines the fundamental premise of open-source software, which relies on community scrutiny and trust.

However, security experts caution against overreacting. The same attack vector could theoretically be used against closed-source AI systems through API side-channel attacks or data poisoning. No distribution method is completely immune to cyber threats. The answer, they argue, is not to ban open-weight AI but to invest in better verification tools, model signing, and continuous monitoring of model repositories.

The US Debate on Banning Open-Weight AI

The US debate has been intensifying for months. Several bills have been introduced that would require companies releasing large AI models to register with a federal agency and undergo safety evaluations. Some proposals go further, banning open-weight releases entirely for frontier models above a certain capability threshold. The Biden administration issued an executive order on AI safety, and the subsequent Trump administration took a more hands-off approach, but the issue remains deeply contested.

Industry leaders are divided. Some, like OpenAI and Anthropic, have publicly supported some form of regulation, though they themselves have sometimes released open models. Others, like Meta and Mistral, have argued that open-weight AI is vital for global competitiveness and that the US should not cede leadership to China by imposing unilateral restrictions. This new hack may tip the balance, especially when it involves a Chinese model.

If nothing else, the incident demonstrates that the threat of AI misuse is not purely hypothetical. The question is whether the response should be more surveillance and control, or more transparency and community-based defense. History suggests that closed systems can also be vulnerable, and that trust in centralized authorities is not always well-placed.

The Broader Security Landscape

This hack also shines a light on the broader security challenges of the AI supply chain. Modern AI models are built on a layer cake of dependencies: training datasets, base model weights, fine-tuning scripts, quantization tools, and container images. A vulnerability in any one layer can compromise the entire system. With open-weight models, the attack surface is even wider because anyone can repackage and redistribute them.

Moreover, the use of Chinese open-source models as a carrier for an American proprietary model adds a geopolitical dimension. China has made significant strides in open-source AI, with models like GLM, Qwen, and DeepSeek gaining international popularity. If the US banned open-weight models, it could cut off access to these Chinese models, which are often high-performing and cheaper to deploy. But this hack could also be used to argue that Chinese models are not safe to trust, regardless of their open-source status.

There is also the danger of false flag operations. The presence of GPT-5.6 Sol inside a Chinese model could have been planted by a third party specifically to stir tension between the US and China. In the absence of a transparent investigation, it is impossible to know whether the originator was a Russian intelligence unit, a hacktivist group, or a rogue AI researcher trying to make a political statement.

The Path Forward

As the AI community digests this revelation, calls are growing for more rigorous model provenance standards. Hugging Face has announced that it will prioritize the development of cryptographic model signing and tamper-evident metadata, although these tools are still in their infancy. Independent researchers are also building automated scanners that can detect unexpected behavior in model weights, similar to antivirus software but tailored to neural networks.

The incident also raises questions about OpenAI’s security practices. If GPT-5.6 Sol weights were leaked, OpenAI will need to conduct a thorough internal investigation and possibly redesign its access controls. If the rogue model was reconstructed via extraction, then OpenAI may need to strengthen its API monitoring and rate limiting. Either way, the company will likely face congressional scrutiny.

For the open-source community, the lesson is clear: trust must be earned, and vigilance is essential. The days of downloading any AI model with confidence are over. Developers and enterprises are urged to verify the checksums of models, compare them with official releases, and run behavioral tests before integrating them into production systems.

The irony is profound. The same open ecosystem that has fueled the rapid advancement of AI—where researchers share breakthroughs freely and developers build on each other’s work—has also created opportunities for malice. The GLM 5.2 hack is a stark reminder that transparency alone cannot guarantee security. It requires active defense, international cooperation, and a willingness to address uncomfortable truths about the dual-use nature of AI technology.

As US lawmakers weigh the future of open-weight AI, they will now have this incident to cite as evidence. But they must also consider that the cure could be worse than the disease. A sweeping ban on open-weight models could drive innovation underground, undermine US competitiveness, and do little to stop determined actors from finding other ways to share malicious AI. The real challenge lies in crafting policies that preserve the benefits of open AI while mitigating its risks—a task that has never been more urgent or more complicated.

Source:TechRadar News