A new study published in the Proceedings of the National Academy of Sciences (PNAS) argues that weak AI safety regulations may actually exacerbate risks rather than mitigate them. The research, conducted by scholars from Cornell University and Carnegie Mellon University, employs game theory and theoretical economics to model how regulation impacts the safety behavior of companies across the AI supply chain. The central finding is that poorly designed rules can lead to a free-rider problem, where general-purpose AI developers offload safety responsibilities onto downstream companies, ultimately reducing overall safety.
The study specifically models a scenario where regulation targets only the companies that apply AI in specific domains—such as medical diagnostics or customer service chatbots—while leaving the core AI model developers (e.g., OpenAI, Google, Anthropic) largely unregulated. At first glance, this seems logical: policymakers might believe it is easier to regulate discrete use cases rather than the broad, foundational models. However, the researchers demonstrate that such a regulatory framework can backfire. When downstream companies bear the full burden of safety, the upstream developers have less incentive to invest in third-party audits, red-teaming, or other safety measures. They assume that the downstream specialists will catch any issues, leading to a collective underinvestment in safety.
“There’s a free-riding behavior that occurs,” said Benjamin Laufer, the study’s principal author. “The regulation acts as a tool for the general provider to offload the safety burden onto the downstream specialist.” The result is a product that may be less safe than if no regulation existed at all, because the lack of clear rules would have forced both parties to independently take precautions.
The paper arrives at a critical moment in the global debate over AI governance. In the United States, the Trump administration has advocated for a light-touch approach, arguing that excessive regulation could stifle innovation and cede the AI race to China. This pro-innovation camp contends that the industry should self-regulate and that any federal guardrails should be minimal. On the other side, safety advocates warn that the profit motive leads companies to underestimate existential and societal risks—from AI-induced job displacement to the strain of data center energy consumption and even potential loss of human control. The two camps are often locked in a polarized argument, with each accusing the other of either being alarmist doomers or corporate shills.
The researchers, however, propose that safety and economic growth are not necessarily at odds. Their model reveals a “sweet spot” where strict, well-designed regulation can improve outcomes for everyone in the supply chain. The dynamics are a classic prisoner’s dilemma: if both the general-purpose provider and the downstream specialist cooperate by making meaningful safety investments, everyone benefits. But without a binding agreement, each party fears the other will defect—cutting costs and offloading risk. The rational choice becomes to defect, leading to a suboptimal equilibrium. Strict regulation that applies to all stakeholders enforces cooperation, forcing both to invest adequately in safety, which in turn increases the overall utility (revenue minus cost) for all players.
“People think of AI as a single object, but actually AI involves a very complicated set of stakeholders and actors that each have their own contributions to the technology,” Laufer said. “To regulate in a thoughtful way, we need to consider the whole supply chain, not just a single provider or entity.”
The study sheds light on a growing concern in AI policy circles: the tendency to focus on downstream applications while ignoring the upstream model creators. Historically, regulations for emerging technologies often target end uses—for instance, drugs are regulated at the point of sale, and vehicles are regulated at the point of use. But AI is different because the same foundational model can serve countless purposes, and its safety properties are largely determined at the training stage. A model that is not robustly tested against adversarial attacks, bias, or failure modes will propagate those flaws into every downstream application. Thus, the authors argue, the most efficient point of intervention is at the model development stage, applying binding safety standards that both producers and deployers must meet.
One of the paper’s most compelling contributions is to show that this approach can be game-theoretically stable. When regulation is strict and evenly applied, the optimal strategy for firms is to comply and invest. The market then rewards safer products, and the innovation race can continue on a level playing field. The alternative—weak or misdirected regulation—creates a race to the bottom, where cutting safety costs becomes a competitive advantage.
This analysis comes as several jurisdictions, including the European Union with its AI Act, are experimenting with tiered regulatory frameworks that classify models by risk level. The EU’s approach already imposes stricter rules on high-risk applications and on the providers of general-purpose AI models. The new study supports such a strategy, but warns against loopholes that exempt certain model sizes or open-source variants. The researchers stress that any exemption can create a free-rider problem, as firms might choose to release models just below the threshold to avoid compliance.
The findings also have implications for the current U.S. policy landscape. The Trump administration has issued executive orders emphasizing voluntary commitments and downplayed the need for mandatory safety testing. Critics argue that this leaves the burden on downstream users, many of whom lack the expertise to evaluate the safety of a massive language model. The PNAS study provides a formal model to back up those criticisms, showing that voluntary, downstream-focused regulation is inherently fragile.
In addition to its policy relevance, the research adds a theoretical foundation to a debate that has largely been driven by anecdote and ideology. By framing AI safety as a collective action problem, the authors point to a clear solution: binding regulations that apply horizontally across the value chain. Without such measures, the industry may find itself in a race not to the top, but to the bottom—where the least safe products become the most profitable, and where the public pays the ultimate price.
The paper has already generated discussion among AI ethicists and policymakers. Some praise it for bringing rigorous economic thinking to a domain often dominated by tech utopianism or dystopian fear-mongering. Others caution that the model is based on assumptions of rational actors and perfect information, which may not hold in the messy reality of corporate politics and regulatory capture. Yet even the skeptics agree that the central insight—that misdirected regulation can be worse than none—deserves serious attention.
As the world races to define the rules for artificial intelligence, this study serves as a timely reminder: when it comes to safety regulation, getting it wrong may be more dangerous than doing nothing at all.
Source:Gizmodo News