California Governor Newsom signed a landmark AI regulation bill requiring red teaming and third-party audits for models with trillions of parameters before deployment.

Author and source: AIBase

California Governor Gavin Newsom has officially signed a landmark package of artificial intelligence regulations. Notably, this new set of laws—designed to strengthen safety risk assessments for advanced large models, enhance transparency disclosures, and prevent catastrophic cyberattacks—received rare endorsement and public support from leading Silicon Valley AI companies, including OpenAI and Anthropic, during the legislative process.

Before launching a trillion-parameter model, it must pass "red team exercises + third-party audits."

In recent years, as the computational scale of generative AI has expanded exponentially and the autonomous capabilities of cutting-edge foundational models have advanced significantly, concerns have grown over the potential misuse of AI for critical infrastructure sabotage, automated biochemical weapons development, large-scale deepfake manipulation, and exploitation of severe cybersecurity vulnerabilities. With federal AI legislation in the U.S. Congress remaining stalled, the California State Legislature has emerged in recent years as the nation’s leading advocate for AI regulation.

The multiple bills signed in this round focus on legally establishing safety testing standards for ultra-large-scale foundational models, requiring model developers to conduct standardized red team adversarial exercises, implement strict catastrophic risk mitigation protocols, and submit third-party independently audited compliance safety reports to state regulatory authorities before training or deploying frontier systems with trillions of parameters.

Unlike the aggressive regulatory proposals that previously caused intense upheaval in Silicon Valley and drew strong opposition from tech giants and venture capital firms, this package of bills deeply incorporated technical recommendations from leading AI laboratories during its drafting and revision. Anthropic and OpenAI had previously expressed serious concerns about overly rigid provisions that could stifle the open-source community or indefinitely expand civil liability; after intensive lobbying and targeted revisions, the final legislation focuses on establishing a transparent “pre-market risk assessment framework” and benchmarks for industry-best safety practices, rather than imposing rigid technical caps or impractical punitive measures, thereby earning collaborative support from major AI research institutions within the legal framework.



Deepfakes must be watermarked, and whistleblowers are protected by law.

In addition to underlying security reviews, the bill also establishes requirements for provenance labeling and watermarking of AI-generated content, mandating platforms and major content distribution channels to embed system-level digital fingerprints on deepfake audio and video content to prevent the spread of false information in elections and malicious impersonation fraud. Simultaneously, the bill provides clearer legal protections for “whistleblowers”—including internal employees and researchers—who identify and disclose critical security vulnerabilities in models, explicitly prohibiting companies from using restrictive confidentiality agreements to deny technical personnel the right to report catastrophic security risks to regulatory authorities.

This move marks a regulatory path in California—the global hub for AI research and development—that has gained relatively broad industry approval for balancing strict safety boundaries with the need to sustain innovation in the sector.