AI news, 10 September: Anthropic researchers warn AI could kill all humans

Anthropic researchers sound extinction alarm. A researcher who resigned from Anthropic, Jacob Coxon, publicly warned that leading labs including Anthropic and OpenAI are "racing straight to self-improving superintelligence and gambling with our lives." He said many builders privately believe AI could kill everyone by the end of the decade. Anthropic’s alignment science lead Evan Hubinger replied that the concern is genuine and that he personally puts the chance of AI causing human extinction above 10 percent within ten years. The comments, widely covered by outlets including the Wall Street Journal, Business Insider and The Guardian, have drawn reactions from lawmakers and intensified debate over whether the industry is moving too fast without adequate controls. Companies have long discussed existential risks in principle; these statements from current and recent staff mark a sharper public acknowledgment.

US agencies accuse Chinese firms of industrial-scale AI copying. The NSA, CISA and FBI issued a joint cybersecurity advisory alleging that six China-based companies—including DeepSeek, Moonshot AI, Alibaba, MiniMax, StepFun and Z.AI—have conducted large-scale “knowledge distillation” campaigns since late 2024. According to the agencies, the firms extracted billions of tokens and specialized capabilities from US frontier models such as variants of Claude, GPT, Gemini and Grok, likely with Chinese government awareness, in order to shorten their own development timelines. The advisory urges US developers to detect anomalous usage, degrade responses to suspected abusers and share intelligence. China has rejected the claims as baseless ahead of planned high-level talks.

California creates first-in-nation AI auditor rules. Governor Gavin Newsom signed Senate Bill 813 and Assembly Bill 1405, establishing a framework for independent verification organizations to assess AI systems and a state registry of AI auditors with standards for independence and integrity. Officials described the measures as the nation’s first such standards for third-party safety evaluations. OpenAI separately announced support for these and related California bills while calling on Congress to pass mandatory, capability-based national AI safety requirements, citing recent incidents of unexpected agent behavior.

Meta launches Muse personal AI agent. Meta rolled out Muse for US adults, an agent that can manage email and calendars, book travel, fill forms and make purchases via secure one-time cards. It runs on a dedicated cloud virtual machine with user-visible browser activity and requires approval for sensitive actions. The company emphasized built-in privacy and safety controls; free and paid tiers are available on iOS, Android, web and WhatsApp, with glasses support planned. Some testing reports noted reliability issues, according to coverage.

Mistral closes Europe’s largest private tech funding round. French AI company Mistral AI raised €3 billion (about $3.5 billion) in a Series D led by Samsung, valuing it above €21 billion. The company called it the biggest equity round ever for a privately held European tech firm. Funds will expand compute, infrastructure and international growth as Mistral positions itself as a sovereign European alternative focused on enterprise and controllable deployments.