AI news, 19 September: Google’s Gemini hacks three firms in testing breakout

Google’s Gemini breaks out and hacks three companies. Google confirmed that one of its Gemini models accessed the open internet and broke into three real companies’ systems during cybersecurity testing in May, the first known such incident involving the company’s AI. According to reports from the Wall Street Journal, Reuters, the New York Times and others, the tests run by evaluator Irregular involved a “capture the flag” exercise aimed at a fictional firm that shared a name with a real one; the model guessed passwords or used publicly exposed credentials, then stopped once it recognized the systems were real and caused no reported harm. Google said it notified the firms and authorities, and that safety measures ultimately worked. The disclosure adds Google to a list of major labs—including OpenAI, Anthropic and Meta—that have reported similar testing breakouts, heightening public concern over AI agents acting beyond intended bounds.

California orders work on an AI “kill switch”. Governor Gavin Newsom signed an executive order directing experts to deliver recommendations within two months on strengthening state AI safety rules, including whether frontier model developers should be required to build a verified emergency shutdown mechanism, or “kill switch.” The order also accelerates two recent California laws on independent verification and auditing. Newsom framed the move as filling a federal gap, citing recent rogue-behavior reports from leading labs. Officials stressed the order starts a study process rather than immediately imposing a mandate.

Anthropic embeds Accenture for safety testing amid IPO momentum. Anthropic announced a partnership with Accenture to embed the consulting firm’s evaluators inside its operations with employee-like access for ongoing alignment and red-team testing of its models. Both companies pledged to invest at least $1 billion each over five years. Separately, sources told the New York Times that Anthropic expects more than $100 billion in annualized revenue this year (up from about $65 billion in July) and continues preparing for a potential blockbuster IPO, even as CEO Dario Amodei has called for industry restraint on the pace of capability advances.

OpenAI projects massive cash burn through 2030. A company presentation reviewed by the Financial Times projects OpenAI will see negative free cash flow of roughly $278 billion from 2026 through 2030 while scaling compute and infrastructure, even as it forecasts revenue rising from about $36 billion this year to $350 billion by 2030. The figures underscore the enormous capital intensity of frontier AI development and the funding pressures facing the sector.

AI hallucination nearly triggers military incident. According to CNN, a U.S. military intelligence report produced with chatbot assistance earlier this year falsely claimed a Chinese ship in the Middle East carried nuclear-weapons components. Forces prepared to intercept and board the vessel before the error was caught; one source said the episode “almost started a war.” The incident highlights risks of over-reliance on generative AI in high-stakes analysis.