How the bad science of AI doomerism is good for big business

The Bulletin of the Atomic Scientists | Sara Goudarzi

Anthropic and OpenAI are leveraging existential 'doomer' narratives to secure massive investments and advocate for industry self-regulation, despite a lack of scientific basis for calculating extinction risks. High-profile staff resignations, including Jacob Coxon's recent departure from Anthropic, have fueled public anxiety while simultaneously boosting the perceived power and market value of these proprietary systems.

This corporate-led alarmism draws heavily from the philosophical movements of effective altruism and longtermism, which assign arbitrary probabilities to human extinction. These figures lack scientific evidence. Critics argue this focus on hypothetical superintelligence distracts from immediate, tangible harms like labor exploitation, environmental degradation, and military miscalculations. For instance, an AI-assisted intelligence report nearly triggered a United States military operation against a Chinese vessel last spring. Meanwhile, legislative efforts like Senator Bernie Sanders' proposed bill to ban superintelligence risk codifying these speculative threats, ultimately reinforcing the tech industry's market dominance and shielding frontier firms from independent public audits.

Comment

The near-miss involving a United States military operation against a Chinese vessel, triggered by an AI-assisted intelligence report, exposes the immediate operational dangers of algorithmic integration in command decisions. Rather than managing speculative existential threats, military planners face the immediate risk of automated confirmation bias within systems like Project Maven. The INDOPACOM near-miss demonstrates how rapid data synthesis can compress decision cycles to the point of accidental escalation before human oversight can verify the underlying telemetry. The reliance on unverified algorithmic outputs directly threatens the integrity of tactical reconnaissance in contested environments like the South China Sea.

Downstream, this operational friction likely forces a costly restructuring of data-validation protocols within the US Defense Intelligence Agency. Command structures face a choice between slowing down target-acquisition pipelines or accepting higher rates of false-positive intelligence during high-tempo operations. Consequently, the deployment of unverified predictive models during maritime patrols risks generating systemic distrust among US Indo-Pacific Command operators toward the automated battle management systems slated for the Pacific Vanguard exercises.

Strategic Question for Discussion

How does the integration of unverified predictive models within Project Maven alter the balance between rapid target acquisition and human-in-the-loop verification during high-tempo maritime encounters?

The trajectory indicates that Project Maven's integration will likely force commanders to accept a higher tolerance for algorithmic error to maintain speed advantages. My assessment is that this shift prioritises sensor-to-shooter velocity over absolute target verification, making accidental engagements more frequent in contested waters. Consequently, the operational reliance on these automated systems will likely degrade tactical patience during critical early-stage crises.

Share your assessment in the comments below.

To read in details click here