How the bad science of AI doomerism is good for big business

Bulletin of the Atomic Scientists | Sara Goudarzi

Anthropic and OpenAI are leveraging unscientific existential risk narratives to inflate their market valuations and secure regulatory advantages, according to industry critics. High-profile resignations of researchers like Jacob Coxon and Jan Leike have fueled public fears of an uncontrollable superintelligence. This doomer narrative serves as a powerful marketing tool for these multi-billion-dollar firms.

Historically, these companies emerged from the effective altruism and longtermism movements, which assign arbitrary probabilities to human extinction. Critics argue that focusing on hypothetical AI apocalypses distracts from immediate, tangible harms. These include environmental degradation, labor exploitation, and flawed military intelligence systems. A recent CNN report revealed that an AI-assisted intelligence report nearly triggered a US military operation against a Chinese vessel. Such real-world failures are already occurring. Yet, tech executives use existential warnings to advocate for self-regulation through aligned bodies like Model Evaluation and Threat Research. This strategy effectively crowds out independent public oversight.

Comment

The integration of unverified algorithmic models into military command structures introduces severe operational hazards. A prime example is the CNN-reported incident where an AI-assisted intelligence report nearly triggered a United States military operation against a Chinese vessel. This near-miss exposes the vulnerability of automated target-recognition and threat-assessment systems to false positives in contested maritime environments. The speed of algorithmic processing outpaces human verification, compressing decision cycles to a dangerous degree.

The underlying mechanism of this vulnerability lies in the training data of these predictive models, which often lack high-fidelity, real-time telemetry from active theatres. When deployed in complex environments like the South China Sea, these systems misinterpret civilian or ambiguous military manoeuvres as hostile intent. Consequently, the integration of unvetted predictive models into INDOPACOM decision-making pipelines risks generating systemic errors that bypass traditional human-in-the-loop safeguards.

Strategic Question for Discussion

If the CNN-reported near-miss involving a Chinese vessel represents a systemic vulnerability in AI-assisted intelligence, how can INDOPACOM validate algorithmic threat assessments without sacrificing the speed advantages of automated systems?

The pattern suggests that validation requires establishing parallel, non-algorithmic verification channels operating independently of the primary sensor feed. My assessment is that INDOPACOM will likely accept slower response times in ambiguous grey-zone scenarios to mitigate the risk of accidental escalation. The available evidence points toward a dual-track system where automated speed is restricted to clear-cut, high-intensity kinetic engagements.

Share your assessment in the comments below.

To read in details click here