If AI Has a 10% Chance of Disaster, What's the Risk of Not Building It?

This week, former Anthropic researcher Jacob Coxon attracted considerable attention after resigning from the company and publicly warning about the potential dangers of increasingly capable AI. Coxon argued that people working at the frontier of AI development genuinely believe systems being built today could eventually threaten humanity itself.

That deserves to be taken seriously.

However, one detail in the subsequent coverage is worth getting right. Coxon himself did not put the probability of human extinction at 10%. The widely reported estimate of greater than 10% came from Anthropic alignment researcher Evan Hubinger in the discussion that followed Coxon's resignation.

That distinction matters, but there is a much bigger question behind it.

What does a 10% chance of AI causing human extinction actually mean?

It isn't a measured probability in the conventional scientific sense. We don't have a historical dataset containing 1,000 civilisations that developed superintelligent AI and 100 that subsequently became extinct.

It is an informed judgement about an event that has never happened.

That doesn't make it meaningless. If somebody with considerable expertise genuinely believes a technology carries even a 1% chance of causing human extinction, dismissing them because the probability cannot be experimentally demonstrated would be reckless.

But there is another side of the calculation that receives considerably less attention.

What is the risk of not developing AI?

The alternative to AI isn't a risk-free world

A lot of discussion about AI implicitly compares two futures.

In the first, humanity develops increasingly powerful artificial intelligence and exposes itself to new risks.

In the second, we don't.

The second future can easily sound like the safe option because it removes the hypothetical AI catastrophe from the equation.

Unfortunately, it doesn't remove everything else.

People still die from cancer.

People still develop Alzheimer's disease.

Antibiotic resistance continues to threaten modern medicine.

Tuberculosis still kills people who could potentially have been diagnosed earlier.

Millions of children receive poor-quality education.

Scientific discoveries still take years or decades.

Natural disasters still kill people.

Climate change still presents enormous technical challenges.

People with disabilities still encounter barriers that technology could potentially reduce.

Poverty, food insecurity and unequal access to expertise remain enormous global problems.

The world without increasingly capable AI is not today's world preserved perfectly in amber. It is a different future with its own risks, missed discoveries and opportunity costs.

That doesn't automatically mean developing AI is worth the risk.

It means both sides of the equation contain risk.

The 10% figure is a judgement, not an observed extinction rate

When an AI researcher assigns a probability to human extinction caused by advanced AI, they are attempting to reason under extreme uncertainty.

There is nothing inherently wrong with doing that. Governments, businesses and individuals make decisions under uncertainty constantly.

The problem comes when a subjective probability begins circulating publicly without its uncertainty travelling with it.

“AI researcher believes there is greater than a 10% chance of extinction” can gradually become “AI has a 10% chance of wiping us out”, which sounds considerably more scientifically established than it actually is.

We simply don't know.

The probability could be substantially higher.

It could be substantially lower.

The catastrophic scenario might depend on assumptions about capabilities that never materialise, or safety research might solve problems that currently appear extremely difficult. Equally, capabilities might develop more quickly than expected and expose risks we haven't anticipated.

Pretending certainty in either direction would be foolish.

My issue isn't with researchers attempting to estimate the risk. It is that we rarely apply the same framework to the benefits we might be giving up.

If we're prepared to discuss the probability of AI ending human life, we should at least acknowledge the possibility that AI could save an extraordinary amount of it.

We don't have to imagine all of the benefits because some are already happening

The strongest argument for AI development isn't that one day we'll have robot servants or that everybody will work three days a week.

It's science.

AlphaFold is probably the clearest example.

For decades, predicting how proteins fold into their three-dimensional structures was an extraordinarily difficult scientific problem. Understanding those structures matters because the shape of a protein is closely connected to what it does, making protein structures enormously useful across biology and medicine.

Google DeepMind's AlphaFold used artificial intelligence to make a major breakthrough in protein-structure prediction.

The AlphaFold database now contains predictions for more than 200 million protein structures, covering almost every catalogued protein known to science. According to DeepMind, more than three million researchers across over 190 countries have used AlphaFold, including more than one million users in low and middle-income countries.

More than 30% of research citing AlphaFold relates to the study of disease.

Researchers are using it in areas including antimicrobial resistance, heart disease, malaria, crop resilience and drug discovery.

This isn't a prediction about what AI might achieve in 2045.

It has already happened.

AI can make expertise available where experts are scarce

The potential becomes particularly interesting when we stop looking exclusively at wealthy countries.

In Britain, we tend to think about AI in healthcare as something that might make an NHS department more efficient.

In parts of the world where there simply aren't enough specialists, the calculation can be much more fundamental.

Tuberculosis provides a useful example.

The World Health Organization has recommended computer-aided detection software for analysing chest X-rays during TB screening since 2021.

Pakistan was one of the early countries to deploy this technology at scale through mobile X-ray units. Between 2017 and 2021, more than 1.2 million people were screened across more than 11,000 mobile camps, resulting in the detection of more than 7,600 cases of tuberculosis.

In 2025, the WHO announced that six computer-aided detection products had met its performance standards for TB screening in people aged 15 and over.

AI isn't replacing some hypothetical perfect healthcare system in these situations.

It can help extend diagnostic capability into places where sufficient human expertise may not otherwise be available.

That distinction matters enormously.

Cancer screening offers another glimpse of what is possible

AI-assisted medical imaging is also producing increasingly interesting evidence in cancer screening.

A 2026 prospective clinical trial published in Nature Medicine involved more than 31,000 women undergoing routine mammography. Researchers compared standard double reading with a partially autonomous AI-supported approach.

The AI-supported strategy reduced radiologists' reading workload by 63.6% while producing a 15.2% higher cancer detection rate, although the recall rate was also higher, which is an important qualification.

That doesn't mean we should remove radiologists from cancer screening tomorrow.

It demonstrates something more interesting: AI may allow scarce medical expertise to stretch further while potentially improving aspects of detection.

In a country struggling with shortages of specialists, reducing the amount of routine work required from each clinician could matter just as much as improving raw diagnostic accuracy.

The future of AI in medicine may therefore be less about replacing doctors and more about dramatically increasing what each doctor can do.

Education could be transformed for a completely different reason

Excellent one-to-one education has always existed.

The problem is that it is expensive.

A wealthy family can employ private tutors, provide individual feedback and adapt education around the needs of one child. A struggling school with 30 children in a classroom cannot reproduce that level of individual attention for everyone.

AI potentially changes the economics of personalised tuition.

A 2025 randomised controlled trial conducted by Harvard researchers compared undergraduate physics students using a carefully designed AI tutor with students receiving conventional active-learning classroom instruction.

The results deserve caution because this was one study, involving one course and a deliberately engineered tutoring system.

They are nevertheless striking.

Students using the AI tutor achieved more than double the median learning gains while spending less time on the material. The median time spent using the AI tutor was 49 minutes compared with 60 minutes of classroom learning time.

That doesn't prove AI tutors are better than teachers.

It certainly doesn't prove children should be educated by chatbots.

What it demonstrates is that properly designed AI-assisted personalised education has genuine potential.

Now consider what happens if that capability becomes cheap enough to give every child access to something resembling individual tuition.

The significance isn't that wealthy families gain another educational tool.

It's that individualised support could become available to people who could never previously afford it.

Weather forecasting is already changing too

Weather prediction is another area where AI has moved beyond an interesting research experiment.

In February 2025, the European Centre for Medium-Range Weather Forecasts made its Artificial Intelligence Forecasting System operational alongside its traditional physics-based forecasting system.

ECMWF reported that its AI model outperformed state-of-the-art physics-based models on a number of measures, including tropical cyclone tracks, with improvements of up to 20%. It also estimated that generating forecasts required roughly 1,000 times less energy.

An ensemble version became operational later in 2025 and can generate forecasts more than ten times faster than the traditional physics-based system while also using dramatically less energy.

The existing physics-based systems remain essential, so this isn't a simple story of AI replacing conventional meteorology.

It is evidence that machine learning can become another extremely powerful tool for understanding the physical world.

Better forecasting has consequences far beyond whether somebody takes an umbrella to work.

Improved predictions of hurricanes, floods, heatwaves and other extreme weather can help governments prepare, allow people to evacuate and give emergency services more time to respond.

Those benefits are difficult to put into one headline number, but they are very real.

The biggest potential benefit may be accelerating science itself

Individual examples such as protein folding or weather forecasting are impressive, but I think they may eventually look relatively small compared with the broader possibility.

Scientific progress is constrained partly by the amount of human intelligence and human time available to work on problems.

There are only so many researchers.

They can only read so many papers.

Run so many experiments.

Analyse so much data.

Consider so many hypotheses.

What happens if every scientist effectively gains hundreds or thousands of extremely capable research assistants?

What happens when an AI can read almost every relevant scientific paper, identify connections between fields, propose experiments, analyse results and help design the next experiment?

Even relatively modest improvements in the productivity of scientists could compound enormously over decades.

Medicine is not the only beneficiary.

More capable AI could accelerate research into batteries, energy storage, nuclear fusion, new materials, carbon capture, agriculture, water purification and countless engineering problems.

This is where the upside becomes difficult to quantify for exactly the same reason that extinction risk is difficult to quantify.

We don't know what hasn't been discovered yet.

Intelligence is an input into almost every problem we care about

This is what makes AI fundamentally different from many previous technologies.

A better engine helps us with transport.

A better solar panel helps us produce energy.

A better antibiotic treats particular infections.

Intelligence helps us develop better engines, solar panels and antibiotics.

Human intelligence is involved in solving almost every technical problem civilisation faces.

If artificial intelligence eventually becomes capable of significantly increasing the effective amount of problem-solving capacity available to humanity, the consequences could be enormous.

This is the strongest version of the positive case for AI.

Not better social media posts.

Not novelty images.

Not asking a chatbot to write an email.

The possibility is that intelligence itself becomes substantially cheaper and more abundant.

If that happens safely, many problems currently constrained by shortages of expertise could become much easier to attack.

There are smaller benefits that matter too

It is easy to become so focused on civilisation-scale arguments that ordinary improvements to people's lives get dismissed as trivial.

They aren't.

AI can already translate information between languages.

It can describe images and visual environments for blind and visually impaired people.

It can transcribe and restructure information for people who struggle with conventional written communication.

It can help somebody understand a complicated document.

It can give a sole trader access to capabilities resembling a copywriter, analyst, researcher, programmer or administrative assistant.

It can help somebody with very little money create a respectable website or understand a spreadsheet.

It can automate repetitive work that people simply don't enjoy doing.

None of these individually justifies taking an existential risk.

Collectively, however, they demonstrate why describing AI solely as a threat misses half of what is happening.

The benefits won't necessarily be distributed fairly

There is an important objection to all of this.

A technology being capable of creating enormous wealth doesn't mean that wealth will be shared equally.

AI could increase economic inequality.

The owners of models, computing infrastructure and intellectual property could capture a disproportionate share of the gains.

Workers could lose bargaining power.

Developing countries could become dependent on technologies controlled elsewhere.

Education could improve while certain professions disappear.

Healthcare systems could become more capable while companies extract enormous profits from them.

A positive technological capability doesn't automatically produce a positive social outcome.

History provides plenty of examples of technologies that increased overall wealth while distributing the benefits extremely unevenly.

So the question cannot simply be whether AI makes society richer.

It also has to include who benefits.

AI creates serious risks that have nothing to do with extinction

There is another danger in concentrating exclusively on hypothetical human extinction.

We can overlook harms that are already much easier to imagine.

AI can make fraud more scalable.

It can produce convincing misinformation.

It can assist cyberattacks.

It can be used for surveillance.

It may make certain biological threats easier to develop.

It can disrupt labour markets.

It can concentrate power in a small number of companies or governments.

It can make people excessively dependent on systems they don't understand.

Some of these risks may be far more likely than extinction even if their consequences are less absolute.

Taking the benefits of AI seriously shouldn't require minimising any of them.

A small probability of extinction is still an enormous problem

This is where I agree with the people sounding the alarm.

Suppose the true probability of advanced AI causing human extinction were genuinely 10%.

That would be an appalling level of risk.

Even 1% would be extraordinarily serious when the potential consequence is the permanent end of humanity.

You cannot dismiss that by pointing towards better cancer screening.

The stakes are too large.

Where I differ from some versions of the argument is the assumption that the solution therefore follows automatically.

If AI also has the potential to prevent enormous amounts of disease, accelerate science, increase global prosperity and help humanity solve problems that themselves kill millions of people, then simply saying "don't build it" carries consequences too.

The real objective should be to reduce the catastrophic risk while preserving as much of the upside as possible.

So what probability would I give AI making humanity better off?

There is no scientifically defensible percentage.

Any number I give here is ultimately a judgement.

With that qualification made very clearly, if humanity continues developing increasingly capable AI while simultaneously taking safety, governance and control seriously, I would personally put the probability of AI leaving humanity substantially better off somewhere around 70% to 85%.

I would assign perhaps 10% to 20% to a more disappointing or mixed future in which AI causes substantial economic and social disruption without delivering the transformative scientific benefits its advocates expect.

For genuinely catastrophic outcomes, I would put the probability somewhere from a few per cent into the low double digits, depending enormously on assumptions about how capable AI becomes, how quickly those capabilities arrive and whether safety techniques keep pace.

I have very low confidence in those exact numbers.

That's important.

If somebody tells you the probability is 9.7%, 17.4% or 82.1%, the decimal places are doing a lot more work than the evidence warrants.

We are attempting to predict the consequences of a technology whose future capabilities we don't yet know.

The numbers are useful for expressing beliefs and comparing risks, but they shouldn't be mistaken for measurements.

There are really two uncertain futures

The AI debate is often framed as though we are choosing between risk and safety.

I don't think we are.

We are choosing between two uncertain futures.

In one, we continue developing increasingly capable AI. We potentially gain extraordinary tools for medicine, science, education, engineering and economic growth, while introducing new risks ranging from unemployment and inequality to the possibility of catastrophic loss of control.

In the other, we deliberately limit or stop development. We reduce some of those new risks but give up some unknown quantity of scientific discovery, medical progress, productivity and problem-solving capability.

Neither future comes with a guarantee.

The mistake is pretending only one side has a cost.

Not developing a cure has consequences too

Opportunity cost becomes uncomfortable when applied to human life because the people affected are invisible.

If a technology causes a disaster, we can count the victims.

If a technology isn't developed and therefore a treatment arrives twenty years later than it otherwise would have, we will never know the names of the people who might have survived.

There is no memorial for discoveries that weren't made.

No statistic records the people who might have been diagnosed earlier if a technology had existed.

We can't know which future scientific discoveries AI will accelerate, which makes assigning a body count to slowing development impossible.

But impossible to count does not mean zero.

World B has a body count too.

Safety and progress don't have to be opposing positions

The debate becomes unhelpful when everyone is forced into one of two camps.

Either AI is an existential threat and development must stop, or AI is an extraordinary opportunity and anyone discussing catastrophic risk is standing in the way of progress.

Neither position seems particularly convincing to me.

Coxon's concerns deserve serious attention precisely because people working close to these systems understand capabilities and failure modes that most of us don't.

Hubinger believing there is a greater than 10% chance of an existential catastrophe should make us pay attention, even if we disagree with his estimate.

At the same time, AlphaFold exists.

AI-assisted TB screening exists.

AI-supported cancer screening exists.

AI tutoring exists.

Operational AI weather forecasting exists.

The upside isn't entirely hypothetical either.

The sensible objective is therefore neither blind acceleration nor blind prohibition.

It is to make the upside as large and widely shared as possible while relentlessly trying to drive the downside towards zero.

That means safety research.

Independent testing.

Sensible regulation.

Security.

International cooperation.

Transparency where possible.

Human oversight where necessary.

And a willingness to slow particular capabilities if the evidence suggests we're losing control of the risks.

None of that requires us to pretend humanity has nothing worthwhile to gain.

Perhaps we're asking the wrong question

I don't know whether advanced AI has a 10% chance of destroying humanity.

Neither does anyone else.

I also don't know how many lives substantially more capable AI could save through discoveries that haven't happened yet.

Nobody knows that either.

That uncertainty should make us humble about both sides of the argument.

What we already know is that artificial intelligence is producing genuine scientific and practical benefits, while increasingly capable systems introduce genuine risks.

We shouldn't minimise either because doing so makes the argument easier.

The question isn't whether AI is dangerous.

Powerful technologies generally are.

Nor is it simply whether AI will benefit humanity.

It already has in some areas.

The much harder question is whether we can capture the extraordinary potential of increasingly capable intelligence while keeping the risks within acceptable limits.

Perhaps we can't.

That possibility deserves to be taken extremely seriously.

But before deciding that not developing the technology is therefore the safer future, we should remember what we're comparing it with.

Humanity isn't currently living in a world without enormous problems.

If AI can help us solve some of them, choosing not to develop it is a decision with consequences too.


Sources and further reading

AI extinction risk and Jacob Coxon

• The Guardian: “AI could kill all humans in next decade, warn experts: but how seriously should we take them?” Covers Jacob Coxon's resignation and Evan Hubinger's greater-than-10% assessment. Read The Guardian article
• Axios: Reporting on Coxon's resignation from Anthropic and his concerns about the direction of advanced AI development. Read the Axios article

AlphaFold and medical research

• Google DeepMind: AlphaFold overview, including its database of more than 200 million predicted protein structures and use by millions of researchers. AlphaFold at Google DeepMind
• Google DeepMind: Further information on AlphaFold's scientific impact and use in disease research around the world. AlphaFold's impact

AI-assisted tuberculosis screening

• World Health Organization: WHO approval of six computer-aided detection software products for analysing chest X-rays for TB screening. Read the WHO announcement
• World Health Organization: Case study covering Pakistan's use of AI-assisted chest X-rays and mobile screening for tuberculosis. Read the WHO case study

AI and breast cancer screening

• Nature Medicine: Prospective clinical trial examining AI-supported mammography and digital breast tomosynthesis screening. Read the Nature Medicine study

AI tutoring and education

• Scientific Reports / Nature: Randomised controlled trial comparing a purpose-built AI tutor with in-class active learning for undergraduate physics students. Read the study

AI weather forecasting

• ECMWF: Announcement of its Artificial Intelligence Forecasting System becoming operational in February 2025. ECMWF's AI forecasts become operational
• ECMWF: Information on its ensemble AI forecasting system becoming operational later in 2025. ECMWF's ensemble AI forecasts