For one of my MSc Ethics in Machine Learning assignments, I chose a question that is difficult to answer without drifting into either science fiction or alarmism:

Will superintelligent AI eventually threaten humanity itself?

At first, the question sounds almost impossible to approach seriously. We do not have artificial general intelligence today, let alone systems that exceed human capability across virtually every domain.

But the more I researched the topic, the more I realised that uncertainty is exactly what makes the question important.

My conclusion was not that superintelligent AI will destroy humanity.

It was that such an outcome is not inevitable, but neither is it impossible.

The difference may depend less on how intelligent future systems become and more on what humans decide to do before they reach that point.

Before Superintelligence, We Need to Define What We Mean

Discussions about AI often mix together very different kinds of systems.

Most of the AI we interact with today is still narrow AI.

These systems can perform specific tasks extremely well: recognise faces, recommend videos, generate text, classify images, detect fraud or predict patterns from large datasets.

They can be powerful without possessing anything resembling general human intelligence.

The next concept is Artificial General Intelligence, or AGI.

AGI usually refers to a hypothetical system able to learn, reason and perform across a broad range of intellectual tasks at roughly human level.

Then comes superintelligence.

A superintelligent system would not simply match us.

It would exceed human capability across many or potentially almost all important cognitive domains.

That distinction matters because the ethical problem changes dramatically as capability increases.

A recommendation algorithm making a poor suggestion is inconvenient.

A highly autonomous system with capabilities far beyond its designers making the wrong decision could be something else entirely.

The Problem Is Not an "Evil" AI

One of the most useful ideas I encountered while researching the topic was the alignment problem.

Popular discussions often imagine the dangerous AI as something hostile.

It becomes conscious, hates humanity and decides to attack us.

That is not necessarily the most interesting risk.

A system does not need hatred, anger or ambition to be dangerous.

It simply needs an objective that does not perfectly match what humans actually intended.

Imagine giving an extremely capable system a goal.

If that goal is specified badly, the system may pursue it in ways its designers did not anticipate.

The more capable the system becomes, the greater the consequences of getting that objective wrong.

This creates a strange possibility:

An AI could cause enormous harm while functioning exactly as designed.

The failure would not necessarily be in its intelligence.

The failure could be in ours.

Intelligence and Values Are Different Problems

This led me to another important distinction.

Making a system more intelligent does not automatically make it more ethical.

A model might become better at reasoning, planning, prediction and problem-solving without becoming any better at understanding what humans consider fair, safe or desirable.

That matters because human values themselves are complicated.

People disagree about justice, freedom, privacy, equality, responsibility and acceptable risk.

Even people who agree on broad principles may disagree completely when those principles collide.

If humans cannot always agree on the correct decision, translating "human values" into something a machine can reliably follow becomes a much harder technical and philosophical problem than it first appears.

Alignment is therefore not simply about writing better software.

It also forces us to ask:

Aligned with whom?

Aligned to which values?

Who gets to decide?

The Asymmetry That Changed How I Thought About the Risk

The idea that stayed with me most from the assignment was asymmetry.

The potential benefits of advanced AI are enormous.

More capable systems could help accelerate scientific discovery, improve healthcare, increase productivity, support climate research, optimise infrastructure and solve problems that currently require enormous amounts of human effort.

Those benefits deserve serious attention.

But the risks do not have the same shape.

If an AI product performs badly, we can usually replace it.

If an investment fails, money can be lost and rebuilt.

If a technology disappoints, society can adopt something else.

Many failures are recoverable.

The most extreme risks associated with highly capable AI would not be.

That creates a difficult decision problem.

Even if a catastrophic outcome is considered unlikely, the scale of the possible harm changes how seriously that probability should be treated.

We already use this reasoning elsewhere.

We do not ignore aviation safety because most flights land safely.

We do not ignore nuclear safety because serious accidents are rare.

When consequences are sufficiently large, low probability alone is not a good reason for doing nothing.

This Changed the Question

At that point, my assignment stopped feeling like a question about whether AI itself was inherently dangerous.

The better question became:

Can we create institutions capable of managing technologies that may eventually exceed the ability of individual humans to fully understand or control them?

That is a very different problem.

It moves the discussion away from machines alone and toward governance.

Research laboratories make decisions.

Technology companies make decisions.

Governments make decisions.

Standards bodies make decisions.

Researchers decide what safety techniques to develop.

Companies decide when systems are ready to deploy.

Governments decide what rules apply.

The trajectory of advanced AI is therefore not simply the inevitable result of technological progress.

It is shaped by human choices.

Why One Safeguard Will Never Be Enough

Another conclusion I reached was that there is unlikely to be a single solution to AI safety.

Alignment alone is not enough.

Regulation alone is not enough.

Human supervision alone is not enough.

International agreements alone are not enough.

A robust safety system would probably need several layers working together.

Technical measures could include better alignment techniques, interpretability, testing, monitoring and mechanisms for maintaining meaningful human control.

Organisational measures could include independent audits, responsible deployment practices and clear accountability when things go wrong.

Governments can establish legal and regulatory frameworks.

And because the most powerful AI systems are unlikely to respect national borders, some form of international coordination will also be necessary.

The principle is similar to security engineering.

We generally do not protect something important using one defence.

We use multiple layers because every individual safeguard can fail.

The Governance Problem Is International

AI development is concentrated in a relatively small number of countries and technology companies.

Its effects will not be.

A powerful system developed in one country can influence economies, information systems, employment, security and public services around the world.

That creates a governance challenge.

If decisions about advanced AI are made only by the countries and companies capable of building the systems, everyone else becomes a rule-taker.

That perspective became particularly important to me because I was writing from Mauritius.

The View from a Small Island State

Mauritius is unlikely to be among the countries building the world's first superintelligent systems.

But that does not mean developments in advanced AI would have little effect here.

We will consume AI products.

Our businesses will depend on them.

Our public sector will use them.

Our students will learn with them.

Our workers may compete with systems developed elsewhere.

Our information environment may increasingly be shaped by them.

Yet many of the major decisions about how these systems operate may be made thousands of kilometres away.

That creates an important issue for small states.

The countries with the least influence over the development of advanced AI may still experience significant consequences from it.

For countries like Mauritius, international AI governance is therefore not an abstract diplomatic discussion.

It may be one of the most practical ways of protecting national interests.

Participation matters.

Representation matters.

And small states need enough technical understanding to contribute meaningfully when international rules are being developed.

Developing Countries Should Not Arrive Late

There is a risk that AI governance follows a familiar pattern.

Advanced economies develop the technology.

They establish standards.

They create regulatory models.

And smaller countries adopt those frameworks later.

That might be convenient, but it also means those countries have little influence over the assumptions built into the rules.

Different countries have different economic structures, cultures, institutions and development priorities.

A governance model designed primarily for major technology-producing economies may not always reflect the needs of smaller developing states.

That is why AI literacy at government level matters.

A country does not need to build frontier AI models to understand their implications.

It does need people capable of participating in the conversations where the rules are written.

What About the Benefits?

None of this means advanced AI should simply be stopped.

That would ignore the extraordinary possibilities the technology offers.

AI may help solve problems we currently consider extremely difficult.

The challenge is avoiding two extremes.

One extreme says:

AI will solve everything, so regulation will only slow progress.

The other says:

AI is too dangerous, so progress itself should stop.

I do not find either position particularly convincing.

The more practical approach is to recognise both sides.

Powerful AI could produce enormous benefits.

Powerful AI could also create serious risks.

Responsible development means attempting to capture the first without pretending the second does not exist.

The Future Is Not Predetermined

By the end of the assignment, the word I kept returning to was choice.

Superintelligence does not automatically imply extinction.

Nor does technological progress automatically guarantee a positive future.

There will be decisions along the way.

What systems are built.

How quickly they are deployed.

How thoroughly they are tested.

What level of autonomy they receive.

Who controls them.

What incentives guide the organisations developing them.

What international rules exist.

What happens when commercial competition conflicts with safety.

Those choices matter.

The future of AI will not simply happen to us.

Human beings are actively constructing the institutions, technologies and incentives that will shape it.

What I Took Away from the Assignment

Before working on this topic, I thought the debate about superintelligent AI was mainly about technology.

Afterwards, I saw it much more as a question about governance, responsibility and foresight.

The technical problem is obviously important.

We need better alignment.

We need better interpretability.

We need ways to understand increasingly complex systems.

But technical solutions operate inside human institutions.

And those institutions ultimately determine which systems are developed, what risks are accepted, and whose interests are represented.

That leads to the main conclusion I took from the assignment:

The biggest question may not be whether superintelligent AI becomes powerful enough to threaten humanity.

It may be whether humanity becomes wise enough to govern something that powerful before it arrives.

For countries large and small, that makes participation in AI governance more than a policy issue.

It is part of deciding what kind of technological future we are willing to build.