AGI, Quasi-Religion and the Very Human Need to Believe

Let me put it this way...

Imagine we’re sitting in a pub, two drinks in, and someone at the next table announces that artificial general intelligence is going to solve climate change, cure cancer, eliminate poverty and usher in an age of abundance.

Five minutes later, someone else says it will destroy humanity, enslave us all and turn the planet into a server farm.

Different prophecies. Same religion.

That's the odd thing about the conversation surrounding AGI. It's started to sound less like a technology debate and more like theology.

There are believers, heretics, evangelists, doomsday prophets and competing denominations. There are sacred texts in the form of research papers, commandments disguised as AI principles and a growing class of high priests who claim to understand what the rest of us cannot.

There is even the promise of salvation.

Build a machine more intelligent than humanity and, depending on which congregation you belong to, it will either save us from ourselves or punish us for our arrogance.

It's a quasi-religion: not quite a faith, not quite a science, but an intoxicating mixture of both.

We have always built gods in our own image

Human beings are not particularly comfortable with uncertainty.

When something is too large, complicated or frightening to understand, we create a story around it. Stories give chaos a shape. They provide heroes, villains, endings and, most importantly, meaning.

AGI is perfect material.

Nobody can say with certainty when it will arrive, what it will look like, how it'll behave or whether the term itself describes a single meaningful threshold. That leaves an enormous blank space, and human beings are exceptionally good at filling blank spaces with belief.

Some people imagine AGI as a benevolent, all-knowing intelligence that'll answer every question and fix every broken system.

Others see it as a digital Antichrist: an intelligence we create but cannot control.

Both views give AGI characteristics we once reserved for gods. Omniscience. Power beyond human comprehension. The ability to judge, reward, punish or transform civilisation.

The machine hasn't even arrived, yet we're already arguing about its commandments.

The rapture, but with better processing power

One of the most religious ideas in the AGI conversation is the concept of an intelligence explosion.

The theory goes that once an AI becomes capable of improving itself, it may rapidly become more intelligent, then more intelligent again, until it leaves humanity behind.

This moment is often described as the singularity.

Strip away the technical language and it has the structure of a rapture narrative. There is a coming event that will divide history into before and after. The faithful will be transformed. The unbelievers may be left behind. Death, scarcity and perhaps even human limitation itself could be overcome.

For some, this is not merely technological progress. It is transcendence.

Silicon Valley has taken some very old human desires - immortality, abundance, perfect knowledge and freedom from suffering - and given them a modern interface.

The robes have been replaced by hoodies, but the sermon is surprisingly familiar.

Then there are the prophets of doom

Every religion needs an apocalypse, and AGI has several.

A superintelligence might decide that humans are an obstacle. It might pursue an apparently harmless objective so relentlessly that it destroys everything around it. It might manipulate governments, financial markets, infrastructure or public opinion before we even understand what it is doing.

These are legitimate risks to examine. Powerful systems should absolutely be subjected to serious governance, testing and scrutiny.

But the tone sometimes moves beyond risk analysis into prophecy.

You start hearing that disaster is inevitable, that only a chosen group truly understands the danger, or that anyone questioning the narrative is dangerously naïve.

At that point, we're no longer discussing engineering alone. We're discussing identity, morality and belonging.

People are not simply expressing a view about AI. They are joining a camp.

The inconvenient reality: most businesses are nowhere near AGI

Here is the less dramatic truth.

While everyone argues about whether a future superintelligence will save or destroy civilisation, most organisations are still struggling to work out which version of a policy document is correct.

• Their SharePoint is full of duplicates.

• Their knowledge base contains conflicting instructions.

• Their customer-service agents cannot find the right answer.

• Their chatbot is confidently quoting guidance that expired 18 months ago.

We are having a philosophical debate about machine consciousness while businesses are feeding today’s AI systems broken links, obsolete procedures, unknown abbreviations and contradictory content.

It is a bit like discussing the moral implications of interstellar travel while your car has three flat tyres.

The immediate problem is not that machines know too much.

It's that organisations frequently don't know what they know.

AI doesn't turn bad knowledge into truth

This is where the religious metaphor becomes dangerous.

People can begin to treat AI as an oracle. They ask it a question, receive a polished answer and assume that fluency is evidence of truth.

But an AI system is not magically purified by intelligence. Its answers remain dependent on the information, context and controls around it.

• Give it duplicated knowledge and it may reproduce the duplication.

• Give it contradictory policies and it may select the wrong one.

• Give it obsolete content and it may deliver an obsolete answer with breathtaking confidence.

The more persuasive the technology becomes, the more dangerous poor knowledge becomes.

That is why the practical conversation about AI should begin long before AGI. It should begin with a much more mundane question:

Can we trust the knowledge our current systems are using?

This is where index fits in

index exists to help organisations answer that question.

We don't ask clients to believe that AI will somehow repair their information estate by itself. We help them examine the knowledge they already have, identify where it's failing and improve it before those failures spread through search tools, chatbots, copilots and automated customer journeys.

• index Scan examines content across systems such as SharePoint, ServiceNow, Confluence, document libraries and knowledge bases. It identifies problems including contradictions, duplicate material, outdated content, broken links, unclear ownership and information that is difficult for machines to interpret.

• index Solve then supports the remediation process: helping teams correct, consolidate, retire or improve problematic content while preserving governance and human oversight.

• index Sustain provides the evidence. It shows whether knowledge quality is improving, where risk remains and how that improvement can affect operational outcomes such as handling time, first-contact resolution, compliance, self-service and AI readiness.

In other words, we are not promising digital salvation.

We are helping clients prevent their existing AI investments from becoming expensive acts of faith.

From belief to evidence

The most important distinction in the next phase of AI will not be between believers and sceptics.

It will be between organisations that treat AI as magic and organisations that treat it as infrastructure.

Magic asks us to believe.

Infrastructure has to be tested, monitored, maintained and governed.

A business should be able to see where an AI-generated answer came from, whether the underlying source is current, whether conflicting guidance exists and who's responsible for fixing it.

It should be able to measure whether introducing AI has actually improved customer outcomes rather than merely creating a more impressive interface.

That is the unglamorous work behind trustworthy AI.

No prophecies. No digital messiahs. Just evidence, accountability and better knowledge.

Perhaps AGI tells us more about ourselves than machines

AGI may eventually become one of the most important developments in human history.

Or it may arrive gradually, unevenly and with far less theatrical drama than either its disciples or its critics expect.

But the mythology surrounding it already tells us something important.

We want technology to rescue us from difficult decisions. We want intelligence without uncertainty, progress without trade-offs and answers without the tedious work of maintaining the information behind them.

Unfortunately, even the cleverest machine cannot compensate indefinitely for organisational confusion.

Before we ask whether an artificial intelligence can understand the whole of human knowledge, we might first want to check whether our customer-service team can locate the correct returns policy.

That may not sound as exciting as building a new god.

But it is probably where the real work begins.

by Paul Tucker - contact@index-ai.net