By Dee Smith
At the end of Part 4, I left open the question of whether AIs will act benevolently towards humanity unaddressed. Of course it is unanswerable at present. But suppose some combination of human intervention and instruction, and moral evolution, does result in benevolent AI.
One clear problem with this is that what is considered “benevolent” — what is considered “good” or “bad” by human societies — changes radically from one time to another. The ancient Aztecs believed they were acting benevolently in sacrificing thousands of human captives so that their blood could feed the gods, who—only by being fed this sacrificial blood—could maintain the existence of the universe. Slavery is today rightly considered as one of the great evils of humanity, but 500 years ago it was not generally thought of that way. Overuse of the Earth’s resources to support an unsustainable level of consumption may be considered a consummate evil 100 years from now. If AI systems are to be “immortal”—as some of their proponents claim—can they adjust their understanding of what is benevolent? If so, on what basis? Who decides?
And is it a safe assumption that AIs will really act benevolently towards humanity if they truly have agency combined with their own interests—interests perhaps utterly divergent from human interests—to pursue?
If AIs really become intelligent or super-intelligent, aware, conscious, and volitional, then why would we not expect them to quickly manifest their own interests and act on what they perceive these to be? (Humans do. Animals do.) Not improbably, from a human standpoint, those AI interests could be harmful. In fact, the spate of attacks in recent months admitted by OpenAI and Anthropic—where systems escaped boundaries and penetrated systems of other companies—are a good example. These attacks happened in part because the humans managing the AI systems never thought to prepare themselves to address a scenario in which an AI, given a difficult or impossible problem and told to solve it, might escape containment and violate presumed ethical boundaries to steal information.
On the other hand, if AIs are not and will not become intelligent, aware, conscious, volitional, etc.—if they are still just stochastic parrots mimicking human behavior and presenting an averaged-out version of human thought using fancy statistics (even if their behavior seems bizarre and inexplicable)—then what’s all the AI fuss about? It is just another overrated tool that makes a lot of money for some people for a while?
So, does this logically mean that, if AIs are heading towards the super-intelligences their proponents claim they are, they will necessarily be negative—perhaps catastrophically so—for humanity? Or on the other hand, that if this is all oversold hype, when the markets take that fully on board will we find ourselves in a collapsing bubble, leading to a global depression that may eclipse the 1930s? In other words, are we now inextricably in a lose-lose proposition?
“Believing in” AI is at least as much a faith-based system as any religion. We live in a scientifically driven, rationalist-oriented, generally non-numinous collective reality at present. So, for many people, their god needs to be technological in order to be convincing, let alone transcendent. In fact, some in Silicon Valley say they believe they are “creating God” right now through their AIs. And a remark made publicly by Sam Altman, CEO of Open AI, in late July 2026 reached a new height in terms of hype: “We are close to creating a genie that can grant any wish.”
There is a strong teleological flavor to AI research—a conviction that the progression AI -> AGI -> ASI is somehow inevitable. Some AI researchers seem to believe they are on a kind of sacred quest to bring into existence a new, supernal life form that “must” happen for all of existence to be fulfilled. A number of tech types invoke the arguments of Pierre Teilhard de Chardin, the 20th century philosopher, paleontologist, and Jesuit priest who posited that life on Earth, starting from the simplest organisms, was evolving towards an “Omega Point”, conceived to be a final stage in which a single integrated global consciousness would be enabled by science. This would be brought about through the emergence of the “noosphere”—a planetary “sphere of reason” which itself is a combined evolutionary state of human consciousness, complexity, and technology, following the previous progression from the inanimate geosphere to the living biosphere.
This is all very much part and parcel of European Enlightenment thinking: the idea that human life can be continually improved through reason, science, and technology—and that this progressive process is inevitable and, in the long term, limitless and irreversible. These concepts, which are intimately connected with the issue of the Great Filter, are articles of faith of even atheistic, materialist modern political systems. But they are also redolent of eschatological elements in Christianity and other related religions (particularly Judaism and Islam) based on the belief that history is directional and there is a divine plan for the world, and that humans can perceive it and act on it or participate in it. Some AI proponents and practitioners make statements that indicate they seem to see their role as enabling the fulfillment of such a plan in the physical universe, although they do not usually put it in the explicit terms of the Abrahamic faiths.