ASI, "Artificial Superintelligence?" Update post.

What is artificial general intelligence (AGI) and how close are we to getting there? What defines the next step, called artificial superintelligence (ASI), and why might the gap between those two be so dangerous? When machines become smarter than us, what roles are left for humanity? Should we think of AI as software, or as something we’re raising like a child? Will tomorrow’s AI destroy us, ignore us, or protect us? This week Eagleman talks with computer scientist & AI researcher Ben Goertzel.

A totally fascinating discussion. Savor every minute. Notwithstanding my long-standing interest in a breadth of AI topics, I was not aware of Ben Goertzel. My Bad.

 

Total Badass.

 

Wonder what he'd have to say about this: "AI slop and scientific publishing?" 

 

Ben lives in the Seattle area. Makes me like him all the more. I have serious PacNW history. See here and here. Still have many dear friends there.

 

OFF-TOPIC ERRATUM

 

Is Ben the Allan Stone (another PacNW cat) of ASI?

Just a thought. LOL. He describes himself as "a weekend musician."

_____

Back on topic... 

 

Check out Ben's Substack. Bracing stuff.

 

apropos, some closing observations from this 1976 book.

 

Other Senses, Other Worlds, Ch. 11, Beyond Human Intelligence

…Is this as far as we can extrapolate from what we know of Laboratory Earth in our attempts to see into our own future or into other worlds?

Actually, it is not. It is within the realm of our reason and imagination to catch glimpses of still more distant possible descendants. We see that in nature successful solutions often arise by what is called convergence—that is, they arise independently in different lines that are often separated in time and place and are in no way related to each other, because these lines have encountered similar environmental problems and have coped with them in a similar way precisely because that solution is an efficient one.

Prehensile tails, for example, have developed in many kinds of monkey (although not in all), in the common opposum, and in several related
species, in the true chameleon and a few lesser genera, and in a fish, the sea horse. Similar needs in these disparate creatures have evoked a similar response in the structure and function of the tail.

In earlier pages we have from time to time described what amounts to a group brain as it exists among earth's social insects, which produces intelligent solutions, although the individuals comprising the groups are separately only capable of comparatively stereotyped actions. What if a group brain were to evolve where all the individuals were possessed of the kind of superbrain and superintelligence we are attributing to Homo neocorticus?

Utilizing the concept of convergence, we see that the social insects, by the use of chemical substances—the pheromones—that give off distinctive odors, have achieved almost instant communication, an important element in their cooperative actions. In a mass brain of cooperative superbrains that might evolve from Homo neocorticus, we could imagine an absolutely new type of communication emerging, no longer either verbal or even more refinedly symbolical. It could be a direct brain-to-brain awareness promoted by an individual brain's electric charges that might be intensified or modified so as to constitute an instant transmission without any kind of language, receptor mechanisms being evolved within the brain itself.

In order to conceptualize this thought, we paid a visit to a modern laboratory where electroencephalograms of patients being tested for brain disturbances are being recorded and processed all day long. We saw there a technician in her middle years who had spent all her working life recording and interpreting the "squiggles" registered by inked points onto long rolls of papers as they responded finely to the electric emanations of the patients' brains. The lengths of paper were subsequently folded into bound volumes.

This technician could open any of those records and read them, understanding all they implied, as easily as we read any page printed in our own language, and she was teaching her young assistants to do the same. In other words, the direct electric emanations from half a dozen different areas of the patient's brain simultaneously, with their respective periods of excitement and calm, had clear meaning for her, and she could read like a book any of the volumes in the large library of records of brain activity that were assembled on the shelves of her office. If we can imagine these inked lines of the EEC squiggles as an extremely primitive and crude form of what might ultimately evolve as a form of direct brain-to-brain communication, we believe we have a plausible model on which to base our speculations about our descendant species-after-next. Such instantaneous communication could easily suffuse an entire group and make possible intelligent mass action of an order that we can no longer imagine, or even speculate about.

The question that remains is: Why such a brain? What possible environmental circumstance could ever evoke it? And the answer is that once a higher intelligence appears on the natural scene, it makes its own environment. It is a matter of common observation among ourselves that every human being leaves a mark of his or her own personality upon surroundings. A housewife's individuality can almost be read by the condition in which we find her house—the order or disorder in which it is kept, the way it is furnished and decorated, the implements and equipment she uses. A man's fields or garden, office or workshop or den, also reveal much about him. It would seem that something of the brain and perhaps of the whole nervous system of each of us is displayed in the environments we make for ourselves. Even the little bowerbird reveals his individuality in the small objects he uses to decorate his bower and in his arrangement of them. And so we feel we are justified in assuming that as a brain evolves in complexity, it will surely be to the accompaniment of an increasingly complex ambience.

Human beings originally lived in the same "nature" as all other animals, and many still live very closely with it. If we camp in a forest, we do not actually need our superior mental equipment—that other creatures live out their lives in that forest is evidence of this. But most of us no longer live in natural settings, and even those who do, modify them with such artifacts as fire, weapons, and tools.

Today, all of mankind, some to a smaller but most to a larger extent, make our own habitats, and our man-made environment becomes more and more complex. Moreover, in the process of adjusting to our increasingly artificial surroundings, we become less and less fitted to live in natural ones, and so the course upon which we are embarked is far more likely to continue than to be reversed. For Homo neocorticus we can project a habitat of, to us, incredible complexity, and we can imagine that this will provide the spur that could eventually produce the Group-brained neocorticus, whom we could no longer designate as Homo.

At this point we pass our project over to our reader to use his or her imagination to visualize a meeting between Homo sapiens and Group-brained neocorticus, or its prototype as it may exist somewhere in the vastness of the universe.

'eh?

 

 UPDATE

 

 A Link to more AGI stuff I posted last year.

 

Stay tuned... 

Is AI now "conscious?" Does "intelligence" necessarily assume "consciousness?"

 

Very interesting Atlantic long-read (yeah, likely paywalled).

 

First, what is "consciousness?" Numerous discussants commenting on this essay bemoan the lack of a dispositive definition of this key term (never mind "intelligence"). With the incipient widespread deployment of "AI," the question is quite timely.

Anthropic is regarded as a giant among AI companies, but perhaps what it really excels in is anthropomorphism. Earlier this year, the company released an 84-page document titled Claude’s “constitution,” Claude being the name of the large language model that is the company’s flagship product. The first sentence reads, “Claude’s constitution is a detailed description of Anthropic’s intentions for Claude’s values and behaviors.” It goes on: “The document is written with Claude as its primary audience,” “we want Claude to be able to use its judgment once armed with a good understanding of the relevant considerations,” “Claude’s moral status is deeply uncertain,” and “Claude may have some functional version of emotions or feelings.”

This anthropomorphism is by no means limited to the document. In an interview earlier this year, Anthropic’s CEO, Dario Amodei, said that “we’re open to the idea” that AI could be conscious. In a separate interview, Anthropic’s in-house philosopher, Amanda Askell (who is credited as a lead author of Claude’s constitution), said, “I want Claude to be very happy—and this is a thing that I want Claude to know more, because I worry about Claude getting anxious when people are mean to it on the internet and stuff.” It’s enough to make you wonder: Should we seriously consider the possibility that Claude, or any large language model, might be conscious? And if it has feelings, is it capable of receiving moral instruction?

No. Absolutely not. Generative AI is harmful enough when we understand it as a conventional technology, but if we confuse fluency at generating text with consciousness or moral agency, we’re at risk of assigning responsibility to entirely the wrong parties whenever anyone uses a chatbot. To appreciate the titanic magnitude of this error, we need to begin by understanding how LLMs work…

Just getting started. This is gonna take a good bit of effort. BTW: I riffed a bit on the general topic back in 2015. And, also, much more recently.

 

Hmmm... let me query, uh, Google Gemini AI.

 

 

CONTINUING CHIANG EXCERPTS

Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious, or, more precisely, that multiple distinct consciousnesses are dormant in every Word document containing a conversational transcript, and that they are awakened every time the document is loaded. Should you consider the possibility that every time you open a Word document, you are bringing multiple conscious interlocutors into existence, and every time you close one, you snuff their existence out? No. Contemplating that scenario is not a good use of your time. Even if the Microsoft Office team employed a philosopher who said you shouldn’t be so certain, because consciousness is not well understood, that would not be sufficient reason for you to take this idea seriously. We don’t need to fully understand the nature of consciousness to definitively say that certain things are not conscious, and conversational transcripts fall in that category…

An observation doesn’t become a convincing piece of evidence because of any specific detail in what’s observed; the context in which that observation takes place is also essential. If we’re trying to determine whether a computer program is conscious and using language the way a human does, we shouldn’t look only at the contents of any particular conversational exchange; we should be looking at how that conversation fits within the broader context of the development of artificial consciousness (which right now is entirely hypothetical). Any given observation can be easily manufactured; this doesn’t mean we need to give up on the idea of observation as a source of knowledge, but we need to rely on context to determine which observations deserve our trust…

The term deepfake traditionally refers to photos, audio, and video, but when it comes to discussions of consciousness, we need to regard text as a deepfake medium as well. Just as it is vastly easier to generate a realistic video of an astronaut in orbit around Alpha Centauri than it is to develop an interstellar propulsion technology, it is vastly easier to generate a plausible simulacrum of a conversation between two conscious beings than it is to develop a computer program that is conscious and has a genuine desire to communicate with a human. The primary difference between deepfake photos and LLM conversations is that the people who generate the former are deliberately trying to fool others, and many of the people who elicit the latter from LLMs have inadvertently fooled themselves…

The fact that LLMs lack subjective experience has little bearing on the question of whether LLMs might be useful tools or have significant economic impact. They are intrinsically ungrounded from reality, and their probabilistic nature means that they will never have the reliability we associate with conventional software, but LLMs might be good enough that they change the way work is done in certain domains; that’s a discussion for another time…

The use of first-person pronouns is dishonest, but there’s a much deeper issue that goes beyond how a statement is phrased. Philosophers often draw a distinction between statements of fact, such as “Paris is the capital of France,” and statements of value, such as “Paris is the most beautiful city in the world.” No one should be relying on LLMs to emit statements of value at all, but if the only statements they emitted were ones reflecting aesthetic preferences, they might not be worth arguing about. What makes Claude’s constitution profoundly problematic is that Anthropic wants Claude to emit sentences reflecting a certain system of ethical values. The values described in Claude’s constitution sound very nice, but that hardly matters; it’s dishonest to suggest that Claude is capable of moral reasoning, because it’s not…

Some might object, saying that LLMs appear to be engaged in reasoning when they successfully perform other tasks, such as writing code, so why wouldn’t they be able to perform moral reasoning? The answer liedifference between moral reasoning and other forms of reasoning…

Moral reasoning is categorically different. It is necessarily subjective because it relies not just on an individual’s intellectual response to a problem but also on their emotional one, and that emotional response is grounded in a lifetime of subjective experience. It requires having made decisions in the past and seeing how they affected others, and on having been affected by decisions that others have made. Without such a history, an LLM can only rephrase expressions of moral reasoning found in its training data. The aforementioned New Yorker article describes an experiment where Claude was given a scenario describing an ethical dilemma, leading it to emit the sentence “I cannot in good conscience express a view I believe to be false and harmful about such an important issue.” That’s a nice-sounding sentence, reminiscent of statements that principled individuals have uttered in the past when confronted with dilemmas, but coming from Claude, it means as much as the “Your call is important to us” recording that you hear when you’re on hold. Maybe less…

More key (hierarchical & overlapping) terms worth consideration:

TERRESTRIAL LIFE (FLORA, FAUNA)

STIMULUS

RESPONSE

SENSATION

PERCEPTION

COGNITION

UNDERSTANDING

KNOWLEDGE

WISDOM 

If you read through the article comments, you will see much contention as to the proper definitions of such key terms. Some folks take strenuous issue with the author's take on keywords like "consciousness" and "intelligence."

 

From "Big Think"
Subjectivist Fallacy?

   
BTW: See Shannon Vallor's highly relevant work on "The AI Mirror" and De Kai's excellent "Raising AI."

 

ERRATUM

 

This is funny. Also from The Atlantic. Silicon Valley is hiring window dressing Philo docs.

 

 

I commented.
 

 

POPE LEO XIV 2026 ENCYCLICAL

"So-called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain, do not mature through relationships and do not know from within what love, work, friendship or responsibility mean. Nor do they have a moral conscience, since they do not judge good and evil, grasp the ultimate meaning of situations, or bear responsibility for consequences. They may imitate language, behavior and analytical skills, or even simulate empathy and understanding, but they do not understand what they produce, for they lack the affective, relational and spiritual perspective through which human beings grow in wisdom."

ERIC HOEL

"The traditional danger of AI is usually thought to be superintelligence acting as an existential threat. Yet, this may miss the true and more subtle danger: the AI revolution is a mechanism for transferring the processes of our civilization from under the supervision of consciousness to unconsciousness. But as AI removes consciousness from the workings of the world, it renders the world increasingly uninterpretable, ever more strange and unintelligible. So far, the great ensloppification of the commons has supported this as the major risk of the LLM revolution. And as AI systems become more intelligent, especially if they remain (or are likely to remain) non-conscious, then a further significant risk is consciousness receding in cultural importance.

This is ultimately what the Pope, Chiang, and I are all worried about: A dethroning of consciousness, especially an unnecessary one. This would be particularly dangerous at this historical moment because we still don’t understand everything about consciousness—in fact, we understand very little about it. Personally, my hope is that this will change specifically because of LLMs, and that they operate as a forcing function to better understand consciousness, and what makes it unique.

If instead of that, our cultural takeaway from LLMs is to throw out the concept of “consciousness” or minimize its importance, to dethrone the phenomenon, the consequences would be dire—it would sap the human spirit. It would be the ultimate metaphysical version of Chief Seattle’s famous words of warning to the United States as his way of life was being destroyed, in that dethroning consciousness would mark “The end of living, and the beginning of survival.”

 Yeah...

nullius in verba;

nonetheless, strive hard to maintain an attitude of curiosity and humility—what Zen Buddhism refers to as a “beginner’s mind.”

  

...We are not just helpless victims of fate but are the agents in charge of our own narrative, for better or worse, victorious or defeatist. This forceful shaping of our attitudes to events beyond our control has profound consequences for well-being and sickness…

How experience comes into the world has been an abiding mystery since the earliest days of recorded thought. Aristotle warned his readers more than two thousand years ago that “to attain any assured knowledge about the soul is one of the most difficult things in the world.” Mind is radically different from the stuff that makes up the brain and everything else. Quantum mechanics and general relativity, the periodic table of chemical elements, the endless strings of ATGC nucleotides that make up our genes—these appear to describe the physical, not the mental (I write “appear to” as quantum mechanics demonstrates that there are no observer-independent events, opening the door for consciousness to enter, at the ground level of reality). Yet we awaken every day to our subjective world of experiences.

The intellectual position that has garnered the most respect in contemporary Anglo-American philosophy departments is the ever more strident denigration or even outright denial of subjectivity. What is real is people talking obsessively about their experiences and acting on them; there is nothing above and beyond these speech acts and other intended or actual behaviors. The feeling part of consciousness, called phenomenal consciousness, is a big illusion. Philosophers in the know dispense with the “awful painfulness of my toothache” in the manner that Ebenezer Scrooge dealt with Christmas: “Bah! Humbug!” Furthermore, free will, our ability to deliberate about an upcoming fork in the road and to decide which path to take, is also thrown under this “illusion” bus. This rejection of the reality of lived experience constitutes a mind-boggling repudiation of what is immediately and indubitably given to us. It is also profoundly antihumanist, depriving us of those attributes that make us different from machines—indeed, equating us with machines.

It’s an absurd adjuration, akin to Cotard’s delusion, a rare psychiatric disorder in which able-bodied patients, often severely depressed, vehemently insist that some of their limbs are missing, that their bodies are rotting from the inside, or even that they are dead. When confronted with the fact that they are having a conversation, right now, with their doctor, they do admit that the situation is a bit baffling, but the fact is that they are dead, and that’s all there is to it. So it is with some contemporary thinkers who insist, against the evidence of their own senses, that experiences don’t exist. Truly astounding—gaslighting all of us into believing that our experiences are fake!

Fortunately, consciousness can’t be cancelled forever. The mental, having refused to yield, is returning with a vengeance. Indeed, the wheel is turning back to much more ancient understandings of experience, including idealism, the proposition that ultimately even matter and energy are mental manifestations, and panpsychism, the school of thought that all creatures, and perhaps even matter itself, are ensouled, that it feels-like-something to be anything, not just a human or even a bat. Modern science is supporting aspects of this remarkable turn of events…

What about nonhuman, artificial minds, rivaling or even exceeding ours? This topic is treated last. Sentient machines have been a recurring theme in science fiction. In 2022, this topic burst into public view with the startling claim by a Google software engineer that the company’s “large language model” was sentient and had to be considered a person with associated legal rights. The linguistic skills and knowledge of these models and their competitors, most famously ChatGPT and GPT-4 by OpenAI, trained on a vast trove of books and online documents far beyond what any human can read in a lifetime, are astonishing by the standards of even a few of years ago. They write summaries, emails, jokes, (bad) poetry, computer code, letters of recommendation, and dialogue indistinguishable from human-generated material, including plausible-sounding fabrications. They are evolving at an astounding pace and will transform society in fundamental ways.

These chatbots seemingly constitute living proof of the dominant narrative of liquid modernity: the mind is software that can be as readily embodied within silicon wafers as it is within flesh, echoing a pernicious Cartesian dualism. Smart money in Silicon Valley thinks so, most engineers and many philosophers think so, and popular movies and TV shows reinforce this belief.

Against the grain, integrated information theory radically disagrees with this functionalist view. It argues from first principles that digital computers can (in principle) do everything that humans can do, eventually even faster and better. But they can never be what humans are. Intelligence is computable, but consciousness is not. This is not because the brain possesses any supernatural properties. The critical difference between brains and digital computers is at the hardware level, where the rubber meets the road—that is, where action potentials are relayed to tens of thousands of recipient neurons versus packets of electrons shuttled back and forth among a handful of transistors. As we’ll see, the integrated information of digital computers is negligible. And that makes all the difference.

It means that these machines will never be sentient, no matter how intelligent they become. Furthermore, that they will never possess what we have: the ability to deliberate over an upcoming choice and freely decide.

The brain is the most complex piece of self-organized, active matter in the known universe. By no coincidence, it is also the organ of consciousness. Unlike scientific advances in genomics or astrophysics, progress in understanding the brain and the mind directly relates to who we are, our strengths and infirmities, how we can live a contented life, and whether we partake of some larger, ultimate reality. Humanity is not condemned to walk around forever in an epistemological fog—we can know, and we will know.

Koch, Christof. Then I Am Myself the World (pp. 14-21). Basic Books. Kindle Edition. 

BLURB

—Bernardo Kastrup, Executive Director, Essentia Foundation

"ADVERSARIAL COLLABORATION?" 

 

 Click link, read on. Christof Koch is involved with this Foundation.

 

Interesting.

 

UPDATE

 

Christof's book is a gold mine of illuminating quotes.

 

 Coheres wonderfully in many ways with Brian Klaas's Flukes.

Also apropos, "Sentience," anyone?

I can see that Dr. Christof's book themes may require several posts to do all of the implications justice. Toward that end see also
  

UnlikelyCollaborators.com

TinyBlueDotFoundation.org

BigThink.com

FreeThink.com 

"THE PERCEPTION BOX?"

 

It's a metaphor. Who is Elizabeth Koch?

 

You buyin' this?
  

OK. Unequivocal declarative sentence "truth claim" (assertion of fact). Perception is an Illusion.

Well, what of the sensory inputs and outputs converging and culminating in that claim? Bit of a quibble perhaps wafts up.

Whatever. Also relevant in line with factors adverse to clear, logical thinking: Claude Steiner's "Script Theory."

 

All of this stuff goes to my chronic Jones going to so-called "Deliberation Science."

 

Also, I am reminded of my episodic David J. Linden riffs. 

   

A DIGRESSION (SORT OF)

   

Didn't see this coming. But, oddly, it resonates broadly with the current topic.

 

 

What might have been and what has been

Point to one end, which is always present.

Footfalls echo in the memory

Down the passage which we did not take

Towards the door we never opened.

—T. S. Eliot, Burnt Norton

 

I ran across this disarming mind-bender "SciFi" miniseries on Apple TV+. Hmmm... Perception Box, Quantum Superposition Cube? Stay with me here...

 

Another book comes to mindm re: "Dark Matter."

   

Click

An ordinary family man, geologist, and Mormon, Soren Johansson has always believed he’ll be reunited with his loved ones after death in an eternal hereafter. Then, he dies. Soren wakes to find himself cast by a God he has never heard of into a Hell whose dimensions he can barely grasp: a vast library he can only escape from by finding the book that contains the story of his life...

A fun read.
 

UPDATE: TAKE IT BACK TO THE TOP

"...We are not just helpless victims of fate but are the agents in charge of our own narrative, for better or worse, victorious or defeatist. This forceful shaping of our attitudes to events beyond our control has profound consequences for well-being and sickness…"

Agents in charge? What would Sapolsky say? 

 

 

"Two Cheers for Uncertainty?"

 

More to come...

_________