So, it can be key for an implementation of the iteration of critical thinking which is crucial for Intelligence ("think, revise, acquire, revise, think, revise, revise...").

I say it would be amazing, it'd also be terrifying. The moment you tread past this cliff its going to change our entire existence, for good or bad.

I'm anxiously enthusiastically nervous and scared shitless.

This may not be necessary. It is pretty fundamental in humans, because factual, ethological decision making apparently needs an emotional drive to actually act on a decision, irregardless of the depth of reasoning that ranks optimality of options (this was found by Damasio, I believe). Those who can reason quite well, but suffer from a disconnection to the lymbic system - so they miss a "trigger" "shaking" them in front of the different options - cannot practically decide.

But for a Decision Support System, what the operator needs is a lucid, informed consultant to query about the options. The decision remains with the operator. Machines can act out of a function. There is no necessary need in a machine to divide "quick synthetic assessment" and "explicit analytical assessment", if in theory the machine has enough computation abilities to stick to the latter, and if the latter is what we want - as we want reliability.

One thing is the implementation of Intelligence, another is emulation (of "Also Potentially Intelligent Systems").

I really don’t understand the enthusiasm for it but I wish I did.

I guess if I create this “oh my god we are doomed and the only way out is through AI domination” I can kind of get excited about it? Like I can invent a crisis that needs solving.

There is no "«need»", if that means "hiding an excuse". But if you want to go "lymbic", as per a parallel post nearby: Intelligence is a Value.

It is a fact: individuals and societies are increasingly in need for Intelligence. When Vannevar Bush conceived the Memex, the need was chiefly for making information more available. Now the overwhelming need is to process it.

In practical terms, it is quite strong how somebody formulated the point in the recent Kissinger's book "Leadership":

-- the world is getting more and more complicated, while administrations are getting less and less capable

where "administrations" there reflect individuals in general. But computers, now largely in pockets, finally dramatically available, were supposed to be the Memex, enhancing tools... The promise must be fulfilled, because the stakes remain high (societally, historically).

--

Update:

> through AI domination [...] invent a crisis

This should be a joke. This seems like living on idle fantasies while there is a real world outside them, that will not run according to wishes just through its own natural running. There are decisions to be made - by humans, by individuals, by administrations, by governments. There are real problems out there: drought, floods, diplomacy, resource management ... what to do, who to marry. There are optimization and decision problems that we cannot just solve through paper (consuming pens and eyes).

We have been studying AI for decades and we have been using aided decision making tools for centuries - automated, for decades. You are using one now, the browser is a DSS information accessor! That means, calculators, not "Overlords"!

these are political problems - a generative algorithm suggesting equitable solutions will not move the needle on whether a dam is built or demolished

let's take the damning of the Nile as a real life example, I don't think there an optimal rate to exchange water in one place for electricity in another, it is a problem that will be solved by whoever is more willing to go to war - how does a superintelligence change this game of chicken?

The basic reply would be that while some decisions are taken out of strong political influence, we still do accounting, for example. Politics is not everything. Analysis influences political action (we speak of "advisors").

More substantially, one could remind that some political actors lament a dire lack of time to get properly informed and to *think*. Tools that facilitate information and its processing could be key to representatives and ministers.

Can you see see how ridiculous this can sound to some people ? We’ve created a ridiculous situation so I guess you’re right, the only way through it is more ridiculousness.

In my opinion that story you’re telling me actually does create a perceived need for AGI.

Edit: It sounds like in your world, this is some type of AI without agency or self preservation goals ? Just like an oracle with an interface or something ? I hate to say this but governments have all the information they need to make better decisions, they make them mostly based on lobbying. Not going to fun when AI can extract more oil at 1000x the speed, crush dissidents or capture your attention to sell you more shit even more effectively.

We have been doing AI for a long time: and we quite normally meant "virtual professional". ("Computer" means "virtual computer [(the job)]"; "AI" stands for "virtual computer [(the job)] with less mechanical, routinary tasks".) This "«agency or self preservation goals»" idea is practically not impossible but an "holliwoodian" drift of the debate - entertaining thoughts replacing practical ones.

If people act badly because of lack of wisdom, because of the hiatus between technological advance and personal advance (as we have been noticing at least for the past three centuries): they shall cultivate themselves. We do. Ignorance is societally the most massive problem, but it is a different problem. (Edit: sorry, not just and precisely "different": it is a related problem, because in fact we use Intelligence to acquire Wisdom. Intelligence is that thing that dissipates ignorance... That makes it a value. Synthetic, real Intelligence is meant to help against the devastating ignorance around.)

That people fall from ladders or use them to hit people in the streets is not a sufficient note to suggest we should do without ladders - and lose the inherent perks.

I guess it depends on our definition of AGI which doesn’t sound aligned

For the rest, you will have to be clearer.

Dr. Chandra: I've erased all of HAL's memory from the moment the trouble started.

Dr. Vasili Orlov: The 9000 series uses holographic memories, so chronological erasures would not work.

Dr. Chandra: I made a tapeworm.

Dr. Walter Curnow: You made a what?

Dr. Chandra: It's a program that's fed into a system that will hunt down and destroy any desired memories.