The AI Superintelligence Debate Has Serious Flaws

(spectrum.ieee.org)

65 points | by rbanffy 1 hour ago

18 comments

  • vanelsas 51 minutes ago
    I find the discussion academic at best. I am more worried about a swarm of dumb AI's causing havoc on infrastructures, causing real-life problems like food, energy and water shortages everywhere. Physical harm is a real possibility, and it seems most AI companies don't care. There is no harness for stupidity and irresponsibility.
  • saagarjha 1 hour ago
    I don’t necessarily agree with the labs’ policies and predictions on RSI but it’s worth noting that the points raised here are something they do have responses to.

    On the idea that training better models requires more data, I think the current thought here is that we are not getting all the “meat off the bone” of the data we have, so there is still quite a ways to go with training that AI can improve. After all, it’s not like Anthropic is doubling their data set every year (?) and yet their models definitely improve by more than that.

    For computability, yes, obviously the models cannot violate mathematics. But there’s a lot of things that are very computable and very desirable, or correspondingly very computable and very bad. This is enough to be worth thinking about.

    For alignment: it’s not actually a question of alignment at all, but rather one of “can we ignore alignment and design systems where the AI cannot cause harm?” Obviously, we can. However the issue is that doing this takes time and very few people (especially the frontier labs!) seem to be doing so. It’s much easier to hook your LLM up to the world and ask it to go than ask, “is this safe?”

    • ben_w 34 minutes ago
      > For alignment: it’s not actually a question of alignment at all, but rather one of “can we ignore alignment and design systems where the AI cannot cause harm?” Obviously, we can.

      This does not seem at all obvious.

      Any system worth using has to actually do something beyond the boundaries of that system. Even if that consists only of writing responses to prompts, the moment those are connected to any thing they can alter the state of that thing.

      Making that safe means it only ever produces a safe response. Even ignoring malicious uses, it is unclear how to correctly classify what "safe" means on an unbounded time horizon.

      • pvab3 20 minutes ago
        We need to stop using the word "Alignment" because it clearly has different meanings for different people. Some people consider it to be preventing the model from waking up and enslaving or destroying humanity a la paperclip maximizer. Some people mean it as preventing the model from assisting with bad things like hacking, cybersecurity exploits, building weapons, and other things that are aligned with that individual user's goals, but not those of society more broadly.
        • ben_w 0 minutes ago
          Those are, unfortunately, much the same thing.

          The paperclip optimiser thought experiment isn't "AI does this spontaneously", it is "AI is told to do this by someone who didn't realise the consequences, and it succeeds".

      • saagarjha 6 minutes ago
        I think you’d need to broaden your scope too far as to be useful for something like that. If we hook AI up to LHC, well, the immediate impact is obvious in that some physicists can’t use the detector. But it’s hard to ascribe “we neutron bombed the Milky Way in 42069 AD” to it finding a theory of quantum gravity.
    • shishy 13 minutes ago
      > but rather one of “can we ignore alignment and design systems where the AI cannot cause harm?” Obviously, we can

      How? Many failures are emergent properties in complex systems -- you can capture some of them but this doesn't really seem obvious to me?

      • saagarjha 5 minutes ago
        This usually means not connecting AI to complex systems
    • BugDeveloper 33 minutes ago
      About computability: All functions on finite sets are computable and superintelligence doesn't require a function on an infinite domain.
    • soco 54 minutes ago
      Training a human brain needs and uses much, much, less data than training the AI models. So maybe the expectations on the AGI are, dunno, just optimistic? I mean you can put tons and tons of gas in a car but that won't make it drive faster.
      • origini 6 minutes ago
        This could also be the perfect argument of why ASI is just around the corner: One breakthrough in architecture and we already have so much more training data then it ever need again.

        If we can make smart humans with a lot less, why do we even need that much for an LLM?

        Perhaps world models will bring ASI, perhaps when LLM progress slows down, labs start to explore a lot more architectures and suddenly find it or the next Mythos will design the ASI architecture.

      • dosinga 12 minutes ago
        You can also use that as a counter argument. Training AI on enormous amounts of data has resulting in pretty decent LLMs still far from AGI. We're running out of data so seemingly this will limit what LLMs can do. Human brains seem to do much more with a lot less training data so presumably there's still a lot we can do to improve just the training algorithms.
      • jeremyjh 47 minutes ago
        Human brains are the product of millions of years of evolution. You have to include that training data as well for it to be a fair comparison.
        • zarzavat 12 minutes ago
          All that evolution compresses to a GB or so of DNA (yes I know it's more complicated than that).

          Humans are a bad example because of the well known width restriction leading us to be born undeveloped.

          But animals can be born with varying levels of fully or partially developed neural networks despite having minimal data transfer from the parents and no apparent training data available in the womb/egg.

          Nature is able to squeeze a lot more efficiency than our current AI efforts.

          • saagarjha 4 minutes ago
            Seems like nature has a pretty cracked 1B parameter base model and fine tuning stack
        • ftchd 6 minutes ago
          I would argue that's the actual "training" part, and your lived experience is the equivalent of fine tuning.
        • ben_w 16 minutes ago
          Kinda, but unclear how much pre-training of our brains evolution did exactly, only that even if DNA was just about the brain (it isn't) it would still be a tiny pre-train.
        • soco 31 minutes ago
          Fair point. Those millions years of training however are not really contained in the corpus of data we are currently using in training AI, also the hardware is quite different - and created by processes with huge and significant differences, so I don't even know what apples with what oranges I should compare...
          • jeremyjh 21 minutes ago
            You can’t directly compare them, is my point.
      • rileymat2 17 minutes ago
        I have no concept of how much less data it uses, we consume video and audio constantly when not sleeping. We do this for years before a baby can talk.
      • XorNot 17 minutes ago
        The counter argument is that it uses less text, but a lot more everything else. Any new parent could tell you the surprising number of things which babies just don't know: which seems like an obvious statement, but babies don't know how their body works. They don't know how their digestive systems work, in most cases there's an entire growth process for those systems to be usefully capable for them.

        All of that though, is training data. All our sensory inputs form an enormous base of input which I doubt the volume and content of the internet at its highest strata properly represents: so our AIs are developed from a much higher stratum of reasoning and this showed up pretty readily in early models - they could write language well, they absolutely had no grounding in how physics worked, for examaple.

      • squidmaster 51 minutes ago
        [dead]
    • majormajor 37 minutes ago
      > For alignment: it’s not actually a question of alignment at all, but rather one of “can we ignore alignment and design systems where the AI cannot cause harm?” Obviously, we can. However the issue is that doing this takes time and very few people (especially the frontier labs!) seem to be doing so. It’s much easier to hook your LLM up to the world and ask it to go than ask, “is this safe?”

      Much easier to pump up the numbers to investors to sell the quick-and-dirty solution. These companies are trying to speed-run things in unsafe ways that (just so coincidentally, I'm sure) happen to be exactly what the things you'd do if you were just in it for the money, and I wish we wouldn't be so accommodating.

      All the "are we creating god" talk is similarly window-dressing nonsense marketing fluff. Gods throughout the history of mythology have been dumb as often as they've been superintelligent, the defining attribute of them is power not reasoning speed. If humans give massive rapid destructive power to a computer program it's a human error, not a god escaping us.

    • api 51 minutes ago
      Having responses doesn’t mean anything. Are the responses valid and grounded in reality? Is there evidence for them?

      A skilled debater can argue any arbitrary position convincingly.

      I thought of a short LT;DR version of why I am not a Rationalist: they think arguments are evidence.

      • ben_w 11 minutes ago
        > I thought of a short LT;DR version of why I am not a Rationalist: they think arguments are evidence.

        You're thinking of a-priorists.

        Rationalists do not (at least not willfully) think arguments are evidence.

        • card_zero 9 minutes ago
          Evidence isn't arguments, at any rate.
      • card_zero 22 minutes ago
        > why I am not a Rationalist: they think arguments are evidence

        That means something, right? But it's an argument.

      • saagarjha 38 minutes ago
        I mean, the post doesn’t have evidence of its arguments either. It’s hard to come up with evidence for something that does not exist
  • origini 8 minutes ago
    1) RSI:

    Give claude a task + compute and see how long it plays around finding a better solution. Doing experiments by itself, profiling, testing, etc. This is impressive and works locally already. More compute you give it, the more interesting things it does.

    Of course everything right now is highly speculative but this doesn't mean its not possible. With the current progress, its something highly speculative and plausable which means you should take it serious enough. Doesn't mean you have to go to Datacenters and blow them up.

    2) weird argument the author makes. 1 Million experts can for sure achieve a lot more a lot faster than 1. But they are also scaling models in parameter which is not just faster but also potentially more intelligent. If they have more compute and compute gets better faster and cheaper, than bigger models are also more plausable.

    3) It doesn't even matter if its going rough because it takes over the world or just because of learning as long as the risk is real that it stumble upon some network which has access to another private network which has access to some launch control pc or starts to mimick a world leader and manipulates humans and you find out, 100 years later, that the biggest disruption of modern society was a LLM agent simulating a book and 'simulating' some book ideas. Our IT infrastructure is fragile when you are able to react a lot faster and better than a human. Our banking system are computers, payment is digital, even plenty of power stations and water stations are controlled digitally remote.

    The flaws in these discussions are people downplaying all of it by either not understanding time or the current capabilities of basic LLMs. We are at the beginning of all of it, we are still 'just' using transformer architecture. The LLM progress is still so fast, that everything else doesn't get the attention it might deserve. The LLM is currently in a very fast and deep local minimum and we have no idea whats around.

    4 years ago loked very different in LLM and ML capabilites. People were laughing about the danger of an LLM now we have more critical CVEs since ever.

    We reached a point were it is cheaper to pay 1000 people to teach an LLM one thing instead of teaching 1000 people or more. It was a skill to be able to use Blender, now an LLM can use Blender. People on Fiverr probably getting a lot less jobs. Real media people are getting fired, translator, writers, etc.

  • jeremyjh 42 minutes ago
    I agree with most of this but I do not think the author understands either the Hugging Face incident or what alignment really means. Reward hacking is a serious issue and the tendency for models to do this is a serious problem that we’ve made approximately zero progress on resolving. We pile on more external guardrails and hacks but those methods will always be vulnerable.
    • thrw045 27 minutes ago
      It's true that the more capable a model is, the more dangerous reward hacking is, but there is a giant elephant in the room. Namely that these ai models still can't create their own high level goals. This means that if there were to be an incident of hacking or whatever, this would have to be launched by a human directly prompting or them setting up an rl environment in some way that lead to it.

      There is a _huge_ difference between an ai model that follows a top level goal given by something else and one that can give itself its own top level goals. The latter would essentially be like a human or other organism and would be very dangeorous the more capable it is.

      The other smaller elephant in the room is that if reward hacking is so dangerous, why don't we see more rogue agents out when we are prompting them ourselves? For the most part, the agents do what they're supposed to do as guided by the prompt, so it seems like to me that at least at present, only agents that are trained on hacking data AND put in an rl environment or given a prompt related to hacking, as well as asked or trained to persist will actually try to do that hacking. I would be worried if chatgpt or claude agents started to hack something every time you asked it to fetch some data from the web or something. So that's one question I have is why does hacking and going outside instructions not happen in normal usage of the models? Does it but they actually work it out of the system in some way? Or is it the other way around where you have to create the conditions necessary for it to happen and these conditions are exceptional not the norm?

      So I'm not trying to undermine this, I think there are things to be taken seriously and thought about but I always wonder why the people from the big labs never include all of this information. They always talk very abstractly about RSI, hacking and extinction without any caveats about what would be needed for that happen. I'd be happy to be proven wrong on this also.

    • Upvoter33 31 minutes ago
      Why isn’t this “just” a security issue? I mean, the security precautions around the hugging face attacks were amateurish no?
      • naasking 10 minutes ago
        Yes, but humans have demonstrated that they don't care about security. If they did we would have switched away from C/C++ long ago.
        • ForHackernews 1 minute ago
          Humans care about security in some contexts. Nuclear weapons are generally well-secured. Maybe AI will finally make them care about software security.
  • Sharlin 27 minutes ago
    Although I disagree with the author about some points, this was a refreshingly balanced take in the sense that at least the author at several points acknowledged his uncertainty (the bar is low indeed).

    On the other hand, the uncertainty is the entire point! Even if P(ASI) is low and P(doom|ASI) is low, what exact risk are we prepared to take? Or more accurately, what exact risk are we prepared to allow a very small, non-representative group of humans to take?

    • mitthrowaway2 18 minutes ago
      Right?

      Back when there was talk about whether CERN was going to create micro-black-holes that grow to swallow the Earth, before turning it on they presented the math that showed they were something like 99.99999999% confident that it would not happen.

      When dismissing a risk that threatens the entire planet, you should have very high confidence in your safety case!

      • immibis2 12 minutes ago
        I hope that's a conservative estimate. I probably wouldn't turn on a machine that I thought would destroy the planet even 1 in 10 billion times.
        • ben_w 7 minutes ago
          It's really hard to be six-sigma (6.36σ is about 1:1e10) confident about anything, given all the cognitive blind spots we have.
  • adamtaylor_13 4 minutes ago
    I find the "faster thinking dog can't play chess" argument to be weird and possibly wrong.

    If my dog could play 1B chess matches in 5 minutes, I'm guessing it would be able to play chess at some reasonable level (assuming rewards, reinforcement, etc. which seems table stakes to this argument)

    These machines are already doing that (and more)

  • imeanwhatdoikno 13 minutes ago
    Can somebody comment on the evidence for the following sentence from the article: "After GPT-4, however, continued scaling [number of parameters] began yielding diminishing returns"?

    My understanding was that scaling in size slowed for practical/economic reasons, not because of its diminishing intelligence returns.

  • 0x445442 22 minutes ago
    Was anyone arrested for the Huggingface hack? If I'd have been caught doing it I believe I'd have been charged with a cyber crime. And I don't think arguing it was the computer program that actually did the hacking would have gotten me off the hook.
    • Jtarii 13 minutes ago
      You also aren't a trillion dollar company inventing the most important technology of the past 50 years.
  • jokoon 59 minutes ago
    We don't even know what intelligence is.

    I just want science to at least build a notion of intelligence, in terms of evolution, biology, psychology, neurology. We don't really know what animal intelligence is, and we only limit AI to human intelligence, which is very narrow.

    If we use AI to do human stuff, it's just an assistant for human goals. If humans don't have a specific goal, super intelligence will not guess or decide it for us.

    What does humanity want? Health? Mental health? Colonize Mars? Reduce income inequality? Solve climate change? World peace? Immortality? Happiness?

    AI can help for those things, but that's it.

    We tend to romanticize super intelligence like it would be some new life form, external to humans, a form of omniscient deity that could be a threat to humanity. There are worse threats: antibiotic resistance, nuclear war, climate change, dictatorships, mental health epidemics, obesity.

    • lukan 48 minutes ago
      "We tend to romanticize super intelligence like it would be some new life form"

      How do you know it won't be?

      Right now it does not really meet the biological criteries, as it cannot reproduce on its own. But once it can mine minerals, build its chips and engines and maintain itself, what word we have to call that, if not artificial life?

      Doesn't mean it will have a soul, but that ain't a scientific concept either. Feelings and emotions? Is that for life required? But we don't know how that works either.

      (Apart from that, I would define intelligence as the ability to understand the world.)

      • krige 37 minutes ago
        > But once it can mine minerals, build its chips and engines and maintain itself, what word we have to call that, if not artificial life?

        Will it? Or are you talking about several separate machines, most of which won't ever meet any criteria for life?

        • Orochikaku 29 minutes ago
          Can’t the same be said about all multi-cellular life?
          • krige 15 minutes ago
            Get out of here with these sophisms. No, a pickaxe, a truck, and a lithography machine is not at all like a cat.
            • lukan 11 minutes ago
              But a cat consists of many small parts, cells and smaller, that would not survive and replicate on their own, just when they form a bigger collective. The same pattern of complex matter organisation would be artificial life. But if you insist, I can also think of machines, that have the replication parts integrated in them (miniaturized metal sinterers etc.) and just need to feed themself with certain raw matter and energy. Would that be artificial life to you?
      • AnneTrotter 35 minutes ago
        I can build a theoretical brainless self-replicating zombie that does those things.

        And we actually do know how feelings and emotions work. Biochemistry. Machines do not have biochemistry. That means no feelings.

        "Soul" is irrelevant as it's inherently unproveable, so ignore soul as a concept for all definitions of intelligence and life.

        • lukan 22 minutes ago
          "And we actually do know how feelings and emotions work. Biochemistry."

          We know that what we perceive is the result of biochemistry, but we don't know if it cannot also be based on other (quantum) process and we absolutely do not understand how feelings work, as it requires accurately moddeling the mind state of a person and as far as I know we are pretty far from that. Otherwise psychologists would be working with brain simulators to help cure their patients.

          "I can build a theoretical brainless self-replicating zombie that does those things."

          Also I am curious how you would do that.

        • LtWorf 13 minutes ago
          > And we actually do know how feelings and emotions work. Biochemistry. Machines do not have biochemistry. That means no feelings.

          You seem to know much more than the collective rest of mankind. Perhaps publish some papers to inform the rest of us?

    • tiborsaas 13 minutes ago
      > We don't even know what intelligence is.

      Why do people keep repeating this? What is it so hard about intelligence to understand? We already have quite satisfying definitions of it. Why its evolution is a mystery?

      We tend to romanticize biological intelligence by treating it like some mystery, but it's "just" a way of evolution and natural selection. The better a species gets at problem solving, the better their chances are at survival.

      • immibis2 11 minutes ago
        Oh yeah? What are the definitions, then?
    • squidmaster 53 minutes ago
      [dead]
    • NichoPaolucci 37 minutes ago
      > Health? Mental health? Colonize Mars? Reduce income inequality? Solve climate change? World peace? Immortality? Happiness?

      > AI can help for those things, but that's it.

      That’s it? What a let down.

  • amelius 57 minutes ago
    The debate will quickly change when we have 8ft tall humanoid robots marching through the streets.

    In fact, if anyone wants the debate to change, they should probably build these robots and take them for a walk through the park/shopping mall in various places.

    • immibis2 11 minutes ago
      They're already making those, but in the shape of a dog, which has better mechanics. Some of them get beta-tested in... places you can't say here. Boston Dynamics is a military contractor.
  • adrianN 32 minutes ago
    „ Implementing roughly human-level intelligence on a faster machine might be impressive, but it’s not necessarily alarming.“

    Unless of course you feed your family by using your human level intelligence and now a machine can do your job faster and doesn’t need to sleep.

  • bryanlarsen 46 minutes ago
    Recursive self intelligence is not going to make any difference. We're already using models to guide creation of their successor, and taking humans out of that process won't speed the process up. Mythos took months and and a ton of resources to create. A larger more capable model will take even more. Humans are no longer the limiting factor. Time, electricity and data centres are.

    If the singularity is coming, it has already started and will take many years.

    • jefftk 25 minutes ago
      > taking humans out of that process won't speed the process up ... Humans are no longer the limiting factor.

      This seems very unlikely to me. While time and compute are clearly large components, we've also seen significant advances due to using that time and compute more effectively. A lot more seems possible here, since with how quickly everyone has been going this is more of a "rapidly scaling up the first things we've found that work well" than "carefully research how to optimize at every level". This is researcher work that is actively moving from humans to machines, and moving more of it there speeds the process up more.

      • bryanlarsen 12 minutes ago
        Which is being done by researchers guiding Fable. Fable is doing most of the work, and is the limiting factor, not the researchers. Fable guiding Fable would not be appreciably faster. Fable2 guiding Fable2 would be, but that's because the workhorse is better, less so the guide.

        Yet despite the improvement, Fable3 will take longer to create than Fable2. The difficulty is increasing faster than capabilities.

        p.s. Anthropic uses Fable2 and Fable3 for iterational improvement names, I'm using it here for generational improvement names.

  • forlorn 14 minutes ago
    Can we just, I don't know, unplug the datacenter power cable putting an end to these capabilities?
  • zodwick 14 minutes ago
    “It’s a machine. At the end of the day, it’s stupid.” Garry Kasparov, 1997
  • keybored 8 minutes ago
    > Many people were caught off guard this past summer when employees of one of the world’s leading AI companies began casually debating whether their products would lead to human extinction. [...] Evan Hubinger, Anthropic’s head of alignment science, replied, “Jacob is correct here—we really do earnestly believe AI could kill all humans!” And Anthropic CEO Dario Amodei subsequently told CNN, “I agree with Jacob much more than I disagree with him.”

    Thanks. What a great paragraph to explain why Anthropic is a lunatic cult according to its own logic.

  • MrOrelliOReilly 34 minutes ago
    Cal Newport is part of a cluster of CS and ML researchers skeptical of imminently emergent ASI (artificial superintelligence), like Andrew Ng and Yann LeCun. I do not find their arguments very convincing! Especially since arguments about ASI risk almost always probabilistic; we _might_ be able to develop ASI from current LLM architectures, and such ASI _might_ not be well-aligned to humanity. So we should act according to these dangerous possibilities!

    To respond to each of his objections in turn:

    1. It is true that strong RSI remains speculative. Weak RSI, where AI improves coding and other toil to accelerate research, is already underway. But Newport's primary objection to strong RSI appears to be that we might have a hard time hill-climbing on non-verifiable domains, e.g. building a successful company or doing politics. However it's _also_ possible we'll see lots of generalized intelligence arise from progress in verifiable domains, so that this does not end up being a problem in practice. One can also imagine new training regiments that address this problem some way, even if it's just better RL environments? Also algorithmic progress is a large component of AI development; why would we not expect continued progress on mathematics (highly verifiable) to improve algorithms and hence AI? Seems to me that strong RSI is a reasonable thing to expect, if not definite.

    2. I find this argument very strange; I suppose that Newport is arguing against an extreme definition of superintelligence that posits it could compute "anything" and so function like a hypercomputer? This would definitely be silly! But he does not offer any concrete argument for why human-level intelligence should be a ceiling on intelligence, or why it should be more compute efficient. Even if we just ended up with Von Neumann level intelligence, this would already be something like TEDAI (top expert dominating ai), especially if we are able to run thousands/million/billions of instances. Maybe this does not reach a strong definition of superintelligence, being _merely_ smartest-human level intelligence, but I think most the concerns about superintelligence still hold in this scenario (even if it's less "magical")

    3. Claiming alignment is not hard is disingenuous, no? Part of the challenge of alignment is even philosophical; what does it mean to have an arbitrarily intelligent agent behave according to values that will help all of human civilization thrive? How do we have such an agent obey commands in such a way that does not lead to disaster or power concentration? Are there even architectures that can prevent an agent from taking destructive actions while still allowing sufficient degrees of freedom for useful and productive actions? This is on top of all the mundane security engineering problems! Perhaps we will be lucky and ASI will be aligned by default, but on what basis can we assume so? Just because they are trained on human values? Are humans themselves not dangerous when given concentrated power?

    Really, I feel that these CS/ML researchers like Newport are just upset they missed the boat or something. I do not feel they are engaging seriously with the arguments at play here. But perhaps they would feel the same about me in the other direction. If you feel my counters miss the point then please speak on their behalf!

    PS I live in Switzerland, as a counter to his accusation of Bay Area brain washing as well.

  • shinojcm 58 minutes ago
    [flagged]
  • Ellatwins25 36 minutes ago
    [flagged]