If it was AI-generated I had no difficulty with it, certainly on par with typical surface level journalist summaries, and vastly better than losing 2 hours of my life to watching some video interviews.. :) AI as we know it may not be real intelligence but it certainly has valid uses
I get a lot from seeing the person talk vs reading a summary. I have gone back and watched a lot of interviews and talks with Ilya. In hindsight, it is easy to hear the future ideas in his words at the time.
That said, I use AI summaries for a lot of stuff that I don't really care about. For me, this topic is important enough to spend two hours of my life on, soaking up every detail.
Say we discover a new architecture breakthrough like Yann LeCun's proposed JEPA. Won't scaling laws apply to it anyway?
Suppose training is so efficient that you can train state of the art AGI on a few GPUs. If it's better than current LLMs, there will be more demand/inference, which will require more GPUs and we are back at the same "add more gpus".
I find it hard to believe that we, as a humanity, will hit the wall of "we don't need more compute", no matter what the algorithms are.
This may not be AGI, but I think LLMs as is, with no other innovation, are capably enough for gigantic labor replacement with the right scaffolding. Even humans need a lot of scaffolding at scale (e.g. sales reps use CRMs even though they are generally intelligent). LLMs solve a “fuzzy input” problem that traditional software struggles with. I’m guessing something like 80% of current white collar jobs can be automated with LLMs plus scaffolding.
My first job out of uni was in creating automated tests to validate some set top box. It involved using library of "blocks" to operate a remote control. Some of the people I have been working with spent their whole career in this narrow area, building those libraries of block and using them for customer and I have no doubts a LLM can today produce the same tests without any human intervention
Replacing labor doesn't require replacing whole jobs, it's enough to only replace specific tasks within those jobs which will reduce the number of workers needed for the same amount of work.
To pick a rather extreme example, the fraction of the population involved in farming is rather lower than in the past. Due to productivity improvements.
It's not clear why your analogy wouldn't have implied the end of white collar work when computers were first invented or when the internet was invented. Both of those should have been massive productivity boosts which meant the workers would have to go elsewhere to feed themselves. Instead Jevon's paradox kicks in every time.
Most QA, most analyst positions, a good chunk of the kludge in intellectually challenging jobs, like medical diagnostics or software engineering, most administrative work, including in education and in healthcare, about 80% of customer success, about 80% of sales, are all within striking distance of automation with current-generation LLMs. And taht's entirely ignoring the 2nd-order effects in robotics and manufacturing.
I see LLMs in a similar way - a new UI paradigm that "clicks the right buttons" when you know what you need, but don't know exact names of the buttons to click.
And from my experience there are lots and lots of jobs that are just "clicking the right buttons".
Such a decision merely tips the scale into a brittle structure territory. It introduces critical points of failure (funneling responsibility through fewer "nodes", stronger reliance on compute, electricity, internet, and more) and reduces adaptability (e.g. data bias, data cutoff dates, unaware of minute evolving human needs, etc).
I agree, AI image recognition is so good already that it can tell what someone is doing or what is happening in a picture. Just have that run at 30 fps and make the robot's movements align with that understanding and bam, you effectively have "AGI" in some sense no? I mean sure, maybe it doesn't really remember anything like a human would and it doesn't learn on the fly yet but it's definitely some kind of intelligent, autonomous thing that will be able to respond to just about anything in the world. Making it able to learn on demand is something people are working on. Chatgpt remembers some stuff already too after all. It's very small and very spotty, weird memory but hey, it's something. As soon as that becomes a tiny bit better you'll already beat humans at memory.
This opinion is not based in reality. The only way to understand that is to go outside and talk to real people who are neither techies nor managers, and, better yet, try to do their jobs better than they do.
All the frontier houses know this too. They also know it will be extremely difficult to raise more capital if their pitch is "we need to go back to research, which might return nothing at all."
Ilya did also acknowledge that these houses will still generate gobs of revenue, despite being at a dead end, so I'm not sure what the criticism is, exactly.
Everyone knows another breakthrough is required for agi to arrive; sama explicitly said this. Do you wait and sit on your hands until that breakthrough arrives? Or make a lot of money while skating to where the puck will be?
I would say the issue is that most of the big AI players are burning a lot more cash than they earn, and the main thesis is that they are doing so because their product will be so huge that they will need 10x-100x infrastructure to support it.
But what we're seeing at the moment, is a deceleration, not an acceleration.
There is no example of a leading company that ships a world-changing product, yet somehow runs out of cash.
Maybe they lose relevance. Maybe they miss the breakthrough. That becomes the reason. So perplexity? Sure. Anthropic, even? Yep. Google? OpenAI? Nah.
Regardless, viewing the unit economics, there are very clear sight lines to profitability if they want it. Just like with Amazon, Tesla, Apple, etc., when you want to grow, hoarding cash is a bad play.
OpenAI is very unlikely to go bankrupt, but they could be in such a difficult financial position that they would have to make painful compromises with Microsoft and/or Nvidia and lose most of their leverage.
Microsoft, largest software company in the world, has (publicly and privately) admitted that it is in their best interest to ensure that OpenAI remains the leading ai company for at least the next seven years.
As for nvidia, if OpenAI has less leverage, that necessitates a different ai company having more. Who would it be?
No matter the unit economics, commoditization and advancement of open models and small startups means that there is at almost a year or two to exploit competitive advantage. If scaling stops, the window to make a profit is extremely narrow.
People at Anthropic have a vested interest in getting you to believe that they are creating new and exciting things. Of course they say that a breakthrough is imminent. Doesn't make it in the least true, though.
> Models look god-tier on paper:
> they pass exams
> solve benchmark coding tasks
> reach crazy scores on reasoning evals
Models don't look "god-tier" from benchmarks. Surely an 80% is not godlike. I would really like more human comparisons for these benchmarks to get a good idea of what an 80% means though.
I would not say that any model shows a "crazy" score on ARC-AGI.
I broadly have seen incremental improvements in benchmarks since 2020, mostly at a level I would believe to be below average human reasoning, but above average human knowledge. No one would call GPT-3 godlike and it is quite similar to modern models in benchmarks; it is not a difference of like 1% vs 90%. I think most people would consider gpt-3 to be closer to opus 4.5 than opus 4.5 is to a human.
Roughly I'd agree, although I don't have hard numbers, and I'd say GPT-4 in 2023 vs GPT-3 as the last major "wow" release from a purely-model perspective. But they've also gotten a lot faster, which has its own value. And the tooling around them has gotten MASSIVELY better - remember the "prompt engineering" craze? Now there are a lot of tools out there that will take your two-sentence prompt and figure out - even asking you questions sometimes - how to best execute that based on local context like in a code repository, and iterate by "re-prompting" itself over and over. In a fraction of the time you could've done that by manual "prompt engineering."
Though I do not fully know where the boundary between "a model prompted to iterate and use tools" and "a model trained to be more iterative by design" is. How meaningful is that distinction?
But the people who don't get this are the less-technical/less-hands-on VPs, CEOs, etc, who are deciding on layoffs, upcoming headcount, "replace our customer service or engineering staffs with AI" things. A lot of those moves are going to look either really silly or really genius depending on exactly how "AGI-like" the plateau turns out to be. And that affects a LOT of people's jobs/livelihood, so it's good to see the hype machine start to slow down and get more realistic about the near-term future.
How do you avoid overfitting with the automated prompts? It seems to add lots of exceptions from what I've seen in the past versus generalize as much as a human would.
I dunno, some of the questions on things like Humanity's Last Exam sure strike me as "godlike." Yes, I'm happy that I can still crush LLMs on ARC-AGI-2 but I see the writing on the wall there, too. Barely over a year ago LLMs were what, single digit percentages on ARC-AGI-1?
I would hope god can do better than 40% on a test. If you select experts from the relevant fields humans, they together would get a passing grade (70%) at least. A group of 20 humans is not godlike.
Didn’t we just see big pretraining gains from Google and likely Anthropic?
I like Dario’s view on this, we’ve seen this story before with deep learning. Then we progressively got better regularization, initialization, and activations.
I’m sure this will follow the same suit, the graph of improvement is still linear up and to the right
While I think there's obvious merit to their skepticism over the race towards agi, Sutskever's goal doesn't seem practical to me. As Dwarkesh also said, we reach to a safe and eventually perfect system by deploying it in public and iterating over it until optimal convergence dictated by users in a free market. Hence, I trust that Google, OpenAI or Anthropic will reach there, not SSI.
> we reach to a safe and eventually perfect system by deploying it in public and iterating over it until optimal convergence dictated by users in a free market
Possibly... but also a lot of the foundational AI advancements were actually done in skunkworks-like environments and with pure research rather than iterating in front of the public.
It's not 100% clear to me if the ultimate path to the end is iteration or something completely new.
Some have been saying this for years now, but the consensus in the AI community and SV has been visibly shifting in the recent months.
Social contagion is astonishingly potent around ideas like this one, and this probably explains why the zeitgeist seems to be immovable for a time and then suddenly change.
People have been saying this before chatgpt and ever since. And they're right.
Its charlatans like sama that muddy the waters by promising the sky to get money for their empire building.
LLMs can make and are great great products. But its sneaky salesmen that are the ones saying scaling is the path to AGI. The reality is that they're just aiming for economies of scale to make their business viable
>What high quality data sources are not already tapped
Stick a microphone and camera outside on a robot and you can get unlimited data of perfect quality (because it by definition is the real world, not synthetic). Maybe the "AGI needs to be embodied" people will be right, because that's the only way to get enough coherent multimodal data to do things like long-range planning, navigation, game-playing, and visual tasks.
> Stick a microphone and camera outside on a robot and you can get unlimited data of perfect quality (because it by definition is the real world, not synthetic).
Be careful with mistaking data for information.
You are getting a digital (maybe lossy compressed) samples of photons and sound waves. It is not unlimited, a camera pointed at a building at night is going to have very little new information from second to second. A microphone outside is going to have very little new information second to second unless something audible is happening close by.
You can max out your storage capacity by adding twenty ML high megapixel cameras recording frames as tiff file but gain little new useful information for every camera you add.
This is where it is a bit confusing for people not familiar with the state of the art.
Some people don't seem to realize how critical the "eval" function is for machine learning.
Raw data is not much more useful than noise for the current recipes of model training.
Human produced data on the internet (text, images, etc.) is highly structured and the eval function can easily be built.
Chess or Go has rules and the eval function is more or less derived or discovered from them.
But the real world?
For driving you can more or less build a computer vision system able to follow a road in a week, because the eval function is so simple. But for all the complex parts, the eval function is basically one bit (you crashed/not crashed) that you have to sip very slowly, and it very inefficient to train such a complex system with such a minimal reward even in simulations.
The real world is governed by physics, so isn't "next state prediction" a sufficient eval function that forces it to internalize a world model? And increasing the timespan over which you predict requires an increasing amount of "intelligence" because it requires modeling the real-world behavior of constituent subsystems that are often black-boxes (e.g. if there is a crow on a road with a car approaching, you can't just treat it as a physics simulation, you need to know that crows are capable of flying to understand what is going to happen).
I don't see how this is any less structured than the CLM objective of LLMs, there's a bunch of rich information there.
> What high quality data sources are not already tapped?
Synthetic data? Video?
> Where does the next 1000x flops come from?
Even with Moore's law dead, we can easily build 1,000x more computers. And for arguments about lack of power - we have sun.
I thought about the huge pile of hard drives in Utah this morning. The TLAs in the USA have a metric shit ton of data that _should_ not be used but _could_ be used.
Even still, we need evolutions in model architecture to get to the next level. Data is not enough.
The price, power, and size. Make it cheap, low power, and small enough for mobile. One way to do this is inference in 4, 2, 1 bit. Also GPUs are parallel, most tasks can be split on several GPUs. Just by adding they you can scale up to infinity. In theory. So datacenters aren't going anywhere, they will still dominate.
Another way is CPU+ + fast memory, like Apple does. It's limited but power efficient.
Looks like with ecosystem development we need the whole spectrum from big models+tools running on datacenters to smaller running locally, to even smaller on mobile devices and robots.
I thought Ilya said we have more companies than we have ideas. He also noted that our current are resulting in models which are very good at benchmarks but have some problems with generalization (and gave a theory as to why).
But I don't recall him actually saying that the current ideas won't lead to AGI.
Then, he starts to talk about the other ideas but his lawyers / investors prevent him from going into detail: https://youtu.be/aR20FWCCjAs?t=1939
The worrisome thing is that he openly talks about whether to release AGI to the public. So, there could be a world in which some superpower has access to wildly different tech than the public.
To take Hinton's analogy of AGI to extraterrestrial intelligence, this would be akin to a government having made contact but withholding the discovery and the technology from the public: https://youtu.be/e1Hf-o1SzL4?t=30
It’s also weird to think that if there is extraterrestrial contact, it will most definitely happen in the specific land mass known as the United States and only the US government will be collecting said technology and hiding it. Out of the entire planet, contact is possible only in the USA.
A problem is that the bulk of the people behind these labs are people that were conditioned from an early age to achieve high scores in standardized tests and conflate that with intelligence. Then apply that mentality to their models resulting in these leaderboards that nobody cares about.
Absolutely this, models acheive very highly on kind problems; ones that you can master with sufficient practice. Which is just remarkable, but the world is a wicked learning environment, and repetition is not enough.
The frenzy around AI is to do with growth fueled cocaine capitalism seeking 'more' where rational minds can see that we don't have that much more runway left with our current mode of operation.
While the tech is useful, the mass amounts of money being shoveled into AI has more to do with the ever escaping mirage of a promised land where there will be an infinite amount of 'more'. For some people that means post scarcity, for others it means a world dominating AGI that achieves escape velocity against the current gridlock of geopolitics, for still others it means ejecting the pesky labour class and replacing all labour needs with AI and robots. Varied needs, but all perceived as urgent and inescapable by their vested interests.
I am somewhat relieved that we're not headed into the singularity just yet, I see it as way too risky given the current balance of power and stability across the planet. The outcome of ever accelerating tech progress at the expense of all other dimensions of wellbeing is not good for the majority of life here.
> The frenzy around AI is to do with growth fueled cocaine capitalism seeking 'more' where rational minds can see that we don't have that much more runway left with our current mode of operation.
When talking with non-tech people around me, it’s really not about “rational minds”, it’s that people really don’t understand how all this works and as such don’t see the limitations of it.
Combine that with a whole lot of FOMO which happens often with investors and you have a whole pile of money being invested.
From what I hear, most companies like Google and Meta have a lot of money to burn, and their official position towards investors is “chances of reaching ASI/AGI are very low, but if we do and we miss out on it, it will mean a huge opportunity loss so it’s worth the investment right now”.
> When talking with non-tech people around me, it’s really not about “rational minds”, it’s that people really don’t understand how all this works and as such don’t see the limitations of it.
What are the limits? We know the limits for naked LLMs. Less so for LLM + current tools. Even less for LLM + future tools. And can only guess about LLM + other models + future tools. I mean moving forward likely requires complexity, research and engineering. We don't know the limits of this approach even without any major breakthrough. Can't predict, but if breakthrough happens it all will be different, but better than (we can foresee) today.
Watch the original Sutskever interview: https://www.youtube.com/watch?v=aR20FWCCjAs
And LeCun: https://www.youtube.com/watch?v=4__gg83s_Do
That said, I use AI summaries for a lot of stuff that I don't really care about. For me, this topic is important enough to spend two hours of my life on, soaking up every detail.
As for being on par with typical surface level journalism. I think we might be further into the dead internet than most people realize: https://en.wikipedia.org/wiki/Dead_Internet_theory
Suppose training is so efficient that you can train state of the art AGI on a few GPUs. If it's better than current LLMs, there will be more demand/inference, which will require more GPUs and we are back at the same "add more gpus".
I find it hard to believe that we, as a humanity, will hit the wall of "we don't need more compute", no matter what the algorithms are.
And from my experience there are lots and lots of jobs that are just "clicking the right buttons".
This is basically all of it.
Kind of how word processors solved the writing is tedious struggle and search solved the "can't curate the internet" struggle.
Ilya did also acknowledge that these houses will still generate gobs of revenue, despite being at a dead end, so I'm not sure what the criticism is, exactly.
Everyone knows another breakthrough is required for agi to arrive; sama explicitly said this. Do you wait and sit on your hands until that breakthrough arrives? Or make a lot of money while skating to where the puck will be?
But what we're seeing at the moment, is a deceleration, not an acceleration.
Maybe they lose relevance. Maybe they miss the breakthrough. That becomes the reason. So perplexity? Sure. Anthropic, even? Yep. Google? OpenAI? Nah.
Regardless, viewing the unit economics, there are very clear sight lines to profitability if they want it. Just like with Amazon, Tesla, Apple, etc., when you want to grow, hoarding cash is a bad play.
As for nvidia, if OpenAI has less leverage, that necessitates a different ai company having more. Who would it be?
So we're rehypothecating CDOs like the last bubble?
Countless examples of companies that strive for profit too early, only to die
They still cost billions to pre-train
Everyone at anthropic is saying ASI is imminent…
[1] https://www.youtube.com/watch?v=MzakqMAaHME
Who exactly is saying this, other than C-level people?
I would not say that any model shows a "crazy" score on ARC-AGI.
I broadly have seen incremental improvements in benchmarks since 2020, mostly at a level I would believe to be below average human reasoning, but above average human knowledge. No one would call GPT-3 godlike and it is quite similar to modern models in benchmarks; it is not a difference of like 1% vs 90%. I think most people would consider gpt-3 to be closer to opus 4.5 than opus 4.5 is to a human.
Though I do not fully know where the boundary between "a model prompted to iterate and use tools" and "a model trained to be more iterative by design" is. How meaningful is that distinction?
But the people who don't get this are the less-technical/less-hands-on VPs, CEOs, etc, who are deciding on layoffs, upcoming headcount, "replace our customer service or engineering staffs with AI" things. A lot of those moves are going to look either really silly or really genius depending on exactly how "AGI-like" the plateau turns out to be. And that affects a LOT of people's jobs/livelihood, so it's good to see the hype machine start to slow down and get more realistic about the near-term future.
I'm not joking.
I like Dario’s view on this, we’ve seen this story before with deep learning. Then we progressively got better regularization, initialization, and activations.
I’m sure this will follow the same suit, the graph of improvement is still linear up and to the right
> The industry is already operating at insane scale.
Sounds a lot like "640K ought to be enough for anybody", or "the market can stay irrational longer than you can stay solvent".
I don't doubt this person knows how things should go but I also don't doubt this will get bigger before it gets smaller.
Possibly... but also a lot of the foundational AI advancements were actually done in skunkworks-like environments and with pure research rather than iterating in front of the public.
It's not 100% clear to me if the ultimate path to the end is iteration or something completely new.
Social contagion is astonishingly potent around ideas like this one, and this probably explains why the zeitgeist seems to be immovable for a time and then suddenly change.
Its charlatans like sama that muddy the waters by promising the sky to get money for their empire building.
LLMs can make and are great great products. But its sneaky salesmen that are the ones saying scaling is the path to AGI. The reality is that they're just aiming for economies of scale to make their business viable
What high quality data sources are not already tapped?
Where does the next 1000x flops come from?
Stick a microphone and camera outside on a robot and you can get unlimited data of perfect quality (because it by definition is the real world, not synthetic). Maybe the "AGI needs to be embodied" people will be right, because that's the only way to get enough coherent multimodal data to do things like long-range planning, navigation, game-playing, and visual tasks.
Be careful with mistaking data for information.
You are getting a digital (maybe lossy compressed) samples of photons and sound waves. It is not unlimited, a camera pointed at a building at night is going to have very little new information from second to second. A microphone outside is going to have very little new information second to second unless something audible is happening close by.
You can max out your storage capacity by adding twenty ML high megapixel cameras recording frames as tiff file but gain little new useful information for every camera you add.
Some people don't seem to realize how critical the "eval" function is for machine learning.
Raw data is not much more useful than noise for the current recipes of model training.
Human produced data on the internet (text, images, etc.) is highly structured and the eval function can easily be built.
Chess or Go has rules and the eval function is more or less derived or discovered from them.
But the real world?
For driving you can more or less build a computer vision system able to follow a road in a week, because the eval function is so simple. But for all the complex parts, the eval function is basically one bit (you crashed/not crashed) that you have to sip very slowly, and it very inefficient to train such a complex system with such a minimal reward even in simulations.
I don't see how this is any less structured than the CLM objective of LLMs, there's a bunch of rich information there.
There is at least one missing piece to the puzzle, and some say 5-6 more breakthrough are necessary.
> Where does the next 1000x flops come from? Even with Moore's law dead, we can easily build 1,000x more computers. And for arguments about lack of power - we have sun.
Even still, we need evolutions in model architecture to get to the next level. Data is not enough.
LLMs can't do jack shit with ciphertext (sans key).
This does not mean he's not an accomplished and very talented researcher.
LeCun was sacked from Meta.
Not sure if it's wise to listen to their advice ...
Earlier:
Ilya Sutskever: We're moving from the age of scaling to the age of research
https://news.ycombinator.com/item?id=46048125
And one of the recent LeCun discussions:
https://news.ycombinator.com/item?id=45897271
Another way is CPU+ + fast memory, like Apple does. It's limited but power efficient.
Looks like with ecosystem development we need the whole spectrum from big models+tools running on datacenters to smaller running locally, to even smaller on mobile devices and robots.
But I don't recall him actually saying that the current ideas won't lead to AGI.
Then, he starts to talk about the other ideas but his lawyers / investors prevent him from going into detail: https://youtu.be/aR20FWCCjAs?t=1939
The worrisome thing is that he openly talks about whether to release AGI to the public. So, there could be a world in which some superpower has access to wildly different tech than the public.
To take Hinton's analogy of AGI to extraterrestrial intelligence, this would be akin to a government having made contact but withholding the discovery and the technology from the public: https://youtu.be/e1Hf-o1SzL4?t=30
It's a wild time to be alive.
Ilya has appeared to shift to closer to Yann's position, though: he's been on the "scaling LLMs will fail to reach AGI" beat for a long time.
Yeah, the actual video with transcripts (YouTube link in bottom of TFA):
https://www.dwarkesh.com/p/ilya-sutskever-2
Ed: TFA is basically a dupe of
https://news.ycombinator.com/item?id=46048125
every LLM easily misaligned, "deceived to deceive" and whatnot and they want to focus on adding MORE ATTACK SURFACE???
and throw more CPU at it?
This is glorious.
time to invest in the pen & paper industry!
While the tech is useful, the mass amounts of money being shoveled into AI has more to do with the ever escaping mirage of a promised land where there will be an infinite amount of 'more'. For some people that means post scarcity, for others it means a world dominating AGI that achieves escape velocity against the current gridlock of geopolitics, for still others it means ejecting the pesky labour class and replacing all labour needs with AI and robots. Varied needs, but all perceived as urgent and inescapable by their vested interests.
I am somewhat relieved that we're not headed into the singularity just yet, I see it as way too risky given the current balance of power and stability across the planet. The outcome of ever accelerating tech progress at the expense of all other dimensions of wellbeing is not good for the majority of life here.
When talking with non-tech people around me, it’s really not about “rational minds”, it’s that people really don’t understand how all this works and as such don’t see the limitations of it.
Combine that with a whole lot of FOMO which happens often with investors and you have a whole pile of money being invested.
From what I hear, most companies like Google and Meta have a lot of money to burn, and their official position towards investors is “chances of reaching ASI/AGI are very low, but if we do and we miss out on it, it will mean a huge opportunity loss so it’s worth the investment right now”.
What are the limits? We know the limits for naked LLMs. Less so for LLM + current tools. Even less for LLM + future tools. And can only guess about LLM + other models + future tools. I mean moving forward likely requires complexity, research and engineering. We don't know the limits of this approach even without any major breakthrough. Can't predict, but if breakthrough happens it all will be different, but better than (we can foresee) today.