I am very new to robots and recently decided to go crazy and buy a humanoid from Unitree. I've spent a while building some tools in order to integrate LLMs with vision capabilities that can control the robot motion in a simple manner... and while I saw Opus and Astra crashing against tables, beds and surfaces (yeah, my LIDAR data isn't being used yet), Argon 4 takes a lot of precautions to avoid crashing against walls and other objects! I told my robot to go to a table, and the guy started walking slowly to the table, started steering at it and described the objects in it. Pretty scary for being something I've vibecoded in a matter of hours at home...
I'm a robotics noob currently building a prototype of a snow sweeping robot (www.frost-e.com) using my own design and I'm encouraged to see similar work coming from real places. Namely:
- Zero shot control using a visual harness and action primitives. I've seen this from a few other places as well (https://tml.stanford.edu/homebody/) and it's essentially having a generalist LLM command a robot via a set of predefined actions. Many robots and many tasks can probably be done with this approach, since not everything needs super intricate 6DoF arms or dexterity
- simulation bring-ups: I have a pipeline that takes in satellite imagery and reconstructs the lots and houses in a 3D environment and snows on it so that my virtual robot can learn to clean. With these latest models, you can probably create a higher fidelity digital twin more directly.
- self-improvement: These models (Argon, Astra, etc) can't operate the robot itself, but would be able to create data to distill into smaller models you can actually utilize.
Assuming that models continue to get 90% more efficient every 18 months, and assuming that hardware continues to get 2x more powerful per watt every 2 years...
Aren't we still like a decade away from an Argon model being able to run on battery power for any reasonable amount of time that would fit on a robot that isn't the size of a school bus?
Add on another couple of years for the speed to get to what it likely needs to be to be generally useful instead of narrowly useful?
I mean - I guess we're maybe 5-7 years away from a reasonably priced robot you could plug into the wall like an old vacuum cleaner (that runs on this type of tech).
I think for 95%+ of applications.. you just need a link to a data center anywhere.
My assumption is the hardware will have one/several smaller models running on it for locomotion, balance, etc.
Then the hardware "phones home" for decision making and planning.
Essentially the planning doesn't need to be close to the device as long as the local models can handle some part of the plan execution and make sure the robot doesn't fall over. The latency shouldn't be an issue. In fact more powerful centralized compute quickly creates an advantage over slower local compute in terms of end to end planning.
> Aren't we still like a decade away from an Argon model being able to run on battery power for any reasonable amount of time that would fit on a robot that isn't the size of a school bus?
Sooner or later, the models will stop improving. After that happens, the best models will be etched into model-specific ASICs with weights fixed in mask ROM (like Taalas is already doing), which will result in massive, at least order of magnitude inference efficiency increase.
They said the same thing for transistor density in new chip nodes. It took 40 years of doubling every 2 years to finally hit a wall.
And although the industry hit a wall in density/area, they switched to scaling in 3D or glueing chiplets together. Moore's Law is still working if you change the definition to transistors/integrated circuit.
You can already run top of the line models on something that isn't the size of a school bus. The couple of KW you need to run them are the bigger problem, though on very high end hardware you might get double the efficiency.
Robots stand-alone AI do not really benefit from things like batch mode so you want to keep your LLM as high level as possible and your robot's subsystems as autonomous as possible, very light coupling between the two, definitely nothing time-critical. And that's the challenge: in robotics almost everything is time critical.
I'm naive, but wouldn't industrial applications be enabled by this anyway? You'll have in-house inference hardware that a robot can talk to via local network.
Also, your model probably doesn't need like a lot of its knowledge to control a robot.
That's only bringing up the horizon maybe 1 year. At industrial scale - I guess something like an Amazon warehouse could start to be interesting with a local datacenter IN the warehouse within 2-3 years.
From a cost perspective, I just can't imagine that heavily, heavily specialized models wouldn't crush a general model, though - which Amazon already does.
I wonder to what degree that holds up for very limited domains. You have to balance model size, power budget and cost which is a completely different ballgame than looking at a stationary server with a near infinite power budget and where the costs can be amortized over much larger numbers of hits.
Robotics tends to be at the worst possible intersection point of all three of those: limited power budget, relatively small model size and cheap enough to be affordable when embedded into an appliance.
- Zero shot control using a visual harness and action primitives. I've seen this from a few other places as well (https://tml.stanford.edu/homebody/) and it's essentially having a generalist LLM command a robot via a set of predefined actions. Many robots and many tasks can probably be done with this approach, since not everything needs super intricate 6DoF arms or dexterity
- simulation bring-ups: I have a pipeline that takes in satellite imagery and reconstructs the lots and houses in a 3D environment and snows on it so that my virtual robot can learn to clean. With these latest models, you can probably create a higher fidelity digital twin more directly.
- self-improvement: These models (Argon, Astra, etc) can't operate the robot itself, but would be able to create data to distill into smaller models you can actually utilize.
Aren't we still like a decade away from an Argon model being able to run on battery power for any reasonable amount of time that would fit on a robot that isn't the size of a school bus?
Add on another couple of years for the speed to get to what it likely needs to be to be generally useful instead of narrowly useful?
I mean - I guess we're maybe 5-7 years away from a reasonably priced robot you could plug into the wall like an old vacuum cleaner (that runs on this type of tech).
My assumption is the hardware will have one/several smaller models running on it for locomotion, balance, etc.
Then the hardware "phones home" for decision making and planning.
Essentially the planning doesn't need to be close to the device as long as the local models can handle some part of the plan execution and make sure the robot doesn't fall over. The latency shouldn't be an issue. In fact more powerful centralized compute quickly creates an advantage over slower local compute in terms of end to end planning.
Sooner or later, the models will stop improving. After that happens, the best models will be etched into model-specific ASICs with weights fixed in mask ROM (like Taalas is already doing), which will result in massive, at least order of magnitude inference efficiency increase.
And although the industry hit a wall in density/area, they switched to scaling in 3D or glueing chiplets together. Moore's Law is still working if you change the definition to transistors/integrated circuit.
Robots stand-alone AI do not really benefit from things like batch mode so you want to keep your LLM as high level as possible and your robot's subsystems as autonomous as possible, very light coupling between the two, definitely nothing time-critical. And that's the challenge: in robotics almost everything is time critical.
Also, your model probably doesn't need like a lot of its knowledge to control a robot.
From a cost perspective, I just can't imagine that heavily, heavily specialized models wouldn't crush a general model, though - which Amazon already does.
Robotics tends to be at the worst possible intersection point of all three of those: limited power budget, relatively small model size and cheap enough to be affordable when embedded into an appliance.
It just seems like jumping the shark a little bit.
It's basically completely unviable for a few years still.
I don't think it's viable in the sense that they are still in a demo stage.
Also, I would assume these models are going to get bigger before they get smaller (but they will get smaller over the long run)
For a set of tasks that almost no one would use a robot for...
I think someone should make a game based on this concept.