Sergey Levine: General robotic foundation models may outperform narrow solutions, the future of medicine involves autonomous robots, and the importance of understanding physical interactions | Invest Like the Best

Key takeaways

• Developing general robotic foundation models may be more efficient than creating narrow, domain-specific solutions.
• Broader data sources can enhance the performance of robotic foundation models compared to narrow solutions.
• Understanding the world is crucial for robotics to generalize across tasks and systems.
• A general-purpose embodied foundation model could trigger a rapid expansion of robotics applications.
• Robots should be designed as tools for specific tasks rather than resembling humans.
• Future robotics in medicine may not be limited to human-like forms or human control.
• Cost-effective training and handling long-tail scenarios are historical challenges in robotic learning.
• Multimodal language models can help robots acquire common sense knowledge, but grounding it in physical situations is challenging.
• AI systems can improve with practice, but creating a general system for new environments is difficult.
• Combining generative AI with reinforcement learning is essential for advancing robotic control.
• Robotic systems need to handle diverse applications efficiently without requiring large amounts of data for each.
• The future of robotics may involve autonomous systems that operate independently of human control.
• Designing robots as specialized tools can lead to more effective solutions for specific tasks.
• The integration of AI in robotics requires overcoming the challenge of applying theoretical knowledge to real-world scenarios.
• The development of versatile robotic systems relies on a foundational understanding of physical interactions.

Guest intro

Sergey Levine is an Associate Professor in the Department of Electrical Engineering and Computer Sciences at UC Berkeley and co-founder of Physical Intelligence. He earned his PhD in Computer Science from Stanford University in 2014 and joined UC Berkeley faculty in 2016. His research pioneered deep reinforcement learning algorithms for robotics, enabling end-to-end training of neural network policies that combine perception and control.

The importance of generality in robotic models

• Developing robotic foundation models for general tasks may be easier than creating narrow, domain-specific solutions. “Part of the thesis of this company is that we believe that doing it at the full level of generality might actually in the long run be easier than trying to special case very specific narrow application domains.” – Sergey Levine
• Robotic foundation models can leverage broader data sources to outperform narrow solutions. “The reason that language models took over for all of those different application domains is because they can leverage much broader sources of data.” – Sergey Levine
• Understanding the world is crucial in robotics, as it allows models to generalize across different tasks and physical systems. “This notion of understanding the world if anything is actually more important in robotics because if you have many different tasks maybe even many different physical systems then you can go from training individual dishwashing specialists or laundry folding specialists and instead train a model that actually understands physical interaction.” – Sergey Levine
• A general-purpose embodied foundation model will enable a Cambrian explosion of robotics applications. “I think something like that might happen in the world of robotics but it can’t happen today because if you want to put together some cool new robotics application… you need to basically solve the intelligence problem.” – Sergey Levine
• Robotic learning has historically struggled with the need for cost-effective training and handling long-tail scenarios. “Historically what has been really difficult in robotic learning is that you need a system that handles the application you want to address that is cost effective to train for the application meaning that you don’t need like a huge amount of data for every single application you want to tackle.” – Sergey Levine
• AI systems can improve continuously through practice, but creating a general system that adapts to new environments is challenging. “I’ve always wanted to really figure out how to get AI systems that get better and better the more they do things… it’s very hard to turn that into a general system that can work in open world settings.” – Sergey Levine
• Combining generative AI with reinforcement learning is essential for advancing robotic control. “The big challenge… has to combine those threads how to bring in all that knowledge that you get with generative ai but also go beyond just human level performance with reinforcement learning.” – Sergey Levine
• Multimodal language models can help robots acquire common sense knowledge, but grounding that knowledge in physical situations remains a challenge. “It turns out that multimodal language models are really good at pulling in knowledge and trying to articulate that knowledge… but they’re not very good at grounding that knowledge in physical situations.” – Sergey Levine

The future of robotics in medicine and design

• In the long run, we could see robots in medicine and surgery that are not limited to human-like forms or human control. “In the long run I think there’s lots of really exciting applications in medicine and surgery where we not only might in the long run not be limited to robots that look like humans we might not be limited to robots that can even be controlled by humans.” – Sergey Levine
• Robots do not need to resemble humans; they should be designed as tools for specific tasks. “The cool thing about being able to build robots is that ultimately they don’t have to be constrained to look like humans at all you can build the right tool for the job.” – Sergey Levine
• The future of robotics may involve autonomous systems that operate independently of human control.
• Designing robots as specialized tools can lead to more effective solutions for specific tasks.
• The integration of AI in robotics requires overcoming the challenge of applying theoretical knowledge to real-world scenarios.
• The development of versatile robotic systems relies on a foundational understanding of physical interactions.
• Future robotics in medicine may not be limited to human-like forms or human control.
• Robotic systems need to handle diverse applications efficiently without requiring large amounts of data for each.
• The potential for autonomy in robotics suggests a future where robots can function without direct human intervention.
• The design philosophy in robotics emphasizes functionality and task-specific solutions over humanoid appearance.
• Autonomous robotic systems could revolutionize fields like medicine by providing new capabilities and efficiencies.
• The shift towards non-humanoid robotic designs reflects a focus on practicality and effectiveness in task execution.

Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our Editorial Policy.