Generative physical AI enables autonomous machines to perceive, understand, and perform complex actions in the real (physical) world.
Physical AI refers to models that understand and interact with the real world using motor skills, and they're often housed in autonomous machines, such as robots or self-driving vehicles.
Generative AI models—large language models such as GPT and Llama—are trained on enormous amounts of text and image data, largely gathered from the Internet. These AIs have astonishing capabilities in producing human language and abstract concepts, but they're limited in their grasp of the physical world and its rules.
Generative physical AI extends current generative AI with understanding of spatial relationships and physical behavior of the 3D world we all live in. This is done by providing additional data that contains information about the spatial relationships and physical rules of the real world during the AI training process.
The 3D training data is generated from highly accurate computer simulations, which serve as both a data source and an AI training ground.
Physically-based data generation starts with a digital twin of a space, such as a factory. In this virtual space, sensors and autonomous machines like robots are added. Simulations that mimic real-world scenarios are performed, and the sensors capture various interactions like rigid body dynamics—such as movement and collisions—or how light interacts in an environment.
Reinforcement learning teaches autonomous machines skills in a simulated environment to perform in the real world. It allows autonomous machines to learn skills safely and quickly through thousands or even millions of acts of trial and error.
This learning technique rewards a physical AI model for successfully completing desired actions in the simulation so the model continuously adapts and improves. With repeated reinforcement learning, autonomous machines eventually adapt to new situations and unforeseen challenges appropriately, preparing them to operate in the real world. Over time, an autonomous machine can develop sophisticated fine motor skills needed for real-world applications, such as neatly packing boxes, helping to build vehicles, or navigating environments unassisted.
Previously, autonomous machines were unable to perceive and sense the world around them. But with generative physical AI, robots can be built and trained to seamlessly interact with and adapt to their surroundings in the real world.
To build generative physical AI, teams need powerful, physics-based simulations that provide a safe, controlled environment for training autonomous machines. This not only enhances the efficiency and accuracy of robots in performing complex tasks, but also facilitates more natural interactions between humans and machines, improving accessibility and functionality in real-world applications.
Generative physical AI is unlocking new capabilities that will transform every industry. For example:
Building the next generation of autonomous machines using generative physical AI involves a coordinated process across multiple, specialized computers: