Use GPT-6 Astra as the robot's "brain"?
It actually works!
Recently, an SO-101 robotic arm, priced at approximately $150 and controlled by GPT-6 Astra, drew a depiction of the Golden Gate Bridge.
This wasn't drawn along a pre-set path—Astra first planned where to begin, then used its camera to observe and adjust in real time as it drew, ultimately completing a red-and-blue Golden Gate Bridge.
This video went viral instantly, and Sam Altman directly shared it: "Give me one too!"
Video link: https://x.com/cdngdev/status/2097339677128982873
In recent years, the robotics community has been discussing a question: “Who will become the OpenAI of robotics?”
Today, the answer to this question may very well be OpenAI itself.
How capable is Astra in practice?
Painting the Golden Gate Bridge looks impressive, but that’s not the most impressive one yet.
Wenli Xiao from CMU Robotics conducted a very intuitive experiment.
They first record a video of a human completing a new task, feed the footage into Codex, and then have GPT-6 Astra control the robotic arm to perform the same task in the same way.
It worked on the first try!
Video link: https://x.com/_wenlixiao/status/2097801944119349455
Wenli Xiao calls it "physical ICL"—while large models previously learned new tasks temporarily from textual and code examples in context, people have now discovered that this capability also works in the physical world.
Another group simply had Astra start working as a "robot engineer."
Lingxiao Guo provided Astra with multi-view video and motion data from a real robot demonstration, allowing it to autonomously handle camera calibration, scene reconstruction, and physical parameter estimation, then transfer the real-world environment into MuJoCo.
The end result is not just a 3D scene that looks similar, but a real2sim environment capable of further contact simulation and motion replay.
Video link: https://x.com/Lingxiao234/status/2096992059731443923
Some people on X have connected Astra to more complex robotic bodies.
Dmytro Hrybov equipped Astra with a Kinova Gen3 robotic arm and a Shadow Hand five-fingered gripper in MuJoCo, enabling it to hold a pencil and draw Picasso’s famous dove.
For the pen to truly make contact with the paper, the robotic hand must also manage the grip posture, friction between the fingers and the pencil, and the contact between the pen tip and the paper surface.
Video link: https://x.com/dimentary/status/2097141042214797801
If the above still seems like internet users pulling pranks, this one truly highlights some issues.
RoboCurve connected GPT-6 Astra to two real I2RT YAM robotic arms.
With every action, Astra sees three camera perspectives—on the top and both wrists—as well as the status of the robotic arms. It then directly outputs the x, y, z, yaw, pitch, roll, and gripper status for the end effectors of both arms, which are subsequently converted into specific joint movements by the underlying inverse kinematics.
The test task was simple: pick up the red block on the table and place it into the bowl beside it. Conducted 20 times, Astra succeeded 19 times, achieving a 95% success rate. In the same test, Fable 5.1 succeeded only 8 times.
No one expected a general-purpose model, not specifically trained for robots, to achieve this level.
And this issue coincides precisely with one of the most popular approaches in the robotics industry over the past few years—training robots with their own dedicated "foundation model brain."
The "robot brain" doesn't need to be trained from scratch.
Over the past few years, as large models have expanded their capabilities into vision, speech, and agents, the robotics industry has repeatedly asked the same question: If GPT has emerged in the language domain, when will robots have their own "GPT"?
Physical Intelligence and Skild quickly became star companies in response to these expectations.
Physical Intelligence aims to train a general robotic foundation model that is truly cross-task, cross-scenario, and cross-robot-body. Its π-series models integrate vision, language, and action into a single framework, enabling robots to learn how to interact with the real world from vast amounts of embodied experience, just as large models understand text.
Skild’s goal is more direct: “Any task, any robot, one brain.” Whether it’s a quadruped robot, a humanoid robot, a desktop robotic arm, or a mobile manipulation platform, all are controlled by the same universal “brain” to understand tasks and plan actions.
The two companies represent the most typical approach in recent years to robotic foundation models: just as language has its own foundation model, robots also need a foundational model trained on vast amounts of robotic data to truly understand bodies and actions.
Therefore, "Who will become the OpenAI of robotics?" has always been an unavoidable question in this space.
But with the emergence of Astra, this question suddenly had a somewhat unexpected answer.
Zeeshan Zia, Principal Scientist at Amazon Alexa and co-founder of Retrocausal, has an observation: “The OpenAI of robotics may well be OpenAI itself, rather than Physical Intelligence or Skild.”
This sounds somewhat provocative, as Astra has not followed the traditional path of robotics companies.
Instead of first collecting large amounts of robotic trajectory data and then training a specialized vision-to-action foundation model, Astra offers another possibility: if general AI is already powerful enough, is it still necessary to build a robot’s brain from scratch?
This issue is particularly nuanced when it comes to OpenAI, as it actually developed robots quite early on.
In 2018, OpenAI introduced Dactyl, enabling the Shadow Dexterous Hand to rotate objects in the real world, and later further enabled the dexterous hand to solve a Rubik’s Cube.
At that time, OpenAI was already researching issues that the robotics industry still cannot avoid today: how to train in simulation and transfer those capabilities to the real world, how to handle friction, occlusion, sensor noise, and complex multi-degree-of-freedom control.
But a few years later, OpenAI shut down its robotics team. At the time, Wojciech Zaremba highlighted a very practical limitation: robots lack access to large-scale data comparable to internet text. Unlike text and images that can be readily obtained in vast quantities from the internet, robotics data is expensive, slow to collect, and difficult to make sufficiently rich and representative of the real world.
Interestingly, OpenAI once stepped away from robotics due to data issues, but may now be circling back through another path.
Over the past few years, OpenAI first scaled models across language, code, vision, and agents, enabling them to accumulate vast knowledge about objects, space, causality, and human behavior. Next, robots need to reliably transform this existing knowledge into physical actions.
OpenAI itself has clearly refocused on this direction. Sam Altman has recently made it clear that OpenAI will develop humanoid robots and explore other robot forms. In his view, the truly critical part remains the "brain" that enables robots to function.
How does Astra control the robot's "body"?
The current solution does not involve Astra directly controlling each motor.
Using RoboCurve’s real-world testing as an example, Astra observes three camera perspectives and the robotic arm’s own state at each step, then directly outputs the target position and orientation for the end effector, including x, y, z, yaw, pitch, roll, and gripper status; these targets are subsequently converted by the underlying inverse kinematics system into specific joint movements.
This is actually a typical division of labor: Astra handles determining "where the hand should go," while the traditional robotic control stack handles "how each joint should move specifically."
Yu Xiang, a former research scientist at NVIDIA, also suggested that a key next direction in robotics may be integrating the most advanced AI models directly into manipulation. Rather than leaving large models at the top level to only decompose tasks and call tools, Astra is already intervening more directly in robotic action decisions.
Link: https://x.com/YuXiang_IRVL/status/2096074174091362388
But as you move forward, issues gradually begin to emerge.
Some even had Astra directly output Unitree Go1’s joint targets, attempting to control the quadruped robot’s walking at 50 Hz like a reinforcement learning policy. The experiment did work, but because Astra’s inference speed couldn’t keep up with real-time control, the physical simulation had to pause between each model call. In the end, 250 inferences yielded only about five seconds of walking.
Link: https://x.com/sri299792458/status/2097349207795335424
Clearly, Astra has already been able to determine what the robot should do next, but giving it direct control over millisecond-level底层 control is still not realistic.
Therefore, a more realistic future形态 may not involve a single large model handling everything from start to finish, but rather maintaining division of labor across different levels: Astra handles environmental understanding, reasoning, and planning, while lower-level policies and controllers rapidly and stably translate those intentions into actions.
The last millimeter
Planning an action is still quite far from actually executing it well.
Still the same set of tests from RoboCurve.
In the task of placing the red block into the bowl, Astra succeeded 19 out of 20 times, achieving a 95% success rate; however, when the task changed to accurately fitting a circular puzzle piece into its corresponding slot, the success rate immediately dropped to 10%, with only 2 successful attempts out of 20.
In the same real-world test, Astra achieved a 95% success rate on the block-into-bowl task, but its success rate dropped to 10% on the puzzle task requiring precise insertion. Source: RoboCurve
Many are just one step away from success. Astra has found the piece, grabbed the knob, and moved it close to the target slot, but struggles to complete the final precise insertion.
These final few millimeters reveal some of the most challenging issues for robots: slight deviations in position or angle prevent parts from aligning; actual contact is disrupted by hardware inaccuracies, making the next action impossible to complete.
This is also why Physical Intelligence recently focused specifically on the "last mile."
In their experiments, it was straightforward for a general-purpose robot model to pick up a screwdriver, but achieving fast and precise alignment with a small M3 screw still requires further optimization through online reinforcement learning; tasks such as inserting an Ethernet cable, plugging in a power cord, and securing zip ties are also stuck at similar stages of fine contact.
Astra's most obvious advantages currently lie in the "mental" tasks of understanding the environment, assessing goals, and planning actions, but when it comes to high-frequency feedback, fine-tuned contact, and force control, the robot's own bodily experience remains crucial.
This also leaves a crucial悬念 surrounding the earlier debate over "OpenAI in the field of robotics."
Robots can certainly use the most powerful models as their "brain," but a physical body to match has not yet emerged.
This article is from the WeChat official account: MachineHeart, authored by MachineHeart.