LLMs Can Select Robot Actions, but That Is Not Enough to Replace Real-Time Control

The Debate Needs to Address Specific Stages of Control

The WeChat account argues that LLMs beginning to generate robot actions could unsettle the division of labor between task planning and action models. The GPT-6 Astra research does provide evidence relevant to this question, but “generating actions” needs to be broken down: in the experiments, a frozen language model selects end-effector targets, which are then executed through a non-learning path-conversion program, after which the model receives observations and error feedback. I therefore see this research as supporting the entry of general-purpose models into action decision-making, but not as sufficient proof that they have taken over the entire control chain. Whether a separate robotics model is still needed depends on which stages have been covered and which capabilities are still supplied by external systems.

Evidence: [1] [7]

The Results Deserve Attention, and the Comparison Conditions Must Be Retained

The 2026-09-21 version of the Astra paper reports an average score of 28.97 and a success rate of 22.48% with medium reasoning in a zero-shot setting. The score captures partial progress, while success requires completing the entire task; the overall metrics are averaged with equal weighting across five capability dimensions. Astra outperformed the entries selected by the paper from the public policy leaderboard dated 2026-09-10. However, Astra received task steps and descriptions of scoring stages compiled from the wiki, while the baselines were not rerun with equivalent text inputs. These results are sufficient to show that the approach merits further research, but they cannot, on their own, answer how much of the advantage comes from the model itself and how much comes from the additional task instructions.

Evidence: [1] [2]

Feedback Already Exists; Reliable Execution Remains Unproven

The WeChat account suggests that better checking and error-correction mechanisms could further improve results. However, the paper's execution framework already returns observations after actions, errors and information about refusals to execute, so the existing results cannot be described as action outputs with no feedback at all. More critically, the paper still reports weaknesses in fine manipulation, dynamic tasks and dual-arm coordination. Physical-robot testing was stopped for safety reasons, and the formal protocol was not completed. RoboDojo's simulation and physical-robot tasks were also not paired task by task. Direct evidence is therefore still lacking for inferring reliability in real deployment from simulation performance; any gains from adding feedback also require experimental confirmation.

Evidence: [1] [2] [7]

Real-Time Performance Is a Capability of the Whole System

Physical Intelligence's RTC work illustrates another type of problem: while the model infers the next action segment, the robot can continue executing its current action chunk, but switching directly may cause discontinuities. RTC preserves actions already committed to execution and coordinates the overlapping portions, allowing inference and execution to run in parallel. This shows that judging whether the next action is appropriate and ensuring that actions connect in time are problems that need to be addressed separately. Its physical-robot experiments used a workstation in the same building and a local area network. These results cannot establish that internet-based cloud control is already reliable, nor can the specific measured latency be generalized into a fixed constraint for all robotic systems.

Evidence: [3] [4]

The Division of Labor Between Models Should Be Tested Against Task Requirements

The original Helix, disclosed by Figure on 2025-02-20, provides a concrete example: onboard System 2 runs at 7–9 Hz and uses asynchronously updated semantic latent variables to guide System 1, which outputs actions at 200 Hz. This division of labor corresponds to different response rhythms, and both systems run onboard. These frequencies are specific to that architecture and cannot serve as universal thresholds for robots. My inference is that even if general-purpose models can take on more action decisions, whether to merge control layers should still be determined by response time, action continuity and task performance. Greater model capability alone cannot prove that any particular architecture will necessarily prevail.

Evidence: [5] [4]

Data Interfaces Still Affect Action Quality

UMI's experiments further show that execution quality also depends on how demonstrations are represented and how they interface with the robot. Its interface uses relative trajectories and matches observation and action-execution latency. Removing latency matching in dynamic throwing made jitter more pronounced; removing the relative pose between grippers in dual-arm clothes folding worsened coordination. These results support a limited judgment: even when the intended action is already specified, the handling of temporal and spatial relationships still changes performance. They do not prove that UMI is superior across all tasks, nor do they validate a universal conversion between hours of human video and hours of UMI data. Determining how much specialized capability general-purpose models can replace still requires comparing complete systems under the same inputs, execution interfaces and task conditions.

Evidence: [6] [1]

What to watch next

• Whether Astra retains its comparative advantage after equivalent task instructions are provided and the baselines are rerun.
• Whether performance in fine manipulation, dynamic tasks and dual-arm coordination supports a deployment judgment once formal physical-robot testing is completed.
• How response time, action continuity and full-task success rates change for unified models and hierarchical control approaches under the same task and hardware conditions.
• Whether the benefits of UMI's latency matching and relative-pose representation can be reproduced across more tasks and robot platforms.