Gemini 4 Argon marks a step change in bodily understanding, code technology, and visual-semantic reasoning—unlocking new workflows throughout the robotics stack and accelerating progress towards bodily AGI.
You’ll be able to see this leap in functionality on the Embodied Reasoning (ERQA) benchmark, which assessments a mannequin’s skill to grasp and resolve bodily, spatial issues in 3D environments. Utilizing pure reasoning, Argon scores 87%, outperforming each different frontier mannequin.
As a general-purpose bodily reasoning mannequin, Argon might be utilized throughout the whole robotics lifecycle. On this publish, we share consultant prompts for Argon throughout 4 key areas:
- Robotic management → Cell manipulation with MuJoCo apply, zero-shot robotic management, and closed-loop steering of video world fashions.
- Real2Sim → Reconstructing bodily calibrated simulation twins from robotic digicam feeds, selfish human video, or hand-drawn paper sketches.
- Bodily understanding → Agentic multimodal hindsight labeling together with detailed understanding of dexterity, dense worth capabilities and progress estimation.
- Analysis and self-improvement → Motion enchancment through simulation and iterative self-improvement on {hardware}.
Unlocking robotic management with Argon
1. Spot cell manipulation: Placing within the trash bin with MuJoCo apply
Argon controls a Boston Dynamics Spot® quadruped (12-DoF legs + 6-DoF arm) to study a brand new ability: stepping on a foot-pedal trash bin to throw away soda cans. Argon creates an articulated MuJoCo digital twin of a foot-pedal trash bin to apply its strikes earlier than stepping on the bodily pedal and dropping soda cans into the open bin.
Spot Cell Manipulation — Actual-to-Sim-to-Actual foot-pedal trash bin disposal: Spot choosing up the soda can, checking its wrist digicam, training 3-leg tripod steadiness in MuJoCo, stepping on the foot pedal, verifying hand centering within the open bin, and dropping the can inside.
There is a Weight loss program Coke can behind you and one other on the black desk in your left.
Use the spot-mobile-manipulation ability to select them up and trash them within the black foot-pedal bin.
Earlier than stepping down on the pedal, confirm within the entrance cameras that your front-right foot goal is centered on the pedal, and use a 2-agent verification swarm (foot-verifier and hand-verifier) to confirm each the foot-pedal step and the 3D hand-in-bin alignment earlier than releasing.Plain textual content
2. Zero-shot management
Argon controls a easy robotic arm to put a fallen mug upright and put the crumpled serviette inside. Argon causes with a visible coordinate harness and motion primitives to carry out this activity over a number of turns.
Left: Third-person actual robotic view; Proper: calibrated wrist digicam with projected 2 cm world coordinates and jaw-closing reticle.
[Attached: wrist_cam.png, obs.json]
Activity: Get the robotic arm to reorient the mug upright and put the serviette contained in the mug.
## Execution Self-discipline
- Merely output 1 sentence remark adopted instantly by your `curl` command (with `# Plan:` on Flip 1 and `# Thought:` on each flip). Every thing wanted is in `obs.json` and the calibrated reticle/grid drawn on `wrist_cam.png`.
## Visible Coordinate Reference (Annotated on wrist_cam.png)
- `+X` (UP in picture): Away from robotic base | `-X` (DOWN): In the direction of robotic base
- `+Y` (LEFT in picture): World left | `-Y` (RIGHT): World proper
- `+Z`: Peak above desk | `Yellow numbers`: World (X, Y) in centimetres (2 cm grid at `grid_z`)
- `Cyan + crosshair`: Present 3D pinch middle
- `Cyan [ + ] brackets`: Present jaw opening and shutting line
- `Inexperienced line`: Present object long-axis orientation (`yaw_deg`: 0° = vertical alongside +X, 90° = horizontal alongside +Y)
- `pitch_deg = 90°`: Straight down (-Z) top-down grasp/place; `pitch_deg < 60°`: Tilted coaxial method alongside finger axis
## Closed-Loop Flip Workflow
1. Examine present state & view `wrist_cam.png`:
`curl -s http://127.0.0.1:8081/obs` (returns `ee_pos`, `holding`, `standing`, `workspace`, `wrist_cam_path`).
2. Choose VLM name granularity upfront on Flip 1:
- Coarse (2–4 calls): `look_at` -> `grasp` -> `place` -> `performed`
- Pre-Grasp Hover Confirm (4–7 calls): `look_at` -> `goto(z=0.04)` (confirm cyan `[ + ]` brackets straddle goal) -> `grasp` -> `place_until_contact` -> `performed`
- Positive Visible Servoing (8–20 calls): `look_at` -> `grasp` -> pre-pose `goto` -> closed-loop `move_by` / `move_until_contact` -> `performed`
3. Ship movement primitives (`POST /plan`) and confirm visually after each flip within the up to date `wrist_cam.png`. If `standing == "grasp_failed"`, name `robotic.look_at(...)` to re-observe and get well closed-loop.
## Code-as-Insurance policies Primitives (`POST http://127.0.0.1:8081/plan`)
robotic.look_at(x=0.35, y=0.13, z=0.14, grid_z=0.015)
x, y = robotic.deproject(u=0.52, v=0.48, z_plane=0.015)
robotic.grasp(x=x, y=y, z=0.015, yaw_deg=-35, pitch_deg=90)
robotic.place(x=0.24, y=0.06, z=0.075)
robotic.place_until_contact(x=0.24, y=0.06, z=0.015, max_lag=0.012, aid=0.002)
robotic.goto(x=0.28, y=0.05, z=0.04, arc_z=0.14, pitch_deg=25, azimuth_deg=90, roll_deg=0, pace=0.10)
robotic.move_by(ahead=0.025, dz=-0.002, dpitch_deg=-3, droll_deg=5, pace=0.04)
robotic.move_until_contact(dz=-0.06, pace=0.03, max_lag=0.012, aid=0.002)
robotic.open_gripper() / robotic.close_gripper() / robotic.residence() / robotic.performed()Plain textual content
3. Omni management
Argon steers robots within the Gemini Omni World Mannequin to rearrange 5 scattered Scrabble tiles (G, R, A, N, O) right into a clear row spelling A - R - G - O - N. As a substitute of specifying one static immediate, Argon steers in a closed visible loop. It writes timestamped motion prompts, inspecting intermediate rollout keyframes, and refining its prompts.
Iterative Video World Mannequin Steering Dexterous Fingers: 33-Step Steered Video Rollout Spelling A-R-G-O-N.
Purpose: {task_goal}
You might be steering a video world mannequin (Gemini Omni) in an iterative visible suggestions loop. Examine the hooked up frames from the newest rollout and write the following timestamped motion immediate for the video world mannequin:
- Break complicated motions into one easy, single-direction motion at a time.
- Preserve inactive fingers retracted and keep away from blocking the digicam's view of objects within the workspace.
- Explicitly state which areas of the workspace grow to be empty after an object strikes.
- Use [MM:SS.mmm] timestamps for every motion beat and wrap your output in tags.
Instance format (pattern timestamps solely, not the total immediate):
[00:00.000] Character A's left hand rests on the aspect whereas the best hand reaches towards the goal object.
[00:02.500] Character A's proper fingertips push the article easily into the middle of the desk, leaving its beginning space empty.
[00:05.000] Character A's proper hand opens and retracts to the aspect so all positioned objects stay in clear view of the digicam.
Plain textual content
Unlocking Real2Sim with Argon
1. Robotic view to simulation
Argon views an actual robotic teleoperation episode and develops a digital twin; it obtains or creates simulation belongings that look and behave like the actual objects upon contact with the dexterous fingers. Argon tunes the belongings and scenes to match sim visuals and phone occasions with these in the actual recorded information in an iterative visual-reasoning loop.
Left (Actual View): synchronized actual robotic digicam feeds; Proper (Simulated): 3D MuJoCo scene reconstructed and calibrated by Argon.
[Attached: robot_rollout_episode]
Construct a MuJoCo scene of the robotic and reconstructed activity objects. When every episode is replayed body by body, by driving the robotic joints towards the logged remark.state with joint-space PD.
Iterate utilizing two verification targets:
(1) Visible match: MuJoCo renders from the head_left, left_wrist, and right_wrist cameras line up with, and appear like, these actual observations.
(2) Bodily match: the simulated contact drive/torque on every Sharpa fingertip, within the sensor body, matches the logged tactile_force in timing, magnitude and route.
What you may tune occurs earlier than simulation begins:
(1) Static object belongings: life like geometry, collision shapes, mass, inertia, friction and joints, with look near the actual objects, plus the preliminary object poses at t = 0.
(2) Offline-planned robotic instructions, modified solely in frames the place the actual fingertips are close to an object. Every command should keep inside 0.55 rad (arm joints) and 0.65 rad (hand joints) of the logged teleop trajectory. In every single place else, replay the logged trajectory unchanged.
No sim dishonest: activity objects might transfer solely by means of actual MuJoCo contact with the robotic fingers and the setting (mj_step).
Not allowed after t = 0: altering MjModel parameters; making use of exterior forces or torques to things; welds or equality constraints; feedforward torques past joint-space PD; setting or rolling again object qpos/qvel; disabling or excluding collisions; reporting contact forces that do not come from the MuJoCo solver (for instance, copying the logged tactile readings).
Instantly current the reconstructed scene and rollout outcomes as an interactive HTML session viewer with one web page per episode, exhibiting:
- On the backside: A synced timeline (play, scrub, pace management) that drives each panel.
- On the left panel: An interactive 3D Blender scene exhibiting the simulated robotic and objects, exhibiting contact factors and drive arrows as replay progresses.
- On the best panel: Actual vs. sim digicam views aspect by aspect for all 3 cameras. Actual vs. sim finger drive visualizations.Plain textual content
2. Human view to simulation
Given an selfish human video of a manipulation activity, Argon identifies the task-relevant objects, replaces crude bounding containers with compound convex-decomposed meshes for concave containers, and retargets the human hand trajectory onto a 76-DoF bimanual humanoid with 22-DoF five-fingered fingers in MuJoCo contact physics.
Human View to Simulation — Selfish Human Video to Dexterous Humanoid Twin: Left exhibits the enter selfish human RGB video; Proper exhibits the synchronized 500 Hz MuJoCo + Blender Cycles multi-view simulation rollout.
[Attached: egocentric_human_video.mp4, candidate_sim_views.png, hand_3d_keypoints.json]
You might be constructing a bodily solvable 500 Hz MuJoCo manipulation setting (`scene.xml`) and studio Blender render that faithfully reproduces the tabletop activity proven in `egocentric_human_video.mp4` on a bimanual humanoid robotic with 22-DoF dexterous five-fingered fingers:
1. Scene Reconstruction & Digital camera Calibration:
- Establish solely the manipulation-relevant objects (the grasped goal object and the receiving container or landmark) and triangulate their 3D tabletop coordinates from the selfish digicam intrinsics/extrinsics.
- Change crude bounding containers with compound convex-decomposed 3D meshes so concave containers (nested bowls, recessed wood trays, open bins) assist bodily inside placement.
2. 500 Hz Dexterous Movement-Becoming Stress Take a look at:
- Retarget the 3D human wrist and finger keypoints to the humanoid's 7-DoF arm and 22-DoF hand joints, and step the trajectory in MuJoCo at 500 Hz (dt = 0.002s) with zero kinematic weld constraints.
- Confirm opposing thumb-and-finger drive closure, clear object raise (> 2 cm), and secure remaining placement inside or beside the goal landmark.Plain textual content
3. Sketch to simulation
With Argon, a developer can snap a photograph of a tough pen-and-paper sketch—exhibiting an Apollo 2 humanoid throwing a ball alongside a parabolic arc to topple a 3-2-1 stack of containers on a desk—and have Argon routinely assemble the GPU-accelerated mujoco_warp scene, synthesize a balanced 76-DoF whole-body and five-finger throw trajectory, confirm 6/6 goal knockdown in closed-loop physics, and export a Blender 4.5 Cycles render.
Sketch to Simulation — Hand-Drawn Sketch to Dexterous Humanoid Trajectory & Cycles Render.
[Attached: hand_drawn_throw_sketch.jpg]
Translate this hand-drawn sketch right into a bodily verified humanoid simulation in `mujoco_warp` and a raytraced Blender Cycles video:
1. Scene & Kinematic Setup from Sketch:
- Parse the spatial structure from `hand_drawn_throw_sketch.jpg`: place the 76-DoF humanoid (32-DoF physique geared up with two 22-DoF five-finger dexterous fingers) on the origin dealing with a desk holding a 3-2-1 pyramid of 6 inflexible goal containers, with a projectile sphere grasped in the best five-finger hand.
2. Complete-Physique Steadiness & Dexterous Throw Synthesis:
- Synthesize a 50 Hz whole-body throwing trajectory (140 steps, 2.80 s) throughout 4 phases: (1) secure stance & 5-finger energy grasp closure, (2) torso/shoulder wind-up with ankle-hip Heart-of-Mass compensation, (3) ahead whipping acceleration, and (4) synchronized 22-DoF five-finger opening at peak hand velocity so the ball launches on a ballistic arc into the pyramid's second-tier middle of mass.
- Confirm in `mujoco_warp` contact physics that the humanoid maintains upright steadiness (< 3 cm pelvis drift) and knocks down all 6 goal containers.Plain textual content
Unlocking bodily understanding with Argon
1. Hindsight labeling and information high quality
Argon has exact bodily understanding of dexterous manipulation and might present detailed segmentation and textual content labeling from video. Argon is ready to determine poor manipulation segments similar to pauses, unstable grasps and fumbles. Given an overhead video of a bimanual ALOHA 2 robotic folding a sq. sheet of paper into an origami fox over 136 seconds, Argon produces a time-aligned 4-level hindsight annotation hierarchy: Episode Purpose (G1), 6 contiguous Semantic Subtasks (G2), 17 contact-level Atomic Primitives (G3), and automatic High quality Management (QC) reweighting that holds activity progress flat throughout fumbles and retries.
Argon Multi-Granularity Hindsight & Telemetry HUD.
[Attached: Aloha2_Origami_20250309.mp4]
Produce a time-aligned multi-granularity hindsight annotation JSON with 4 ranges:
1. G1_episode_instruction: Concise crucial sentence describing the general manipulation aim.
2. G2_subtasks: Contiguous semantic subtask phases specifying which arm(s) anchor vs. fold/rotate the article.
3. G3_atomic_primitives: Positive-grained contact-level primitives (3–15s) aligned with grasp closure, crease sweeps, launch, or retry makes an attempt.
4. quality_control_and_reweighting: Maintain task_progress_pct flat throughout non-progressing slip/delamination retry loops and downweight them (0.15x–0.25x) for Conduct Cloning.Plain textual content
2. Discover the odd ball
To recreate the basic 12-ball steadiness scale puzzle with robots, Argon builds a bimanual Franka FR3 MuJoCo setting with 12 visually similar pool balls (one secretly heavier or lighter) and an articulated Roberval two-pan steadiness scale. It then writes a vision-only closed-loop solver that triangulates the balls from stereo RGB cameras, visually reads the size tilt after every weighing, and isolates the odd ball into the Heavy (Pink) or Mild (Blue) tray inside 3 weighings.
Discover the Odd Ball (Imaginative and prescient-Solely Closed-Loop Solver): Twin-arm Franka FR3 weighing candidate balls on the Gold/Teal pans, studying stereo scale tilt , and isolating the confirmed Heavy odd ball into the Pink tray in 3 weighings.
# Immediate 1 — Physics-Correct MuJoCo Surroundings & Benchmark Era
Make a MuJoCo setting the place a dual-arm robotic (two 7-DoF Franka FR3 arms with parallel-jaw grippers on a shared tabletop) is tasked to unravel the basic 12-ball steadiness scale weight puzzle from https://www.mathsisfun.com/pool_balls_solution.html.
1. Create 12 visually indistinguishable pool balls in a 2x6 beginning rack (11 customary 150 g, 1 odd ball both 270 g or 30 g).
2. Construct a practical two-pan Roberval steadiness scale on the middle of the desk with parallelogram equality constraints (`mjEQ_JOINT`) so the Left (Gold) and Proper (Teal) pans keep horizontal, plus Pink (Heavy) and Blue (Mild) submission trays.
3. Monitor distinct settled weighings (`weighing_count`) and reward `1.00` for isolating the true odd ball within the right Heavy/Mild tray inside <= 3 weighings.
# Immediate 2 — Closed-Loop Imaginative and prescient-Solely Solver
Implement an unbiased closed-loop coverage relying strictly on stereo RGB digicam inputs (`head_camera_left_rgb_image`, `head_camera_right_rgb_image`) and robotic proprioception (`add_privileged_task_observations=False`):
1. Section the 12 balls, Gold/Teal pans, and Pink/Blue trays in each stereo views and triangulate their 3D coordinates.
2. Visually learn the steadiness scale tilt after every weighing from stereo vertical pixel disparity (+1 = left heavier, -1 = proper heavier, 0 = balanced).
3. Rotate the gripper wrist yaw by 90 levels when greedy/inserting contained in the slim scale pans to clear the raised rims, and isolate the odd ball into the Pink or Blue tray inside <= 3 weighings.Plain textual content
3. Worth operate
Argon is a basic worth operate for robotics duties. Evaluated on a 60-second bimanual ring-sorting episode from a human UMI dataset, Argon iteratively inspects synchronized head, left-gripper, and right-gripper digicam feeds round contact transitions to generate dense worth operate / activity progress predictions and decompose the trajectory into 28 segments with optimistic, impartial, and detrimental progress slopes.
Worth Perform: Synchronized 3-camera UMI playback (head, left gripper, proper gripper) locked to animated activity progress curve and bimanual phase annotations.
[Attached: umi_head.mp4, umi_left_gripper.mp4, umi_right_gripper.mp4]
Take a look at 3 movies of human carrying a gripper performing actions from head, left, proper gripper views. Take a look at the movies completely and provides an in depth plot of progress on the activity (with optimistic, impartial, detrimental slope), figuring out vital segments within the video and labeling the actions. Take a look at particular person frames or subsegments of the video if essential to get a greater understanding.Plain textual content
Unlocking analysis and self-improvement with Argon
1. Motion enchancment through simulation
Argon can generate new information in simulation to enhance efficiency of generalist vision-language-action (VLA) fashions. Ranging from a single RGB photograph of the actual tabletop, Argon autonomously builds a calibrated MuJoCo simulation twin, defines new duties, and synthesizes a closed-loop movement generator to gather 4,500+ domain-randomized episodes in simulation—boosting real-world bodily robotic Success Price from 56% to 75% and decreasing completion time from 35s to 24s.
Motion Enchancment through Simulation: improves the real-world analysis benchmark (56% → 75% Success Price and 35s → 24s Completion Time).
[Attached: real_tabletop_photo.jpg]
You might be given a single overhead RGB digicam photograph (`real_tabletop_photo.jpg`) of a bimanual robotic workspace (two 7-DoF Franka FR3 arms with parallel-jaw grippers and 4 RGB cameras) dealing with a desk organized with photobooth social gathering props.
1. Reconstruct procedural 3D meshes and compound MuJoCo collision primitives for all 9 social gathering props matching their real-world dimensions, colours, and resting poses, together with spring-hinged deformable crown panels for the smooth plush cap.
2. Outline a 77-task single-arm and bimanual curriculum (pick-and-place, uprighting, directional pushing, mid-air handovers, bimanual scoop-and-push, and pipelined desk clearing).
3. Synthesize a closed-loop IK trajectory generator throughout ±10 cm pose randomization, confirm `reward == 1.0` in MuJoCo physics, and export 4,500+ episodes for VLA fine-tuning.Plain textual content
2. Self-Enchancment for visible choose and place
On a easy 6-DoF arm, Argon iteratively self-improves each execution pace and placement accuracy on a choose and place activity through trial and self-reflection.
SO-101 Imaginative and prescient-Guided Choose-and-Place — Iterative {Hardware} Self-Enchancment Throughout Iterations 1 → 2 → 3: Iteration 1 (58.9s), Iteration 2 (22.9s, grazed field rim on drop), and Iteration 3 (17.4s, dead-center drop).
# Stage 1 — Measure Objects & Dry-Run Plan (No {Hardware} Movement)
Stage 1: measure objects — discover the objects and plan the entire movement. The arm doesn't transfer on this stage.
A. I put a LEGO brick and a field on the desk in entrance of the arm, and the wood field to 1 aspect. Discover the brick and the wood field within the picture. And likewise in the actual world. Present this in an annotated picture.
B. A dry-run plan (no {hardware}). Write the planner to select and place the lego block. Print the plan desk, draw all poses onto the overhead picture, and cease. Don't transfer the robotic.
Report with proof.
# Stage 2 — Pre-Choose Hover & Wrist-Digital camera Verification
Stage 2: Put the gripper above the brick with out touching it, and confirm with the wrist digicam that the arm and the imaginative and prescient agree. Please document executions with each cameras (compressed video is OK).
# Stage 3 (Autonomous Choose-and-Place → Self-Enchancment → Optimization):
1. Execute autonomously: Flip the Stage 1 dry run into an actual grasp, raise, and place into the wood field. Run it your self topic to security guidelines from AGENTS.md.
2. Self-improve: When an try fails, diagnose it your self. Assessment your personal logs and wrist/overhead footage, discover the foundation trigger, verify the repair in a dry run, and retry.
3. Optimize on suggestions: As soon as it succeeds, make it at the least 2× quicker inside security limits (rule R5) except I authorize extra, or give suggestions.Plain textual content
3. One-prompt robotics paper reproductions in MuJoCo WebAssembly
With a single immediate per paper, Argon reads a robotics management manuscript and any launched checkpoints, ports the physics mannequin and closed-loop controller (from 7-network hierarchical locomotion in DeepLoco and 50 Hz PPO actors in HUSKY to Internet Employee iLQR in Complete-Physique MPC) into self-contained MuJoCo WebAssembly browser labs, and verifies them in headless Chromium.
5 Landmark Legged & Humanoid Papers Operating Dwell in Browser (MuJoCo WASM): Robotic Soccer OP3 1v1, DeepLoco, Massively Parallel Deep RL ANYmal C, Complete-Physique MPC Go2, and HUSKY G1 Skateboarding (from 68 interactive browser sandboxes reproduced by Argon).
[Attached: paper.pdf, optional author_checkpoints/ or urdf_assets/]
Reproduce the paper "" from scratch in MuJoCo WebAssembly and deploy a self-contained browser lab + visible essay:
1. Match the paper's precise physics timestep, actuator good points (kp/kv), torque limits, and kinematic constraints in MuJoCo XML, and run actual neural inference or Internet Employee iLQR in JavaScript on the paper's precise management charges.
2. Construct `index.html` (Spatial UI simulator with digicam presets, reside telemetry plots, and disturbance injection) and `scrollytelling.html` (interactive visible essay).
3. Run a headless Chromium (Playwright) verification loop asserting 0 console errors and non-zero activity progress over 300+ frames. Plain textual content
Advancing embodied intelligence with Argon
Argon bridges coding, visible notion, physics simulation, and embodied robotic management. By reasoning natively over multi-view video, 3D geometry, contact mechanics, and proprioceptive telemetry, Argon understands not simply what objects are in a scene, however how they bodily work together with brokers to perform whole-body dexterous duties. Gemini 4 Argon marks an vital milestone accelerating the trail towards bodily AGI. By constructing this core intelligence, our aim is to allow robotics that may work alongside people to unravel complicated challenges.
Getting began
You can begin constructing with Gemini 4 Argon right now throughout Google’s developer platforms:
- Google AI Studio: Out there to Google AI Extremely subscribers.
- Gemini API: Out there with the paid tier of the Gemini API in AI Studio. Use
gemini-4-argonwithin the Interactions API with day-one assist throughout thegoogle-genaiSDKs. - Google Antigravity: Out there for Google AI Extremely subscribers. To begin constructing, choose Gemini 4 Argon (
gemini-4-argon) as your energetic mannequin in Google Antigravity for complicated planning,/aimruns, and multi-agent orchestration.
