任务执行失败Door threshold + milk grasp
README play config · microwave + milk
repro_readme_nonpriv_hf
FAIL
最终判定门发生明显运动但 native Open=false；milk 随后空抓，最终严格谓词 0/2。
失败阶段Door threshold + milk grasp
Causal trace audit
Failure Mode Analysis
判定双重失败：门发生运动但未跨过 Open 阈值；随后 milk 又独立空抓。
最早失效点Step 4 · microwave door only partially opened; native Open(microwave)=false
近端物理原因
固定世界坐标拉门轨迹不足以满足铰链阈值；milk 的单一 grasp candidate 随后没有形成承重夹持。
传播原因
代码在两个动作后都无条件写 success，没有检查门关节状态、milk 是否随抬升移动，继续空夹放置。
AGPT-5.5 代码 / 状态机38%
硬编码门弧；动作返回即判成功；门/抓取均无 post-condition gate。
BLIBERO / TCP 标定27%
门的铰链阈值与弧线需要几何标定；但这不是“代码完全正确、只差一个数”。
CMolmo / SAM3 感知5%
Molmo 离线是混杂项；现有运动证据不支持 SAM3 把对象认错。
D抓取 / 3-D / 控制接口30%
milk 候选位置合理但抓取失稳；top-k、夹爪接触与实际 TCP 都未暴露。
决定性证据
- 504 帧完整轨迹：frames 86–216 门确实运动，但 native Open=false；LIBERO 的 is_open 阈值为 qpos < -1.3 rad（约 74.5°）。
- frames 246–357：夹爪到达 milk 附近并抬升，milk 始终留在桌面。grasp position=[0.8885,-0.0424,0.0985] 与目标区域一致。
- 该轮 Molmo 服务离线，但 SAM3 text fallback 仍把机器人引到 handle 与 milk 附近，因此不能把失败解释为 gross detector miss。
仍不能确认
没有保存 SAM mask、top-k grasp、实际 TCP 和 microwave qpos 曲线，故 milk 的 Contact-GraspNet 与 TCP 残差、门弧与接触滑脱仍无法完全拆开。
最小反事实实验
固定感知后读取 qpos 并闭环拉门；对同一 milk mask 比较 top-down/宽侧 oracle grasp 与当前 argmax，再做 ±1–3 cm TCP sweep。
责任百分比用于表达该条 trace 的工程因果权重，不是统计概率；没有受控重放的结论标为 strongly supported，而非 confirmed。
01
Task setting
Kitchen tabletop; microwave starts closed; milk carton starts upright on the table. Goal predicates: Open(microwave) AND In(milk, microwave_heating_region).
02
失败根因
双重失败：门发生运动但未跨过 Open 阈值；随后 milk 又独立空抓。
Evidence
视觉证据
Why
证据与推理
- 504 帧完整执行：门有明显运动，但没有跨过 LIBERO 的 Open 阈值。
- milk 抓取是第二个独立失败：闭爪/抬升后 milk 仍在初始桌面位置。
- 两个动作都被代码无条件记为 success，最终 native predicates 为 0/2。
03
真实 API 审计
model=gpt-5.5 · elapsed=22.3s · finish_reason=stop · source: 0005_p506268_policy_writer._generate_once.json
发送给 Policy Writer 的完整 prompt
===== SYSTEM =====
You are a deterministic policy writer for a code-as-policy robot system. Write ONLY Python code. Do not use while True loops. Use only the provided imported primitive functions and learned helper functions.
===== USER =====
You are a deterministic policy writer for a code-as-policy robot system.
Write Python code that executes the following plan.
AVAILABLE IMPORTED FUNCTIONS (use ONLY these, not env.<method>):
close_gripper, get_object_3d_points_and_masks_from_language, get_object_pose, get_observation, get_oriented_bounding_box_from_3d_points, goto_home_joint_position, goto_pose, open_gripper, sample_grasp_pose
API DOCUMENTATION:
get_observation() -> dict[str, typing.Any]
Doc:
Get the observation of the environment.
Returns:
observation:
A dictionary containing the observation of the environment.
The dictionary contains the following keys:
- ["agentview"]["images"]["rgb"]: Current color camera image as a numpy array of shape (H, W, 3), dtype uint8.
- ["agentview"]["images"]["depth"]: Current depth camera image as a numpy array of shape (H, W), dtype float32.
- ["agentview"]["intrinsics"]: Camera intrinsic matrix as a numpy array of shape (3, 3), dtype float64.
- ["agentview"]["pose_mat"]: Camera extrinsic matrix as a numpy array of shape (4, 4), dtype float64.
- ["robot0_eye_in_hand"]["images"]["rgb"]: Current wrist camera image as a numpy array of shape (H, W, 3), dtype uint8.
- ["robot0_eye_in_hand"]["images"]["depth"]: Current wrist camera depth image as a numpy array of shape (H, W), dtype float32.
- ["robot0_eye_in_hand"]["intrinsics"]: Wrist camera intrinsic matrix as a numpy array of shape (3, 3), dtype float64.
- ["robot0_eye_in_hand"]["pose_mat"]: Wrist camera extrinsic matrix as a numpy array of shape (4, 4), dtype float64.
- ["robot_cartesian_pos"]: Current end-effector (panda_hand) pose in the robot/world frame as a numpy array of shape (8,), dtype float64. The first 3 elements are the robot's end-effector XYZ, the next 4 elements are the quaternion wxyz, and the last element is the gripper position normalized, 0 (closed) to 1 (open).
- ["robot_joint_pos"]: Current joint positions as a numpy array of shape (7,), dtype float64. The last element is the gripper position normalized, 0 (closed) to 1 (open).
get_object_pose(object_name: str, use_multiview: bool = True) -> tuple[numpy.ndarray, numpy.ndarray] | tuple[None, None]
Doc:
Get the pose of an object in the environment from a natural language description.
The quaternion from get_object_pose may be unreliable, so disregard it and use the grasp
pose quaternion OR (0, 0, 1, 0) wxyz as the gripper down orientation if using this for
placement position.
It is possible that get_object_pose is sometimes unreliable and will return None for both
position and quaternion.
Args:
object_name: The name of the object to get the pose of, in underscore separated lowercase words.
use_multiview: If True, uses the wrist camera as well as the main camera for segmentation. If False, only uses the main camera for segmentation.
Returns:
position: (3,) XYZ in meters (world frame).
quaternion_wxyz: (4,) WXYZ unit quaternion (world frame).
sample_grasp_pose(object_name: str, use_multiview: bool = True) -> tuple[numpy.ndarray, numpy.ndarray]
Doc:
Sample a grasp pose for an object in the environment from a natural language description.
Uses multiview point clouds and plan_grasp_from_point_clouds for more reliable grasp planning.
Do use the grasp sample quaternion from sample_grasp_pose.
Args:
object_name: The name of the object to sample a grasp pose for, in underscore separated lowercase words.
use_multiview: If True, uses the wrist camera as well as the main camera for segmentation. If False, only uses the main camera for segmentation.
Returns:
position: (3,) XYZ in meters (world frame).
quaternion_wxyz: (4,) WXYZ unit quaternion (world frame).
goto_pose(position: numpy.ndarray, quaternion_wxyz: numpy.ndarray, z_approach: float = 0.0) -> None
Doc:
Go to pose using Inverse Kinematics.
There is no need to call a second goto_pose with the same position and quaternion_wxyz after calling it with z_approach.
Args:
position: (3,) XYZ in meters.
quaternion_wxyz: (4,) WXYZ unit quaternion.
z_approach: (float) Z-axis distance offset for goto_pose insertion approach motion. Will first arrive at position + z_approach meters in Z-axis before moving to the requested pose. Useful for more precise grasp approaches. Default is 0.0.
Returns:
None
open_gripper() -> None
Doc:
Open gripper fully.
Args:
None
close_gripper() -> None
Doc:
Close gripper fully.
Args:
None
get_oriented_bounding_box_from_3d_points(points: numpy.ndarray) -> dict[str, typing.Any]
Doc:
Get the oriented bounding box from 3D points.
Args:
points: np.ndarray: The 3D points to get the oriented bounding box from.
Shape: (N, 3), dtype float64.
Returns:
dict[str, Any]: The oriented bounding box. The dictionary contains the following keys:
- "center": np.ndarray: The center of the oriented bounding box in point cloud frame.
- "extent": np.ndarray: The extent of the oriented bounding box.
- "R": np.ndarray: The rotation matrix of the oriented bounding box in point cloud frame.
Example:
>>> points = np.random.randn((100, 3))
>>> obb = get_oriented_bounding_box_from_3d_points(points)
get_object_3d_points_and_masks_from_language(text_prompt: str, use_multiview: bool = True) -> dict[str, typing.Any]
Doc:
Segment an object using text prompt across multiple views (agentview and wrist).
Uses Molmo to locate the object in both camera views, then SAM3 for
segmentation. Returns the masks and 3D points that appear in both views.
Args:
text_prompt: Text description of the object to segment.
use_multiview: If True, uses the wrist camera as well as the main camera for segmentation.
Returns:
dict containing:
- "agentview_mask": np.ndarray of shape (H, W), dtype bool
- "wrist_mask": np.ndarray of shape (H, W), dtype bool (None if single view)
- "points_3d": np.ndarray of shape (N, 3), 3D points in world frame
- "agentview_score": float, SAM3 confidence score
- "wrist_score": float, SAM3 confidence score (None if single view)
goto_home_joint_position() -> None
Doc:
Return the arm to its reset joint configuration with high manipulability
LEARNED HELPER FUNCTION REFERENCES (already defined and callable — use them directly):
No learned helper functions available yet.
REQUIREMENTS:
- Use the primitive functions listed in AVAILABLE IMPORTED FUNCTIONS above AND any LEARNED HELPER FUNCTION REFERENCES shown above.
- The learned helper functions are pre-defined and available in your execution scope. Call them directly.
- Do NOT invent wrapper functions on env or low_level_env.
- Do NOT use while True loops or any unbounded retry patterns. Use bounded for loops instead.
- Set RESULT to a structured dict describing success/failure for each step.
- Add explicit comments before each plan step.
- Wrap the executable body for EACH plan step in the runtime marker context
`with step_context("<step id>", "<step description>", step_index=<0-based index>):`.
This is a logging marker, not a robot primitive. It lets the verifier save
exact per-step API traces, motion frame ranges, and intermediate artifacts.
- Define reusable sub-functions for non-trivial skill sequences that could be added to the skill library.
- On retries, follow the diagnoser's requested edit scale. If a top-of-retry "## EDIT SCALE: argument-level" banner is present (or the prose says argument-level), keep the same primitive/helper call sequence and change only the named arguments first (target text, verify_label, pose/offset, approach axis, release height, retry count, or gate threshold). Rewrite helper bodies, replace primitives, or add new wrappers only when the banner / diagnosis says rewrite-needed because arguments cannot express the fix, the API/function is wrong for the task, the sequence is structurally wrong, or there is a real code/logic bug.
IMPORTANT: Use ONLY functions from the AVAILABLE IMPORTED FUNCTIONS list and LEARNED HELPER FUNCTION REFERENCES above.
Do NOT call any function not in those lists. Check the API DOCUMENTATION for correct function signatures.
Extract object names from the GOAL description below.
General rules:
- Read the API DOCUMENTATION carefully for the correct function signatures and return types.
- When retry feedback names a primitive/helper call, preserve that call and tune its arguments before changing the overall code structure, unless the feedback explicitly calls for a larger rewrite.
- For pick-and-place: perceive object location, move to grasp pose, close gripper, move to target, open gripper. Use the available perception functions to locate objects.
- Use bounded for loops (not while True) for retry patterns.
- For multi-step tasks (pick A then place on B), chain the patterns sequentially.
- Never leave a successfully grasped object suspended because a pre-release wrist check failed. Once the placement target has been localized, execute a cautious descend, open the gripper, retreat, and then verify the final resting state.
- NumPy array truthiness: localization primitives often return ndarrays or lists of candidates. NEVER write `if pos:` or `if not result:` on an ndarray — raises "truth value of an array is ambiguous". Instead use `if pos is None` / `if len(pos) == 0` / `if result.get("verified")` on explicit scalars. This pattern has burned many past attempts.
# === TASK-SPECIFIC INPUTS BELOW (vary per attempt; everything above is stable for cache reuse) ===
SCENE: i_want_to_open_the_microwave_and_put_the_milk_inside_it
GOAL: I want to open the microwave and put the milk inside it.
OBJECT SCOPE: {}
PLAN:
- step-1: Get an updated observation and confirm the scene: the microwave is on the right side of the table with its door closed, the vertical handle is visible on the front-left edge of the microwave door, and the milk carton is upright near the center-front area of the table. | skills: get_observation | notes: Use the agent-view image to verify the microwave door state before planning the opening motion. The milk appears unobstructed and graspable.
- step-2: Localize the microwave handle and estimate the microwave door/face geometry for grasping and opening. | skills: get_object_pose, get_object_3d_points_and_masks_from_language, get_oriented_bounding_box_from_3d_points | notes: Segment/localize the microwave handle using a prompt such as 'microwave handle' or 'vertical handle on microwave door'. If handle pose is unreliable, segment the microwave door/handle region and use the visible vertical handle on the left/front edge of the microwave as the grasp target.
- step-3: Open the gripper, move to a pre-grasp pose in front of the microwave handle, then grasp the handle. | skills: open_gripper, sample_grasp_pose, goto_pose, close_gripper | notes: (a) APPROACH DIRECTION: approach from the front of the microwave, perpendicular to the door face, targeting the exposed vertical handle; avoid approaching from above because the gripper may collide with the microwave top or door edge. (b) PULL/PUSH DIRECTION: after grasping, pull the handle outward away from the microwave body and slightly leftward following the door hinge arc, since the visible handle is on the left/front edge and the door should swing open outward. (c) GRASP SELECTION CRITERION: select a grasp whose approach axis is roughly perpendicular to the microwave front face and whose fingers can wrap around the vertical handle from the front, with the gripper aligned to clamp the handle thickness rather than pinching the flat door surface.
- step-4: Pull the microwave door open by moving the grasped handle outward and leftward along a short arc, then release the handle once the door is sufficiently open. | skills: goto_pose, open_gripper | notes: A dedicated articulated-opening skill would improve reliability by maintaining a compliant grasp and following the hinge arc. Use a conservative motion: pull outward from the microwave face, then continue leftward/outward until the interior opening is exposed enough to place the milk. Keep clearance from the table and microwave body.
- step-5: Move the arm away from the opened microwave door area and get a fresh observation to verify that the microwave is open and the interior cavity is reachable. | skills: goto_home_joint_position, get_observation | notes: This prevents collision with the open door and confirms the prerequisite state before picking the milk.
- step-6: Localize the milk carton on the table and sample a stable grasp pose. | skills: get_object_pose, sample_grasp_pose | notes: The milk carton is visible upright near the center-front of the table. Use multiview perception for grasp sampling if available. Prefer a side grasp around the carton body that keeps the carton upright.
- step-7: Pick up the milk carton from the table. | skills: open_gripper, goto_pose, close_gripper | notes: Approach the sampled grasp pose from above with a small z_approach, close the gripper on the carton, then lift vertically to clear the table.
- step-8: Localize the open microwave interior and compute a placement pose inside the cavity. | skills: get_object_pose, get_object_3d_points_and_masks_from_language, get_oriented_bounding_box_from_3d_points | notes: Use prompts such as 'open microwave interior', 'inside of microwave', or 'microwave cavity'. The placement point should be centered inside the open cavity, above the microwave floor, with enough clearance from the open door and front lip.
- step-9: Move the milk into the open microwave and place it on the microwave floor. | skills: goto_pose, open_gripper | notes: A dedicated place-inside-container skill would help handle the tight opening and avoid collisions. Use a gripper-down or upright-preserving orientation, insert through the open front of the microwave, lower to the interior floor, release the milk, and retreat outward along the same approach path.
- step-10: Retract the arm to a safe pose and verify that the milk is inside the open microwave. | skills: goto_home_joint_position, get_observation | notes: Final verification should check that the milk is no longer on the table and is visible or plausibly placed inside the microwave cavity. The goal only requires opening the microwave and putting the milk inside; it does not require closing the microwave.
## LESSONS FROM PAST FAILURES (advisory — NOT ground truth)
The block above (if non-empty) lists distilled lessons in "WHEN ... | WRONG ... | DO ..." form, each followed by a metadata footer like `(confidence=0.80, applied=5× helped=1×, last_helped=iter4 (2 iters ago), distilled=iter2 (4 iters ago))`. Use that footer to weigh how much to trust each lesson.
PRIORITY ORDER when signals conflict:
0. The SUB-AGENT DIRECTIVE block above (if non-empty) — this is the parallel sub-agent's committed approach assignment, a HARD constraint, and dominates every other guidance source below.
1. The current attempt's RETRY CONTEXT (diagnoser feedback + visual evidence + PREVIOUS CODE below) — this saw what actually happened on this exact task and is ground truth for what to fix.
2. A lesson with high reliability (helped/applied ≥ 0.5 with applied ≥ 2) AND recent `last_helped` (≤ 2 iters ago) — strong prior, follow its DO recipe.
3. A lesson with low reliability (helped/applied < 0.3) OR stale (`last_helped=never` or ≥ 5 iters ago) — background prior only; don't let it override the current diagnosis.
Empty block means no relevant prior failures.
API 返回并实际执行的 robot control code
RESULT = {
"step-1": {"success": False, "details": ""},
"step-2": {"success": False, "details": ""},
"step-3": {"success": False, "details": ""},
"step-4": {"success": False, "details": ""},
"step-5": {"success": False, "details": ""},
"step-6": {"success": False, "details": ""},
"step-7": {"success": False, "details": ""},
"step-8": {"success": False, "details": ""},
"step-9": {"success": False, "details": ""},
"step-10": {"success": False, "details": ""},
"overall_success": False,
}
def make_offset_pose(position, dx=0.0, dy=0.0, dz=0.0):
new_position = position.copy()
new_position[0] = new_position[0] + dx
new_position[1] = new_position[1] + dy
new_position[2] = new_position[2] + dz
return new_position
def safe_localize_object(object_name, use_multiview=True, attempts=2):
found_position = None
found_quaternion = None
for _ in range(attempts):
position, quaternion = get_object_pose(object_name, use_multiview=use_multiview)
if position is not None:
found_position = position
found_quaternion = quaternion
break
return found_position, found_quaternion
def safe_segment_obb(text_prompt, use_multiview=True, attempts=2):
segmented = None
obb = None
for _ in range(attempts):
segmented_candidate = get_object_3d_points_and_masks_from_language(text_prompt, use_multiview=use_multiview)
points = segmented_candidate.get("points_3d")
if points is not None and len(points) > 0:
segmented = segmented_candidate
obb = get_oriented_bounding_box_from_3d_points(points)
break
return segmented, obb
def sample_grasp_with_retries(object_name, use_multiview=True, attempts=2):
grasp_position = None
grasp_quaternion = None
for _ in range(attempts):
grasp_position, grasp_quaternion = sample_grasp_pose(object_name, use_multiview=use_multiview)
if grasp_position is not None and grasp_quaternion is not None:
break
return grasp_position, grasp_quaternion
def pick_object_from_sampled_grasp(object_name, z_approach=0.08, lift_height=0.16):
grasp_position, grasp_quaternion = sample_grasp_with_retries(object_name, use_multiview=True, attempts=2)
if grasp_position is None or grasp_quaternion is None:
return False, None, None
open_gripper()
goto_pose(grasp_position, grasp_quaternion, z_approach=z_approach)
close_gripper()
lifted_position = make_offset_pose(grasp_position, dz=lift_height)
goto_pose(lifted_position, grasp_quaternion, z_approach=0.0)
return True, lifted_position, grasp_quaternion
# step-1: Get an updated observation and confirm the scene.
with step_context("step-1", "Get updated observation and confirm microwave and milk scene", step_index=0):
observation_initial = get_observation()
RESULT["step-1"]["success"] = observation_initial is not None
RESULT["step-1"]["details"] = "Updated observation acquired for microwave door, handle, and milk carton scene."
# step-2: Localize the microwave handle and estimate the microwave door/face geometry.
with step_context("step-2", "Localize microwave handle and estimate handle geometry", step_index=1):
handle_position, handle_quaternion_from_pose = safe_localize_object("microwave_handle", use_multiview=True, attempts=2)
handle_segment, handle_obb = safe_segment_obb("vertical handle on microwave door", use_multiview=True, attempts=2)
if handle_position is None and handle_obb is not None:
handle_position = handle_obb["center"]
RESULT["step-2"]["success"] = handle_position is not None
RESULT["step-2"]["details"] = "Microwave handle localized using pose and/or segmented oriented bounding box."
# step-3: Open the gripper, move to a pre-grasp pose in front of the microwave handle, then grasp the handle.
with step_context("step-3", "Approach and grasp microwave handle from the front", step_index=2):
handle_grasp_position, handle_grasp_quaternion = sample_grasp_with_retries("microwave_handle", use_multiview=True, attempts=2)
if handle_grasp_position is None or handle_grasp_quaternion is None:
handle_grasp_position, handle_grasp_quaternion = sample_grasp_with_retries("vertical_handle_on_microwave_door", use_multiview=True, attempts=1)
if handle_grasp_position is None and handle_position is not None:
handle_grasp_position = handle_position
handle_grasp_quaternion = (0.0, 0.0, 1.0, 0.0)
if handle_grasp_position is not None and handle_grasp_quaternion is not None:
open_gripper()
goto_pose(handle_grasp_position, handle_grasp_quaternion, z_approach=0.10)
close_gripper()
RESULT["step-3"]["success"] = True
RESULT["step-3"]["details"] = "Gripper closed on microwave handle."
else:
RESULT["step-3"]["details"] = "Failed to obtain a usable microwave handle grasp pose."
# step-4: Pull the microwave door open by moving the grasped handle outward and leftward along a short arc, then release.
with step_context("step-4", "Pull microwave door outward and leftward, then release handle", step_index=3):
if RESULT["step-3"]["success"]:
pull_pose_1 = make_offset_pose(handle_grasp_position, dx=-0.10, dy=0.00, dz=0.00)
pull_pose_2 = make_offset_pose(handle_grasp_position, dx=-0.16, dy=0.10, dz=0.00)
pull_pose_3 = make_offset_pose(handle_grasp_position, dx=-0.20, dy=0.16, dz=0.00)
goto_pose(pull_pose_1, handle_grasp_quaternion, z_approach=0.0)
goto_pose(pull_pose_2, handle_grasp_quaternion, z_approach=0.0)
goto_pose(pull_pose_3, handle_grasp_quaternion, z_approach=0.0)
open_gripper()
retreat_from_handle = make_offset_pose(pull_pose_3, dx=-0.04, dy=0.04, dz=0.08)
goto_pose(retreat_from_handle, handle_grasp_quaternion, z_approach=0.0)
RESULT["step-4"]["success"] = True
RESULT["step-4"]["details"] = "Microwave handle pulled outward/leftward and released."
else:
RESULT["step-4"]["details"] = "Skipped opening motion because handle grasp failed."
# step-5: Move the arm away from opened microwave door area and get a fresh observation.
with step_context("step-5", "Return home and observe opened microwave", step_index=4):
goto_home_joint_position()
observation_after_open = get_observation()
RESULT["step-5"]["success"] = observation_after_open is not None
RESULT["step-5"]["details"] = "Arm moved home and fresh observation acquired after door-opening attempt."
# step-6: Localize the milk carton on the table and sample a stable grasp pose.
with step_context("step-6", "Localize milk carton and sample grasp", step_index=5):
milk_position, milk_quaternion_from_pose = safe_localize_object("milk_carton", use_multiview=True, attempts=2)
milk_grasp_position, milk_grasp_quaternion = sample_grasp_with_retries("milk_carton", use_multiview=True, attempts=2)
if milk_grasp_position is None or milk_grasp_quaternion is None:
milk_grasp_position, milk_grasp_quaternion = sample_grasp_with_retries("milk", use_multiview=True, attempts=1)
RESULT["step-6"]["success"] = milk_grasp_position is not None and milk_grasp_quaternion is not None
RESULT["step-6"]["details"] = "Milk carton localized and grasp pose sampled."
# step-7: Pick up the milk carton from the table.
with step_context("step-7", "Pick up milk carton", step_index=6):
if RESULT["step-6"]["success"]:
open_gripper()
goto_pose(milk_grasp_position, milk_grasp_quaternion, z_approach=0.10)
close_gripper()
milk_lift_position = make_offset_pose(milk_grasp_position, dz=0.18)
goto_pose(milk_lift_position, milk_grasp_quaternion, z_approach=0.0)
RESULT["step-7"]["success"] = True
RESULT["step-7"]["details"] = "Milk carton grasped and lifted from table."
else:
RESULT["step-7"]["details"] = "Skipped milk pickup because grasp sampling failed."
# step-8: Localize the open microwave interior and compute a placement pose inside the cavity.
with step_context("step-8", "Localize open microwave interior and compute placement pose", step_index=7):
interior_position, interior_quaternion_from_pose = safe_localize_object("open_microwave_interior", use_multiview=True, attempts=2)
interior_segment, interior_obb = safe_segment_obb("open microwave interior", use_multiview=True, attempts=2)
if interior_position is None and interior_obb is not None:
interior_position = interior_obb["center"]
if interior_position is None:
interior_position, interior_quaternion_from_pose = safe_localize_object("inside_of_microwave", use_multiview=True, attempts=1)
if interior_position is not None:
placement_position = interior_position.copy()
placement_position[2] = placement_position[2] + 0.03
pre_place_position = make_offset_pose(placement_position, dx=-0.10, dy=0.00, dz=0.10)
placement_quaternion = milk_grasp_quaternion
RESULT["step-8"]["success"] = True
RESULT["step-8"]["details"] = "Open microwave interior localized and placement pose computed."
else:
RESULT["step-8"]["details"] = "Failed to localize open microwave interior."
# step-9: Move the milk into the open microwave and place it on the microwave floor.
with step_context("step-9", "Place milk inside open microwave", step_index=8):
if RESULT["step-7"]["success"] and RESULT["step-8"]["success"]:
goto_pose(pre_place_position, placement_quaternion, z_approach=0.0)
goto_pose(placement_position, placement_quaternion, z_approach=0.04)
open_gripper()
retreat_position = make_offset_pose(pre_place_position, dx=-0.06, dy=0.00, dz=0.04)
goto_pose(retreat_position, placement_quaternion, z_approach=0.0)
RESULT["step-9"]["success"] = True
RESULT["step-9"]["details"] = "Milk inserted into microwave cavity, released, and gripper retreated."
else:
RESULT["step-9"]["details"] = "Skipped placement because milk was not picked or microwave interior was not localized."
# step-10: Retract the arm to a safe pose and verify that the milk is inside the open microwave.
with step_context("step-10", "Retract and verify final state", step_index=9):
goto_home_joint_position()
final_observation = get_observation()
RESULT["step-10"]["success"] = final_observation is not None and RESULT["step-9"]["success"]
RESULT["step-10"]["details"] = "Arm retracted home and final observation acquired after placing milk inside microwave."
RESULT["overall_success"] = (
RESULT["step-1"]["success"]
and RESULT["step-2"]["success"]
and RESULT["step-3"]["success"]
and RESULT["step-4"]["success"]
and RESULT["step-5"]["success"]
and RESULT["step-6"]["success"]
and RESULT["step-7"]["success"]
and RESULT["step-8"]["success"]
and RESULT["step-9"]["success"]
and RESULT["step-10"]["success"]
)API 返回的失败诊断原文
```json
{
"visual_success": false,
"failed_step": "step-7",
"failure_mode": "grasp_failure",
"confidence": 0.82,
"plan_issue": false,
"plan_issue_reason": "",
"subagent_skill_target": null,
"subagent_skill_target_reason": "",
"subagent_approaches": [],
"edit_scale": "argument_level",
"visual_predicate_status": [
{
"step_id": "step-1",
"description": "Get an updated observation and confirm the scene: the microwave is on the right side of the table with its door closed, the vertical handle is visible on the front-left edge of the microwave door, and the milk carton is upright near the center-front area of the table.",
"visually_satisfied": true,
"evidence": "The initial scene/verifier evidence shows the microwave on the right with the vertical handle visible and the milk carton upright on the table. This was an observation-only step and is consistent with the trajectory setup."
},
{
"step_id": "step-2",
"description": "Localize the microwave handle and estimate the microwave door/face geometry for grasping and opening.",
"visually_satisfied": false,
"evidence": "The handle is visible and the robot later moved to the handle area, but the final segment does not visually prove that the handle/door geometry was localized. The stdout also shows repeated perception-service connection failures and no segmentation/pose artifacts for the handle."
},
{
"step_id": "step-3",
"description": "Open the gripper, move to a pre-grasp pose in front of the microwave handle, then grasp the handle.",
"visually_satisfied": false,
"evidence": "The robot appears to have interacted with the handle earlier, but by the final segment the handle has been released, so the grasp itself is no longer observable. This step was likely achieved transiently but is not directly satisfied in the terminal view."
},
{
"step_id": "step-4",
"description": "Pull the microwave door open by moving the grasped handle outward and leftward along a short arc, then release the handle once the door is sufficiently open.",
"visually_satisfied": true,
"evidence": "The terminal wrist/scene view shows the microwave door displaced open relative to the body, with the handle/door no longer closed flush. The verifier also reports the door was pulled open and released."
},
{
"step_id": "step-5",
"description": "Move the arm away from the opened microwave door area and get a fresh observation to verify that the microwave is open and the interior cavity is reachable.",
"visually_satisfied": false,
"evidence": "Although the arm retreated, the terminal image does not clearly show a verified reachable interior cavity. The view is close/oblique and does not establish that the open microwave interior was usable for placement."
},
{
"step_id": "step-6",
"description": "Localize the milk carton on the table and sample a stable grasp pose.",
"visually_satisfied": false,
"evidence": "The milk carton remained visible on the table, but there are no reliable localization/grasp artifacts. Stdout shows repeated perception-service connection failures for 'milk_carton', and the later grasp did not result in holding the carton."
},
{
"step_id": "step-7",
"description": "Pick up the milk carton from the table.",
"visually_satisfied": false,
"evidence": "The milk carton was not lifted and the gripper is empty by the end. The code executed close_gripper and a lift after sampling a milk grasp, but visually the carton remained on the table, indicating the gripper closed off-target or failed to secure the carton."
},
{
"step_id": "step-8",
"description": "Localize the open microwave interior and compute a placement pose inside the cavity.",
"visually_satisfied": false,
"evidence": "The final view does not show a confirmed computed placement pose or clear localization of the microwave floor/cavity. Stdout shows repeated failures for 'open_microwave_interior' and 'open microwave interior', with no useful segmentation artifacts."
},
{
"step_id": "step-9",
"description": "Move the milk into the open microwave and place it on the microwave floor.",
"visually_satisfied": false,
"evidence": "The milk was never transported into the microwave. The gripper appears empty after the milk pickup attempt, and the verifier reports the carton remains upright on the table rather than inside the cavity."
},
{
"step_id": "step-10",
"description": "Retract the arm to a safe pose and verify that the milk is inside the open microwave.",
"visually_satisfied": false,
"evidence": "The arm retracted, but the required final state is not met: the milk is not inside the microwave. The terminal view/verification indicates an empty gripper and no placed carton in the microwave."
}
],
"policy_feedback": "argument_level: keep the overall sequence of opening the microwave, then grasping the milk, then placing it. The door-opening portion appears to have worked; do not rewrite that part. Change the milk pickup call in step-7: the sampled grasp for `sample_grasp_with_retries(\"milk_carton\", use_multiview=True, attempts=2)` / `goto_pose(milk_grasp_position, milk_grasp_quaternion, z_approach=0.10)` did not actually close on the carton. Use a more explicit target text such as the visible upright milk carton body/broad side, sample or bias the grasp toward the carton side faces rather than an arbitrary point, and increase retries. Approach the carton with a side or slightly top-down grasp centered on its rectangular body, then close and lift only after the gripper is aligned around the carton width. Also reduce reliance on the failing language-service calls shown in stdout: if `get_object_pose`/segmentation returns no points for `milk_carton`, retry with the simpler label `milk` or a fresh observation before executing the grasp pose. The code currently marks step-7 successful immediately after close/lift even though the gripper is empty, so add a post-lift visual/width check that retries the milk grasp before proceeding to microwave placement."
}
```source: 0018_p506268_failure_diagnoser._query_llm_text.json
Agent 调用时间线 展开原始调用索引
| # | 调用方 | 耗时 | 结果 | 证据 |
|---|---|---|---|---|
| 1 | task_proposer.propose_novel_candidates | 12.9s | stop | embedded above |
| 2 | environment_creator._generate_bddl | 3.7s | stop | embedded above |
| 3 | planner.plan | 21.0s | stop | embedded above |
| 4 | planner_verifier.verify_plan | 7.9s | stop | embedded above |
| 5 | policy_writer._generate_once | 22.3s | stop | embedded above |
| 6 | per_step_verifier._verify_step_package | 6.5s | stop | embedded above |
| 7 | per_step_verifier._verify_step_package | 6.5s | stop | embedded above |
| 8 | per_step_verifier._verify_step_package | 9.0s | stop | embedded above |
| 9 | per_step_verifier._verify_step_package | 7.2s | stop | embedded above |
| 10 | per_step_verifier._verify_step_package | 14.4s | stop | embedded above |
| 11 | per_step_verifier._verify_step_package | 6.8s | stop | embedded above |
| 12 | per_step_verifier._verify_step_package | 6.5s | stop | embedded above |
| 13 | per_step_verifier._verify_step_package | 8.5s | stop | embedded above |
| 14 | per_step_verifier._verify_step_package | 7.7s | stop | embedded above |
| 15 | per_step_verifier._verify_step_package | 6.5s | stop | embedded above |
| 16 | verifier._query_llm_json | 3.0s | stop | embedded above |
| 17 | verifier._query_llm_json | 7.2s | stop | embedded above |
| 18 | failure_diagnoser._query_llm_text | 25.2s | stop | embedded above |