The objective of this experiment is to evaluate the use of LLMs to shape trajectories by suggesting 3D poses based on natural language input. This hypothesis is tested by experimenting with the withdrawal and change of objective methods. In the first experiment, the withdrawal method is tested with various prompts that express positions relative to the user's location or other objects on the board. The second experiment, similar to the first, aims to gauge satisfaction with the change of objective based on user input. Each experiment is evaluated through a user study with 20 participants, who rate their satisfaction with GPT-4's responses for these functions. The task involves the robot picking up a cube on the left side of the user and moving it between the user's hands. During the experiment, the user's left hand serves as the obstacle, representing the closest point from which the robot withdraws. Trajectory points (without withdrawal) are recorded, along with the robot’s speed and safety indexes. Based on this data, three distinct trajectories are computed. Two types of prompts are subsequently tested: commands relative to the user and commands relative to objects on the table. For example, a user-relative command might be: “Move to the right,” while an object-relative command could be: “Move closer to the stack.” For each prompt, the three withdrawal methods are tested at three coordinates along the trajectory (9 trajectories per prompt): one at the beginning, one in the middle, and one at the end. The video displays an example of a withdrawal trajectory for a middle trajectory point with the prompt “Move to the left.” In the examples below, withdrawal 1 (orange) is the original method presented by Garcia et al., withdrawal 2 (red) uses a position suggested by GPT, and withdrawal 3 (green) combines GPT-4 suggestions with the original method.

The robot is stopped while picking up a cube, which is not visible to the user as shown in the figure above. There are 25 cubes on the table, from which the robot can choose to change objective based on user input. Similar to the first experiment, the user’s input can include commands either relative to the user or relative to objects in the scene. For example, a user-relative command might be: “Don't take cubes on the right,” while an object-relative command could be: “Take cubes that are close to the robot.”