PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents
PhoneCLI turns repeated app navigation into deterministic commands that mobile agents can call without paying a VLM cost each step.
The paper says it builds those commands by exploring an app externally and mapping screens, elements, and navigation paths. At runtime, the agent chooses a compiled command, verifies it, and executes the replay path in under a second when it holds. If the path fails or the task needs open-ended interaction, it falls back to the embedded VLM agent. The authors report higher task success on AndroidLab, fewer steps and tokens, and efficiency gains when transferred to AndroidWorld’s M3A agent. ArXiv · AI/CL/LG's note
The paper says it builds those commands by exploring an app externally and mapping screens, elements, and navigation paths. At runtime, the agent chooses a compiled command, verifies it, and executes the replay path in under a second when it holds. If the path fails or the task needs open-ended interaction, it falls back to the embedded VLM agent. The authors report higher task success on AndroidLab, fewer steps and tokens, and efficiency gains when transferred to AndroidWorld’s M3A agent. ArXiv · AI/CL/LG's note
score 5