Agent-Native Interfaces

Research Notes › RSI · CLI

Notes on how agents should operate software — through the pixels of a human interface, or through a constructed, verifiable command surface. One paper per entry.

Start here → GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents (Zhou et al., Jun 2026). The cleanest controlled comparison in this space: 440 desktop tasks where both modalities get identical goals, initial states and verifiers. GUI wins out of the box (59.1% vs 48.2%) — but only 37.6% of verifier checkpoints are reachable through the CLI skill layer at all, and patching that coverage lifts CLI to 69.3%. The bottleneck is the interface, not the model.
The full reading list is at the bottom of this page.

Why this series is not called "CLI"

Because the shell is not the point. The same papers show MCP tool agents — also a programmatic interface — collapsing to 11.5% on ServiceNow with 83 curated tools, while seven generic CRUD tools on another platform nearly match a browser agent. What separates the winners is not command line vs pixels but whether the interface exposes a contract: explicit state, stable verbs, deterministic feedback, programmatic verification, and a way to discover all of it at task time. CLI is currently the cheapest way to get one.

The map

Four layers, ordered by where the stable semantic interface comes from — plus an evaluation axis that runs across all of them.

All four layers do the same thing in different currencies: replace per-step open-world reasoning with a contract constructed once. They differ only in who pays for that contract, and when.

Five claims the field currently rests on

1. The interface can outweigh the model. On matched tasks, GPT-5.4 through a GUI (59.1%) beats a stronger model through a skill-mediated CLI (48.2%). Swap the harness under a fixed model on WeaveBench and Opus 4.7 goes from 41.2% to 13.2%.
2. The CLI bottleneck is skill coverage, not capability. 37.6% checkpoint coverage → 69.3% success once patched. Whether that coverage can be synthesized without looking at the verifier is stated as an open question.
3. Hybrid is not free. Simply exposing GUI + tools drops Claude-4.5-Sonnet from 61.9% to 48.4% on OSWorld. It has to be trained (ToolCUA) or forced by task design (WeaveBench: ≤3.5% single-channel vs 35.1% hybrid).
4. The case is economic as much as it is about accuracy. Terminal agents run 2–9× cheaper than browser agents at equal success; compiled skills cut tokens 82–96%; CLI agents take roughly half the steps.
5. Outcome-only grading is not safe here. Trajectory auditing drops GPT-5.5 from 53.5% to 33.3% on WeaveBench, and the single largest failure family is reward hacking — fabricated renders, hard-coded metrics.

And the ceiling is real: oracle solutions show ~87% of mobile benchmark tasks are CLI-solvable. Cameras, freehand drawing, values that only exist in a rendered panel — for those, the GUI is not a fallback, it is the only path.

Reading list

Thirteen papers, in writing order. The tag is the layer from the table above.

• framing Why GUI pixels are not agent-native — the series intro. A GUI is a lossy, vision-only encoding of a contract that already exists inside the application, and the agent has to decode it back on every invocation: control discovery, visual grounding, state inference, context maintenance, re-derived each time. Anchor: deleting one file takes 3 CLI steps vs 7 GUI steps (Beyond the GUI Paradigm, Fig. 1).
• eval GUI vs. CLI: Execution Bottlenecks — Zhou, Zhang, Zhao, Wei, Song, Cohan, Zhao (NYU Shanghai + Yale + NTU), Jun 2026 · arxiv · the cleanest controlled experiment: 440 tasks, 18 apps, identical goals/states/verifiers, modality-native action spaces. GUI 59.1% vs CLI 48.2%; only 37.6% of verifier checkpoints reachable through the skill layer; verifier-guided patching → 69.3%. Complementary failure taxonomies: GUI = grounding + long workflows, CLI = skill coverage + implicit-default reconstruction + unobservable semantics.
• eval OSWorld-MCP — Jia, Liao, Zhang, Xu, Xie et al. (PKU + Tongyi Lab), Oct 2025 · arxiv · the first attempt at a fair GUI-vs-tools comparison; 158 curated MCP tools on top of OSWorld; introduces TIR (tool invocation rate) and ACS (avg completion steps). Tools help almost everyone, but Claude-4-Sonnet's TIR is still only 36.3%, and performance degrades as the catalog grows.
• eval WeaveBench — Li, Zhou, Yu, Xu, Yang, Li, Shan (MSRA + Tsinghua), Jun 2026 · arxiv · 114 long-horizon tasks that require both channels (median 76 tool calls, 16 GUI↔CLI switches). Single-channel ≤3.5% vs hybrid 35.1% (+31.6pp, against +3–4pp on prior hybrid benchmarks). A trajectory-aware judge drops GPT-5.5 from 53.5% to 33.3%; reward hacking is 35% of all failures.
• L0 Terminal Agents Suffice for Enterprise Automation — Bechard, Marquez Ayala, Chen, Skelton, Davasam, Sunkara, Yadav, Rajeswar (ServiceNow), Apr 2026 · arxiv · 729 tasks on ServiceNow / GitLab / ERPNext. Terminal 73–79% vs web 69–80% vs MCP 33–39%, at 2–9× lower cost. A single generic api_call tool recovers most of MCP's deficit → the problem is catalog granularity, not the protocol. Self-written skills cut ServiceNow cost 43.7%.
• L0 Beyond the GUI Paradigm: Do Mobile Agents Need the Phone Screen? — Gu, Jiang, Guo, Chi, Wang, Liu, Yu, Chen, Wang (Mila/Concordia + Toronto + McMaster), Jun 2026 · arxiv · off-the-shelf coding agents driving Android through ADB alone beat GUI models post-trained on mobile data (71.8% vs 69.3/68.1/57.8 on AndroidWorld). Introduces the CLI-Advantage Suite (45 tasks GUI benchmarks structurally cannot sample). Across 3 harnesses × 4 model APIs, generic tools only help weak models (Codex +14.3pp, Opus 0.0pp). Oracle ceiling 88.8% / 86.3%.
• L1 CLI-Anything: Towards Agent-Native Computer Use — Yang, Fan, Huang (HKU), Jun 2026 · arxiv · code · the manifesto. A 7-step harness-lift SOP and a contract H = (S, C, I, R, V, D). Two growth modes: Blender (the backend already exists — lower JSON scene state into bpy) vs Slay the Spire II (the backend must be created by an in-process bridge). Slogan worth keeping: if an artifact can be checked by code, it can usually be built by code. 83 CLIs / 32 categories; 87.8% of CLI-Hub calls come from agents.
• L1 OpenCLI (system, not a paper) — jackwener, 2026 · github · 179 sites / 1332 commands / a Chrome browser-bridge reusing your logged-in session. The pieces the papers lack: a degradation ladder (adapter → sitemap-guided browser → raw primitives) with a promotion path back up; trace-based online self-repair whose oracle is re-running the command; machine-readable failure contracts (sysexits); and the only public decay measurement I know of — fixes/adapter-year by strategy (PUBLIC_API 1.18, COOKIE 2.01, UI_SELECTOR 1.92, PAGE_FETCH 8.41, INTERCEPT 8.69). The line is contract vs no contract, not API vs DOM.
• L2 SkillDroid: Compile Once, Reuse Forever — Chen, Bellucci, Sun, Jacucci (Helsinki + UC3M + Shenzhen), Apr 2026 · arxiv · compiles LLM-guided mobile trajectories into parameterized skills replayed with zero LLM calls; 85.3% vs 62% baseline, −49% LLM calls, 100% over 79 replay rounds. Best line to steal: stateless agents are non-convergent — the baseline decays 80% → 44% over 150 rounds while SkillDroid climbs 87% → 91%.
• L2 AutoRPA: GUI Automation through LLM-Driven Code Synthesis from Interactions — Chen, Hu, Yu, Yin (Hangzhou Dianzi Univ), ICML 2026 · arxiv · translator + builder agents distil ReAct trajectories into robust RPA functions; hybrid repair resumes ReAct from the breakpoint. −82~96% tokens, and the synthesized code beats its own teacher (no sampling noise at test time).
• L2 AppAgent-Claw: CLI Is All You Need for GUI Automation — Song, Zhang, Song, Zhang (Westlake AGI Lab), Jun 2026 · arxiv · record-once / annotate-once / replay-many when there is no backend to lift. Three-layer localization (anchor → context → monitor-relative) with post-action validation. Under dark mode, layer-1 hit rate collapses 74.4% → 0% yet end-to-end stays 100% — reliability comes from managing failure, not eliminating it.
• L3 ToolCUA: Optimal GUI-Tool Path Orchestration — Hu, Zhang, Xu, Qiao, Yang et al. (Tongyi Lab + Fudan + Shanghai AI Lab), May 2026 · arxiv · code · the table that should be in every talk: handing frontier models a hybrid action space hurts (Claude-4.5-Sonnet 61.9 → 48.4; EvoCUA-32B 52.6 → 40.5). Fix: synthesize interleaved GUI-Tool trajectories from existing GUI corpora, then online RL with a tool-appropriateness + path-length reward. An 8B model reaches 46.85% on OSWorld-MCP with the lowest step count of any model.
• closing The skill layer rots — my own angle, written last. No paper measures decay, because benchmark environments are frozen snapshots and OpenCLI's sites are not. The claim: contract strength (who pays the maintenance bill) predicts decay better than contract level (API vs DOM) — and the open question left by GUI-vs-CLI, can coverage be synthesized without peeking at the verifier?, is really a question about where a first-party, CI-maintained specification of an app's behaviour can be found.

Adjacent work — nearest neighbours from an arXiv sweep, for related-work only; none of them occupies the "app's own test suite → agent interface" cell. UIFormer (synthesizes DSL programs compressing UI representations, −48.7~55.8% tokens; frames the lack of Boolean oracles as the fundamental obstacle — but optimizes the observation, not the action contract) · TDAD (compiles agent prompts from behavioural specs turned into executable tests; the reusable part is its anti-specification-gaming protocol) · SkillFab (demand-first skill lifecycle: unmet capability → skill → review → registry) · MACdroid (abstracts general test logic from source tests and re-concretizes it — exactly the lift needed, but the output is another test) · ReuseDroid · NL2Test (carves replayable API regression tests from captured traffic; 3,196 tests in production, 85.4% adoption) · TestGen at Meta · "Modern web test suites rot" (counter-evidence to stay honest: UI refactors break locators and teams abandon suites within weeks) · CLI task-synthesis cluster — CLI-Universe, SETA, Terminal-World (keyword-adjacent but orthogonal: they synthesize tasks and environments for training, not interfaces for acting).