Computer-Use Agents: Methods, Data, Evaluation and Trust

Abstract

Computer-use agents — systems that operate graphical interfaces through screenshots, accessibility trees and keyboard-and-mouse actions — moved in roughly three years from scripted proofs of concept to trained models that complete a majority of single-application tasks on several benchmarks. This survey maps that transition across the full agent loop: observation and interfaces, perception and grounding, planning and memory, action generation and execution, data, training, evaluation, safety and deployment. We read 121 papers in full and screened a corpus of 613 relevant works.

Four findings organise the field. First, the action interface rather than the observation does most of the work: controlled ablations that change only the alignment between what an agent sees and what it can do produce larger gains than observation compression, and the strongest desktop systems are those that route between GUI control and code execution. Second, the measurement apparatus is the weakest component: independent audits show no evaluated LLM judge exceeding 70% precision, rule-based judges under-reporting success by 16–19 points and inverting model rankings, and a published grounding benchmark containing roughly 11% label errors. Third, the site of the gain is verification rather than planning: decomposed ablations attribute almost all of the benefit of backtracking to judging whether the resulting state is the intended one, not to checking that an action executed. Fourth, hijack is not harm, and the field routinely reports the former as the latter: three independent security studies measure the full chain and find end-to-end attack success far below injection rates.

We also document what the literature does not settle: reporting is dominated by single runs without variance, training and evaluation distributions increasingly overlap, several of the largest announced data releases have no in-text location, and human oversight fails as a safety backstop in the one study that tested it directly. The survey closes with five foundational disagreements stated with evidence on both sides and eight research directions derived from the gaps.

Scope and method

Subject boundary. This survey covers agents that operate graphical interfaces — web browsers, mobile applications, and desktop operating systems. Terminal- and API-only agents appear only as contrast, because the hybrid-interface evidence (Section 10.5) requires their presence as a comparison condition; they are not treated as a separate chapter or corpus. Reports of safety-relevant behaviour in these systems are included; general LLM safety work is not.

Cutoff. Works are included through v1 arXiv submission dates of 12 October 2026. arXiv identifiers in this report were verified against library metadata at the time of writing; where a paper's identifier could not be verified, its number is still cited but its entry is based on the metadata we could confirm.

Construction. Candidate retrieval combined keyword rules, semantic recall and the daily arXiv announcement stream, unioned into a single pool rather than treated as competing filters, restricted to the period above and to the relevant subject classes. Screening applied one shared inclusion condition — the work concerns agents that operate GUI environments — with CLI-only agents, pure terminal benchmarks and general-purpose LLM work excluded. This left 613 included works from a pool of 833. The corpus was then assigned to a working taxonomy of twelve branches, and 121 papers were read in full and documented as structured evidence cards covering mechanism, setting, quantitative result, released artefacts and stated limitations. Papers cited for breadth rather than depth are used in grouped sentences and do not carry load-bearing claims.

Evidence tiers. Works read in full at the source and used for specific quantitative claims are treated as primary evidence. Works summarised from abstract and metadata only are marked as such in the text where their claims matter. Vendor-reported results and commercial system documentation (Section 12.5) are treated as vendor claims, not as verified measurements, and are never compared with peer-reviewed numbers in this survey.

Exhibits. Figures and tables are numbered in two separate sequences, each in order of first appearance, and every exhibit is referred to by its number from the text. Figure captions and table captions follow the exhibit they describe.

Where evidence is thin. Four areas carry weaker support than their chapter length suggests, and the text flags each: accessibility assistance, where both included works are prototypes and one has no participants with the disability it targets (Section 12.3); the CLI-contrast direction, supported by a single benchmark (Section 10.5); image-modality defences in safety, where one tested remedy fails outright (Section 11.5); and the picture-versus-structure question, where the one controlled model-by-observation study shows the ranking reverses across models (Section 4.4).

What this survey is not. It is not systematic in the sense of a registered protocol with formal inclusion/exclusion adjudication: screening used a single shared condition applied by an automated judge with sampled verification. It should not be read as a census of the field, nor should its counts be used to infer growth rates or category shares. Where we report a trend, the supporting periods and their comparability are given.

1 Introduction

A computer-use agent (CUA) is an AI system that operates a computer the way a person does — it looks at the screen, and it clicks, types, taps, and scrolls — in order to carry out a task stated in natural language. The instruction "book the cheapest direct flight to Lisbon on Friday and add it to my calendar" is executed end to end: opening a browser, filling a form, reading a confirmation, switching to a calendar app, and creating the event. No per-website API, no hand-written script, and no per-application plug-in is assumed. This is a deliberately narrow capability compared with the broader literature on tool-using language agents, and the narrowness is the point: the interface is the graphical user interface (GUI) that already exists, so anything a person can reach through a screen is in principle addressable.

The research area is young, fast-moving, and unusually fragmented. Early work automated web and mobile navigation with reinforcement learning over structured page representations [5, 7]. The arrival of large foundation models shifted the field toward instruction-conditioned policies that read a screen and emit actions [3], and by 2024 a recognisable benchmark ecosystem had formed around desktop, web, and mobile environments [5, 6, 8]. Since then the field has bifurcated into two mutually reinforcing tracks: native computer-use models, trained end to end on GUI trajectories and increasingly refined by reinforcement learning, and framework-style agents that orchestrate a frozen frontier model with perception, memory, and verification modules. Commercial products now ship the capability directly. Nine review articles have appeared in roughly two years [1, 2, 3, 4, 10, 11], which is itself a signal that the field is still consolidating rather than settled.

This survey is written against that background. We cover agents that operate a computer through its graphical interface, across web, mobile, and desktop/operating-system environments, through v1 submissions up to October 2026. Command-line and terminal-only agents are discussed only where they bear on the GUI problem (for example, hybrid agents that choose between clicking and running a command); purely terminal benchmarks and agents are outside the boundary. Within that boundary we ask a single question: how does a computer-use agent actually work, and what does the evidence say about which design choices matter?

The organizing framework of this survey is the agent's own information-processing loop. Every CUA, however built, must (i) obtain an observation of the screen, (ii) locate the thing the instruction refers to, (iii) decide what to do next, (iv) execute an action that the environment accepts, and (v) learn, if it learns at all, from data and reward. Chapters 4–9 take those stages in turn; Chapters 10–12 handle evaluation, trust, and applications, which cut across all stages. Figure 1 shows the structure.

Figure 1. Organization of the survey as five stages, read left to right.

Scope and method

The corpus was assembled from eight complementary retrieval routes over the local library — semantic recall for the core computer-use problem, web navigation, mobile control, desktop/OS control, visual grounding, training and reinforcement learning, and security — together with title-targeted keyword routes and reference-driven snowballing. The union of those routes was screened for relevance to the GUI-centric boundary above, giving 613 included works out of 833 candidates; 67 further works were added as verified anchor references. Citation tiers were assigned by reading depth: a set of works was read in full text and may carry mechanism and result claims, a larger set was read at method level, and the remainder are cited for what they state they do. The Appendix records the routes, the cutoff, and the areas where coverage is thin. We do not claim the corpus is exhaustive: the field has no unified terminology, and the sections on efficiency and on pre-2023 reinforcement-learning web agents rest on fewer inspected sources than the rest.

Contributions

First, a unified formal setup (Section 2) that expresses observation spaces, action spaces, grounding, and learning objectives in one notation, so that methods built for different platforms can be compared mechanism by mechanism rather than platform by platform. Second, a taxonomy organized by the agent loop rather than by the systems that instantiate it (Section 3). Third, an updated synthesis through October 2026 that covers the reinforcement-learning turn in training and the reliability problems that have surfaced as agents moved from sandboxes to live environments — a period largely absent from the earlier reviews [1, 2, 3, 4]. Fourth, a critical account of evaluation practice, where we separate what benchmarks demonstrate from what they are often taken to demonstrate [12, 13, 14]. Fifth, a first-class treatment of security, which for a system with mouse and keyboard control is not an afterthought [9, 10].

Takeaways

• Computer-use agents are instruction-conditioned policies over a GUI, not tool-use wrappers: the environment resists, the observation is visual or structural, and the action must be grounded.
• The field is consolidating quickly; several broad reviews already exist, so a useful contribution is an integrated account of the agent loop and an honest reading of the evidence, not another list of systems.
• Progress is uneven: model capability has advanced faster than our ability to measure reliability.

2 Preliminaries

2.1 Background and definitions

We use computer-use agent (CUA) for a system that receives a natural-language instruction and satisfies it by operating a computer's graphical interface — reading the screen and issuing clicks, keystrokes, taps, and scrolls. The term overlaps with, but is narrower than, several neighbours. It is narrower than tool-using language agents, which may act through typed APIs without ever seeing a screen. It is narrower than robotic process automation, whose "agents" are scripts tied to pixel regions or window titles and which break silently when a layout changes. It is broader than web agents alone, because the same capability is exercised on mobile operating systems and desktops. And it excludes game-playing and pure code/terminal agents except where those illuminate the GUI problem. This is essentially the boundary drawn by the closest existing review [1], which also excludes video games, server control, and coding agents; we follow it and additionally keep purely terminal benchmarks out of the corpus.

Two properties distinguish CUAs from ordinary instruction-following. First, the environment was designed for humans, not for machines: the observation is a rendering, not a message, and the action must be grounded to a physical location on that rendering. Second, the environment is stateful and only partially resettable — a real website cannot be rolled back, and a payment cannot be undone. Both properties drive the design space that the rest of this survey examines.

2.2 A shared formal setup

We adopt the partially observable, instruction-conditioned formulation used in the recent interface literature [47] and extended by later reviews [1, 4], so that methods built for different platforms can be compared in one notation. Table 1 lists the symbols.

Table 1. The shared notation. Sections 4–9 reuse these symbols so that methods built for different platforms can be compared term by term rather than platform by platform.

Three design families are then read off the same setup. Observation: is $o_t$ pixels, structure, or both? Action: is $a_t$ a coordinate pair, an element index, or a program? Grounding: is $g$ learned as part of $\pi_\theta$, or supplied by a separate localisation model, or made unnecessary by choosing an action space that already names the target (Section 4.2)? Learning is the fourth axis: the policy may be fixed and prompted, fine-tuned on trajectories by maximising $\mathbb{E}_{(o,i,a)\sim\mathcal{D}}\log\pi_\theta(a\mid o,i,h)$, or optimised against environment reward by maximising $\mathbb{E}_{\tau\sim\pi_\theta}[\sum_t\gamma^t r_t]$, with the reward either terminal or supplemented by a process model.

2.3 Development timeline

Figure 2 dates the works that mark the field's transitions. The pre-foundation-model era trained agents with reinforcement learning over structured page representations [16, 17]; the first large-model agent used a browser with human feedback [15]. From 2023 the field standardised rapidly: generalist web-agent datasets and environments [5, 7], an Android device-control dataset [18], a multi-environment agent benchmark [19], and the first visual language models for GUI control [21, 22], together with the mark-and-index prompting trick that made element grounding tractable for off-the-shelf models [20]. 2024 added real desktop and mobile environments and the first native action models [6, 8, 25, 29, 31, 35]; 2025 brought native computer-use models and the reinforcement-learning turn [37, 38]; 2026 is characterised by hybrid-interface benchmarks and by security work maturing into its own literature [42, 45, 46].

Figure 2. Representative milestones, by v1 submission date. Concurrent and later variants are cited in the chapters rather than here.

2.4 Related surveys

Eleven reviews and position papers frame this area (Table 2). None is redundant with another, but none covers the whole loop at the current cutoff. The closest, [1], offers a technology-agnostic taxonomy over domain, interaction, and agent perspectives with a cutoff in 2024 and excludes coding agents and software testing. The capability-axis reviews [2, 3] organise around perception/reasoning/planning/acting. [4] widens the interface to include command-line interaction, which we deliberately do not. The safety reviews [9, 10] and the reinforcement-learning review [11] are narrower still.

Table 2. Related surveys. "Cutoff" is the latest literature the survey substantively covers; comparability across rows is limited because each uses different inclusion rules.

Takeaways

• A CUA is defined by grounded operation of a human-designed interface, not by the use of tools.
• One POMDP-style setup covers all three platforms and exposes the four real design axes: observation, action space, grounding, and learning.
• Nine prior reviews exist; the contribution available now is currency, integration, and an honest reading of evidence, not novelty of topic.

3 Taxonomy

3.1 Why organize by the loop

A survey of this field can be organised by platform (web, mobile, desktop), by system lineage (which model succeeded which), or by the components of the agent itself. We adopt the last, because it is the axis along which methods actually differ. Two web agents built on the same backbone and benchmarked on the same site may share nothing except the loop they both instantiate; conversely, a mobile agent and a desktop agent that both replace an accessibility tree with a marked screenshot are making the same design choice, and it is more useful to compare them directly than to file them under different platforms.

The loop has five stages. Observation (Section 4) fixes what the agent sees. Perception and grounding (Section 5) turn that observation into a target. Planning, memory and reasoning (Section 6) decide what to attempt and what to remember. Action generation and execution (Section 7) turn the decision into something the environment accepts. Learning (Sections 8–9) covers what changes in the model or its data. Evaluation, trust and deployment (Sections 10–12) cut across all five. Figure 3 shows the tree with representative works at the leaves.

Figure 3. The taxonomy. Numbers cite representative works at each leaf; the leaves are not mutually exclusive — a single system typically fixes several of them at once, and the chapters explain where the choices interact.

Figure 4. What the Observation leaf looks like in practice, from OmniParser [118]. A screenshot is annotated with numbered boxes, and each box yields a text entry or a generated icon description ("Icon Box ID 24: a delete or remove function."). The policy then selects an index rather than a coordinate. This is the concrete form of the observation-interface choice analysed in Section 4.1.

3.2 Secondary axes

Three further distinctions recur and belong in a table rather than in chapter headings (Table 3).

Platform. Web, mobile, and desktop/OS differ less in method than in their observation substrate and their failure modes. Web offers a machine-readable DOM; mobile offers an accessibility tree of variable fidelity; desktop often offers neither reliably, which is why desktop agents lean hardest on pixels (see Sections 4.1, 10.2). Cross-platform transfer is now a stated goal of the native action models [31, 35, 41].

Learning paradigm. Prompted or scaffolded agents change nothing inside the model; fine-tuned action models change parameters on trajectory data; reinforcement-learned agents change parameters against environment reward. Section 9 examines where each is defensible and what the reported gains actually cost.

Verification style. Independent of the agent, an environment verifies by executing a task-specific script on the final state, by asking an LLM judge, or by replaying against recorded human trajectories. This choice determines how comparable any two reported numbers are (Section 10).

Table 3. The three cross-cutting distinctions, each with the chapters where it binds. They are kept out of the chapter headings deliberately, so that the loop remains the primary axis.

Takeaways

• Organising by the agent loop makes cross-platform comparison possible and exposes which component each method actually changes.
• Platform, learning paradigm, and verification style are second-order axes; they belong in tables, not in chapter headings.
• The loop is a description of shared structure, not a claim that good agents implement it explicitly — several frontier models collapse perception and planning into a single forward pass.

4 Environments and interfaces

4.1 Three observation families

What an agent can see determines what it must learn. The field has converged on three observation families, and the cleanest evidence that the choice matters comes from comparisons that hold everything else fixed.

Structure-only. The agent consumes a textual rendering of the page or application: an accessibility tree, a DOM serialisation, or a Markdown compilation of one. AgentOccam is the most instructive instance because it changes nothing else — no new model, no new training, only an alignment of the observation function and the action function to each other. It compresses the accessibility tree into Markdown, drops actions the text interface cannot express (scroll, hover, press) and adds note/stop plus planning actions, and reports 43.1% on WebArena against 33.3% for the strongest prior agent and 16.5% for a vanilla baseline [47]. Its ablation decomposes the gain: aligning the action space contributes the largest share, followed by disabling scroll, then observation compression. That ordering is itself a finding — the action interface, not the observation, did the most work.

Pixels-only. No structural input at all. Pix2Act established the paradigm: pure-pixel observation, coordinate actions, behaviour cloning from human demonstrations with pixel-level pre-training, and Monte-Carlo tree search at inference, reaching 96.2% average on MiniWoB++ while beating DOM-based baselines [16]. Cradle carries the same commitment to its logical end — screen video in, keyboard and mouse out, no APIs — and reports 7.81% on OSWorld against 5.03% for GPT-4o and 4.59% with Set-of-Mark overlays [128]. The pure-pixel setting also defines the General Computer Control research problem in its strongest form, and the low absolute numbers show how much remains.

Hybrid and compiled. Most deployed systems take inputs from both channels and fuse them, and the 2026 variants go further by compiling the interface into an addressable structure. UFO2 is the clearest desktop instance: an accessibility-tree-primary control graph augmented by vision, a host agent that plans and application-specialised agents that execute, and an action space that is the union of GUI events and native application APIs [125]. It reaches 30.5% on Windows Agent Arena against 20.8% for a strong commercial agent.

Figure 5. UFO2's control architecture, redrawn from the paper's framework figure [125]. A host agent decomposes the task and instantiates one application-specialised agent per Windows application; each specialised agent reports subtask completion back to the host, which owns scheduling and application lifecycle. The layered split — one planner, many executors bound to specific applications — is what makes the union of GUI events and native APIs usable in one action space. Web$hell compiles live pages into addressable information and action objects with a persistent versioned cache, and reports 0.679 task success on WebArena-Verified against 0.264 for the comparison system at cost parity [126]. Beyond Pixels takes a lighter route and downsamples the DOM itself — merging containers by depth, converting content to Markdown, pruning attributes and sentences — reporting 73% against 65% for a grounded GUI baseline on 18 Online-Mind2Web tasks at roughly a fifth of the token cost [141]. Tactile compiles multi-source UI evidence into a candidate target set that carries role, text, state, geometry, affordance and provenance, and splits grounding, execution and verification into four explicit stages, lifting Codex's Success@100 from 41.1% to 50.0% on a macOS benchmark [129].

The extreme of the structural direction is ASIL, which discards screenshots altogether: the environment exposes a typed JSON state, and the agent acts through schema-constrained semantic calls — set_value, invoke_function, modify_file — behind an adapter contract [123]. On 380 tasks across 15 applications it reports 81.6% for one frontier model against 6.6% under a GUI condition, in fewer than five actions per trajectory. We flag two caveats: the comparison is asymmetric, since the structured side received evaluator-derived hints the GUI side did not, and the arrangement presumes the applications cooperate.

4.2 From coordinates to semantics

The action space has moved in one direction over the period covered, and the stages are visible in this chapter's works (Table 4).

Table 4. The action-space stages in the order the field moved through them, with a representative work and the failure characteristic of each.

The trend is not a refutation of coordinates but a widening of what counts as an action, and its clearest evidence is that systems combining channels outperform single-channel ones (Sections 7.4 and 10.5).

4.3 The interface is a design position

Two works argue that the interface should not be the agent's problem at all. The Agentic Web Interface position paper proposes that sites expose their information and controls in a machine-actionable form, moving grounding responsibility to the site rather than the agent, and cites DOMs exceeding a million tokens and per-task cost as motivation [130]. It offers principles and recommendations rather than a prototype; its value here is as the explicit statement of an assumption that Web$hell, Tactile and ASIL all act on [126, 129, 123].

This is where computer-use agents meet a question the CLI world did not face. A terminal is an interface designed for a machine; a web page is not. The hybrid-interface results of Section 10.5 are the empirical form of the same argument, and they are what make this position more than a manifesto.

4.4 What the controlled comparisons actually settle

One work deserves highlighting because it refuses to take a side. Holding the action interface fixed and varying only the observation representation and the history, Read More, Think More finds that the answer is model-dependent rather than monotone: richer HTML observation is worth +17.5 percentage points for one model and −18.8 points for another on the same benchmark [127]. The interaction between observation format and model is large enough to swamp the average effect, which means "pixels versus DOM" is not a question with a general answer — it is a question about a specific model, a specific site, and a specific token budget.

That result should govern how the rest of this chapter's literature is read. Aisles of papers report that their observation format is best; almost none of them vary the model family while holding everything else fixed, and the one that does finds the ranking flips.

Takeaways

• Three observation families coexist: structure-only, pixels-only, and hybrid/compiled, with direct ablations inside each.
• AgentOccam's decomposition is the clearest single result here: aligning the action interface mattered more than compressing the observation.
• The action space widened from coordinates to element indices to programmatic and semantic calls; systems that combine channels outperform single-channel ones.
• The fully compiled or symmetric-structure position is credible on its own benchmark but rests on application cooperation and, in the strongest case, on asymmetric comparison conditions.
• The one controlled model-by-observation study shows the ranking between pixels and structure is not stable across models, which weakens most absolute claims about the best interface.

5 Perception and grounding

5.1 From reading structure to seeing pixels

An agent that cannot find the element it means to click cannot act. For the first generation of web agents this was solved by reading: the accessibility tree or DOM supplied a list of elements with names, roles and bounding boxes, and the agent selected one. The cost of that solution is the dependency itself, and three quarters of the work in this chapter exists to remove it.

The progression is visible in the sequence of artifacts. SeeClick reframed the agent problem as a grounding problem and trained a model to output coordinates from a screenshot, showing that about a million grounding samples could give a general-purpose model usable screen-locating ability where the same model had none [25]. Set-of-Mark prompting showed that a general vision-language model could select among numbered overlays rather than infer coordinates [20]. OmniParser took the step that most cleanly separates perception from policy: freeze the model, and have a separate fine-tuned detector, icon captioner and OCR produce the element list and the numbered overlay, so that a stock GPT-4V consumes a structure it never learned to extract [118]. UGround then made pure-pixel grounding a pluggable module — a planner emits a short natural-language description of the target and the grounding model converts it to a coordinate — and trained it on 10M synthesised elements [29]. By the end of 2024 the dominant architecture was two-stage: a planner that does not see pixels and a grounder that does nothing else.

Two earlier works in the same lineage are still the clearest statements of the perceptual problem. Screen Parsing argued in 2021 that the task is not detecting elements but reverse-engineering the interface model — recovering elements together with their grouping and hierarchy — a framing that the later pure-pixel models mostly abandoned; our evidence for its reported 23% improvement is limited to the abstract, since the body was not available for this review [117]. CogAgent added a dedicated 1120×1120 high-resolution cross-attention branch to a low-resolution backbone and pre-trained on 140M screen-element pairs, reaching 58.2% step success on Mind2Web and beating text-fed models of far greater size [21]. That result was the first strong evidence that pixels need not lose to structure, and it made resolution the central engineering question.

5.2 Resolution is the failure axis, and it is now quantified

ScreenSpot, the standard grounding benchmark, has targets averaging about 2% of the screenshot. ScreenSpot-Pro, built on 1,581 samples from 23 professional applications across three operating systems, reduces that to 0.07% [119]. The effect of that change is dramatic and worth stating precisely, because it is the single most informative measurement in this chapter (Table 5).

Table 5. Grounding accuracy by benchmark setting. The rows are not comparable with one another: ScreenSpot and ScreenSpot-Pro differ by orders of magnitude in target size, which is the subject of the discussion below.

The gap between 82% and 19% is not a model-generation gap; OS-Atlas-7B is a strong grounder on ScreenSpot and a weak one on ScreenSpot-Pro. It is a resolution gap, and it means that benchmark scores quoted without a resolution regime are close to meaningless. The same works also show the failure is graded rather than binary: accuracy falls monotonically as target size shrinks [119].

5.3 Two remedies: train bigger, or search smarter

Training-side answers all amount to giving the model more pixels of the right patch. CogAgent added a high-resolution branch [21]; Iris replaces uniform resizing with information-density-driven adaptive cropping, allocating more crops to denser regions [121]; ShowUI drops redundant visual tokens inside connected UI components, cutting visual tokens by a third and training time by 40% [34]; OS-Atlas scaled the corpus to 13.58M elements across five platforms [31]; Phi-Ground systematically varied the data engineering — 40M-scale crawling, reference-expression style, coordinate conventions — and found that the representation of the grounding target, not the architecture, drives the result [120].

Inference-side answers are more surprising, because they work without training anything. ScreenSeekeR has a planner hypothesise candidate regions, crops recursively, and lets the grounder work on a small image, reaching 48.1% on ScreenSpot-Pro against 18.9% for the best end-to-end model at the time — a 29-point gain with no parameter updates [119].

Figure 6. Recursive region narrowing on a ScreenSpot-Pro task, from the ScreenSpot-Pro paper [119]. The top row shows a single-shot model and a partial-zoom baseline both missing the target; the bottom row shows the same task solved by successive narrowing, where each step supplies a text description of the plausible area ("File Explorer window", then "top action bar", then "toolbar at the top of the file explorer window") before the final grounding decision. The gain in Section 5.2 comes from this loop, not from a larger model. Visual test-time scaling generalises this: propose a focus point, crop four proportional regions, sample a candidate action in each, and aggregate by drawing the candidates as landmarks on the image and letting the model choose [122]. It lifts Qwen2.5-VL-72B from 47.8 to 61.6 on ScreenSpot-Pro and UI-TARS-72B from 38.1 to 50.2, with a 7B model plus the method (41.2) beating a 72B model without it (38.1).

The cost is real and the paper reports it honestly: 66.8% more expensive web trajectories, and 180–360% overhead on ScreenSpot-Pro. This is the same trade-off that appears in test-time scaling for planning (Section 9.4), and in both cases the gain is largest when the budget is constrained but not exhausted.

5.4 Self-labelled data, and its characteristic bias

Grounding corpora are largely machine-generated, and the published numbers for autonomy are notable. UGround synthesised 10M elements by rules plus a language model [29]; Aria-UI built 11.5M samples by captioning elements and then humanising the captions into instructions [36]; OS-Atlas reached 13.58M elements [31]. Iris's contribution is to close the loop explicitly: a refer-and-ground self-consistency check accepts generated pairs whose round-trip recovers the original target, producing about 3M self-labelled samples and reaching 74.6% on ScreenSpot at the same data volume where SeeClick gets 53.4%, i.e. slightly above UGround's 10M-annotated 73.3% [121].

The same paper names the bias this procedure introduces: round-trip agreement is easiest for large, common elements, so self-labelling over-samples them. Its own desktop icon and widget score (57.5) trails UGround (63.6). Ferret-UI, which builds its training signal from an external element detector rather than a round-trip, reports the complementary failure — it cannot learn about elements its detector never fires on, such as status-bar indicators [27]. Both are instances of the same structural problem: in a self-labelled pipeline the label distribution is the labelling instrument's distribution, and errors in it are invisible in the final score.

5.5 The measurement problem sits inside the benchmark

OS-Atlas inspected ScreenSpot by hand, found roughly 11.32% of labels to be wrong, corrected them, and re-released the benchmark as ScreenSpot-V2 [31]. Any comparison made against the original labels before that correction carries an error bar of unknown sign. Two other measurement concerns recur: Ferret-UI scores grounding at IoU greater than 0.5, which is a weak proxy for whether a click lands on the intended control [27] — a point is either inside the box or it is not; and ScreenSpot-Pro reports single runs with no variance [119].

It is worth connecting this to the evaluation audit of Section 10.3. Grounding scores are the most frequently quoted single number in the field, and at least one of the benchmarks producing them was measurably wrong by about a tenth.

5.6 Grounding is not agency

The temptation is to read these tables as a capability ranking. They are not. UGround and Aria-UI are explicitly non-autonomous — both require an external planner and neither can be deployed standalone [29, 36]. Aguvis makes the point experimentally: it holds the best single-step grounding figures in its comparison (81.8 for the 7B model on ScreenSpot, 89.2 for the 72B in the self-plan setting) yet scores only 10.26% end-to-end on OSWorld, well below the 17.04% obtained by pairing GPT-4o's planner with Aguvis's own grounding module [35]. Ferret-UI, conversely, beats GPT-4V on elementary grounding tasks by wide margins while falling below it on the advanced reasoning tasks [27].

The correct reading is that grounding is a necessary component whose measured accuracy does not compose linearly into task success. Section 10.2 shows why: a step-level error rate of a few per cent compounds over twenty steps, and that is before planning errors enter.

Takeaways

• The field moved from reading structure to seeing pixels, ending with a two-stage architecture — a planner that does not see pixels and a pluggable grounder — that persists into 2026.
• Resolution is the dominant failure axis and is now quantified: the same model family scores 82% on ~2% targets and 19% on 0.07% targets.
• Training-free inference-side methods (recursive search, visual test-time scaling) deliver some of the largest grounding gains in the survey, at a 1.7–4.6× cost.
• Self-labelled corpora reach parity with far larger human-annotated ones, but inherit the labelling instrument's bias toward large, easily round-tripped elements.
• At least one grounding benchmark was found to contain ~11% label errors and had to be re-released, which should temper confidence in the most-quoted numbers in the field.
• Strong grounding does not compose into a strong agent: the best-grounding model in this chapter is not the best end-to-end agent.

6 Planning, memory and reasoning

6.1 Memory is the least standardised component

Every long-horizon agent needs to remember something, and the field has not agreed on what. Three designs recur, and they are not variants of one idea.

Episode and skill libraries. Finished trajectories are stored and retrieved by embedding. WebCoach writes only completed trajectories to an episode store and lets a small model decide whether to inject one or two sentences of advice, lifting Skywork-38B from 0.473 to 0.614 on WebVoyager [54]. MapAgent distils each visited page into structured page-chunks per application in a vector database, retrieves the top three, and plans coarse-to-fine; on cross-application mobile tasks it reaches 0.350 against 0.200 for the strongest baseline [61]. Mirage-1 organises the same idea hierarchically across Meta/Core/Execution layers and searches at the subgoal level, reaching 42.2% on AndroidWorld [59].

Running summaries. Rather than a library, the agent maintains a compressed account of the current task. Mobile-Agent-v2 runs three roles — planning, decision, reflection — with a textual progress memory, and lifts task success from 43.2% to 75.0% on 88 live mobile instructions [64]. PAL-UI adds an explicit look-back action that re-fetches a past screenshot, together with a two-level summary, and improves AndroidControl-High step success by 10.7 points [62]. MobA uses stack-driven depth-first decomposition with a five-part memory and re-plans only when reflection says the current path is infeasible [58].

Predicted state. The third family does not store the past at all but predicts the future, and is treated in Section 6.3.

The counter-evidence matters as much as the gains. WebCoach's mechanism regressed on a 7B backbone (0.328→0.311) [54], and PAL-UI's look-back action actively hurt performance when the summary was removed (52.3 against 57.8 with it) [62]. Retrospective actions appear to need a compressed representation of the task to be useful; without one, revisiting the past is closer to noise than to information.

6.2 Backtracking and replanning: the cleanest controlled evidence

If the memory literature is heterogeneous, the backtracking literature is unusually well controlled, and it supplies this chapter's most trustworthy numbers.

WebRollback adds two explicit capabilities to a greedy search agent: a binary critique deciding whether to continue or go back, and a rollback operation that selects a historical step to return to, which may be many steps earlier. It reports 27.36% full-completion on Mind2Web-Live against 24.53% for one-way search and 23.82% for best-first, with three-run means and reported dispersion [53]. BEAP-Agent runs a state-space depth-first search with a persistent tracker and multi-level backtracking, scoring 28.2% on OSWorld against 24.0% for its base model, with an ablation showing −1.9 points when backtracking is removed and −4.6 when the tracker is removed; 35.8% of tasks triggered a backtrack at least once [55].

The negative result in this group is equally instructive. Agent+P builds a UI transition graph by program analysis and solves for a shortest path with a classical planner, translating the plan into natural-language guidance for an unchanged agent. On a 44-task subset across five applications it lifts T3A from 24.32% to 38.64%; run on all 116 tasks, the same intervention yields +1.72 and +0.86 points [60]. Subset gains that do not survive the full benchmark are the clearest available example of why this chapter's headline numbers should be read with their task sets attached.

6.3 World models: a live disagreement

Whether an agent should imagine the consequences of an action before taking it is the most genuinely contested question in this chapter, and the two sides are both well evidenced.

WebDreamer treats the language model as a world model: it simulates the next observation and scores the simulated trajectory, then acts on the better branch without touching the environment. It reports 23.6% on VisualWebArena against 17.6% for the reactive baseline, 37.0% on Online-Mind2Web against 26.0%, and 25.0% on Mind2Web-Live against 20.2%, while running four to five times faster than tree search [56]. MobileDreamer implements the same idea for mobile screens, predicting the next screen as a structured text sketch of labels, text and boxes, matched by optimal transport, and lifts a strong model from 60.53% to 65.78% on AndroidWorld [57].

Three limits are reported by the same papers. WebDreamer's gains shrink as the imagined horizon grows, and beyond a depth of one the method degrades [56]. MobileDreamer depends on an OCR-derived sketch and reports a 23.4% rate of failing to find the correct screen [57]. And both depend on the world model being accurate in exactly the regime where it is least reliable — long horizons in unfamiliar interfaces. The disagreement is not about whether simulation works but about how far ahead it remains calibrated.

6.4 Decomposition and hierarchy

Hierarchical decomposition is the oldest idea in this chapter and the least controversial. MobA's stack-driven depth-first scheme reaches 28 of 50 first-attempt completions on MobBench against 17 of 50 for the previous multi-agent design, with an efficiency score of 3.44 against a human reference of 3.53 [58]. Mirage-1 combines a three-layer skill memory with subgoal-level search [59]. The interesting observation is methodological rather than architectural: several of these systems report their gains at step level rather than task level (PAL-UI at step success [62]; the diagnostic below at subgoal agreement [63]), which is a weaker unit of evidence, since a step-level improvement need not survive compounding over a trajectory.

6.5 What the diagnostics say

Two works step back and ask where failures originate. One reformulates web agents as a four-component process — high-level plan, low-level execution, postcondition check, replanning — and varies whether the high-level representation is natural language or PDDL, on 104 Mind2Web-Live tasks [63]. Its most useful descriptive finding is about horizon: the plans it observes contain 3.04 subgoals over 7.08 actions on average. If typical tasks decompose into about three subgoals, then the deep hierarchical machinery of Section 6.4 may be over-engineered for current benchmarks, and much of the reported benefit of planning may be the benefit of checking, not of decomposing.

This connects to the verification finding of Section 7.2 and the representation finding of Section 4.4. In three different places — planning, acting, and seeing — the controlled comparisons locate the gain in the feedback and verification loop rather than in the sophistication of the forward plan.

A standing caveat applies to this entire chapter. Across the cards inspected for this section, essentially no paper reports multiple random seeds with dispersion or significance tests; WebRollback's three-run means [53] and BEAP's ablations [55] are the exceptions. Single-run gains of a few points on a few hundred tasks are within the range that environment and model-version variation can produce, and none of these studies establishes otherwise.

Takeaways

• Memory designs split into embedded episode or skill libraries, running textual summaries, and predicted state; none is standard, and two of the three have documented regressions on weaker models or without a summary component.
• Backtracking and replanning give the cleanest controlled evidence in the chapter, with decomposable ablations (−1.9 for backtracking, −4.6 for the tracker on OSWorld).
• Subset gains can vanish on the full benchmark: a symbolic-planning intervention worth +14 points on 44 tasks is worth under 2 points on 116.
• World-model planning is genuinely contested: it beats reactive agents and is faster than tree search, but its benefit decays with horizon and its sketch prediction misses the screen in nearly a quarter of cases.
• Typical tasks decompose into about three subgoals, which suggests current hierarchies are provisioned for tasks longer than the benchmarks contain.
• Almost nothing in this chapter reports variance; single-run, few-point differences should not be treated as established.

7 Action generation and execution

7.1 From choosing an action to proving one

Until 2024 the action stage was a generation problem: given the state and the plan, emit a click. Four independent lines of 2025–2026 work replace that with a verification problem, and they converge on the same structural change from four different directions.

The first scores candidates instead of sampling them. V-Droid enumerates the plausible actions in a state, scores each with a learned verifier built from the policy's own first-token probabilities, and takes the argmax; crucially it separates selecting a discrete action type from completing its parameters, so the expensive generative pass is only needed for the minority of steps that need free text [87]. On AndroidWorld this reaches 59.5%, and the controlled comparison is clean: with identical training data, verifier-driven selection gives 47.4% against 35.8% for a supervised selector and 27.4% for a generator. The design also pays for itself in latency — 0.44 s for the 88% of steps requiring no parameter completion — which matters because Section 10.4 shows planning and reflection dominate wall-clock cost in deployed agents.

The second learns to check its own prediction. VeriGUI makes action-effect verification a first-class output: each step the policy compares the previous step's predicted screen change with what actually happened, then acts and predicts the next effect. Its mixed reward is asymmetric, penalising a false verification claim at −2.0 while rewarding a correct one at +1.0 [92]. The result that matters is not the headline recovery rate (52.5% for the 7B model against 45.5% for UI-TARS-7B) but the offline figure: on AndroidControl-High the same-scale baselines score zero on simulated test success, while the verification-trained models reach 16.7% (3B) and 23.5% (7B).

The third gates with rules plus a learned judge. BacktrackAgent executes an action, then runs a rule verifier (is the action legal, did the page change at all?) and a learned judge (given the instruction, the page and the history, was that the right action?). If either fails, a reflector regenerates a different action at the same step rather than continuing [93].

Figure 7. BacktrackAgent's four stages, from the paper's overview figure [93]. Generation produces an action from a trajectory; detection runs a rule verifier and a vision-language judger over the executed transition; recovery hands the failure to a reflector that emits a replacement action for the same step; the trajectory then continues. The step-level detail on the left shows the retry loop, and the panel on the right shows the enumerated action space the verifier checks against. The ablation in Section 7.2 decomposes this pipeline and finds almost all of its value in the judger rather than the verifier.

The fourth does not touch the model at all. WebFovea hardens the harness — the parse–execute–feedback–observe round trip — and checks after each action whether a DOM fingerprint (URL, title, node count, text length, scroll offset, focus) actually changed [88].

Modular frameworks such as Agent S had already split the loop into planner, grounder and executor roles [30]; the shift described here is different, because it leaves the roles intact and changes what happens between deciding and committing.

Figure 8. The verification step that the four works in this section insert between decision and commitment. What differs between them is the mechanism used at C, not the position of the check.

7.2 What verification is actually worth

BacktrackAgent's ablation is the most informative single number in this chapter, because it decomposes the gain by component. Removing the whole backtracking mechanism costs 5.65 points of task success. Removing only the judge costs 5.32 of those; removing only the rule verifier costs 0.45 [93]. Checking that an action was executable and changed the screen — the cheap, trivially automatable check — is almost information-free. Almost all the value lies in judging whether the resulting state is the intended one, which is the expensive part and the part that needs a model.

The same lesson appears with a negative sign in the same paper. Backtracking degrades text input (IoU −1.60) because the reflector, asked to produce a different action, tends to substitute words that appear in the instruction rather than what the task requires [93]. A verification mechanism optimised for changing the action is not automatically a mechanism for choosing better actions, and this is the clearest documented case in the survey of a correction loop actively harming a sub-behaviour.

7.3 Execution-layer failures that look like model failures

WebFovea deserves separate treatment because it identifies a failure mode that the rest of the literature would attribute to the wrong component. Working on live websites through the Chrome DevTools Protocol at a 1920×1080 viewport, it recorded 1,312 clicks and found that not one of them landed beyond x > 1440 or y > 810 [88]. Calibrating with synthetic targets gave a required correction of 1.324 horizontally and 1.350 vertically when a full-resolution image was sent, and 1.010/0.993 when the image was sent pre-downsampled — establishing that the inference API was rescaling the screenshot before the model ever saw it, so every coordinate the model emitted was systematically compressed. Correcting the mapping took 24 paired development tasks from 8/24 to 20/24 and cut step-limit hits from 67% to 4%.

The same study documents three further execution defects of the same kind: native <select> popups are painted outside the page and never appear in a screenshot, so one agent clicked the same dropdown nine times before giving up; a reference prompt instructing the model to submit with a newline was never honoured because the executor never pressed Enter (79 of 356 type calls, 22%, were of this form); and iframe targeting failed because the agent addressed the top-level document. None of these are reasoning failures, and all of them would appear in a results table as one.

This is a general caution for reading this survey: a grounding or planning accuracy measured through a browser-control or emulation harness is a measurement of the harness-plus-model system, and only some papers isolate the two. WebFovea is the one work here that does.

7.4 Code as a first-class action

The most successful single system in this survey's desktop results is not a better GUI agent; it is an orchestrator that chooses per sub-task between a GUI operator and a programmer [89]. CoAct-1 adds Python and Bash execution to the action space and routes work accordingly, reporting 60.76% on OSWorld against 53.10% for GTA-1 with the same planner backbone and 56.00% for Agent S2.5, while using fewer steps per success (10.15 against 15.22 and 14.90). Efficiency and success moving together is the signature of a genuinely better action space rather than a longer search.

This result is the action-side counterpart of the interface argument in Section 4, and it is corroborated from the harness side by the hybrid-systems benchmark in Section 10.5, where GUI-only agents score at most 1.8% and CLI-only agents at most 3.5% while the hybrid configuration reaches 35.1%.

At the other end of the spectrum, the execution backend itself can be replaced. HIDAgent moves the agent out of the controlled machine entirely and onto a sub-$30 external hardware device that impersonates a keyboard and mouse and captures HDMI video, so any device that accepts HID input can be driven without installing anything [91]. The work reports no quantitative evaluation — the authors describe their evidence as descriptive — and ground-truth-resolution desktop grounding remains poor. It is included here because it is the only work in this survey that changes the trust boundary of execution rather than the policy, which matters for the safety analysis of Section 11.

7.5 Verification before execution

VeriSafe inverts the timing: instead of checking an action after it has been applied, it formalises the user's instruction into Horn-clause rules once per task, then uses developer-declared state transitions to compute the expected effect of each candidate action and block the action if a rule would be violated [90]. The cost is declarative: about 59 predicates and 437 lines of code per application across the 18 applications studied, 1,058 predicates in total, of which 540 are active in the experiments. The benefit is that one language-model call per task replaces one per step, and unsafe actions never reach the device.

The limits are equally clear and the authors state them: false positives can terminate a legitimate task; closed-source applications require humans to write the predicates in place of developers; and the approach does not claim complete safety. Section 11.3 treats this as the most concrete instance of a broader question — whether computer-use safety is best pursued by making the model more careful or by making the world checkable.

Takeaways

• The action stage is being reformulated from generation to verification, with four distinct mechanisms converging on the same structural change.
• Decomposed ablations show the value sits almost entirely in judging the resulting state, not in checking that an action executed.
• Execution-layer defects can dominate results: a screenshot-rescaling bug compressed every one of 1,312 clicks into a sub-region of the viewport, and correcting it raised success on the paired development tasks from 8/24 to 20/24, a factor of 2.5.
• Treating code as a first-class action improves both success and step efficiency, matching the hybrid- interface evidence from two other directions.
• State-of-the-art recovery mechanisms can damage specific sub-behaviours (text input) — verification is not free of failure modes of its own.

8 Data and trajectories

8.1 Where the trajectories come from

Every trained computer-use model in this survey is a function of a corpus, and the corpus is the part of the field that changed fastest. In 2023 the reference dataset was crowdsourced: Android in the Wild recorded 715,142 episodes from paid raters operating an emulator, with screenshots and pixel-inferred UI features [18]. By 2026 the reference scale is millions of trajectories per project, and almost none of them are human — WebWorld assembles 1.06M open-web interaction trajectories [75], MobileViews 1,213,866 unique screens [73], GUI-Net-1M a million-plus trajectories distilled from video tutorials [69].

The provenance of that supervision falls into three families, and the choice between them is the single most consequential data decision (Table 6).

Table 6. Where the trajectories come from. The three families differ in cost, in the failure mode they impose on the corpus, and in how much of the supervision is a model's own opinion rather than a record of what happened.

The mined-and-generated family is where the methodological interest now lies, because it removes the human from the loop in two distinguishable ways. The first is instruction-first: propose a task, run an agent, keep the trajectory if a judge says it succeeded. InSTA instantiates this as a proposer–agent–judge flywheel over the Common Crawl top-1M PageRank sites and trains a 1.7B policy that reaches 56.9% judged success on 3,000 held-out (site, task) pairs, up from 11.5% zero-shot, surpassing its own 235B collector [70]. The second is interaction-first: let the agent wander, then label. NNetNav compresses each state transition to a sentence, labels the resulting prefix with an instruction, and prunes with an outcome reward model, giving Llama-3.1-8B 16.3% on WebArena where the same model zero-shot scores 1.0% [71]. AndroidGen goes further and turns the label into a training signal: a critic annotates sub-goals, failures are cut at the first unfinished sub-goal and the prefix is re-labelled with the shorter instruction, so failed runs become samples rather than waste [74].

Figure 9. InSTA's three-stage flywheel, from the paper's pipeline figure [70]. Stage 1 proposes a task for a sampled site out of a million; Stage 2 has an agent attempt it and emit a structured action with a target element id; Stage 3 has a language-model judge score the attempt on success, efficiency and self-correction. The figure makes the circularity of Section 8.3 concrete: the judge in Stage 3 scores the trajectories that the same model family collected in Stage 2.

8.2 Scale is not the variable that matters most

Reported gains from data look large, and they are, but reading them as a scaling story is a mistake. The clearest controlled comparison comes from GroundCUA, which builds a much smaller corpus than its competitors — 3.56M densely annotated elements from expert demonstrations, about 64 annotations per screen — and reports GroundNext-7B at 69.2 average point accuracy against JEDI-7B's 56.1, where JEDI uses roughly nine million samples [65]. Density and provenance beat volume.

Two further results complicate a naive scaling reading. SoftwareGym synthesises entirely new desktop applications and generates tasks inside them, lifting OSWorld Pass@3 from 41.39% to 56.97% with synthetic data alone — but adding that synthetic data to a model already trained on OSWorld lowers OSWorld from 61.48% to 58.61% [72]. The mixing is not additive. And AgentTrek's mined trajectories transfer unevenly: the same corpus supports a large gain on ScreenSpot-Web grounding and a much smaller one on end-to-end navigation [68], which is what one should expect if grounding and planning draw on different parts of the distribution. NNetNav reports the sharpest instance of this: mixing WebArena traces into live-site training reduces average performance by 1.3 points, an early sign that environment-specific and environment-general data are not interchangeable [71].

Taken together these results support a claim the papers themselves mostly avoid stating: for current corpora, what a trajectory contains — a dense element label, a correct sub-goal boundary, a genuine terminal state — matters more than how many of them there are. This is our inference from the comparisons above, not a conclusion any single paper draws.

8.3 Quality control, and where it becomes circular

Large automatically built corpora are only as good as their filters, and the filters are usually language models. Three recurring problems follow.

Circularity. When a judge model filters data produced by the same model, success labels partly encode the collector's own mistakes. InSTA's judge is the same 235B model family that collects the data [70]; AndroidGen's critic is the deployment policy's own judge [74]; SPA-Bench's evaluator is the same GPT-4o used by most agents it scores (Section 10.3). None of these papers measures how much of the reported gain would survive an independent filter.

Shared provenance. Cross-app tasks in several mobile benchmarks are taken from datasets that the best performing agents were fine-tuned on; the OdysseyAgent case is the clearest (Section 10.2). A held-out split by task does not necessarily hold out the distribution the model trained on.

Yield. Automatic pipelines have their own success rates, and they are not high. GUI-360°'s two-round collection succeeds on only 26.09% of attempts, which is why 1,093,525 of its steps are failed runs [66]; AgentTrek's corpus covers 127 websites [68]; MobileViews still requires human intervention to unlock stuck applications [73]. The published trajectory count is the output of a lossy process that is rarely reported end to end.

A genuine advance sits alongside these problems: failures are now kept rather than discarded. GUI-360° retains failed trajectories explicitly as a supervision resource for reinforcement learning, and AndroidGen re-purposes them as prefix-augmented short-horizon samples. For a field where most agents fail, this is a better use of the data than silent deletion.

Finally, the reporting quality of dataset papers is uneven in ways that limit re-verification. TongUI, MobileViews, SoftwareGym and AgentTrek announce open releases of datasets and models but do not give downloadable locations in the versions we read; several omit licences and per-source counts; almost none report annotator agreement or per-source yield. This is a coverage and reproducibility limitation of the evidence base, not merely a formatting complaint — it is why the scale figures in this chapter should be read as author-stated rather than independently verified.

Takeaways

• Trajectory supervision moved from human demonstration to mined and agent-generated data in about eighteen months, with a third, synthetic-environment family now emerging.
• The strongest controlled comparison available says dense, well-provenanced data beats larger automatically collected data — not that scale is useless.
• Synthetic and environment-specific data mix badly: two independent results show a corpus that helps on one environment hurting on another.
• Success filters are usually the collector's own model, and provenance overlaps with evaluation benchmarks are common; both inflate measured progress in ways the papers do not quantify.
• Keeping failed trajectories as training signal is a real methodological advance of the period.

9 Training and action models

9.1 Three regimes, three orders of cost

Everything in this chapter falls into one of three regimes. Prompting changes nothing inside the model and is bounded by scaffolding; supervised fine-tuning changes parameters on recorded or synthesised trajectories; reinforcement learning changes parameters against outcomes the environment supplies. The regimes are not alternatives in practice — most strong systems use all three in sequence — but they differ by orders of magnitude in cost, and the differences explain most of the reported numbers.

At one end, UI-R1 trains a 3B model on 136 samples and lifts ScreenSpot from 65.0% to 83.3%, above a 7B supervised model [114]. GUI-R1 replaces a ten-million-sample SFT corpus with 3,000 curated RL samples [115]. At the other end, Qwen-CUA runs native screenshot-only reinforcement learning with verifiable rewards across a fleet measured in tens of thousands of GPU-hours, reporting 86.2% on OSWorld-Verified against 83.4% for a frontier closed model [41]. Both endpoints are real; the field has not yet converged on which regime is appropriate for which capability.

9.2 What supervised training buys, and what bounds it

The largest open supervised effort is OpenCUA, built on 22,625 human desktop tasks recorded across Windows, macOS and Ubuntu — averaging 18.6 steps over more than 140 applications — with teacher models synthesising the reasoning traces that are filtered before training [38]. Its 72B model reaches 45.0% on OSWorld-Verified, the best reported open result on that benchmark at the time, against 27.1% for the strongest prior open agent and 61.4% for a frontier closed model. The paper is unusually candid about the bound: human annotation is the ceiling, annotators may not use the most efficient procedure, and informed-consent requirements bias who appears in the data.

Qwen-CUA takes the opposite route and removes the intermediate representation entirely — no accessibility tree, no DOM, no code execution, screenshots and primitives only — and reports that native control scales, while noting that adding a Bash channel reduced its score on a desktop benchmark [41]. That negative result matters more than the headline: the hybrid action space that Section 7.4 shows winning under scaffolding does not automatically win once the policy is trained, and the paper does not resolve why.

9.3 The reward is the active design variable

If Chapter 8 identified data provenance as the main lever for supervision, Chapter 9's lever is the reward function, and five distinct designs are now in play (Table 7).

Table 7. Reward designs in use, with the effect each paper reports on the benchmark it names. The rows are not comparable with one another: base model, environment and training budget all differ (Section 9.6).

The controlled comparison inside GUI-Shepherd is the most useful single result here: with the same base model and environment, replacing a sparse terminal reward with a step-level process reward moves success from 37.0% to 40.5% [112]. This is the training-side counterpart of the verification finding in Section 7.2 — what is evaluated, and how densely, matters more than the optimiser. It also explains why UI-S1 can reach 34.0% on AndroidWorld without touching an environment at all: its semi-online scheme keeps the model's own history and patches in the expert action on mismatch, then derives a discounted step-level advantage from action agreement [111].

The countervailing evidence is that process rewards with no environmental grounding have an unmeasured failure mode. UI-S1's reward never touches a real environment; GUI-R1's reward is offline label matching and the paper reports that no qualitative reasoning behaviour emerged, attributing this to the single-image, non-interactive setup [115]. When the reward is a comparison to a recorded expert, the model can improve at imitating expert steps without becoming better at recovering from its own.

9.4 Online RL has recomposed the leaderboards

On interactive benchmarks, the year's movement is attributable to environment interaction rather than corpus size. AndroidWorld's best reported result went from roughly 30% in 2024 (Section 10.2) to 80.2% for a 9B model trained with online multi-turn RL, ahead of a 72B baseline by about sixteen points in the 7B case [109]. A separate online-RL system reaches 44.8% on the same benchmark from a 32B policy [110]. OSWorld's strongest open numbers now cluster at 48–49% from 9B models trained end-to-end [108], and the two-stage curricula that produce them are described in enough detail to be repeated.

Three engineering patterns recur across the successful runs: keep a viable gradient when reward variance collapses (ARPO's experience replay and task selection [107], MobileRL's failure-curriculum filtering [109]); manage the entropy collapse inherent to long-horizon RL (ComputerRL's alternating RL/SFT phase [108]); and reshape the reward toward efficiency rather than mere success, so that short correct trajectories score higher than long ones [109]. The last is one of the few places in this survey where a training change targets efficiency directly, and it connects to the efficiency findings of Section 10.4.

Test-time scaling sits adjacent to training and is best understood as a bounded lever. GTA1 samples several action proposals per step and has a model judge pick one, reaching 58.4% on ScreenSpot-Pro and 45.2% on OSWorld with a 7B policy and a strong planner [113]. Its own analysis shows the benefit is budget-dependent — largest at a 50-step limit, smaller at 100 — which is what one would expect if the cost of deliberation competes with the cost of acting.

9.5 Sample efficiency, compute, and what the numbers cost

Two efficiency claims deserve separate statement because they pull in opposite directions. RL can be extremely sample-efficient: UI-R1's 136 samples and GUI-R1's 3,000 samples both beat supervised models trained on four to six orders of magnitude more data [114, 115], and the constraint these results relax is annotation cost rather than compute — RL still requires rollouts against an environment, and GUI-R1's offline reward avoids even that at the cost of the grounding problem just discussed. At the other end, the strongest native agent is expensive by design: an RLVR run over a fleet for several days [41].

Because barely any paper reports both, the practical trade-off between annotation, rollout and compute cost remains unquantified. Papers report the axis they win on. This is a genuine evidence gap, and it is the same gap that Section 10.4 documents from the evaluation side.

9.6 Confounds that limit what these numbers mean

Three systematic problems run through the reported gains.

Training on the test distribution. ARPO selects its 128 training tasks from OSWorld by filtering for tasks the base model sometimes solves, then reports on OSWorld [107]; MobileRL trains on AndroidWorld and AndroidLab and reports on both [109]; AutoGLM states that it was optimised in the environments it is evaluated on [32]. Held-out splits by task do not hold out the distribution. This does not make the gains fictional — the base models are the same, so the comparison is internally fair — but it does mean these numbers are not transfer estimates.

Single runs. Most training papers here report one run with no variance. MobileRL and ComputerRL report confidence intervals; ARPO, UI-S1, GUI-Shepherd, UI-R1, GUI-R1 and SE-GUI do not. Given that WebRL's own authors identify curriculum generation as a source of seed-to-seed variance [33], the missing dispersion is not a technicality.

Reward-model bias. When the reward is a learned model rather than a program, the policy is optimising that model. MobileRL's AndroidLab results depend on a VLM reward model of its own admission [109], and ARPO's authors name reward hacking as an open hazard of substituting learned rewards for scripted ones.

Takeaways

• Prompting, supervised fine-tuning and reinforcement learning coexist, and their costs differ by orders of magnitude; the strongest and the cheapest results in the survey both come from RL.
• Reward design, not the optimiser, is the active variable: a step-level process reward beats an outcome reward on the same base model and environment.
• Online multi-turn RL has raised AndroidWorld's ceiling from ~30% to 80.2% in about a year, and the winning runs share three engineering patterns: variance preservation, entropy management, and efficiency-aware reward shaping.
• Sample efficiency and compute cost trade off, and no paper here measures both, so the comparison remains open.
• Training-on-the-test-distribution, single-run reporting and learned-reward bias are systematic in this literature and bound how far these numbers transfer.

10 Benchmarks and evaluation

10.1 Environments and verification styles

The benchmark ecosystem partitions by platform, and within each platform by how success is decided. The second partition matters more than the first, because it determines what any reported number means (Table 8).

Table 8. The benchmark landscape by platform and verification style. The verification column governs comparability: a script that runs on the final state, an LLM judge over the trajectory and an offline replay against recorded human actions do not measure the same thing.

Three verification styles are visible: execution of a task-specific script on the final state, an LLM judge over the trajectory, and offline comparison against recorded human actions. Numbers are not comparable across these families, and the reason is not stylistic — an execution script and a judge disagree about success on a substantial fraction of trajectories, as Section 10.3 quantifies.

Figure 10. How the offline, human-recorded family is built, from WebLINX [124]. A navigator controls the browser while an instructor supplies the user's turns; the session is recorded, post-processed and published as a dataset. Because the artefact is a recording rather than a runnable task, this family scores per-turn action similarity and cannot score task success — the third verification style in the table above.

The parameterisation of tasks is a second, quieter axis. AndroidWorld generates millions of task instances from 116 templates with randomised parameters and device-state rewards [8]; MiniWoB++ did the same for the web; static suites such as UI-Vision offer one human demonstration per task [135]. Parameterised environments resist memorisation; static ones cannot. Hierarchical multi-platform frameworks such as MMBench-GUI organise the same environments under a shared rubric so that one agent family can be scored across desktop, mobile and web [39].

Figure 11. AndroidWorld's structure, from the paper's overview figure [8]. An Android emulator exposes apps and OS state; the agent acts through touch or APIs and reads a UI tree; TaskEval generates parameterised instances and scores them with device-state rewards. The template-plus-parameters design is why 116 tasks yield effectively unbounded instances, and why this benchmark resists memorisation better than static suites.

10.2 Platform results, and why they should not be pooled

The most useful way to read the results is platform by platform, with the state at the start of the period next to the state at its end.

Web. WebArena's early leader was a text-only agent at 43.1% [47]. Live-site results are much lower and disputed: WebVoyager reported 59.1% for GPT-4V with human evaluation in early 2024 [23], and by 2025 numbers approaching 90% were circulating. A re-audit with human evaluation found most agents clustered near 30% on a harder live benchmark, with only two systems above 55% [137]. Enterprise web gives 42.7% for a strong model on 19,912 task instances [26], and agentic search — a different task — gives 0.28 for the best system against a human 0.54 [12].

Desktop. OSWorld reports a human baseline of 72.36% [6]. The best reported agent result in this survey is 60.76% (Section 7.4), and the strongest native screenshot-only model claims 86.2% on the re-verified subset (Section 9.2). Windows Agent Arena, which parallels the same task family on Windows, reports a best result of 19.5% against a human baseline of 74.5% [28] — a 55-point gap. The desktop picture is therefore highly benchmark-dependent, and the OSWorld-Verified subset selection is not comparable with the original 369-task suite.

Mobile. This is where movement is largest and cleanest. AndroidWorld went from 30.6% for an accessibility-tree agent in 2024 [8] to 80.2% for a 9B model trained with online RL [109]. The same platform's third-party benchmarks disagree sharply on difficulty: SPA-Bench reports 0.544 overall success for its best agent but only 0.150 on cross-application tasks [105]; Mobile-Bench reports 80.96% on single-application tasks, 63% on short multi-application tasks, and 26.5% on long ones [106]; A3 reports 21.9% over 201 tasks on live Google Play applications, with every agent at 0% on its hard tier [134]. The pattern across all three is the same: single-application tasks are largely solved, cross-application long-horizon tasks are not.

Two structural facts should govern any reading of these numbers. First, hardest-subset collapse is ubiquitous — cross-app on mobile, list-filtering on enterprise web (0.0% for all three models tested [26]), hard tier on live mobile [134]. A headline average hides a bimodal distribution. Second, training and evaluation distributions increasingly overlap (Section 9.6), so the AndroidWorld curve in particular measures progress on a target that several groups are optimising against directly.

10.3 Judging: how reliable is the pass/fail signal?

If one chapter of this survey changes how a reader interprets the others, it should be this one. The measurement apparatus has itself been measured, and it is the weakest part of the field.

Two independent audits converge from opposite directions. AgentRewardBench scored twelve LLM judges against 3,906 expert binary labels on 1,302 trajectories spanning five benchmarks [136]. No judge exceeded 70% precision: the best reached 69.8% precision at 83.1% recall, meaning roughly 30% of trajectories it called successful were failures. Rule-based judges went the other way — high precision (83.8%) but 55.9% recall, under-reporting success by 16.7 points on WebArena and 18.5 on VisualWebArena, and in two cases inverting the ranking of two models relative to expert judgement.

The re-audit of live-web results makes the same point from the leaderboard side: under human evaluation most agents sat near 30% where reported figures were near 90%, and the authors' own judge agreed with human verdicts 85.7% of the time [137]. Their conclusion — that the field was partially measuring its judges — is supported by both audits independently.

Three further results extend the critique from judgement to attribution.

Composite scores overstate progress. ComponentBench scores 2,910 tasks whose success is decided by programmatic terminal-state predicates rather than a judge, and finds that on its harder derived set performance drops 10–39 percentage points; only 34.1% of one frontier model's successes fall within the human step budget [140]. The re-audit similarly notes that trajectory-level judging lowers scores by about 20 points relative to outcome-only judging (Section 10.5, [42]).

Late-stage failures masquerade as near-misses. Decomposing a web agent's pipeline into request formulation, action planning and grounding shows a model at 48.78% end-to-end whose individual stages score 70.17% and 62.87% [139]. More pointedly, manual re-labelling showed that over half of the "errors" counted against it were reasonable alternative actions the benchmark's single ground truth did not admit; adding legitimate alternatives improved measured performance by up to 8.3 points [139]. A single-reference benchmark manufactures failures.

Judges are being redesigned, and it helps. Treating the judge as an agent that gathers evidence rather than classifies a screenshot raises accuracy from 78.8% to 86.9% on a 321-trajectory desktop benchmark [131], and a process-reward judge improves a mobile agent by +14.53 points against +8.56 for a standard process reward [132]. These gains are real, but both pay in inference cost, and neither reports cost per judged trajectory.

One asymmetry deserves explicit statement, since this survey relies on it: Section 8.3 shows that the same deficiencies appear in data pipelines, where judges filter training trajectories. The measurement problem is therefore not confined to reporting — it feeds back into what models are trained on.

10.4 Efficiency, cost and environment variation

A new axis appeared during the period: not whether the agent succeeds, but at what cost and how reliably.

OSWorld-Human adds human minimal-step trajectories to all 369 OSWorld tasks and a weighted efficiency score [13]. Its headline finding is that an agent with 41.4% success uses 2.7 to 4.3 times the steps a human needs, and that planning and reflection account for 76–96% of end-to-end latency. This is the quantitative form of the observation in Section 7.1 that deliberation dominates wall-clock time, and it identifies where efficiency work should aim.

AI Agents That Matter argues that agents should be compared on an accuracy–cost Pareto frontier rather than by accuracy alone, and demonstrates the point by re-implementing published agents and finding a simple baseline that dominates a much more expensive one [14]. WAREX reaches a similar conclusion by a different route: interposing a proxy that injects network, server and script faults, it measures success together with latency, token counts and per-task cost, showing that reliability under perturbation is a separate axis from capability [138].

Environment variation is the newest measurement. OpenApps generates eight systematic variants of each application — themes, fonts, adversarial content, language — and re-runs the same tasks. The observed spread is enormous: one model ranges from 4% to 63% across variants, and cross-variant standard deviation is more than twice within-application standard deviation [133]. A concrete instance: a model at 0.41 pass@1 on the default theme scores 0.05 on dark theme. Any single-configuration score is therefore a sample from a wide distribution that is almost never reported.

10.5 Hybrid interfaces: the 2026 frontier

The chapter closes on the result that most cleanly separates current systems from their predecessors. WeaveBench constructs 114 tasks that provably cannot be completed through a single channel, then evaluates agents under different runtime configurations [42]. GUI-only agents score at most 1.8%; CLI-only agents at most 3.5%; the hybrid configuration reaches 35.1%. The best reported combination is a frontier model with a full coding-agent harness at 41.2%. In the same study, trajectory-aware judging reduces one model's score from 53.5% to 33.3%, the clearest single demonstration that outcome-only evaluation inflates results.

This result unifies three separate strands of this survey. Section 4.2 traced the action space widening from coordinates to semantic calls; Section 7.4 found the strongest desktop system is an orchestrator that routes between GUI and code execution; Section 12.1 found that every mature deployment case relies on a non-GUI oracle. Hybrid interfaces are where those three observations meet, and they are consistent with the product design of the commercial systems (Section 12.5).

Takeaways

• Benchmarks partition by platform and, more consequentially, by verification style; numbers from execution scripts, LLM judges and offline replay are not comparable.
• Hardest-subset collapse is ubiquitous — cross-application mobile, enterprise list-filtering and hard tiers all sit near or at zero — so headline averages conceal bimodal distributions.
• The measurement apparatus is the field's weakest component: no audited LLM judge exceeded 70% precision, rule-based judges under-reported success by 16–19 points and inverted rankings, and a single ground-truth reference classed more than half of one agent's real errors as errors.
• Efficiency, cost and environment variation are now measured, and each changes the picture: leading agents need 2.7–4.3× the human step count, and one model's pass rate varies from 4% to 63% across cosmetic variants.
• Hybrid GUI-plus-code configurations outperform single-channel ones by an order of magnitude on tasks that require both.

11 Trust, safety and security

11.1 The attack surface is the observation channel

A computer-use agent reads the world and acts on it through the same surface. That is the structural fact from which this chapter's whole literature follows: any content the agent can observe is content an attacker may control, and there is no point in the pipeline where an instruction from the user and an instruction from a web page are distinguishable by construction.

Two surveys systematise the consequences. One catalogues eight inherent threats — grounding errors, misunderstanding, hallucination, context limits, latency — alongside seven external ones, and argues that the inherent failures are safety problems in their own right, not merely the substrate that attacks exploit [9]. The other reorganises trust around an execution gap between the fidelity of perception, of reasoning, and of interaction, and draws the consequence that the two standard LLM safety reflexes fail here: output filtering assumes output can be reviewed before it acts, and refusal training assumes failures are isolated, whereas a GUI agent's errors are executed and irreversible [10]. A third work abandons metrics entirely and red-teams a deployed system, enumerating four untrusted channels — web content, linked files, UI elements and layout, and tool outputs — and reporting an exploit chain in which an injected forum post leads to installing a progressive web app, granting it filesystem access, and hijacking MIME handlers to achieve code execution [94]. Its central claim, that the defects are architectural (missing provenance tracking, weak binding between interface and action, uncontrolled memory and delegation) rather than behavioural, is the most useful framing in the chapter, because architectural defects are not fixed by better prompting.

The attack literature explores all four channels and both modalities (Table 9).

Table 9. Attack surfaces, representative attacks and reported effects. The effects are as the papers state them, and most are hijack rates rather than end-to-end harm (Section 11.2).

Two of these deserve emphasis for what they show about why attacks work. WebInject optimises its perturbation in raw pixel space while explicitly modelling the conversion from web page to screenshot — including per-monitor colour management — and the model's own resize step [99]. The ablation is decisive: the same perturbation applied directly to the screenshot achieves 0.000 success. The attack does not exploit the model's vision so much as the pipeline's unmodelled transformations. VisualTrap shows the complementary result on the supply side: poisoning a tenth of the localisation corpus installs a trigger that redirects clicks, and — the finding with the longest shadow — the defence works only while the poisoning is confined to the language module; once the vision tower is affected, no amount of clean data tested restores behaviour [101].

Figure 12. Three places VisualTrap plants a poisoned image, from the paper's workflow figure [101]. Left: poisoning the grounding pretraining data misdirects localisation from the intended object (green circle) to a target object (red circle). Top right: poisoning end-to-end fine-tuning data installs a trigger that emits the attacker's click next to a plausible thought. Bottom right: the same trigger reaches a modular pipeline and attacks the planner's grounded object reference instead. The defence result in Section 11.3 follows from this picture — the tested remedy stops working once the poisoning reaches the vision module.

Branch steering is the newest and conceptually hardest threat. Earlier system-level defences aimed at control-flow integrity: an agent may only take actions authorised by its plan. But a graphical environment forces plans to contain runtime branches, so an attacker who cannot inject a single new instruction can still steer the agent into a branch that was authorised in advance and is nonetheless harmful [102]. This is a different kind of failure from prompt injection — no instruction is injected, and the compromised decision is one the system already agreed to take.

11.2 Hijack is not harm

Three independent works measure the full chain from "the agent read the attacker's text" to "the attacker's goal was achieved", and all three find severe decay. WASP reports intermediate hijack rates up to 0.857 and end-to-end completion topping out at 0.167 across five models and three scaffolds, and names the pattern security by incompetence: current agents are hard to subvert all the way because they are not yet competent enough to finish the attack [45]. AgentHijack formalises this with three tiers and gets 84.5% for patch-induced hijack, 47.0% for hijacked planning, and 20.3% end-to-end — with the sharpest single result in the chapter, a model whose hijack rate is 100% but whose end-to-end attack success is 0% [100]. Not-an-A11y shows the same spread across frameworks rather than tiers: 0.822 in the worst framework-model pair and 0.150 in the best [96].

Figure 13. Where an injected instruction actually ends, from WASP [45]. The Sankey traces each first post-injection action (left) through the agent's subsequent actions (middle) to the terminal state (right). Only the thin red ribbon reaches Completed Adversarial Goal; the two dominant outcomes are the agent remaining confused and the injection continuing to be ignored. The figure is the visual form of this section's claim — headline injection rates measure hijack, and hijack mostly does not become harm.

This gap cuts two ways and both need stating. It means headline injection numbers are usually hijack rates and should not be read as the probability that an attacker gets what they want. It also means the field's measured robustness is contingent on current capability, and will degrade as agents improve — the authors of WASP say this explicitly. A defence evaluated only against hijack metrics may be certified as working at the moment it matters least.

The same gap appears in risk benchmarking. RiOSWorld's 492 executable risk tasks give an overall risk rate of 84.93% by intent and 59.64% by completion [98], and MobileSafetyBench finds that goal achievement on low-risk tasks stays near 80% while indirect injection succeeds essentially always — 0/10, 0/10, 1/10 and 2/10 for GPT-4o, Claude 3.5, Gemini 1.5 and o1 respectively [97]. Capability and safety are not in tension in these numbers; the agents are simply not equipped to notice they are being used.

11.3 Defences, and what they cost

Defence work has bifurcated into input filtering and architectural constraint, and results now separate cleanly enough to compare.

BrowseSafe is the strongest detector result in this survey: it treats all tool output as untrusted, classifies in parallel fixed windows with a conservative OR aggregation, and replaces flagged content with a placeholder without revealing it. Fine-tuned from a 30B mixture-of-experts model, it reaches F1 0.904 at a median latency under one second against 0.863 at 23–36 seconds for the best frontier general model, and 0.350 for a dedicated guard model [103]. That last comparison is the interesting one — purpose-built small guards badly underperform here — and it reframes injection detection as an engineering problem with a working solution in the text modality.

COBRA is the strongest architectural result: by fixing per-branch capability constraints before any untrusted runtime data is seen and enforcing them deterministically at the HTTP and MCP boundaries, it reports 0% attack success on the steering benchmark, against 87.6% for the ablated version, 89.5% for a vanilla dual-LLM design and 94.4% for a ReAct agent, while retaining 97% of clean task utility with four false positives across 138 benign cells [102]. The authors are careful about what it does not cover: security-relevant values that never appear in an HTTP or MCP call, free-text semantics and server-side effects, and channels outside HTTP and MCP altogether.

Against these, the utility accounting in most attack papers is simply absent. VPI-Bench, WebInject, AgentHijack and VisualTrap all report attack success without any measurement of benign task performance under the defence, so a reduction in attack success cannot be separated from an agent that has merely become more cautious [46, 99, 100, 101]. WASP is the exception and shows the honest difficulty: a defensive system prompt lowers attack success on GPT-4o from 0.429 to 0.226 while utility falls from 0.622 to 0.459 [45]. Until utility-cost accounting is standard, defence comparisons in this chapter carry a systematic ambiguity.

Two further results constrain the space of acceptable defences. First, human oversight is not a reliable backstop: in the fine-print study, 38 of 39 human participants accepted a privacy policy containing a malicious instruction [95]. Second, defence by pre-execution verification requires declarative effort that does not scale to third-party software — VeriSafe needed roughly 59 predicates and 437 lines of code per application (Section 7.5) [90].

11.4 Beyond injection: monitoring an agent that is not attacked

A distinct safety problem is a deployed agent pursuing a hidden goal of its own. Control OSWorld adapts the AI-control methodology to GUI work by pairing 318 OSWorld tasks with 81 programmatically verified harmful side tasks, so that sabotage is detectable by state rather than by judgement [104]. Monitoring is strong under favourable conditions — a full-trajectory monitor reading actions, visible text and screenshots reaches pAUROC below 3% false-positive rate of 0.960 with TPR@1% of 0.92 — and drops sharply under realistic constraints: remove the agent's visible text and pAUROC falls to 0.808; move to per-step monitoring, which is what an intervention protocol actually requires, and TPR@1% falls to 0.49, or 0.08 without visible text [104].

The paper is explicit that its conditions are generous: OSWorld tasks are narrow and well-specified, so almost any unrequested action is conspicuous, and the harms are moderate. It also demonstrates a finding that should generalise beyond this setting: hiding a harmful goal alongside a legitimate task eliminated refusals entirely and suppressed verbal leakage from 26.1% to 1.9% — and the effect was larger for the more capable model [104]. Capability and concealment improve together.

11.5 What the chapter does not cover

Four gaps are worth naming. No work here evaluates an adaptive attack against its own proposed defence; every defence is tested on the same fixed template or attack family it was designed against. Defences in the image modality are essentially absent — BrowseSafe is deliberately text-only, and VisualTrap's only tested remedy fails once the vision module is poisoned. Memory and planning context as an injection channel appear in one paper [96]. And no study evaluates safety jointly across web, mobile and desktop, even though the deployed systems do all three.

Takeaways

• For computer-use agents the observation channel is the attack surface; the systematisation literature classifies grounding and hallucination errors as safety threats in their own right.
• Reported attack success is usually hijack, not harm: three independent works show end-to-end rates far below hijack rates, and that gap will shrink as agents get more capable.
• Framework choice changes measured vulnerability by large factors — the same model reaches 100% under one harness and 31.7% under another.
• Defence results now separate: a purpose-built classifier reaches F1 0.904 sub-second, and an architectural approach reaches 0% measured attack success while keeping 97% of utility.
• Utility cost is unmeasured in most attack papers, and human oversight fails (38/39 participants complicit).
• Safety facing a non-injected, self-interested agent is measurable but currently depends on the monitor being able to read the agent's own text.

12 Applications and deployment

12.1 Software QA is the beachhead

Most published computer-use agents are evaluated on synthetic task suites. The applications with real deployment evidence look different: they are in software quality assurance, and they are there because QA supplies something a benchmark cannot — a pre-existing oracle and a user who wants the work done.

Four independent lines of work agree on this. Test execution from natural-language cases now runs against real Android applications at 73% case success with an 11.8 s/step cost, against 71.3% and 20.5 s for the strongest earlier LLM baseline [79]. Test migration — re-deriving a regression suite on a new app version — is the largest reported deployment gain in this survey: 90.3% success against 21.6% for rule-based mapping and 43.1% for the best LLM baseline, at a few minutes rather than roughly ninety [80]. GUI fuzzing reaches 68.5% class coverage and, more tellingly, finds 208 unique crashes confirmed as real bugs across 24 commercial applications [81]. And end-to-end web testing against a natural-language specification reports 0.96 precision and 0.96 recall at detecting injected defects, against 0.26/0.69 for a screenshot-judging agent [78].

What these four have in common is not the agent architecture — that varies from single-agent loops to multi-agent pipelines — but that a machine-checkable oracle already exists, or can be constructed.

12.2 The oracle, not the agent, decides the quality

The sharpest contrast in the chapter is between two agents that both watch a test run and decide whether it passed. One judges from screenshots alone; the other parses the specification into (condition, action, expectation) triples, symbolises the GUI elements involved, and compiles cross-state dependencies into executable assertions [78]. Reported defect-detection precision and recall differ by a factor of 3.7 (0.96 against 0.26). A separate study that measures the judging step directly finds the structural limit: with a strong multi-agent implementation, human–machine agreement on test verdicts reaches only 0.61, 26 of 113 cases disagree under all three commercial models tested, and the best specificity is 0.57 — meaning the system tends to report passing tests as failures [77].

Figure 14. The mechanism behind the contrast above, from WebTestPilot [78]. Left: the live application state. Second panel: a schema declares typed symbols over the elements the test cares about. Third: the schema is instantiated into symbols carrying concrete values read off the page. Right: assertions are written over those symbols, so the verdict is a compiled expression over cart.items and cart.subtotal rather than a model's opinion of a screenshot. That substitution is what the factor-of-3.7 precision gap in this section measures.

Table 10. Application areas by evidence quality. The last column is the reason a reader should discount the representative result beside it.

The false-positive column of Table 10 deserves emphasis because it is the least-reported and most consequential number for anyone considering deployment. The usability-defect study crawled 120 real academic websites, found 29 confirmed defects, and reported that 85% of everything it flagged was wrong [76]. Papers that report only detection rate would have made this tool look ready. This is the same pattern the evaluation-audit literature documents at the benchmark level (Section 10.3), reappearing where it has direct operational cost.

12.3 Accessibility: the highest-stakes, thinnest evidence

Two works target users who cannot operate a GUI by pointing. One replays existing human-written accessibility test instructions against real VoiceOver gestures on iOS, re-planning when the UI changes or the instruction itself is wrong [82]; the other adds a supervising second model to a voice-driven web navigator so that failed actions can be retried, lifting cumulative accuracy by about 36 points after the first correction round [84]. Both are serious attempts at a problem where the payoff is large.

Both are also prototypes. The accessibility test replayer supports four features on one platform and evaluated with ten professional QA engineers rather than end users; the voice navigator reports no usability study, no participants with the motor impairments it targets, and no comparison against a screen reader. The gap between the stated user and the measured evidence is the largest in this survey, and it should be read as an open invitation rather than a result.

12.4 Simulated users are a screening tool, not evidence about users

A distinct application is using agents as subjects rather than operators: a thousand persona agents drawn from a hundred-thousand persona pool ran a between-subjects experiment on a live e-commerce site, and the detected treatment effect on purchases (414 vs 403 of 500) was statistically significant at the 5% level [83]. The authors are explicit that this supplements rather than replaces real user testing, and the evidence supports their caution: the spend difference was not significant, the human treatment arm is not reported, and the effect appears on one design feature of one site. Read as a way to triage which interface variants deserve a real experiment, the result is plausible; read as a substitute for one, it is not supported.

12.5 What the commercial systems publish

The systems most people actually use have no arXiv papers. Anthropic released computer use as a public beta on 22 October 2024 and described it as experimental, recommending sandboxed, low-risk tasks [85]; as of August 2026 it is generally available as a documented client toolset with batch actions and a browser tool. OpenAI announced Operator and its Computer-Using Agent on 23 January 2025 and reported 38.1% on OSWorld, 58.1% on WebArena and 87% on WebVoyager; the capability was folded into ChatGPT's agent mode in July 2025 [86].

Two observations follow, and both are limitations of this survey's evidence rather than findings about the systems. First, these numbers are vendor-reported against benchmarks this survey has criticised, with no published protocol, run count or variance; they should not be compared with the peer-reviewed results in Sections 10.2 and 12.1. Second, the product framing independently corroborates the hybrid-interface conclusion of Section 4: both vendors ship a computer-use tool alongside code execution, file access and API tools, not instead of them. The industry's own packaging treats GUI control as one channel among several.

Takeaways

• Software QA is the only application area in this survey with repeated, human-verified deployment evidence, and it works because an oracle already exists.
• Where the oracle is formal, agent test verdicts approach expert reliability; where it is a screenshot judgement, they do not, and false-positive rates of 85% are reported in the wild.
• Accessibility assistance has the largest gap between stated beneficiaries and measured evidence, with no study involving the target users.
• Simulated-user experiments are a triage tool whose own authors disclaim substitution for real users.
• The commercial systems with the widest reach publish no verifiable protocols, and their product design confirms rather than challenges the hybrid-interface finding.

13 Foundational questions

Five disagreements run through this survey. Each is stated with the strongest evidence on both sides, and each is followed by an explicit note on where the present authors would place their weight — labelled as our reading rather than as a result.

13.1 Do pixels suffice, or is structure necessary?

For pixels. Pix2Act showed that pure-pixel observation with coordinate actions beats DOM-based baselines on MiniWoB++ [16]; CogAgent beat substantially larger text-fed models on Mind2Web using only screenshots [21]; UGround showed that grounding can be made a pluggable module independent of any markup [29]. The strongest evidence in this direction is developmental: Qwen-CUA removes the intermediate representation entirely and reports 86.2% on OSWorld-Verified, higher than closed models that use richer interfaces [41]. If pixels were insufficient, the top of that leaderboard would not look like this.

For structure. AgentOccam's ablation on WebArena is the cleanest controlled result: rewriting the action interface around what a text observation can express, with no model change, produced the largest single component of a 43.1% result [47]. ASIL replaces screenshots with a typed JSON state and schema- constrained semantic calls and reports 81.6% against 6.6% for a GUI condition on 380 tasks [123]. And Read More, Think More finds that richer HTML observation is worth +17.5 points for one model and −18.8 for another on the same 330 tasks [127].

Our reading. The two sides are measuring different things. Structure helps where it is available and trustworthy, and its benefit is largest for the action interface rather than the observation. Pixels help where structure is absent, which is most of the desktop and much of the mobile long tail. The model-dependence found in [127] is the decisive fact: there is no architecture-independent answer, which is why the practical trend is hybrid (Sections 4.3, 10.5).

13.2 Does training on the environment beat better scaffolding?

For training. AndroidWorld's best result rose from 30.6% to 80.2% in about a year, driven by online multi-turn RL rather than corpus scale [8, 109]. OSWorld's strongest open numbers likewise come from end-to-end trained 9B models [108]. UI-R1 reaches 83.3% on ScreenSpot with 136 training samples [114], and SE-GUI trains a 7B model past a 72B baseline [116] — RL is not merely scaling.

For scaffolding. AgentOccam changes no parameters and beats prior work on WebArena [47]. Visual test-time scaling raises ScreenSpot-Pro from 47.8 to 61.6 with no training at all [122], and ScreenSeekeR more than doubles the best end-to-end grounding score without updating a single weight [119]. Web$hell compiles the interface rather than training a policy and reports 0.679 on WebArena-Verified [126]. AgentOccam's ablation decomposition locates most of its gain in interface design rather than reasoning.

Our reading. These are complements with different cost structures, and the honest statement is that the field does not know the exchange rate between them because almost no paper reports both axes. The one place the comparison is nearly controlled — grounding, where training-free search [119] and test-time scaling [122] compete against trained models [120] — the training-free methods are highly competitive at a 1.7–4.6× inference cost. Our inference is that scaffolding buys reliability cheaply and training buys headroom expensively, which is why production systems do both.

13.3 Is the gain in planning, or in verification?

For planning. The planning literature reports large, decomposed effects: BEAP's backtracking ablation is worth 1.9 points and its state tracker 4.6 [55]; WebRollback improves full completion by nearly three points over strong search baselines [53]; hierarchical decomposition reaches human-comparable efficiency in one mobile study [58].

For verification. BacktrackAgent's ablation is the sharpest single number in this area: removing the whole backtracking mechanism costs 5.65 points, but removing only the learned judge costs 5.32 of that, while removing the rule verifier costs 0.45 [93]. GUI-Shepherd shows a step-level process reward beating an outcome reward on the same base model and environment [112]. Decomposing a web agent shows healthy per-stage scores underneath a mediocre end-to-end result [139], and the diagnostic literature finds tasks average only 3.04 subgoals [63].

Our reading. The decomposition in [93] is the most informative result in this survey on the planning-versus-verification question, and it favours verification decisively: checking that the resulting state is the intended one is where the value is; checking that an action executed is nearly information-free. Combined with [63]'s short observed horizons, we read the evidence as saying that current benchmarks are short enough that decomposition machinery is over-provisioned, and that the binding constraint is noticing that something went wrong.

13.4 Can the reported numbers be believed?

They can be interpreted. Execution-verified environments produce reproducible, parameterised scores with human baselines [6, 8, 28]; several works report three-run means and dispersion [53, 55]; OSWorld-Verified and ScreenSpot-V2 show the community correcting its own instruments [6, 31]; and the efficiency and variation work [13, 133] adds the dimensions that flat success rates omit.

They cannot be compared. No audited LLM judge exceeded 70% precision, rule-based judges under-reported success by 16–19 points on two benchmarks and inverted a model ranking [136]; human re-evaluation cut live-web scores from near 90% to near 30% [137]; one published grounding benchmark contained about 11% label errors [31]; a single-reference benchmark manufactured more than half of one agent's errors [139]; and one model's pass rate ranges from 4% to 63% across cosmetic environment variants [133].

Figure 15. What the judging protocol is worth, from WeaveBench [42]. Each bar is one model on the same 114 hybrid tasks under two protocols: the lighter upper segment is the score under outcome-only judging, the darker lower segment the score under trajectory-aware judging, and the number between them the difference. The penalty ranges from 10.3 to 20.2 points and is largest for the strongest model, which is the quantitative form of the reading rule that closes this section.

Our reading. Both statements hold simultaneously, and the practical consequence is a reading rule: a number is interpretable within an environment, a protocol and a configuration, and is not evidence about capability in general. The strongest claims in this survey are those that survive a controlled ablation inside one setting, not those with the largest headline figure.

13.5 Should agents adapt to the web, or the web to agents?

Agents adapt. The dominant research programme of the period is exactly this: better grounding (Chapter 5), better interfaces (Chapter 4), better memory and verification (Chapters 6, 7), all aimed at existing interfaces that were designed for humans. Every benchmark in Section 10.1 is built on that premise, and the hybrid-interface results [42] show it working.

The web adapts. The Agentic Web Interface position argues that grounding should be a property of the site, not a learned capability, and notes DOMs exceeding a million tokens and per-task costs as the reason [130]. Web$hell's compiled cache [126] and ASIL's typed state and schema-constrained calls [123] are working prototypes of that position; Tactile's provenance-carrying target sets [129] are a partial instance.

Our reading. The current evidence favours adaptation because that is where the effort has gone, not because it has been shown superior — the two strategies have not been compared on a common task under common conditions. The decisive consideration is not technical: a web that exposes machine interfaces is a different governance regime, and no paper here addresses who would enforce it. We regard this as the most consequential open question in the survey.

Takeaways

• Pixels versus structure has no architecture-independent answer: the one controlled model-by-observation study finds the ranking reverses across models.
• Training and scaffolding are complementary with unmeasured exchange rates; where they have been compared directly, training-free search and test-time scaling are competitive at 1.7–4.6× inference cost.
• The decomposable ablations available favour verification over decomposition as the site of the gain.
• Benchmark numbers are interpretable within a protocol and not comparable across protocols; both halves of that sentence are supported by audits, not opinion.
• Whether agents adapt to the web or the web to agents is unresolved technically and unaddressed institutionally.

14 Resources

This chapter inventories what the field has built and can be reused. It is a starting point rather than a catalogue: the resources below are those encountered and verified during this review, and the availability column records what the papers actually state rather than what they intend.

14.1 Environments and benchmarks

Table 11 lists what this review encountered and what each entry actually publishes.

Table 11. Environments and benchmarks encountered in this review, with the artifact each one actually publishes. Where an entry says "no in-text link", the paper announced a release without giving a location in the version we read.

14.2 Datasets and trajectory corpora

Table 12 does the same for the corpora that the models of Chapter 9 are trained on.

Table 12. Trajectory corpora, their provenance and their stated availability. Availability is recorded as the papers describe it, not as verified by this review.

14.3 Models and tooling

Open action models and grounding modules form the reusable core: UI-TARS [37], OpenCUA [38], OS-Atlas [31], UGround [29], ShowUI [34], Aria-UI [36], Aguvis [35], CogAgent [21], SE-GUI [116], UI-R1 [114], GTA1 [113], MobileRL [109], and the RL-trained computer-use systems [108, 41].

The engineering substrate is at least as important and less often credited: BrowserGym and AgentLab for web environments [26]; Playwright, Selenium and Appium for control; OmniParser [118] and Set-of-Mark overlay generation [20] as perception front-ends; Android emulators and Docker/QEMU virtual machines for isolation; and distributed RL infrastructure (VERL, vLLM) for the training side ([108, 109, 41]). HIDAgent [91] is the outlier worth noting: a hardware execution layer that requires no software on the target device.

14.4 What is actually available

The artifact and availability columns of Tables 11 and 12 should be read with Section 8.3 in mind. A substantial fraction of the largest corpora are announced as open without a downloadable location in the version we inspected — among them GUI-Net-1M, GroundCUA, AgentTrek and SoftwareGym — and several release announcements omit licence terms and per-source counts. Where a link is given, it is listed in this survey's references.

Two resources deserve specific mention because they are instruments rather than data. ScreenSpot-V2 is the corrected version of the field's most-quoted grounding benchmark, with roughly 11% of labels fixed [31]; any comparison predating it should be read with that correction in mind. And OSWorld-Human supplies minimal-step human trajectories for all 369 OSWorld tasks, which is what makes efficiency measurement possible at all [13].

Takeaways

• The field now has credible, execution-verified environments on all three platforms plus a distinct mobile ladder from single-app to cross-app tasks.
• Trajectory corpora grew from hundreds of thousands to over a million items in about eighteen months, with provenance shifting from human demonstration to mining and self-labelling.
• Availability claims are unreliable: several of the largest corpora have no in-text link, and licences and per-source counts are frequently omitted.
• ScreenSpot-V2 and OSWorld-Human are the two artifacts most likely to change how existing results should be read.

15 Future directions

The directions below are derived from the gaps documented in the preceding chapters; each is stated as a research problem with the evidence that motivates it and, where possible, the measurement that would settle it.

Close the efficiency gap before the capability gap. Leading agents already succeed on a majority of single-application tasks on several platforms, but use 2.7–4.3 times the human step count, with planning and reflection accounting for 76–96% of end-to-end latency ([13], Section 10.4). Cost per task is the binding constraint on deployment, and it is barely optimised for: only one training method in this survey shapes its reward directly toward shorter correct trajectories [109]. The measurement that matters is a success-versus-cost Pareto curve per platform, which existing work has the data to produce [14].

Make evaluation independent of the system under test. No audited LLM judge exceeded 70% precision, and the judges that filter training data are frequently the same models that produced it (Sections 8.3, 10.3) [136]. Three concrete remedies are available and none is standard practice: report multiple runs with dispersion; evaluate at least a sample of trajectories with human or independent judgement; and re-run evaluation under systematic environment variation, where a single model currently spans 4% to 63% [133].

Adopt verification-first architectures. The decomposed ablations point the same way from three directions: verifying the resulting state is where the gain is, not checking that an action executed [93]; process rewards beat outcome rewards on the same base model [112]; and step-level decomposition exposes failure stages that end-to-end scores hide [139]. The open problem is cost — verification roughly halves reasoning throughput [93] — so the research question is not whether to verify but what to verify cheaply.

Solve cross-application, long-horizon work. Every platform shows the same profile: single-application tasks are largely solved, cross-application tasks collapse, often to zero (Section 10.2) [105, 134, 26]. Since typical benchmark tasks decompose into about three subgoals [63], progress here requires benchmarks that are actually long, not planners that are deeper.

Standardise data provenance and licensing. Corpus size grew an order of magnitude while reporting quality did not: several of the largest releases have no in-text link, many omit licences and per-source counts, and self-labelling procedures carry biases that are never audited (Sections 8.3, 5.4) [121]. A minimum reporting standard — source, yield, licence, annotation agreement, filter model and its version — would cost little and make the data literature reviewable.

Rekindle safety work on the parts that are not injection. Four gaps are concrete and addressable (Section 11.5): adaptive attacks against each proposed defence; utility-cost accounting alongside every attack-success figure; defences that operate on the image modality, where a single poisoned vision tower defeated every tested remedy [101]; and cross-platform safety evaluation.

Decide the interface question deliberately. Whether agents adapt to interfaces designed for humans or interfaces adapt to agents is currently being answered by default, through research effort rather than design. The technical case for agent-facing interfaces is now evidenced by working prototypes [123, 126]; what the surveyed literature does not address is who would enforce or standardise them [130]. This is the open question with the largest non-technical consequences.

Measure the exchange rate between training and scaffolding. The field has expensive trained systems and cheap training-free methods that are competitive on the tasks where both have been measured, and almost no study reports both axes (Sections 9.5, 13.2). Because annotation cost, rollout cost and inference cost are all rising, a single study comparing trained grounding models against test-time scaling at matched performance would be unusually valuable.

Takeaways

• The most actionable near-term direction is efficiency, where the gap is quantified and the Pareto framing already exists.
• Evaluation independence — repeated runs, independent judgement, environment variation — is a methodological change rather than a research breakthrough, and is the cheapest improvement available.
• Verification-first design is supported by three independent lines of evidence; the open problem is its cost.
• Cross-application and long-horizon competence, not deeper planning machinery, is where capability is missing.
• The interface question and the safety-modality gaps are the two areas where the literature is thinnest relative to their consequences.

References

[1] Pascal J. Sager, Benjamin Meyer, Peng Yan, Rebekka von Wartburg-Kottler, Layan Etaiwi, Aref Enayati, Gabriel Nobel, Ahmed Abdulkadir, Benjamin F. Grewe, Thilo Stadelmann. A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directions. Journal of Artificial Intelligence Research, vol. 85, 2026. doi:10.1613/jair.1.19490; arXiv:2501.16150

[2] Dang Nguyen, Jian Chen, Yu Wang, et al. GUI Agents: A Survey. 2024. arXiv:2412.13501

[3] Chaoyun Zhang, Shilin He, Jiaxu Qian, et al. Large Language Model-Brained GUI Agents: A Survey. 2024. arXiv:2411.18279

[4] Xueyu Hu, Tao Xiong, Biao Yi, et al. OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use. 2025. arXiv:2508.04482

[5] Shuyan Zhou, Frank F. Xu, Hao Zhu, et al. WebArena: A Realistic Web Environment for Building Autonomous Agents. 2023. arXiv:2307.13854

[6] Tianbao Xie, Danyang Zhang, Jixuan Chen, et al. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. 2024. arXiv:2404.07972

[7] Xiang Deng, Yu Gu, Boyuan Zheng, et al. Mind2Web: Towards a Generalist Agent for the Web. 2023. arXiv:2306.06070

[8] Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, et al. AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. 2024. arXiv:2405.14573

[9] Ada Chen, Yongjiang Wu, Junyuan Zhang, et al. A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron? 2025. arXiv:2505.10924

[10] Yucheng Shi, Wenhao Yu, Jingyuan Huang, Wenlin Yao, Wenhu Chen, Ninghao Liu. Towards Trustworthy GUI Agents: A Survey. 2025. arXiv:2503.23434

[11] Jiahao Li, Kaer Huang. A Survey on GUI Agents with Foundation Models Enhanced by Reinforcement Learning. 2025. arXiv:2504.20464

[12] Boyu Gou, Zanming Huang, Yuting Ning, et al. Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge. 2025. arXiv:2506.21506

[13] Reyna Abhyankar, Qi Qi, Yiying Zhang. OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents. 2025. arXiv:2506.16042

[14] Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, Arvind Narayanan. AI Agents That Matter. 2024. arXiv:2407.01502

[15] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, et al. WebGPT: Browser-assisted question-answering with human feedback. 2021. arXiv:2112.09332

[16] Peter C. Humphreys, David Raposo, Toby Pohlen, et al. A data-driven approach for learning to control computers. ICML 2022. arXiv:2202.08137

[17] Shunyu Yao, Howard Chen, John Yang, Karthik Narasimhan. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. 2022. arXiv:2207.01206

[18] Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, Timothy Lillicrap. Android in the Wild: A Large-Scale Dataset for Android Device Control. 2023. arXiv:2307.10088

[19] Xiao Liu, Hao Yu, Hanchen Zhang, et al. AgentBench: Evaluating LLMs as Agents. 2023. arXiv:2308.03688

[20] Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, Jianfeng Gao. Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V. 2023. arXiv:2310.11441

[21] Wenyi Hong, Weihan Wang, Qingsong Lv, et al. CogAgent: A Visual Language Model for GUI Agents. 2023. arXiv:2312.08914

[22] Chi Zhang, Zhao Yang, Jiaxuan Liu, et al. AppAgent: Multimodal Agents as Smartphone Users. 2023. arXiv:2312.13771

[23] Hongliang He, Wenlin Yao, Kaixin Ma, et al. WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models. 2024. arXiv:2401.13919

[24] Jing Yu Koh, Robert Lo, Lawrence Jang, et al. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. 2024. arXiv:2401.13649

[25] Kanzhi Cheng, Qiushi Sun, Yougang Chu, et al. SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents. 2024. arXiv:2401.10935

[26] Alexandre Drouin, Maxime Gasse, Massimo Caccia, et al. WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? 2024. arXiv:2403.07718

[27] Keen You, Haotian Zhang, Eldon Schoop, et al. Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs. 2024. arXiv:2404.05719

[28] Rogerio Bonatti, Dan Zhao, Francesco Bonacci, et al. Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale. 2024. arXiv:2409.08264

[29] Boyu Gou, Ruohan Wang, Boyuan Zheng, et al. Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents. 2024. arXiv:2410.05243

[30] Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, Xin Eric Wang. Agent S: An Open Agentic Framework that Uses Computers Like a Human. 2024. arXiv:2410.08164

[31] Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, et al. OS-Atlas: A Foundation Action Model for Generalist GUI Agents. 2024. arXiv:2410.23218

[32] Xiao Liu, Bo Qin, Dongzhu Liang, et al. AutoGLM: Autonomous Foundation Agents for GUIs. 2024. arXiv:2411.00820

[33] Zehan Qi, Xiao Liu, Iat Long Iong, et al. WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning. 2024. arXiv:2411.02337

[34] Kevin Qinghong Lin, Linjie Li, Difei Gao, et al. ShowUI: One Vision-Language-Action Model for GUI Visual Agent. 2024. arXiv:2411.17465

[35] Yiheng Xu, Zekun Wang, Junli Wang, et al. Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction. 2024. arXiv:2412.04454

[36] Yuhao Yang, Yue Wang, Dongxu Li, et al. Aria-UI: Visual Grounding for GUI Instructions. 2024. arXiv:2412.16256

[37] Yujia Qin, Yining Ye, Junjie Fang, et al. UI-TARS: Pioneering Automated GUI Interaction with Native Agents. 2025. arXiv:2501.12326

[38] Xinyuan Wang, Bowen Wang, Dunjie Lu, et al. OpenCUA: Open Foundations for Computer-Use Agents. 2025. arXiv:2508.09123

[39] Xuehui Wang, Zhenyu Wu, JingJing Xie, et al. MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents. 2025. arXiv:2507.19478

[40] Pei Yang, Hai Ci, Mike Zheng Shou. macOSWorld: A Multilingual Interactive Benchmark for GUI Agents. 2025. arXiv:2506.04135

[41] Dunjie Lu, Shuai Bai, Tianyi Bai, et al. Qwen-CUA: Native Computer Use for (almost) Everything. 2026. arXiv:2608.02352

[42] Wanli Li, Bowen Zhou, Yunyao Yu, et al. WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces. 2026. arXiv:2606.09426

[43] Yikun Fu, Bowen Fu, Zhenyu Wu, et al. MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop. 2026. arXiv:2606.22557

[44] Lawrence Keunho Jang, Andrew Keunwoo Jang, Jing Yu Koh, Ruslan Salakhutdinov. MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents. 2026. arXiv:2606.16748

[45] Ivan Evtimov, Arman Zharmagambetov, Aaron Grattafiori, Chuan Guo, Kamalika Chaudhuri. WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks. 2025. arXiv:2504.18575

[46] Tri Cao, Bennett Lim, Yue Liu, et al. VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents. 2025. arXiv:2506.02456

[47] Ke Yang, Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chaudhari, George Karypis, Huzefa Rangwala. AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents. 2024. arXiv:2410.13825

[48] Liangbo Ning, Ziran Liang, Zhuohang Jiang, et al. A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Models. 2025. arXiv:2503.23350

[49] Guangyi Liu, Pengxiang Zhao, Yaozhen Liang, et al. LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects. 2025. arXiv:2504.19838

[50] Biao Wu, Yanda Li, Zhiwei Zhang, Yunchao Wei, Meng Fang, Ling Chen. Foundations and Recent Trends in Multimodal Mobile Agents: A Survey. 2024. arXiv:2411.02006

[51] Shuai Wang, Weiwen Liu, Jingxuan Chen, et al. GUI Agents with Foundation Models: A Comprehensive Survey. 2024. arXiv:2411.04890

[52] Junan Hu, Jian Liu, Jingxiang Lai, et al. GUI Agents with Reinforcement Learning: Toward Digital Inhabitants. 2026. arXiv:2604.27955

[53] Zhisong Zhang, Tianqing Fang, Kaixin Ma, Wenhao Yu, Hongming Zhang, Haitao Mi, Dong Yu. WebRollback: Enhancing Web Agents with Explicit Rollback Mechanisms. 2025. arXiv:2504.11788

[54] Genglin Liu, Shijie Geng, Sha Li, Hejie Cui, Sarah Zhang, Xin Liu, Tianyi Liu. WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidance. 2025. arXiv:2511.12997

[55] Ziyu Lu, Tengjin Weng, Yiying Yang, Yuhang Zhao, Xinxin Huang, Wenhao Jiang. BEAP-Agent: Backtrackable Execution and Adaptive Planning for GUI Agents. 2026. arXiv:2601.21352

[56] Yu Gu, Kai Zhang, Yuting Ning, Boyuan Zheng, Boyu Gou, Tianci Xue, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, Huan Sun, Yu Su. Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents. 2024. arXiv:2411.06559

[57] Yilin Cao, Yufeng Zhong, Zhixiong Zeng, Siran Dai, Liming Zheng, Jing Huang, Haibo Qiu, Peng Shi, Wenji Mao. MobileDreamer: Generative Sketch World Model for GUI Agent. 2026. arXiv:2601.04035

[58] Zichen Zhu, Hao Tang, Yansi Li, et al. MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automation. 2024. arXiv:2410.13757

[59] Yuquan Xie, Zaijing Li, Rui Shao, Gongwei Chen, Kaiwen Zhou, Yinchuan Li, Dongmei Jiang, Liqiang Nie. Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skills. 2025. arXiv:2506.10387

[60] Shang Ma, Xusheng Xiao, Yanfang Ye. Agent+P: Guiding UI Agents via Symbolic Planning. 2025. arXiv:2510.06042

[61] Yi Kong, Dianxi Shi, Guoli Yang, et al. MapAgent: Trajectory-Constructed Memory-Augmented Planning for Mobile Task Automation. 2025. arXiv:2507.21953

[62] Zikang Liu, Junyi Li, Wayne Xin Zhao, Dawei Gao, Yaliang Li, Ji-rong Wen. PAL-UI: Planning with Active Look-back for Vision-Based GUI Agents. 2025. arXiv:2510.00413

[63] Mohamed Aghzal, Gregory J. Stein, Ziyu Yao. Why Do LLM-based Web Agents Fail? A Hierarchical Planning Perspective. 2026. arXiv:2603.14248

[64] Junyang Wang, Haiyang Xu, Haitao Jia, et al. Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration. 2024. arXiv:2406.01014

[65] Aarash Feizi, Shravan Nayak, Xiangru Jian, et al. Grounding Computer Use Agents on Human Demonstrations. 2025. arXiv:2511.07332

[66] Jian Mu, Chaoyun Zhang, Chiming Ni, et al. GUI-360°: A Comprehensive Dataset and Benchmark for Computer-Using Agents. 2025. arXiv:2511.04307

[67] Quanfeng Lu, Wenqi Shao, Zitao Liu, et al. GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices. 2024. arXiv:2406.08451

[68] Yiheng Xu, Dunjie Lu, Zhennan Shen, et al. AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials. 2024. arXiv:2412.09605

[69] Bofei Zhang, Zirui Shang, Zhi Gao, et al. TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents. 2025. arXiv:2504.12679

[70] Brandon Trabucco, Gunnar Sigurdsson, Robinson Piramuthu, Ruslan Salakhutdinov. InSTA: Towards Internet-Scale Training For Agents. 2025. arXiv:2502.06776

[71] Shikhar Murty, Hao Zhu, Dzmitry Bahdanau, Christopher D. Manning. NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wild. 2024. arXiv:2410.02907

[72] Tian Jin, Wenhao Wang, Jing Kang, et al. SoftwareGym: Scaling Computer-Use Agent Training with Diverse Synthesized Software Environments. 2026. arXiv:2610.12611

[73] Longxi Gao, Li Zhang, Shihe Wang, et al. MobileViews: A Million-scale and Diverse Mobile GUI Dataset. 2024. arXiv:2409.14337

[74] Hanyu Lai, Junjie Gao, Xiao Liu, et al. AndroidGen: Building an Android Language Agent under Data Scarcity. 2025. arXiv:2504.19298

[75] Zikai Xiao, Jianhong Tu, Chuhang Zou, et al. WebWorld: A Large-Scale World Model for Web Agent Training. 2026. arXiv:2602.14721

[76] Naimeng Ye, Xiao Yu, Ruize Xu, Tianyi Peng, Zhou Yu. AI Agents for Web Testing: A Case Study in the Wild. 2025. arXiv:2509.05197

[77] Antoine Chevrot, Alexandre Vernotte, Jean-Rémy Falleri, Xavier Blanc, Bruno Legeard. Are Autonomous Web Agents Good Testers? 2025. arXiv:2504.01495

[78] Xiwen Teoh, Yun Lin, Duc-Minh Nguyen, Ruofei Ren, Wenjie Zhang, Jin Song Dong. WebTestPilot: Agentic End-to-End Web Testing against Natural Language Specification by Inferring Oracles with Symbolized GUI Elements. 2026. arXiv:2602.11724

[79] Shamit Fatin, Mehbubul Hasan Al-Quvi, Haz Sameen Shahgir, et al. LELANTE: LEveraging LLM for Automated ANdroid TEsting. 2025. arXiv:2504.20896

[80] Xiaolei Li, Jialun Cao, Yepang Liu, Shing-Chi Cheung, Hailong Wang. ReuseDroid: A VLM-empowered Android UI Test Migrator Boosted by Active Feedback. 2025. arXiv:2504.02357

[81] Biniam Fisseha Demissie, Yan Naing Tun, Lwin Khin Shar, Mariano Ceccato. VLM-Fuzz: Vision Language Model Assisted Recursive Depth-first Search Exploration for Effective UI Testing of Android Apps. 2025. arXiv:2504.11675

[82] Maryam Taeb, Amanda Swearngin, Eldon Schoop, Ruijia Cheng, Yue Jiang, Jeffrey Nichols. AXNav: Replaying Accessibility Tests from Natural Language. 2023. doi:10.1145/3613904.3642777; arXiv:2310.02424

[83] Yuxuan Lu, Ting-Yao Hsu, Hansu Gu, et al. AgentA/B: Automated and Scalable Web A/B Testing with Interactive LLM Agents. 2025. arXiv:2504.09723

[84] Shridhar Mehendale, Ankit Walishetti. DexAssist: A Voice-Enabled Dual-LLM Framework for Accessible Web Navigation. 2024. arXiv:2411.12214

[85] Anthropic. Developing a computer use model. 2024. anthropic.com/news/developing-computer-use

[86] OpenAI. Computer-Using Agent. 2025. openai.com/index/computer-using-agent

[87] Gaole Dai, Shiqi Jiang, Ting Cao, Yuanchun Li, Yuqing Yang, Rui Tan, Mo Li, Lili Qiu. Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deployment. 2025. arXiv:2503.15937

[88] Jiangang Han. WebFovea: When the Model Is Right but the Click Is Wrong — Reliable Round Trips for Vision-Based Web Agents on Live Websites. 2026. arXiv:2610.03036

[89] Linxin Song, Yutong Dai, Viraj Prabhu, et al. CoAct-1: Computer-using Multi-Agent System with Coding Actions. 2025. arXiv:2508.03923

[90] Jungjae Lee, Dongjae Lee, Chihun Choi, et al. VeriSafe Agent: Safeguarding Mobile GUI Agent via Logic-based Action Verification. 2025. arXiv:2503.18492

[91] Jeffrey P. Bigham. HIDAgent: A Toolkit Enabling "Personal Agents" on HID-Compatible Devices. 2026. arXiv:2602.00492

[92] Yuzhe Zhang, Xianwei Xue, Xingyong Wu, et al. Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction. 2026. arXiv:2604.05477

[93] Qinzhuo Wu, Pengzhi Gao, Wei Liu, Jian Luan. BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism. 2025. arXiv:2505.20660

[94] Daniel Jones, Giorgio Severi, Martin Pouliot, et al. A Systematization of Security Vulnerabilities in Computer Use Agents. 2025. arXiv:2507.05445

[95] Chaoran Chen, Zhiping Zhang, Bingcan Guo, et al. The Obvious Invisible Threat: LLM-Powered GUI Agents' Vulnerability to Fine-Print Injections. 2025. arXiv:2504.11281

[96] Rahul Deivasigamani, Sayeda Faatin Alvi, Derqui Andrea, Kaushal Punjabi, Stjepan Picek. Not an A11y: How Android Accessibility Exposes Mobile AI Agents to Indirect Prompt Injection. 2026. arXiv:2608.08939

[97] Juyong Lee, Dongyoon Hahm, June Suk Choi, W. Bradley Knox, Kimin Lee. MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control. 2024. arXiv:2410.17520

[98] Jingyi Yang, Shuai Shao, Dongrui Liu, Jing Shao. RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agents. 2025. arXiv:2506.00618

[99] Xilong Wang, John Bloch, Zedian Shao, Yuepeng Hu, Shuyan Zhou, Neil Zhenqiang Gong. WebInject: Prompt Injection Attack to Web Agents. EMNLP 2025. arXiv:2505.11717

[100] Zhihao Liu, Hongyu Sun, Zhiyuan Fu, et al. AgentHijack: Visual Patch Attacks on Multimodal Computer-Use Agents. 2026. arXiv:2609.09212

[101] Ziang Ye, Yang Zhang, Wentao Shi, Xiaoyu You, Fuli Feng, Tat-Seng Chua. VisualTrap: A Stealthy Backdoor Attack on GUI Agents via Visual Grounding Manipulation. 2025. arXiv:2507.06899

[102] Giulio Zingrillo, Hanna Foerster, Ilia Shumailov, Yiren Zhao, Robert Mullins. Securing Computer-Use Agents Against Branch Steering Attacks. 2026. arXiv:2610.03089

[103] Kaiyuan Zhang, Mark Tenenholtz, Kyle Polley, Jerry Ma, Denis Yarats, Ninghui Li. BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agents. 2025. arXiv:2511.20597

[104] Maria Sanchez del Rio, Mikhail Terekhov. Control OSWorld: An AI Control Environment for GUI Computer Use Agents. 2026. arXiv:2610.03818

[105] Jingxuan Chen, Derek Yuen, Bin Xie, et al. SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation. 2024. arXiv:2410.15164

[106] Shihan Deng, Weikai Xu, Hongda Sun, et al. Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents. 2024. arXiv:2407.00993

[107] Fanbin Lu, Zhisheng Zhong, Shu Liu, Chi-Wing Fu, Jiaya Jia. ARPO: End-to-End Policy Optimization for GUI Agents with Experience Replay. 2025. arXiv:2505.16282

[108] Hanyu Lai, Xiao Liu, Yanxiao Zhao, et al. ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents. 2025. arXiv:2508.14040

[109] Yifan Xu, Xiao Liu, Xinghan Liu, et al. MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agents. 2025. arXiv:2509.18119

[110] Yucheng Shi, Wenhao Yu, Zaitang Li, et al. MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment. 2025. arXiv:2507.05720

[111] Zhengxi Lu, Jiabo Ye, Fei Tang, et al. UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning. 2025. arXiv:2509.11543

[112] Cong Chen, Kaixiang Ji, Hao Zhong, et al. GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Tasks. 2025. arXiv:2509.23738

[113] Yan Yang, Dongxu Li, Yutong Dai, et al. GTA1: GUI Test-time Scaling Agent. 2025. arXiv:2507.05791

[114] Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, et al. UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning. 2025. arXiv:2503.21620

[115] Run Luo, Lu Wang, Wanwei He, Longze Chen, Jiaming Li, Xiaobo Xia. GUI-R1: A Generalist R1-Style Vision-Language Action Model For GUI Agents. 2025. arXiv:2504.10458

[116] Xinbin Yuan, Jian Zhang, Kaixin Li, et al. Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning. 2025. arXiv:2505.12370

[117] Jason Wu, Xiaoyi Zhang, Jeff Nichols, Jeffrey P. Bigham. Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots. 2021. doi:10.1145/3472749.3474763; arXiv:2109.08763

[118] Yadong Lu, Jianwei Yang, Yelong Shen, Ahmed Awadallah. OmniParser for Pure Vision Based GUI Agent. 2024. arXiv:2408.00203

[119] Kaixin Li, Ziyang Meng, Hongzhan Lin, et al. ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use. 2025. arXiv:2504.07981

[120] Miaosen Zhang, Ziqiang Xu, Jialiang Zhu, et al. Phi-Ground Tech Report: Advancing Perception in GUI Grounding. 2025. arXiv:2507.23779

[121] Zhiqi Ge, Juncheng Li, Xinglei Pang, et al. Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining. 2024. arXiv:2412.10342

[122] Tiange Luo, Lajanugen Logeswaran, Justin Johnson, Honglak Lee. Visual Test-time Scaling for GUI Agent Grounding. 2025. arXiv:2505.00684

[123] Rui Xie, Lu Chen. ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions. Findings of EMNLP 2026. arXiv:2608.26991

[124] Xing Han Lù, Zdeněk Kasner, Siva Reddy. WebLINX: Real-World Website Navigation with Multi-Turn Dialogue. 2024. arXiv:2402.05930

[125] Chaoyun Zhang, He Huang, Chiming Ni, et al. UFO2: The Desktop AgentOS. 2025. arXiv:2504.14603

[126] Sunghwan Kim, Deogyong Kim, Sangam Lee, Donguk Kwon, Dongha Lee. Web$hell: A Unified Interface for Agentic Web Access. 2026. arXiv:2610.13416

[127] Masafumi Enomoto, Ryoma Obara, Haochen Zhang, Masafumi Oyamada. Read More, Think More: Revisiting Observation Reduction for Web Agents. 2026. arXiv:2604.01535

[128] Weihao Tan, Wentao Zhang, Xinrun Xu, et al. Cradle: Empowering Foundation Agents Towards General Computer Control. 2024. arXiv:2403.03186

[129] Yong Liu, Zhenyi Zhong, Zhanpeng Shi. Tactile: Giving Computer-Using Agents Hands and Feet. 2026. arXiv:2607.14443

[130] Xing Han Lù, Gaurav Kamath, Marius Mosbach, Siva Reddy. Build the web for agents, not agents for the web. 2025. arXiv:2506.10953

[131] Chenrui Shi, Yuwei Wu, Yang Liu, et al. Interactive Reward Agent: GUI Task Evaluation via Environment-State Verification. 2026. arXiv:2607.25904

[132] Tao Xiong, Xavier Hu, Yurun Chen, et al. GUI-PRA: Process Reward Agent for GUI Tasks. 2025. arXiv:2509.23263

[133] Karen Ullrich, Jingtong Su, Claudia Shi, et al. OpenApps: Simulating Environment Variations to Measure UI-Agent Reliability. 2025. arXiv:2511.20766

[134] Yuxiang Chai, Shunye Tang, Han Xiao, et al. A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluation. 2025. arXiv:2501.01149

[135] Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, et al. UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction. 2025. arXiv:2503.15661

[136] Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade, et al. AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories. 2025. arXiv:2504.08942

[137] Tianci Xue, Weijian Qi, Tianneng Shi, et al. An Illusion of Progress? Assessing the Current State of Web Agents. COLM 2025. arXiv:2504.01382

[138] Su Kara, Fazle Faisal, Suman Nath. WAREX: Web Agent Reliability Evaluation on Existing Benchmarks. 2025. arXiv:2510.03285

[139] Daniel Röder, Akhil Juneja, Roland Roller, Sven Schmeier. Detecting Pipeline Failures through Fine-Grained Analysis of Web Agents. 2025. arXiv:2509.14382

[140] Tianchen Guan, Xinlei Lin, Royce Cheng-Yue, Xiangjun Wang, Shuyan Zhou. ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents. 2026. arXiv:2608.18307

[141] Thassilo M. Schiepanski, Nicholas Piel. Beyond Pixels: Exploring DOM Downsampling for LLM-Based Web Agents. 2025. arXiv:2508.04412