Gym-Anything: Interactive Paper

Real-world workflows are long-horizon and take place in heterogeneous environments, often requiring hundreds of steps across diverse software configured with domain-specific data. Analysing a medical imaging dataset requires a radiology tool set up with annotated clinical CT scans. Reconciling financial records requires an ERP system populated with transaction histories. No existing benchmark provides this kind of domain-grounded evaluation.

Existing benchmarks fall into two categories, each addressing the problem from a different angle but with different trade-offs:

Static datasets (Mind2Web, AITW, AndroidControl, OmniACT) collect thousands of episodes but evaluate via action-matching, penalizing valid alternative strategies. An agent that accomplishes the same goal through a different but equally correct sequence of actions is marked as wrong.

Interactive benchmarks such as OSWorld, WebArena, WindowsAgentArena, and AndroidWorld pioneered execution-based evaluation in realistic environments, but are constrained in scale: OSWorld covers 9 software with 369 tasks, WebArena covers 6 websites with 812 tasks, WindowsAgentArena covers 11 software with 154 tasks, and AndroidWorld covers 20 apps with 116 tasks. No single existing benchmark provides training data, long-horizon tasks, and broad occupational coverage simultaneously.

The pipeline proceeds in seven steps:

1. Estimate GDP per occupation using employment count × mean wage, scaled to national GDP. 2. Discover software used per occupation using an LLM with web search, producing ~16,600 software across ~1,400 categories. 3. Clean the catalog by deduplicating and removing hallucinated entries via web-grounded verification. 4. Attribute GDP to individual software via the formula below.

Where p_computer is the fraction of work involving computers (from occupational surveys), s_category is the share attributed to a software category (e.g., "spreadsheets" for an accountant), and s_software is the individual software's share within its category.

5. Filter to sandboxable: self-hostable, free, GUI-based, no specialized hardware. Only ~3,400 of ~16,600 pass. 6. Substitute: high-GDP non-sandboxable software (e.g., Bloomberg Terminal at $79.5B, Salesforce, Epic EHR) is replaced with the closest free alternative. 7. Tiered selection across five tiers to balance economic impact and diversity.

Each environment is defined by a simple JSON spec:

{
  "env_id": "moodle_env",
  "name": "Moodle LMS",
  "base": "ubuntu-gnome-systemd",
  "resources": { "cpus": 4, "memory": "4G" },
  "hooks": {
    "pre_start": "scripts/install.sh",
    "post_start": "scripts/configure.sh"
  },
  "observations": ["rgb_screen"],
  "actions": ["mouse", "keyboard"]
}

Agents interact through a standard Gymnasium-style API:

import gym_anything

# Create environment with a specific task
env = gym_anything.make("moodle_env", task="enroll_student")
obs = env.reset()

for step in range(200):
    actions = agent.step(obs)  # Your agent decides actions
    obs, reward, done, info = env.step(actions)
    if done:
        break

result = info["verification"]
print(f"Score: {result['score']}, Passed: {result['passed']}")
env.close()

Previous works such as OSWorld and VisualWebArena demonstrated the value of interactive evaluation by manually constructing environments through direct OS interaction and VM snapshots. While this approach produces high-quality setups, the resulting snapshots are difficult to inspect, version-control, or partially reuse, making it challenging to scale to hundreds of applications.

In Gym-Anything, the staged design enables caching at each stage boundary: creating new tasks for an already-configured application only requires re-running the task-specific setup script. Combined with network-process-file isolation between containers, this enables massive parallelization: in our experiments we run 400+ concurrent environments across 1,600 CPUs.

The library supports multiple compute backends. Docker for standard deployments. Apptainer for rootless systems like SLURM clusters. QEMU (with and without Apptainer) for full OS virtualization including Windows, with support for QEMU savevm for instant VM state restoration. AVD (Android Virtual Device) for Android environments. AVF (Apple Virtualization Framework) for macOS with Rosetta translation.

moodle_env/
  env.json            # Environment specification
  scripts/
    install.sh        # Software + dependencies
    configure.sh      # Realistic data + settings
  tasks/
    enroll_student/
      task.json       # Task description + success criteria
      setup_task.sh   # Set specific starting state
      export_result.sh# Post-task data export
      verifier.py     # Programmatic verification
    create_course/
      ...

def verify(traj, env_info, task_info):
    # Pull data from running container
    result = env_info['copy_from_env']('/tmp/enrollment.json')
    data = json.loads(result)
    enrolled = any(s['name'] == 'Jane Smith' for s in data['students'])
    return {
        "passed": enrolled,
        "score": 100 if enrolled else 0,
        "feedback": "Student enrolled" if enrolled else "Not found"
    }

The creation agent is a coding agent (Claude Opus 4.5/4.6 via Claude Code) equipped with bash, python, and computer-use tools. Given a new software name, it: researches how the software should be configured; finds and downloads real-world data (e.g., public medical imaging datasets for radiology software, published email corpora for messaging clients); studies similar previously created environments; implements setup scripts; launches the environment; takes screenshots; uses visual grounding to verify the application reached the expected state; and iteratively debugs failures.

Separating creation and audit roles provides three benefits: (a) removes self-confirmation bias, (b) the written audits ensure interpretability, and (c) the adversarial framing catches self-misleading claims. The audit agent analyses screenshots and logs, inspects script files, and if necessary actually runs the environment. It outputs a structured audit detailing what is correct and what the critical issues are.

A recurring challenge is context fatigue: after processing hundreds of thousands of tokens, the creation agent often declares the task done prematurely. We address this with a re-prompting technique — whenever the agent stops, we re-prompt it to reread the setup guidelines and checklists and complete any requirements it may have skipped.

The proposer follows guidelines across three dimensions: realism (does the instruction reflect genuine real-world use?), difficulty (does the task require multi-step, long-horizon interaction?), and diversity (do tasks cover varied functionality of the software?).

The agentic loop in the proposer is necessary because the model must actively run the software, search for and download realistic data, interact via the GUI, and verify the resulting state. This expensive step occurs only ~5 times per software but is essential for producing high-quality seeds.

During amplification, tasks are generated sequentially: the model receives all previously generated instructions as context when producing each new task, ensuring diversity. Semantic similarity filtering then discards near-duplicates. After generation, automated filtering launches each task, captures the starting state, and a VLM checks whether it matches the task description. Tasks failing this check are discarded. The pipeline yields 12,103 tasks total after filtering.

The verification formula: V(τ) = Σ w_i · VLM(τ, c_i, I), where each c_i is a subtask to verify and w_i is its point value. The VLM (Gemini 3 Flash) receives the full trajectory τ, the subtask description, and privileged information I, yielding a score from 0-100 rather than binary pass/fail.

Privileged information examples:

A separate integrity checklist ensures agents did not bypass the intended workflow. Three conditions: (a) the intended software was actually used, (b) application state was reached through the GUI (not by directly editing files), (c) no environment artifacts were exploited. Failing any condition zeroes the entire score.

Click any screenshot to enlarge. Showing 36 of 200+ environments:

Contamination filtering: Tasks are nodes in a similarity graph with LLM-graded similarity scores (1-8 scale) as edges. Any pair scoring 4+ is flagged. Connected components are assigned to a single split, ensuring no contaminated pairs cross the train/test boundary.

CUA-World-Long: 200 tasks (one per software) designed through trajectory-guided analysis of agent failure modes. The agent receives 8 quality guidelines covering real-world relevance, objective evaluability, and realistic data. All 200 tasks are manually verified. These tasks often require 200+ steps for humans and 500+ steps for models, with a mean of 425 steps across all Long trajectories.

Click an occupation group to see representative tasks:

Click a trajectory to enlarge: