Agents

The agents/ folder contains reference agents and the programs used to run them.

You only need it if:

• you want to run one of the reference agents
• you want a starting point for your own agent
• you want to evaluate an agent across benchmark tasks

The Two Main Parts

Most of this section comes down to two folders:

What Lives In agents/agents/

agents/agents/ contains the agent classes.

An agent is the code that looks at the current observation and decides what actions to take next.

We currently ship these agents:

• ClaudeAgent
• Gemini3Agent
• GeminiComputerUseAgent, GPT54ComputerUseAgent (provider-native computer-use tools)
• Qwen3VLAgent
• KimiAzureAgent
• ClaudeCodeAgent, CodexCliAgent (external CLI harnesses, see below)

These classes are exported through agents.agents, which is why the evaluation commands refer to names such as --agent ClaudeAgent.

What Lives In agents/evaluation/

agents/evaluation/ contains the programs that actually run an agent against environments and tasks.

The two main ones are:

• run_single.py: run one agent on one task
• run_batch.py: run one agent across many tasks

If you only want to try an agent once, start with run_single.py.

What An Agent Receives

Our current agent interface is built around four methods:

• __init__
• init(task_description, display_resolution, save_path)
• step(obs, action_outputs)
• finish(...)

The important one is step(...).

That method receives:

• the latest observation from the environment
• the outputs from the previous actions

and it returns one or more action groups.

Each action group contains:

• tool_id
• actions

The actions list can contain normal environment actions such as mouse and keyboard input. It can also contain built-in control actions such as:

• {"action": "screenshot"}
• {"action": "wait", "time": 1.5}

Those control actions are handled by the environment layer, not by the evaluation loop.

GeminiComputerUseAgent accepts either the normal obs["screen"]["path"] or a chronological obs["frames"] list. When several frames are present, every frame is sent in order in the initial request and in later computer function responses. Its request deadline and total retry attempts can be set with request_timeout_seconds and request_attempts in agent_args.

Autonomous Agents (External CLI Harnesses)

Some agents are not step-based. ClaudeCodeAgent and CodexCliAgent wrap existing coding CLIs (Claude Code, Codex) and let the CLI run the whole episode on its own. These set a class flag autonomous = True and implement run_episode(env, task_description) instead of step(...). run_single.py detects the flag, calls run_episode once, and then runs the same mark_done verification as every other agent.

The CLI never runs inside the task VM. It runs in a throwaway, GUI-less scratch container and controls the task through an in-process action gateway. The prompt gives the CLI the gateway's authenticated HTTP contract; the CLI may call it from Python, JavaScript, shell scripts, loops, image-analysis code, or the optional act convenience command. The gateway accepts one action from the same vocabulary the other agents use and returns screenshot data plus the remaining step budget. Paused mode retains its configured chronological frame window. All three live modes configure the runner for one zero-duration frame and return exactly that frame; screenshot-only calls do not spend the action budget. In paused mode the task freezes on the final returned frame after each observation window, and the next action is applied to that final frame's state.

The gateway has independent action and observation lanes. A target-time priority queue orders actions and waits without owning the injection lane. The lane is occupied only while env.step(..., capture_observation=False, settle_after_actions=False) injects an action and receives its acknowledgement. Observation capture is serialized by a different lock, so a waiting or newly arrived action can proceed while another request retrieves and encodes its frame. The action response still waits for its own post-ack frame before it is returned.

The gateway is the contract boundary, not a prescribed client or control-loop implementation. The task environment is a separate VM whose files and credentials are not exposed to the CLI container. The shared machinery lives in agents/shared/cli_harness.py.

Frame-window environments expose four timing contracts through one evaluator flag:

Timestamped responses report frame_captured_at_s for the image and current_time_s near response serialization. These are separate because frame retrieval and encoding can take time. Action responses also report the acknowledgement as action_executed_at_s, the time from the request's last observed frame to that acknowledgement, and recent observe-to-execute latencies. In live_timestamped_execution, a mouse or keyboard action may add an absolute execute_at_s on the same clock. The queue wakes the injection lane when that target is due; the response reports measured lateness between the requested time and acknowledgement. A time that has passed executes immediately. The other three modes reject that field. This contract is implemented by the shared CLI harness, so it applies to every CodexCliAgent model, including gpt-5.6-terra and gpt-5.6-luna.

The sandbox is backend-agnostic and auto-selected the way env runners are (GYM_ANYTHING_AGENT_SANDBOX override, then detect), in agents/shared/agent_sandbox.py: Apptainer (rootless, no daemon, the primary path on clusters) or Docker (daemon machines, using a bridge network). There is deliberately no no-isolation fallback: if neither is available the run fails rather than launch a permission-bypassed CLI unsandboxed. File and process isolation hold on both backends, and the env is a separate VM either way, so the agent has no filesystem path into it. Network egress is not an isolation boundary: rootless Apptainer shares the host network and Docker can reach the host gateway because both the model API and action gateway must remain reachable. The harness withholds task-VM addresses and credentials; it does not enforce a destination allowlist.

These agents need Apptainer or Docker on the host. Claude Code reads ANTHROPIC_API_KEY. Codex uses the host's existing CLI login: the harness copies only $CODEX_HOME/auth.json (default ~/.codex/auth.json) into a mode-0700 host scratch directory mounted as the container's private CODEX_HOME for each run. It does not mount the host Codex home or store the copied credential with episode artifacts, and it deletes the private scratch directory during teardown. Pass codex_auth_path in agent_args to select another login file. Codex sessions are ephemeral by default. Pass persist_session=true to retain only the session JSONL under the episode's cli_harness/codex_home/sessions/ directory; the copied login file remains private and is discarded with the container.

The Fastest Way To Try One

gym-anything benchmark moodle --task enroll_student --agent ClaudeAgent --model claude-opus-4

That command loads the environment, resets it, creates the agent, lets it act until the run finishes, and writes run artifacts.

If You Want To Add Your Own Agent

The simplest workflow is:

1. copy a nearby file in agents/agents/
2. implement the same basic methods
3. export the new class from agents/agents/__init__.py
4. run it with gym-anything benchmark

If you're starting from scratch, read:

1. agents/agents/base.py
2. one concrete agent such as agents/agents/claude.py
3. agents/evaluation/run_single.py

That gives you the agent interface first, then one implementation, then the program that drives it.