Testing

We have a mix of fast contract tests and heavier integration tests.

A good workflow is:

1. run the most relevant targeted tests while you're working
2. run a broader set before you finish
3. only run the heavier execution tests when the change actually affects that area

Fast Broad Check

If you want one general pass over the Python test suite:

python -m pytest tests -q

Tests By Area

Core runtime

Use these when changing environment lifecycle, session info, action handling, or public API surface:

python -m pytest \
  tests/test_public_api_contract.py \
  tests/test_env_runtime_behaviors.py \
  tests/test_cli_contract.py -q

Benchmarks and verification

Use these when changing benchmark loading, task validation, or verifier behavior:

python -m pytest \
  tests/test_benchmark_registry.py \
  tests/test_verification_system.py \
  tests/test_verification_status.py -q

Remote cluster

Use these when changing the remote client, master, worker, or dashboard paths:

python -m pytest \
  tests/test_remote_client.py \
  tests/test_remote_module_layout.py \
  tests/test_worker_reset_policy.py \
  tests/test_remote_cluster_integration.py -q

Agents

Use these when changing the agent interface or the evaluation loop:

python -m pytest \
  tests/test_agents_module_layout.py \
  tests/test_agent_evaluation_contract.py -q

Runners and execution support

Use these when changing runner selection, compatibility reporting, or real execution behavior:

python -m pytest \
  tests/test_compatibility.py \
  tests/test_doctor.py \
  tests/test_qemu_dbus_fast_io.py \
  tests/test_runner_execution_contracts.py -q

Real Execution Tests

GYM_ANYTHING_RUN_EXECUTION_TESTS=1 \
GYM_ANYTHING_EXECUTION_RUNNERS=avf \
python -m pytest tests/test_runner_execution_contracts.py -q

Screenshot Latency Benchmarks

When changing QEMU screenshot capture, benchmark the integrated API boundary, not only runner internals:

PYTHONPATH=src GYM_ANYTHING_RUNNER=qemu \
python benchmarks/cua_world/tools/measure_screenshot_latency.py \
  --env-dir benchmarks/cua_world/environments/google_earth_env \
  --task take_screenshot \
  --samples 50 \
  --warmup 10 \
  --interleave \
  --methods qmp_ppm,api_capture_observation,api_step_noop,api_step_screenshot \
  --use-cache \
  --cache-level post_task \
  --use-savevm

To benchmark the optional D-Bus backend:

PYTHONPATH=src GYM_ANYTHING_RUNNER=qemu \
GYM_ANYTHING_QEMU_FAST_IO_BACKEND=dbus \
python benchmarks/cua_world/tools/measure_screenshot_latency.py \
  --env-dir benchmarks/cua_world/environments/google_earth_env \
  --task take_screenshot \
  --samples 50 \
  --warmup 10 \
  --interleave \
  --methods api_capture_observation,api_step_noop,api_step_screenshot,qmp_ppm \
  --use-cache \
  --cache-level post_task \
  --use-savevm

Input Latency Benchmarks

When changing QEMU keyboard or mouse input, benchmark both the fast input dispatch boundary and the full env.step(...) boundary. The input benchmark also writes guest device-list evidence, XInput event evidence, semantic application checks, and review screenshots for move, click, right-click, middle-click, double-click, triple-click, drag, scroll, text, and hotkey behavior.

Linux fast keyboard input requires the uinput agent and should show GymAnything Fast Keyboard in the guest device evidence. The QMP keyboard path is only an explicit experimental backend; it uses temporary 5 ms hold / 10 ms inter-key pacing, so passing latency numbers there do not by themselves prove production keyboard correctness. Use GYM_ANYTHING_QEMU_FAST_KEYBOARD_BACKEND=qmp-experimental to select it, and tune its temporary timing margins with GYM_ANYTHING_QEMU_QMP_KEY_HOLD_MS and GYM_ANYTHING_QEMU_QMP_KEY_GAP_MS.

The dispatch timing starts before runner.inject_action(...) and ends after the active fast input backend acknowledges the command sequence, so it includes socket flush and backend response time. It does not claim that the target app has consumed the event. For app-consumption evidence, inspect the generated Google Earth search readback, gedit semantic checks, and contact-sheet screenshots. Do not use raw pixel deltas as pass/fail evidence for input correctness:

PYTHONPATH=src GYM_ANYTHING_RUNNER=qemu \
GYM_ANYTHING_QEMU_FAST_IO_BACKEND=dbus \
python benchmarks/cua_world/tools/measure_input_latency.py \
  --env-dir benchmarks/cua_world/environments/google_earth_env \
  --task take_screenshot \
  --samples 50 \
  --warmup 10 \
  --use-cache \
  --cache-level post_task \
  --use-savevm \
  --output-json benchmarks/cua_world/environments/google_earth_env/artifacts/fast_input_latency_qmp_dbus.json \
  --evidence-dir benchmarks/cua_world/environments/google_earth_env/artifacts/fast_input_evidence

When validating QEMU device-topology changes, avoid reusing old savevm checkpoints. QEMU memory snapshots include device state, so an old snapshot can fail to load after adding or removing input devices. Run without --use-savevm or regenerate the checkpoint.

Docs Checks

For docs changes:

cd docs
npm run build

A Practical Rule