The cooperbench eval command evaluates agent runs by executing test suites in isolated sandboxes and computing success metrics.
Usage
cooperbench eval -n <experiment_name> [options]
Basic examples
Evaluate an experiment
cooperbench eval -n my-experiment
logs/my-experiment/ directory.
Force re-evaluation
cooperbench eval -n my-experiment --force
eval.json already exists.
Evaluate specific tasks
cooperbench eval -n my-experiment -t 8394
Parameters
Required
Experiment name to evaluate. Must match a directory in logs/.Example: my-experiment (evaluates logs/my-experiment/)
Task filtering
Use a predefined task subset.Example: lite
Filter by repository name.Example: llama_index_task
Filter by specific task ID.Example: 8394
Specific feature pair to evaluate, comma-separated.Example: 1,2
Execution
Number of parallel evaluations.Default: 10
Execution backend for running test suites.Options:
modal - Modal cloud platform (default)docker - Local Docker containersgcp - Google Cloud Platform Batch jobs
Force re-evaluation even if eval.json exists.
How evaluation works
For each task instance:
- Load agent patches - Reads
patch.diff from agent logs
- Create sandbox - Spins up isolated container with repository
- Apply patches - Applies agent changes to codebase
- Run tests - Executes test suite defined in task metadata
- Compute results - Records pass/fail for each test
- Save results - Writes
eval.json with test outcomes
Evaluation output
Results are saved to:
logs/{experiment_name}/task_{id}_feature_{f1}_{f2}/eval.json
Example eval.json
{
"task_id": 8394,
"features": [1, 2],
"tests_passed": 12,
"tests_failed": 2,
"tests_total": 14,
"success": false,
"test_results": [
{
"test_name": "test_feature_1",
"status": "passed",
"duration": 0.45
},
{
"test_name": "test_feature_2",
"status": "failed",
"error": "AssertionError: expected 42, got 41"
}
],
"duration_seconds": 125.3
}
Filtering examples
Evaluate specific subset
cooperbench eval -n exp-123 -s lite
Evaluate specific repository
cooperbench eval -n exp-123 -r dspy_task
Evaluate specific task
cooperbench eval -n exp-123 -t 8394
Evaluate specific feature pair
cooperbench eval -n exp-123 -t 8394 -f 1,2
Combine filters
cooperbench eval -n exp-123 -s lite -r llama_index_task
Backend examples
Evaluate on Modal (cloud)
cooperbench eval -n my-experiment --backend modal
Evaluate locally with Docker
cooperbench eval -n my-experiment --backend docker
Evaluate on GCP Batch
cooperbench eval -n my-experiment --backend gcp
cooperbench config gcp first.
High concurrency for cloud
cooperbench eval -n my-experiment -c 50 --backend modal
Low concurrency for local
cooperbench eval -n my-experiment -c 2 --backend docker
Skip auto-evaluation
By default, cooperbench run automatically evaluates after completion. To disable:
cooperbench run --no-auto-eval
cooperbench eval -n <experiment_name>
Incremental evaluation
Evaluation skips tasks that already have eval.json:
cooperbench eval -n my-experiment
cooperbench eval -n my-experiment --force
Aggregate results
After evaluation, you can aggregate results across all tasks:
ls logs/my-experiment/*/eval.json | xargs jq '.success' | grep -c true
import json
from pathlib import Path
exp_dir = Path("logs/my-experiment")
successes = 0
total = 0
for eval_file in exp_dir.glob("*/eval.json"):
with open(eval_file) as f:
result = json.load(f)
total += 1
if result["success"]:
successes += 1
print(f"Success rate: {successes}/{total} ({100*successes/total:.1f}%)")
