harbor is a framework for packaging agentic tasks: datasets, agent harnesses, sandboxed execution, and verifiers that score the result. HarborEnv runs harbor trials as castform rollouts, so anything already expressed as a harbor task can be trained on directly.
it inverts the BaseEnv contract: there are no tools to register and no reward function to write. the harbor agent, sandbox, and verifier own the rollout loop, and HarborEnv preserves the verifier’s reward components (including rewardkit partial credit when available). most users configure HarborEnv directly instead of subclassing it.
configuration
configure HarborEnv with harbor’s native dataset, agent, sandbox, and verifier models:
from benchmax.envs.harbor import HarborEnv, HarborTrialTemplate, ModalCredentials
from harbor import (
DatasetConfig,
EnvironmentType,
TrialAgentConfig,
TrialEnvironmentConfig,
TrialVerifierConfig,
)
env = HarborEnv(
dataset=DatasetConfig(name="org/dataset", ref="latest"),
eval_ratio=0.1,
trial=HarborTrialTemplate(
agent=TrialAgentConfig(name="mini-swe-agent"),
environment=TrialEnvironmentConfig(type=EnvironmentType.MODAL),
verifier=TrialVerifierConfig(
env={"JUDGE_MODEL": "openai/judge-model"},
),
),
sandbox_credentials=ModalCredentials(
token_id="...",
token_secret="...",
),
)install benchmax[harbor] while authoring, and include the sandbox provider in the bundle’s pip_dependencies, e.g. harbor[modal]>=0.18,<0.19 or harbor[daytona]>=0.18,<0.19.
datasets and splits
harbor’s DatasetConfig resolves local directories, harbor packages, registries, and git repositories into a content-addressed snapshot: the same benchmax dataset type every environment returns. choose evaluation data based on how the results will be used:
| configuration | train | eval |
|---|---|---|
eval_dataset=... | complete primary dataset | complete explicit dataset |
eval_ratio=0.1 | remaining 90% | deterministic 10% holdout |
eval_ratio=0 | complete primary dataset | disabled |
an explicit eval_dataset takes precedence over eval_ratio. ratio splits use canonical task hashes, so source ordering and machine paths never change which split an example lands in.
execution and scoring
for each rollout, harbor runs the configured agent in its sandbox against one task; the model being trained is served to the agent as an OpenAI-compatible endpoint. when the trial finishes, the verifier (or rewardkit) scores it, and those components become the rollout’s reward.
benchmax owns rollout-group concurrency. max_concurrent_trials can additionally cap how many sandbox trials the environment runs at once.
sandbox_credentials become part of the environment bundle, so use dedicated, revocable credentials rather than personal ones.
agents and harnesses
the trial’s agent config selects the harness that drives the model: one of harbor’s built-ins, or any harness of your own.
whichever you use, disable compaction and summarization: training assumes a linear chat history, where each model call extends the previous messages. a harness that rewrites its context mid-rollout produces a transcript that can’t be trained on.
custom harnesses
a custom harness bundles its python source and adjacent resources, so its import path doesn’t depend on your checkout:
from pathlib import Path
from benchmax.envs.harbor import BundledAgentSource, BundledHarborAgent
from harbor import TrialAgentConfig
agent = BundledHarborAgent(
config=TrialAgentConfig(import_path="my_agent:MyAgent"),
source=BundledAgentSource.from_directory(
Path(__file__).parent,
files=("my_agent.py", "helpers.py", "prompts/system.txt"),
),
)only explicitly listed files are captured. use package-relative imports between captured modules, and locate adjacent resources from the agent module’s __file__.
validate and launch
the path is the same as any environment (dump_bundle, upload, launch), with one difference: omit train_dataset and eval_dataset at upload, since the environment resolves its splits at runtime. validation works unchanged; run it before every launch.
two complete harbor environments live in the benchmax repo: