tldr; we’ve added support for harbor! you can train on your harbor tasks with castform in just a few lines of code.

here’s an example with the harvey lab dataset.

# example harvey environment
class HarveyLabHarborEnv(HarborEnv):
    """Harvey's latest LAB dataset on Modal; the agent harness defaults to Harvey's own."""

    def __init__(
        self,
        *,
        sandbox_credentials: ModalCredentials,
        harbor_verifier_env_var: Mapping[str, str],
    ) -> None:
    
    super().__init__(
          dataset=DatasetConfig(name="harveyai/lab", ref="latest"),
          trial=HarborTrialTemplate(
              agent=harvey_harness(),
              environment=TrialEnvironmentConfig(
                  type=EnvironmentType.MODAL,
                  kwargs=environment_kwargs,
              ),
              verifier=TrialVerifierConfig(env=harbor_verifier_env_var),
          ),
          sandbox_credentials=sandbox_credentials,
      )

check out more e2e examples here.

what this means

with harbor, the agent ecosystem has a common format for defining environments, harnesses & reward functions. with this integration, these tasks can directly be used as training environments with castform. this allows for the following:

train on your harness, unchanged

most rl stacks make you reimplement your agent with their environment abstraction before you can train on it (which means there is drift between the thing you train on vs what you ship). with the harbor integration, you can point the trainer at whatever harness you run in prod (e.g. terminus, claude code, etc.)

in ~20 lines

all you need to do is point at the taskset, declare your harness & sandbox and you are good to go. everything else (e.g. sandbox tunnels, tokenizer edge cases, token-in-token-out, rollout tracking) is handled for you by us.

how it works

harbor’s flexibility is also what makes it hard to train on. while harbor has integrations with other trainers, none of them fulfill both these requirements:

  1. support training with any arbitrary harness

  2. maintain TITO* for training stability

    • TITO a.k.a. token-in-token-out means that the trainer is trained on the exact tokens generated by the model in multi-turn rollouts. this principle is usually violated when tokens are decoded and then re-encoded, because encode(decode(x)) ≠ x. read here for more details. ensuring a perfect match is crucial for training stability.

tinker + harbor supports TITO but is limited to the terminus harness. skyrl + harbor works with any harness but does not support TITO. this makes sense since supporting both is tricky: TITO requires the harness to operate with tokens instead of the standard text chat messages that most harnesses use.

to work around this constraint, we moved the token tracking task to a gateway, similar to the implementation in rllm & nvidia’s polar trainer. this TITO gateway serves as a proxy, communicating with the harness with the standard oai chat message format while using tokens directly with the trainer.

harness                    gateway                    trainer
  |<========= text ==========>|<======== tokens =========>|
  |                           |                           |
  |----- chat request ------->|                           |
  |   (full text history +    |                           |
  |    new tool result)       |                           |
  |                           |                           |
  |                     match text history                |
  |                       against stored                  |
  |                        token cache                    |
  |                           |                           |
  |                    encode only new msgs               |
  |                      [text -> tokens]                 |
  |                           |                           |
  |                           |----- prompt tokens ------>|
  |                           |                           |
  |                           |                           |
  |<------------- streamed, decoded response -------------|
  |   (text/tool calls)       |    incremental decode     |
  |   (as tokens arrive)      |                           |
  |                           |                           |
  |                     [stream ends]                     |
  |                   store text <-> token map            |
  |                           |                           |
  |  (appends its tool        |                           |
  |   result, sends next      |                           |
  |   turn as new request)    |                           |

the main logic of our gateway was implemented by referencing rllm’s gateway, but last-mile customizations were still tricky. some examples:

  • enabling agents in a remote sandbox to reach our trainer by adding publicly accessible & secure tunnels
  • adding streaming / keep-alive heartbeat to avoid harness timeout
    • streaming of text and tool calls meant that we must incrementally decode tokens while working around quirks and edge cases of each model’s output
    • e.g. gemma does not follow the standard tool call format - you must retain the sentinel tokens for parsing to work, yet you have to eventually strip these tokens before returning the chat responses with streaming enabled
  • multimodal
    • the tokenizer needs to pad the token input with right-sized placeholders so when the images are converted into actual tokens by the tokenizer, they can be substituted correctly.

for simplicity, our harbor integration currently supports linear agent trajectories with no summarization / compaction. support for non-linear trajectories is coming soon!

start training your harbor task on castform today!