castform now lets you train vision models to see better.
with castform, you can now post-train vision-language models. this means being able to train with pictures, diagrams & screenshots as input. vision models have been somewhat under-appreciated, despite their wide range of applications. here’re some use-cases:
convert technical diagrams into structured output: circuit diagrams, floor plans → json. these are often highly domain specific and use conventions that general models haven’t quite seen, so post-training can be quite useful.
video/multi-frame reasoning. sports mechanics, assembly line steps, security footage. you can post-train model to judge what happened across frames → something general models aren;t great at.
the environment contract remains the same. there’re two ways you can feed images → as part of a prompt or as results from a tool call.
image inputs
an input message’s content can be a list of parts instead of a plain string. mix text and image_url freely - urls or data uris both work.
content = [
{"type": "image_url", "image_url": {"url": image_data_uri}},
{"type": "text", "text": problem},
]
prompt_messages = [
{"role": "system", "content": system_prompt},
{"role": "user", "content": content},
]tool outputs
run_tool can return content parts too, so a tool call can present a new image to the model mid-rollout. vision chat templates render the image_url part as real image tokens so the model actually sees it.
an obvious use is zooming.
async def run_tool(self, rollout_id, tool_name, **tool_args):
crop = self._crop(self._images[rollout_id], tool_args)
return [
{"type": "image_url", "image_url": {"url": to_data_uri(crop)}},
{"type": "text", "text": "zoomed view of the requested region."},
]example training run: dominant-color
code: https://github.com/castform-ai/benchmax/tree/main/examples/dominant-color
training run on castform: https://app.castform.com/train/11a57ad8-c09b-4834-94bf-38c7f74305db
in the task, the model needs to view three checkered tile images (in each, ~55% of each grid shares one dominant color, drawn from a fixed sixteen color palette). image 1 arrives in the prompt; to view images 2 and 3 the model needs to make asee_next_image tool call.
user: [image 1: noisy checkered tile, mostly red] This is image 1 of 3.
tool: see_next_image() → [image 2: mostly teal]
tool: see_next_image() → [image 3: mostly purple]
assistant: \boxed{red, teal, purple}reward is all-or-nothing on naming every dominant color in the order seen.