import data
castform auto-generates training data from the data you already have - existing corpuses, agent traces, and more.
from benchmax.rag.qa_generation.pipeline import Pipeline
from benchmax.rag.qa_generation.pipeline_config import PipelineConfig, PlatformConfig
from benchmax.rag.corpus.postgres.source import PostgresChunkSource
# supports turbopuffer, pinecone, chroma, postgres
source = PostgresChunkSource(corpus_name="GitLab Handbook")
source.populate_from_existing_corpus_name("GitLab Handbook")
cfg = PipelineConfig(
platform=PlatformConfig(api_key="sk_..."),
)
rag_dataset = Pipeline(cfg, source_factory=lambda _: source).run()from benchmax.traces import TracesPipeline
# supports braintrust, langfuse, langsmith
pipeline = TracesPipeline(
traces=traces,
min_completion_chars=40,
tool_relay=0.8,
dedup=0.9,
target_examples=1000,
output_dir="outputs/agentic",
)
result = pipeline.run()| prompt | ground truth |
|---|---|
| Which internal Slack channel is for questions and requests for the
CEO? |
#ceo — a public channel for questions and requests for the CEO [Source:
Communicating with the Field].
|
| When must a company-issued laptop be wiped before it's handed off
or after a security incident? |
Whenever it's lost or compromised during a security incident, and before
it's donated, returned, or transferred to another team member [Source:
Laptop Remote Management and Monitoring].
|
| What should I use to capture meeting notes and keep absent
teammates informed? |
Google Docs — a real-time collaborative tool to take notes, share
decisions, and keep all stakeholders informed [Source: How do you
collaborate and whiteboard remotely?].
|
| When scheduling a recurring leadership call, what sign-in path and
calendar availability setting avoid blocking customer meetings? |
Sign in to Zoom via SSO (Okta), then set the calendar event's
availability to Free so it won't reserve the time [Source: Support
Leadership Sync].
|
| prompt | trajectory |
|---|---|
| How can I run the individual steps of a CI job on my own machine,
using the pipeline definition stored locally? |
search_corpus("run CI job steps locally pipeline definition") →
read_doc("Step Runner for executing GitLab Steps") → respond("use Step
Runner to execute a job's steps locally…") [Source: Step Runner for
executing GitLab Steps]
|
| After a performance test run, how do I remove all the sample data
it created, and are the two cleanup methods treated differently? |
search_corpus("performance test cleanup delete generated data") →
read_doc("GitLab Performance Tool (GPT) Quick Start") → respond("run the
GPT cleanup step; Method 1 and Method 2 differ…") [Source: GitLab
Performance Tool (GPT) Quick Start]
|
| How should I request help from the team that produces technical
promotional materials and product demos? |
search_corpus("technical marketing request demos promotional materials")
→ read_doc("Technical Marketing") → respond("follow the Technical
Marketing content request process…") [Source: Technical Marketing]
|
| What should I review to get better at uncovering a prospect's needs
and clearly explaining our differentiation in customer
conversations? |
search_corpus("discovery methodology prospect needs differentiation") →
read_doc("Effective Discovery") → respond("leverage the listed discovery
resources and GitLab-tailored methodology…") [Source: Effective
Discovery]
|
define tools & rewards
plug in the same tools your agent uses in production, and define quality with composable reward functions
from benchmax.envs.base_env import BaseEnv
class SearchEnv(BaseEnv):
system_prompt = """you are a retrieval agent..."""
async def run_tool(self, name, **args):
if name == "search_corpus":
return self.corpus.search(args)
async def compute_reward(
self, rollout_id, completion, ground_truth
):
return {
"correctness": await compute_correctness(),
"conciseness": await compute_conciseness(),
"citation": await compute_citation(),
"tool_call_efficiency": await compute_tool_call_efficiency(),
}search_corpus monitor training
watch reward curves climb in real time and inspect qualitative model outputs. catch reward hacking and regressions early.
beat the frontier
achieve frontier model performance at a fraction of the cost with a model you own and control.
questions?
do i need ml or rl expertise?
no. castform is designed for engineers and researchers alike: it works out of the box with no ml expertise, and exposes advanced controls for those who want it. you bring your data and define what success looks like, and we handle the rl algorithms, environment scaffolding, distributed training, and infrastructure.
what use cases do you support?
virtually anything rl fine-tuning. if you can define verifiable success metrics for your task, we provide the algorithms and infrastructure to fine-tune a model to optimize for those.
you just need to set up the two things the trainer needs: an environment (what the model has access to, and the reward signals that define how the model is scored) and a dataset (the examples it trains on).
to make it easier, we provide automated dataset generation and environments for training rag agents and fine-tuning on production agent traces.
how do i get access?
you can start training right away by signing up at app.castform.dev. it is fully self-service and pay-as-you-go, with enough free credits for new users to trial their first training run.
who owns my data and the trained model?
you do. we train readily available open-source models on your data, and you can export the trained weights at any time. apply them and deploy the new model wherever you like. any data you upload for training is stored securely and used only for your training runs, and can be deleted at any time.
how does pricing work?
castform is pay-as-you-go: you only pay for the compute you use during training runs. there are no seats, no monthly minimums, and no lock-in. new users receive free credits to trial their first run.
see the full breakdown on our pricing page.
how can i learn more?
check out our docs for guides, examples, and the python sdk reference.
prefer to talk it through? reach out at castie@castform.com and we'll help you figure out whether rl fine-tuning is a fit for your use case.