qa generation | castform docs

rl training a rag agent requires a dataset of question-answer pairs grounded in your corpus. we provide an automated pipeline for generating this dataset. you can point the pipeline at an indexed corpus and it produces a train/eval split of questions, their answers, and the chunks that support each answer.

the output is a train/eval split in jsonl, ready to launch a training run.

quickstart

you can generate qa pairs with the default pipeline settings, just provide a corpus and target sample count. expect the pipeline to generate around ~5-10 questions per minute.

from castform.platform import ensure_session
from castform.rag.qa_generation.pipeline_config import PipelineConfig, PlatformConfig, CorpusConfig, TargetsConfig
from castform.rag.qa_generation.pipeline import Pipeline

ensure_session()

cfg = PipelineConfig(
    platform=PlatformConfig(),
    corpus=CorpusConfig(corpus_name="my-docs", corpus_id="..."),
    targets=TargetsConfig(total_samples=200),
)

pipeline = Pipeline(cfg)
result = pipeline.run()

train_data = result["train_dataset"]
eval_data = result["eval_dataset"]

with the configured castform endpoints, corpus and model requests use the active local session. to use an external model provider instead, set llm_base_url and its explicit llm_api_key on PlatformConfig; platform credentials are never reused as model credentials.

if you haven’t chunked anything yet, you can pass docs_path instead of corpus_id to chunk and index documents in one step. the run is resumable: rerun with the same output directory and it picks up from the last checkpoint.

how it works

the pipeline runs four stages:

profile: it first samples your corpus to learn the domain, summarizing the content and extracting the entities and jargon that recur in it. this grounds the generated questions in your actual terminology (it uses your corpus description and example queries here too, if you provide them).
generate: it samples chunks from your corpus and uses an llm to write questions answerable only from those chunks, each paired with its answer and the supporting chunks. you control the mix of question types based on the types of questions you want the model to be able to answer, from simple lookups to multi-hop reasoning across documents.
filter: every pair is quality-checked. pairs whose answer isn’t supported by the chunks, or that a plain keyword search already answers (so there’s nothing for the model to learn), are dropped or regenerated with feedback.
transform: each question is annotated with the query style it matches (keyword, natural language, expert shorthand), and the train/eval split stratifies across styles and question types.

basic customization

corpus context

telling the pipeline about your domain can improve question quality. a description and a few example_queries are used during profiling to summarize your corpus and understand your terminology.

field	default	description
`description`	`""`	plain-text description of your corpus
`example_queries`	`[]`	example search queries users would ask

question mix

targets.primary_type_distribution controls how many questions of each type the pipeline generates. weight it toward the kinds of questions your model needs to answer. the weights sum to 1.

type	default	description
`lookup`	`0.30`	single-chunk fact lookup
`multi_hop`	`0.70`	questions spanning linked chunks

lookup is the cheapest (one llm call); multi_hop requires chunk linking first, so it costs more. targets.reasoning_mode_distribution separately mixes reasoning styles (factual 0.45, temporal 0.15, inference 0.30, sequential 0.10).

question styles

questions are generated across the styles real users type:

style	example
`keyword`	`k8s pod memory limits`
`natural`	`how do I set memory limits on kubernetes pods?`
`expert`	`configure resource requests and limits in pod spec`

the mix is fixed per question type: lookup questions lean keyword (35% keyword / 45% natural / 20% expert) while multi_hop questions lean natural language (10% / 60% / 30%). each row records its style in metadata, and the train/eval split stratifies on it.

output

control the train/eval split and where files land with split and output.

field	default	description
`split.train_ratio`	`0.8`	fraction of data for training
`split.stratify_by`	`["qa_type", "style"]`	balanced splits across these columns
`output.dir`	`"outputs/pipeline"`	output directory
`output.train_jsonl`	`"train.jsonl"`	training data filename
`output.eval_jsonl`	`"eval.jsonl"`	eval data filename

putting it together

a customized run combines the settings above into one config:

from castform.rag.qa_generation.pipeline_config import (
    PipelineConfig, PlatformConfig, CorpusConfig, CorpusContextConfig,
    TargetsConfig, SplitConfig, OutputConfig,
)
from castform.rag.qa_generation.pipeline import Pipeline

cfg = PipelineConfig(
    platform=PlatformConfig(),
    corpus=CorpusConfig(corpus_name="my-docs", docs_path="./my-docs"),
    # corpus context: ground questions in your domain
    corpus_context=CorpusContextConfig(
        description="internal engineering docs for acme corp",
        example_queries=["how do I configure the auth middleware?"],
    ),

    # question mix: weight toward what your model needs to answer
    targets=TargetsConfig(
        total_samples=200,
        primary_type_distribution={
            "lookup": 0.4,
            "multi_hop": 0.6,
        },
    ),

    # output: train/eval split and where files land
    split=SplitConfig(train_ratio=0.8),
    output=OutputConfig(dir="outputs/my-docs"),
)

result = Pipeline(cfg).run()
train_data, eval_data = result["train_dataset"], result["eval_dataset"]

see launching a training run to start a training job with the result.

advanced customization

chunk linkers

linkers find related chunks for multi-hop questions. select one via linker.type.

metadata (default) links chunks that share file structure and header metadata, gated by coherence and similarity checks. no LLM calls.

field	default	description
`max_candidates`	`10`	candidate chunks considered per seed
`max_secondaries`	`3`	max related chunks linked per seed
`min_chunk_chars`	`400`	skip chunks shorter than this
`filter_same_file`	`true`	drop candidates from the seed chunk’s own file
`min_coherence`	`0.15`	drop candidate links below this coherence

wiki links via the corpus’s entity graph: candidates share entities with the seed chunk, scored for coherence and complementary information. use it when your corpus has entity-graph preprocessing.

generators

llm_direct (default) makes a direct LLM call per QA pair. fast.

field	default	description
`model`	`"gpt-5.4"`	generation model
`max_concurrent`	`8`	parallel generation requests
`batch_enabled`	`true`	enable batch processing

tips:

chunk size: 1024-2048 chars works well. too small gives low context, too big gives noisy questions.
spread seeds: more chunks with fewer questions each beats fewer chunks with many questions.

filters run in sequence, cheapest to most expensive. each marks items as passed, rejected, or needs_refinement; items that need refinement get regenerated with feedback.

1. deterministic guards catch format and length issues: empty answers, single-word questions, missing references.

field	default	description
`min_question_chars`	`12`	minimum question length
`min_answer_chars`	`50`	minimum answer length
`min_reference_chunks`	`1`	minimum reference chunks

2. quality_gate rejects fragments, structural questions, and guide-pointer questions with cheap heuristics, and flags thin answers for refinement. no LLM calls.

3. retrieval_too_easy_llm checks if naive BM25 can already find the answer. if so, the question won’t teach the model anything via RL. marks as needs_refinement rather than rejecting.

field	default	description
`overlap_threshold`	`0.5`	chunk overlap threshold for flagging
`too_easy_confidence_threshold`	`0.85`	confidence above this = too easy

4. grounding_llm uses an LLM judge to check whether the answer is actually supported by the reference chunks. the most important filter.

5. hop_count_validity uses an LLM judge to check that a multi-hop question genuinely needs all of its linked chunks (leave-one-out by default).

refinement loop. failed items get regenerated with the failure reason as feedback:

filters run on all QA pairs
needs_refinement items get regenerated with feedback
regenerated items go through filters again
repeat until all pass or budget runs out

field	default	description
`max_refinements_per_item`	`2`	max fix attempts per pair
`max_same_seed_attempts_before_reanchor`	`2`	failures before switching to a different seed chunk
`max_rounds`	`4`	max filter-refine cycles
`max_total_regenerations`	`total_samples * 2`	global budget cap

if a seed chunk keeps producing bad questions, reanchoring to a different chunk is more productive than retrying.

checkpointing. results are saved after each filter round. resume: true (the default) picks up from the last completed round on restart.

full config example

every option, with its default. you only need the handful shown in the quickstart; this is the exhaustive reference.

random_seed: 42
verbose: true
resume: true

platform:
    api_key: 'sk_...' # corpus/control-plane credential
    base_url: 'https://api.castform.com'
    llm_api_key: 'sk_...' # explicit model credential; may be different
    llm_base_url: 'https://llm.castform.com/v1'

corpus:
    docs_path: './my-docs'
    corpus_name: 'my-docs'
    min_chunk_chars: 400

corpus_context:
    enabled: true
    description: 'internal engineering documentation for acme corp'
    example_queries:
        - 'how do I configure the auth middleware?'
        - "what's the retry policy for failed jobs?"
    num_top_level_samples: 4
    num_random_samples: 4
    generate_entity_patterns: true

targets:
    total_samples: 200
    primary_type_distribution:
        lookup: 0.30
        multi_hop: 0.70
    reasoning_mode_distribution:
        factual: 0.45
        temporal: 0.15
        inference: 0.30
        sequential: 0.10

linker:
    type: 'metadata'
    metadata:
        max_candidates: 10
        max_secondaries: 3
        min_chunk_chars: 400
        filter_same_file: true
        min_coherence: 0.15

generation:
    mode: 'llm_direct'
    llm_direct:
        model: 'gpt-5.4'
        max_completion_tokens: 4096
        max_concurrent: 8
        batch_enabled: true

filtering:
    deterministic_guards:
        enabled: true
        min_question_chars: 12
        min_answer_chars: 50
        min_reference_chunks: 1
    filters:
        - 'quality_gate'
        - 'retrieval_too_easy_llm'
        - 'grounding_llm'
        - 'hop_count_validity'
    grounding_llm:
        judge_model: 'gpt-5.4'
    retrieval_llm:
        judge_model: 'gpt-5.4'
        overlap_threshold: 0.5
        too_easy_confidence_threshold: 0.85

refinement:
    enabled: true
    max_refinements_per_item: 2
    max_same_seed_attempts_before_reanchor: 2
    max_rounds: 4

split:
    train_ratio: 0.8
    stratify_by: ['qa_type', 'style']
    seed: 42

output:
    dir: 'outputs/pipeline'
    train_jsonl: 'train.jsonl'
    eval_jsonl: 'eval.jsonl'