a beginner english learner asks how to order coffee. the tutor responds with three clauses, two idioms, and a phrasal verb. the learner is lost. frontier language models are excellent english speakers. turns out, that can be a problem when you’re teaching english.

in this post, we’ll share the work we did with elsa to fix that: taking real production conversations between learners and a tutoring product & post-training a model to meet beginner learners where they actually are (while also being half the cost & latency).

context

elsa is an ai-powered english learning app that helps learners improve speaking, pronunciation, and fluency. learners interact with a chatbot across grammar, vocab & role-play lessons. here’s a synthetic conversation to give you a flavor:

tutor: hi mateo! today we’ll practice vocabulary for talking about food and cooking. ready to start?

user: yes, ready.

tutor: great! can you make a sentence using the word “fresh”? it means recently made or picked, not old.

user: i buy fresh bread every morning.

tutor: nice work! your sentence uses “fresh” correctly to describe bread that was just made. you also showed a daily habit, which is a great way to use the word in context. keep it up!

tutor: let’s try another one. can you make a sentence using the word “spicy”? it means having a strong, hot flavor.

user: my mom cooks very spicy food.

tutor: excellent! you used “spicy” correctly to describe the kind of food your mom makes. that’s a clear, natural sentence. well done!

concretely, the failure modes were as follows:

  • long, multi-clause sentences, which can be confusing to a beginner learner
  • using vocabulary that’s out of range for a beginner
  • conversational clauses that felt natural to a more advanced speaker but didn’t give the learner enough scaffolding to respond

we set out to post-train a model to fix these issues.

the boring part: data curation

elsa is widely-used across the world, which meant we had access to a very large, rich corpus of production chatbot traces. this put us in a very good spot, because this allowed us to train a model that could capture all of the real-world noise and variability that you can never get from synthetic data.

we optimized for the following:

  • keeping only conversations that involved sustained interaction between user & ai tutor
  • elsa’s ai tutor handles quite a variety of tasks → vocabulary, grammar, free-form conversations. each of these categories also has many sub-tasks. the distribution of these was skewed towards the more popular features (e.g. freeform dialogue). we ensured to sample in a way that ensured meaningful representation across task types.
  • elsa was also always adding new tasks. we wanted to make sure that the post-trained model could generalize to new task/instruction types. we ensured there was a distinct split between task types across train & eval

after curating the conversations, we needed to convert them to training examples. specifically, we decomposed each trace into individual turns. each turn becomes one training example: the conversation history up to that point is the prompt. we keep the model’s response at that turn as a reference.

reward construction

for any rl post-training run, reward construction’s key to ensuring good model behavior. we built a set of rubrics across different tasks, grading things that elsa’s team cared about. this included:

  • whether corrections were pedagogically useful
  • conversational scaffolding i.e. did the tutor leave the learner an opening to respond
  • instruction following
  • hint quality

at the same time, we used heuristic rewards to measure how “beginner”-friendly a piece of text was. this included components like:

  • complexity of vocabulary
  • number of interlinked clauses in a tutor response

just using the length of a response (i.e. reward shorter sentences vs longer ones) was difficult to calibrate → the model would just collapse to trivial 5-6 word responses.

reward hacking

using the above rubrics alone wasn’t good enough → the model learned quickly to hack its judges. an example: the model started excessively using emojis and punctuation like “** —”.

just iteratively patching rubrics based on how the model learned to reward hack wasn’t efficient. after some trial and error, we found the following recipe to be the most robust.

the core change: instead of asking the judge for an absolute score on each response in isolation, we score by comparison. for each prompt we sample a group of responses and ask the judge to rank them against each other on one rubric at a time. the judge returns a tiered ranking (ties allowed), and we convert positions in that ranking into rewards in [0, 1] — best tier ≈ 1, worst tier ≈ 0, ties share the midpoint of the positions they cover.

ranking is harder to hack than absolute scoring. emoji-spam or em-dash padding only “wins” if it actually looks better than the alternatives in a head-to-head — when all the siblings carry the same trick, none of them gets credit for it. the judge is also doing an easier task (compare these n) than the one we kept asking before (assign a calibrated number), which makes its signal less noisy.

we then anchor the scale with the reference response — the original tutor turn we kept from the trace during data curation. on each step we drop that reference into the ranking as an extra, unlabeled response, so the judge can’t tell which one came from the human tutor and just ranks it alongside the model samples. once the ranking comes back, the reference’s tier becomes the anchor: responses ranked above it get rewards in (0.5, 1.0], those tied with it get 0.5, and anything ranked below it falls into [0, 0.3) — best-below-ref approaches 0.3 from beneath, worst-ranked goes to 0. that last band is a deliberate discontinuity — “worse than the human reference” is strictly worse than “as good as the human reference”, no matter how close. without this anchor, a judge graded leniently in absolute terms can let the whole group drift toward hacky outputs; tying rewards to the real tutor turn keeps the model honest.

diversity

we also noticed some mode collapse in the model outputs over training time. given a particular conversational history, the model was losing its ability to come up with diverse types of responses with a certain level of depth both in terms of content and style.

to get around this, we scale a response’s reward by how different it is from other responses sampled from the prompt. specifically, we use another llm to look at the group of responses and cluster them into similar groupings. we then divide a particular response’s reward by the size of the cluster it belongs to

results

the final post-trained model performs better on the dimensions that matter for elsa’s product. learners get responses with simple, relevant sentences matching their level. the responses have the kind of scaffolding that nudges learners towards more engaging, longer conversations.

and the model allows for all of this whilst also being quite small. this means lower costs and latency. a tutoring ai tool like elsa runs for many turns per user session and these token economics add up pretty fast.

your turn: the broader lesson is that when a frontier model is failing at a domain-specific task, the fix usually isn’t a bigger model. it’s a smaller model that has actually been shown what the task looks like.

you can do this in seconds. spin up your own model → app.castform.com