prompt
learner: "i want one coffee please"
semantic space · 5 sampled responses
cluster llmgroups near-duplicates
cluster a · |a|=3
cluster b · |b|=1
cluster c · |c|=1
resp 0
"try: 'i'd like a coffee, please.'"
resp 1
"close! say 'i'd like a coffee'"
resp 2
"good — 'i'd like a coffee.'"
resp 3
"could you ask more politely?"
resp 4
"what size would you like?"
reward ÷ cluster size
r' = r / |cluster| → crowded clusters get diluted
resp 0
÷3
0.8 → 0.27
0.27
resp 1
÷3
0.7 → 0.23
0.23
resp 2
÷3
0.6 → 0.20
0.20
resp 3
÷1
0.5 → 0.50
0.50
resp 4
÷1
0.4 → 0.40
0.40
lone responses keep full credit · the model is pushed off the popular mode and toward the empty regions of the space