Counting letters is how the training-data picker measured diversity

Flow

Where the export goes soft, and where I'd start

Start at the cutoff: chart real distances next to it.

1. 1Grab sessions⌄

   Takes the newest finished sessions, not the best ones.

2. 2Score them⌄

   A checklist that rewards looking careful, which isn't the same as being right.

3. 3Measure by spelling⌄

   Counts runs of letters, so a login fix and a button fix come out almost the same.

4. 4Skip anything too close⌄

   A default cutoff nobody checked. Too strict and it stops early with no reason given.

5. 5Write the file⌄

   Word for word, with nothing removing passwords, and long answers chopped off.

Picking a varied set of training examples is what the new fine-tuning export says it does. The part that decides what counts as varied is a letter counter.

It came in as one big drop with docs and no tests, and I think an agent wrote most of it. My stack records coding-agent sessions. The export turns them into training data for a small model add-on: grab finished sessions, score them, cut them into question and answer pairs, pick a few hundred good and different ones, and write a file. I read the code. I haven't run it on real data.

Why a login fix looks like a button fix

To compare two examples, the code counts short runs of letters and turns the counts into a list of numbers. Two examples that use similar letters look close. Two that use different letters look far apart.

That tells you how text is spelled, not what a session did. A note about a login fix and one about a button color come out almost the same, because they share most of their letters. I'd guess most real examples look nearly identical to this ruler.

Get the cutoff right

The picker starts with the best-scored example. Then it keeps adding whatever scores best on quality plus distance, and skips anything closer than a cutoff to a pick it already has.

If you build one of these, the cutoff is the thing to get right. Here it's a default number nobody checked against real distances. If it's too strict, the picker stops early and hands back fewer examples, with no reason given. It also favors examples that are far from everything, and those are often junk.

A cutoff taken from the data itself is a one-liner.

Take the tenth percentile of the distances between sampled pairs in the pool.

Careful-looking isn't correct

Quality is a checklist: did the session finish, did tests seem to pass, did the agent read before it edited. That rewards sessions that look careful, which isn't the same as being right. And the fetch step grabs the newest sessions, not the best.

Passwords go in as typed

Prompts and answers go into the file word for word. Nothing removes passwords or private data. Long answers get chopped off mid-sentence.

I'd start with a chart of real distances next to that cutoff. It's an afternoon of work, and it shows whether this ruler can tell sessions apart at all.