In my May 28 announcement, I asked: “You have Codex, I have Codex — why do we have different leaderboard scores?”

That was the question behind our Data Fest 2026 discussion about ML competitions in the era of LLM agents, with Mikhail Kamenshchikov, Alexander Ryzhkov, and Anton Legchenko. Research, coding, debugging: agents were being tried at every step. The less obvious part was telling useful progress from another pile of spent tokens.

Two identical agent cores connect to different investigation paths and distinct prediction structures.
The same agent core can sit inside different sequences of experiments and checks.

The question still interests me. Naming the agent says very little about the experiment around it. What problem was it given? What information could it use? Which result counted as progress? When a run produced a better score, what changed in the solution?

I find those questions more useful than keeping a tally of how much code an agent wrote. A generated feature, a repaired pipeline, and an explanation of a suspicious validation result are different kinds of help. They deserve different checks. Even a disappointing run can leave something useful if it makes the next experiment clearer.

The discussion recording starts around 5:13:00. This is a short note around the question from my announcement; the conversation itself is in the recording.