# same agents, different scores.

Notes from our Data Fest discussion on context, experiments, and the work that still falls to us.

Author: Vlad Kostoglodov. Language: en.
Archive date: 2026-05-28. Current version prepared: 2026-10-14.
Date basis: Date of my original Telegram announcement; this retrospective was expanded in October 2026.
Status: ready.
[Profile and technical work JSON](https://vk0st.com/facts.json): public profile, concrete work and source references.
[Human-readable article](https://vk0st.com/writing/same-agents-different-scores/).

I expected to keep writing code and reviewing my teammates' changes myself. Then the changes started arriving faster than I could read them. By the time I had a question about one version, the work had already moved on.

One of my first uses for an agent was to make sense of what other people's agents had produced. I needed to recover enough context to contribute. This was a slightly ridiculous way to discover that my familiar way of keeping up no longer worked.

In my May 28 announcement, I asked: **you have Codex, I have Codex — why do we have different leaderboard scores?** At Data Fest 2026, I discussed that question with Mikhail Kamenshchikov, Alexander Ryzhkov, and Anton Legchenko. At the time, I had been working with agents for about two weeks. I brought a fresh set of problems rather than a settled recipe.

![An amber illustration of four seated panelists; I am third from the left, speaking into a microphone.](https://vk0st.com/media/data-fest-panel-amber.webp)

Our Data Fest 2026 panel. An artistic interpretation of a real photograph, with the original underneath. Watch the discussion ↗

[Original panel photograph](https://vk0st.com/media/data-fest-panel-original.webp)

Looking back, I keep returning to the decisions around the agent. What should it investigate? What should it read? Which experiments deserve compute? What would convince us that a higher score represents a better solution?

## Experience goes into the setup

Alexander described putting his competition experience into the agent's starting context. Validation, feature engineering, target encoding: you can explain known traps before the agent spends a run rediscovering them.

That is already one answer to the question in the title. Two people can use the same model and give it very different starting points. One has a vague task description. The other has a validation protocol, a useful baseline, and a list of approaches that have already failed.

My own instinct was to make the pipeline more like Lego: separate parts that an agent can investigate and replace. Alongside them, keep notes about experiments, metrics, and what we learned. I wanted to be able to pick up the work without reconstructing the whole conversation.

Consider a recommendation pipeline with candidate generation followed by ranking. Suppose the question is whether a new candidate source finds useful items that the existing sources miss. That is an experiment you can describe: keep the validation split and ranker fixed, change the candidate pool, and inspect both candidate recall and the final ranking metric. If you also change the split, features, and ranker, the score might improve, but the original question remains unanswered.

This is an example of the kind of boundary I mean, rather than a recipe from a particular competition. The agent needs somewhere to make a change, and I need a way to understand its effect. A note that says which component changed, against which baseline, is much easier to revisit than a chat full of promising ideas.

## A bad feature can leave a useful question

Anton brought up an example involving data leakage in a public competition solution. Investigating why the invalid feature helped led his team to reconsider the task and the training setup.

I liked that distinction. Removing a leaking feature is necessary. Understanding why the model depended on it can still be useful. It might expose a weakness in the training distribution or a part of the task the baseline handles badly. Those are questions to investigate with a valid experiment; the leaking score itself cannot establish an improvement.

This also changes what I would ask an agent to do. “Remove the leak” gives it a repair job. “Explain what the model was getting from this feature, then suggest a valid way to test that explanation” gives it an investigation. The second task needs a person who notices that there may be something worth learning from the failure.

## Choosing what the agent gets to read

Anton also described using previous competition winners' talks and slides as material for agents to work through. There is a useful human decision before any summarisation happens: choosing solutions worth studying.

A winning solution can show where a team spent its effort, which candidates or features mattered, and what did not survive into the final pipeline. It gives the agent something more concrete than a request for general advice about a task.

I talked about a similar approach to recent papers and implementations. The interesting question for me was where an idea might fit in the current pipeline. Which block could it replace? Does it use information we actually have? Is there a reasonably small experiment that would tell us whether to continue?

Reading becomes much more useful when it has a destination. A new model can be elegant and irrelevant to the bottleneck in front of you. An old technique can be exactly what the pipeline needs.

Mikhail raised the question of novelty, and Anton distinguished an idea that is new to the person trying it from one that is new to the field. That feels especially relevant when an agent produces a convincing explanation. Before getting excited, it helps to find out what you have actually rediscovered.

## Compute still changes the race

The panel did not settle on one explanation for different results. Alexander discussed the experiment budget and running work in parallel. Anton emphasised experience in choosing which experiments to run. Mikhail pointed out that faster implementation does not make training disappear.

Those considerations coexist. More compute lets you test more hypotheses. Experience helps you spend it on better ones. Faster code generation can make the training queue longer while leaving the same person responsible for interpreting its results.

I also talked about competition formats with practical resource and inference constraints. Those appeal to me because they connect to the recommendation systems I like working on. The quality of a model comes with a cost: memory, training time, and the work required to serve its predictions. A leaderboard that reflects some of that makes the engineering decisions more interesting.

## Keeping up with the experiments

I started with a very ordinary problem: there was more code than I could comfortably read. An agent helped recover the context. But that raised another question: what context should we leave behind so that recovering it is easier next time?

For me, the useful record is the experiment. What question led to it, what changed, which comparison we trust, and what the result makes us want to try next. Even a failed run can earn its place in that record. It can rule out a tempting direction or reveal that the validation setup needs work.

I do not have a formula for turning access to an agent into a competition result. The discussion made me more interested in the work around it: selecting evidence, drawing boundaries around experiments, and making results understandable enough to act on.

If the next batch of code arrives before I have finished reading this one, I still want to be able to explain why we ran the experiment and what we learned.

## The discussion

The [full recording](https://m.vkvideo.ru/video-152990965_456240297?list=ln-R3aw78rMCkP7NtPNzW) includes the panel at approximately **05:13:00–06:16:48**. These are positions in the full event video:

| Position | Topic |
| ---: | --- |
| 05:19:16 | Keeping up with code produced by agents |
| 05:21:11 | Alexander on putting experience into the setup |
| 05:25:16 | Anton's example of investigating a leaking feature |
| 05:31:05 | Modular pipelines, notes, and experiment records |
| 05:50:11 | Anton on compute and learning from previous solutions |
| 05:56:12 | Competition formats with practical constraints |