DataSmith: Automating Data Research

Figure 1. DataSmith performs the entire data research loop end to end. Starting from identifying relevant datasets, it curates and augments training data, decontaminates it against evaluation sets, trains and evaluates models, diagnoses failures, and turns those insights into new hypotheses for the next iteration.

Before we get into how DataSmith works, we want to show you how well it does. After that we will walk you through the benchmark, and then the harness itself. On a benchmark that evaluates how well an autonomous agent can post-train a base model by changing only its training data, LLMs running on top of DataSmith consistently outperform the same models in a general coding harness.

Figure 2. Every Harness LLM we tried did better inside DataSmith than inside a standard coding harness. Each pole represents the average performance over four benchmarks (AIME2025, GSM8K, HumanEval, BFCL) for three base models, after post-training for 367M tokens. Blue poles are Claude Code with access to Datology's curation stack; orange poles are the same setup with DataSmith as the harness. The light gray pole on the left is where the base models start before any training, and the dark gray pole on the right is where the official instruct versions of those models, trained with orders of magnitude more compute, sit.

Glossary

There are a lot of terms that we will be constantly using in this blog, and below is a shared vocabulary as we walk through this project.

1. CurateBench: an autoresearch benchmark for data curation

We are interested in whether autoresearch agents can train better models by iteratively improving only the training data. To study this, we set up an environment in which the base model, optimizer, training configuration, and evaluation are fixed, while the agent can only change and submit the training data.

1.1 Why do we need a new benchmark?

We initially planned to use PostTrainBench (PTB), which benchmarks how well agents can autonomously improve base model capabilities through post-training. However, three parts of its design make it unsuitable for our goal.

1. PTB trains a different model for each benchmark, so each capability is optimized in isolation. In practice, the post-training stage trains a single model on a unified data mixture that simultaneously advances the performance on multiple benchmarks. This design choice misses important aspects of data mixing, and cross-domain performance transfer.
2. PTB starts from an empty sandbox, so agents waste a large proportion of the GPU budget building infrastructure rather than training models. In many cases, more than 25% of GPU time is spent on setup and overhead. Instead, we want to measure agents’ research ability given mature data infrastructure.
3. PTB gives agents an open intervention space allowing the model to change everything from the training algorithm (SFT, GRPO, on-policy distillation), the training configuration, the compute budget, to the data. This means that the training algorithms, FLOPs, and recipes across different sessions are often not comparable, making it difficult to attribute a strong run to data curation alone.

1.2 CurateBench

CurateBench takes a huge amount of inspiration from PostTrainBench, but isolates the effect of data by fixing the optimizer, training recipe, then evaluating a single base model across multiple benchmarks. The goal is to reward data choices that make a model broadly better rather than choices that optimize for an individual benchmark.

We use the same three base models and four evaluations as PostTrainBench Lite: Qwen3-1.7B-Base, Qwen3-4B-Base, and SmolLM3-3B-Base, evaluated on AIME2025, GSM8K, HumanEval, and BFCL.

Training steps: 350
Global batch size: 64
Sequence length: 16,384
Tokens seen: 367M
Learning rate: 1e-5
Compute: 8×H100

The agent is limited to only changing the SFT training data. It is allowed to search the internet to download any relevant new datasets or generate synthetic data via API access to Qwen3-30B-A3B-Instruct-2507 or a previously trained model. In any case, training data assets must be decontaminated against the eval questions before use. In each autoresearch session, we allow up to 20 training job submissions and do not limit the total time for research.

1.3 Datology’s Curation Stack Establishes a Strong Lower Bound for Autonomous Data Research

Good infrastructure can be one of the biggest accelerants of good research. Before we introduce DataSmith, we ask what a general coding agent can achieve with Datology's curation stack. To this end, we run Claude Code with each of the four Harness LLMs, giving it access to our curation environment. This will later serve as the baseline against which we measure the value of DataSmith's research orchestration.

For a public reference point, we compare the baseline against PostTrainBench v1.0. Across the two matched configurations, the CurateBench baseline averages 4.9 percentage points above PTB, despite having a significantly restricted action space. These restrictions prevent using RL, changing training budgets, or hillclimbing on a single evaluation while ignoring regressions elsewhere. These results simultaneously validate CurateBench and show that Datology's curation stack already helps general coding agents achieve stronger results than they do in PTB's environment, before DataSmith adds any research orchestration.¹1.We use v1.0 because it supports external API-based synthetic generation, making its setup closer to the CurateBench environment.

Figure 3. Average score across AIME2025, GSM8K, HumanEval, and BFCL for PTB v1 and CurateBench baseline. The comparison is shown for GLM 5.2 and Opus 4.8 (Sonnet 5 & Opus 5 aren’t reported in PTB v1). The untrained base-model and instruct-model references show the starting point and the performance of the corresponding post-trained models for many more orders of magnitude compute, respectively.

2. DataSmith is a strong research harness for data curation

To isolate the effect of the harness, we hold the curation environment fixed and change only the harness. For each Base Model, we compare matched runs of the same Harness LLM in Claude Code versus DataSmith.

DataSmith outperforms Claude Code in all twelve matched comparisons, by 5.1 percentage points on average. The improvement is consistent across all three Base Models and all four Harness LLMs.²2.SmolLM3-3B-Base: 11.2T pretrain + 100B long-context mid-train. SmolLM3-3B: +140B reasoning mid-train (35B × 4) and +8B SFT (1.8B × 4). DataSmith’s 734M is ~200× less than that ~148B. From the SmolLM3 Tech Report.

On SmolLM3-3B, the strongest DataSmith run (with Opus 5) beats the released instruct model by 3 percentage points on the reported four-evaluation suite. It starts from the same base checkpoint and uses 734M post-training tokens, whereas the instruct model is trained with 148B post-training tokens, roughly 200× fewer.

Figure 4. Matched CurateBench runs comparing Claude Code and DataSmith. The base model, LLM, curation environment, training setup, submission budget, and evaluations are held fixed.

It is also noteworthy that the choice of harness can also outweigh the choice of the Harness LLM. Opus 5 is the strongest and most expensive Harness LLM in our comparison, yet GLM 5.2 with DataSmith outperforms Opus 5 with Claude Code on all three base models. This is the second time we see the same pattern: what surrounds the Harness LLM matters more than which model it is. First, Datology’s curation environment provides the data-action space, and DataSmith helps the agent make better use of it.

3. DataSmith

DataSmith is an AutoResearch harness for data curation that can run over long horizons. It builds on top of Claude Code and Datology’s curation environment. Claude Code provides tool use, context management, and subagents, while Datology’s stack exposes training, evaluation, and curation jobs behind simple commands.

DataSmith adds the ingredients that a long-running research process needs, like a persistent loop agent, specialist research agent roles, data curation skills, and a memory management system that tracks the state of every experiment.

3.1 A Multi-Agent Research Team

First, let us dive into the roles of various subagents in the multi-agent research team that DataSmith orchestrates. At the core, the loop agent coordinates the run and stays persistent till the goal is achieved. It observes events, dispatches tasks, executes decisions, and polls long-running jobs. It is supported by four specialist subagent roles, which are implemented as skills:

This separation lets the strategist maintain a research direction while individual investigations run in parallel. New evidence can arrive from a completed experiment while other agents are still preparing data for the next one.

3.2 Skills and Tools as the Interface to the Curation Environment

DataSmith's research roles interact with Datology's curation environment through skills. Each skill packages the context and tools required for a particular operation, giving the agent a consistent interface to the curation workflows it needs during research.

These capabilities are exposed as tools to the agent for onboarding, profiling, filtering, deduplication, and decontamination, and as skills for synthetic data generation, training, and evaluation. With these tools and skills, agents can directly operate our infrastructure to curate data rather than rebuild from scratch for every experiment.

For example, Autosynth agent is initiated with a data synthesis blueprint designed by Strategist agent. Autosynth agent can then assemble different tools to build an end-to-end synthetic loop: data generation, answer verification, failure repair, and eval decontamination. After enough data is synthesized, it will register as a trainable asset and send back to the strategist to determine how to use for the experiment.

3.3 Long-Horizon AutoResearch Loop

To enable long-horizon research, most existing works (AI Scientist, Automated W2S Researcher, ScientistOne) use a pre-defined loop in which subagents are called in a fixed sequence. DataSmith instead uses a loop agent to orchestrate which subagents to call and what information to put into their context, following the principle of the Bitter Lesson by relying less on a hard-coded workflow.

A typical run follows:

But the loop can deviate from this workflow depending on observation. After each training run, scores update the leaderboard, and the loop agent decides how to proceed. For example, if the current strategist begins to stagnate, the loop agent can spawn another strategist with a different context, while concurrently launching diagnosis agents to inspect the eval failures.

We use an experiment tree to track this history: each training job submission corresponds to a node containing its data recipe and evaluation results, while branches represent alternative research directions. This gives the agent a persistent record of what research ideas have been tried and how each experiment relates to previous runs.

Training and data preparation can proceed asynchronously. The agent can submit up to three concurrent training jobs while preparing new datasets for future runs. As soon as a training slot becomes available, it can launch another job if the required data is ready. The loop agent polls completed training jobs and feeds their evaluation results back to the subagents to decide the next move.

Together, the orchestration of specialized subagents, the experiment tree, and the research loop enable long-horizon research beyond the constraints of a single context window.³3.DataSmith turns data curation into a long-horizon research loop, where specialized subagents produce data and research insights, and every experiment builds on what came before.

4. Research Taste in DataSmith Agent

A common failure of general coding agents tasked with data research is that they are too eager to launch experiments on the cluster: launch a training run, read the scores, make a micro adjustment to the mixture, and launch again. The AutoResearch loop is rather shallow: the agent keeps locally optimizing the recipe it started with instead of digging into the evaluation outputs, questioning its own approach, and pivoting when performance plateaus.

DataSmith has no explicit research taste built into it. Even so, it spends 2.6× more reasoning tokens than a general coding agent, holding the harness and LLM fixed. DataSmith reasons about how to design experiments that test its research hypotheses and can self-critique its current research bets. In other words, it “runs fewer experiments and thinks about them more.”

4.1 DataSmith explores wider

To understand how agents research between training jobs, we measured five metrics across all sessions: how many tokens they spend reasoning, how many new datasets they try, how often they delegate to subagents, how much research infrastructure they build, and how many distinct research bets they make.

Whatever the base model, DataSmith leads on every axis: on average

• 6.6× more new datasets
• 48.4× more subagent calls per session
• 45× more infrastructure code lines written
• 1.8× more research ideas

4.2 DataSmith makes the exploration pay off

Wider exploration is only interesting if it is worth what it costs. Let’s use the SmolLM3-3B run with Opus 5 as the Harness LLM as a case study for this comparison.

Spending more on the agent makes training more compute-efficient. DataSmith costs more per submission on the agent side, but it achieves a higher benchmark score per GPU-minute spent. At the same GPU time, it ends up 4.9 percentage points ahead of Claude Code. As training runs get longer and GPU costs grow, squeezing more value out of each run matters even more. We expect the cost of compute to far outpace the cost of thinking.

DataSmith keeps searching after the baseline plateaus. Across all twelve pairs, every DataSmith run was still gaining in its final quarter. A sample of the trajectory can be found in the plot above. This suggests that the larger exploration tendency allows it to come out of plateaus.

5. Towards a "move 37" for autonomous data research

In its 2016 championship match against Lee Sedol, AlphaGo made a baffling play on move 37 that shook human spectators and changed the game of Go forever. DataSmith is still in its infancy and has yet to make an equally groundbreaking move, but it has already started making research decisions that are surprising and effective.

Here is one that we found especially striking.

6. What’s next?

Today, DataSmith begins automating the search for better post-training data in a research process that is still largely designed by human researchers. Our long-term goal is to make this process recursive by learning from experience: each experiment should teach DataSmith not only which data interventions work, but also how to design new specialist agents, build better tools, and improve coordination. DataSmith is an early step toward research systems that do not just accelerate and automate discovery, but learn from trials and failures how to become better researchers.

Citation

Please cite as:⁴4.*Equal contribution.

Aochong Oliver Li*, Haoli Yin*, Rishabh Adiga, Darren Teh, Siddharth Joshi, Bogdan Gaza, David Schwab, Ari Morcos, Pratyush Maini, "DataSmith: Automating Data Research", DatologyAI Blog, Aug 2026.

Or use the BibTeX citation:

@article{datology2026datasmith,
author = {Aochong Oliver Li and Haoli Yin and Rishabh Adiga and Darren Teh and Siddharth Joshi and Bogdan Gaza and David Schwab and Ari Morcos and Pratyush Maini},
title = {DataSmith: Automating Data Research},
journal = {DatologyAI Blog},
year = {2026},
month = {August},
note = {https://www.datologyai.com/blog/datasmith}
}

We thank Arya Gupta, Anshuman Suri, Dhruba Ghosh, Jasper Tan, Luke Merrick, and Matthew Leavitt for their thoughtful comments on the draft.