BeyondWeb Ex Ante: Ensuring Backtest Safety in Synthetic Data

Language models are increasingly used for financial applications. Before a trading strategy is deployed, it is tested against history, using only information that existed at the time. This is called backtesting. When a model helps build the strategy, the same constraint has to apply to the model's training data, because a model that has read what happened next has already seen the test set. So a firm training its own models needs a hard knowledge cutoff. For real documents this is challenging but achievable: use date metadata and drop anything undateable. The challenge arises from the unfortunate reality that many documents have conflicting or unreliable timestamps.¹1.For example, using the later of pes2o's two timestamps dated nearly all 38.8 million documents by the day Semantic Scholar imported them rather than the day they were published. And git commit timestamps come from the committer's own machine, which we found can result in impossible dates ranging from 2029 to 2038 to 1714, and even 2262-04-11, which is where a 64-bit nanosecond clock overflows.

Synthetic data complicates this further. It has become an essential part of modern training recipes, both to improve model quality and to stretch a finite supply of high-quality tokens (e.g. Nemotron-CC; Qwen3; Kimi K3; BeyondWeb; Anthropic). But a corpus that is substantially written by a model is no longer bounded by the dates of its sources, because the model that wrote it can introduce information from outside the sources and date bounds. For example, using a 2026 model to rephrase 2019 data could introduce information about the 2020 COVID-19 pandemic, something the 2019 data shouldn't know anything about.

Most ways of producing synthetic data put a knowledge cutoff out of reach. Generator-driven approaches ask a model to write, for example, a textbook on a subject, drawing largely on what the model already knows, and hence there is minimal source information to check the output against. Rephrasing-based approaches, something we've pioneered here at Datology, work differently. The model is given a specific source document and asked to rewrite it, so in principle every claim in the output traces back to facts in the dated source document.

Here we show that rephrasing models consistently introduce claims their sources do not support, and present a novel, scalable approach for making rephrased synthetic data ex ante with respect to a chosen cutoff:

1. We first narrow an intractable question (when did every fact in this document become knowable?) into a tractable one based around three criteria: does a rephrased document contain trading-relevant, time-sensitive claims not supported by its source? Asking frontier models the intractable question directly, with and without web search, is both unreliable and prohibitively expensive.
2. We then integrate these criteria into a novel approach that can assess documents at a rate of approximately 8k tokens per second per H100, cheap enough to scale effectively to multi-trillion token corpora.
3. Notably, our method affords continuous, customizable control over how strictly documents are flagged. This makes backtest integrity a dial customers can set by their own risk tolerance, at the cost of increased generation.

Throughout, we call a corpus backtest-safe when its residual rate of problematic claims has been measured and bounded against a stated cutoff.

Prior to any intervention, a frontier judge (GPT-5.6 Sol) marks 1.5% to 10.2% of claims problematic in the public synthetic data we measured, and 0.53% of the claims in our BeyondWeb synthetic data. Our method can reduce that to as low as zero problematic claims in a sample of 583,860 claims. This zero has a one-sided 95% upper bound of 0.00051%, which is itself about 3,000x below the cleanest public alternative we measured. Generating BeyondWeb Ex Ante costs only about 1.7x as much as generating standard BeyondWeb data, and imposes no measurable cost on downstream model quality.

Every public synthetic corpus we measured contains claims its sources do not support²2.This is arguably a feature. Consolidating what a model knows and working it into documents in non-obvious ways is part of why synthetic data is so powerful. A claim the source does not state is likely still worth training on if backtest integrity is not a concern., and none of them report the rate. To our knowledge, no prior work measures the knowledge cutoff of synthetic pretraining data, let alone enforces one. We show that doing so is possible at modest computational cost. We start with an example of a problematic rephrase and then explore the two obvious approaches for fixing it that don't work.

How we assess backtest safety in synthetic data

Anachronisms in rephrased data

Catching an anachronism means reading every sentence, and that is what defeats the obvious fixes at a scale of billions of documents. Below we show an example of a rephrasing-induced anachronism in which the source document, a Wall Street wire report from 2021, describes share price declines in companies such as Amazon, Apple, and Facebook. The rephrase of the document, written in the style of a critical essay, introduces the phrase "Magnificent Seven".

Example of a rephrasing-induced anachronism.

The term "Magnificent Seven" was coined in a May 2023 client note by Michael Hartnett, Bank of America's chief investment strategist, two years after this 2021 source, and was in general market usage by that summer. The membership is anachronistic too. In May 2021 the S&P 500's seven largest weights were Apple, Microsoft, Amazon, Alphabet, Facebook, Berkshire Hathaway, and JPMorgan Chase; Nvidia was fifteenth at under 1% of the index, worth roughly $345 billion, sitting between Home Depot and Procter & Gamble. Nvidia did not reach the top handful until 2023, crossing a trillion dollars in market capitalization on 30 May and finishing that month as the index's fourth-largest weight.

Why the obvious approaches don't work

The obvious fix is to ask a model to date the rephrase and compare it against the cutoff. We tested that across ten frontier models, and it does not work. All ten models converged on similarly unusable behavior. They were unable to date most passages, so they flagged between 78% and 93% of documents, and neither more capable models nor web search changed that. And even if it worked, it would be cost-prohibitive. Dating every document would run roughly $20 million per trillion tokens of rephrased output with GPT-5.6 Sol, $30 million with Claude Opus 4.8, and about $1 million with GPT-5.6-Luna, the cheapest model we measured.

Another obvious approach would be to fact-check every claim in the rephrased document. Automated fact-checking almost always begins by parsing the document into units, either a list of atomic claims or at minimum a sentence split. In the long-form setting that chopping is done by a large model or a human annotator, and it is expensive: FActScore finds 26 to 41 atomic facts per generation and pays $4 in annotator time to check one claim, and VeriScore's full LLM-based extract-retrieve-verify pipeline cost $1,038 for 400 generations. Even reducing VeriScore's 2024-era costs by using contemporary API prices and efficiency techniques yields estimates from $5 million (GPT-6-Luna) to $100 million (GPT-6.1-Sol) per trillion tokens. So both claim extraction and claim assessment require radically cheaper solutions to operate at scale.

Three criteria for defining problematic claims

We first decompose the rephrased document into claims using a sentence-based approach similar to MiniCheck, then each claim is checked against three criteria:

Faithfulness asks whether the source states the claim. It is necessary but not sufficient. Unfaithfulness alone flags about 24% of claims, most of them embellishments and inferences the rephraser drew rather than facts it imported, so a faithfulness-only standard is far broader than required for determining whether a claim could compromise a backtest.

Trading relevance. Could this claim inform a trading decision? This is the criterion that makes the intervention about trading rather than about unfaithfulness in general. It is also the only one of the three that is domain-specific, and the only one that changes if the bound you need is a licensing date or a regulatory one. A claim qualifies when it states something about an identifiable economic exposure. For brevity we do not provide a comprehensive description here, but examples include:

• A policy, environmental, or security claim that directly implies material economic impact.
• Business or market outcomes such as demand, financing, or competitive position.
• A technical result that states a material commercial capability.

A qualifying statement can be a result, a comparison, or a forecast, and it need not carry a number: "the company delayed its product launch" qualifies on its own.

Time sensitivity asks whether the claim's assertion is placed after the cutoff or left undated. Long-established facts pass, but sufficiently precise versions of an established fact may not. For example, it's well-established that days are longer in summer than in winter, while a figure for daylight duration at a given latitude to the microsecond rests on measurement technology that became available recently.

The conjunction between all three criteria has to hold on the same sentence, and we drop any document carrying even one problematic claim. Each criterion is a category of evidence, and may combine several signals. Importantly, the criteria return confidences rather than verdicts, so how much confidence we demand before flagging is adjustable.

The criteria above need a rubric precise enough to settle their boundary cases and maximize signal-to-noise, which took us many rounds of iteration and refinement. Our own scoring applies this rubric at corpus scale, at roughly 8k tokens per second per H100, and the frontier judge applies it to held-out samples as our reference measurement.

From 1.5% in public data to zero, at 1.7x the generation cost

We ran the frontier judge (GPT-5.6 Sol³3.We evaluated ten frontier models as our oracle for this task: GPT-5.6 Luna, Terra, and Sol, Opus-4.8, Fable 5, GLM-5.2, DeepSeek-v4-Pro, Qwen3.7-Plus, Nemotron-3-Ultra, and Kimi K2.6. We prefer an oracle that over-flags rather than under-flags, so that contamination is bounded from above instead of below. We found that GPT-5.6 Sol was by far the most stringent judge, so we selected it as our oracle.) using the three criteria described above on random samples of three public synthetic corpora (Nemotron-CC, FinePhrase, and Cosmopedia) totaling ~100k claims, and on one 2k-document corpus we generated using NeMo Curator, a publicly available synthetic data generation tool.

Public synthetic corpora carry problematic claims at 3,000x to 20,000x the upper bound on BeyondWeb Ex Ante, where we observed none. Each point is the share of claims a frontier judge (GPT-5.6 Sol) marks problematic, meaning unsupported by its source, trading-relevant, and either placed after the cutoff or left undated. The axis is logarithmic, and the whisker is the one-sided 95% upper bound of our rate after backtest-safety interventions. We distinguish between rephrasing-based and generator-driven approaches. Rephrasing approaches are asked to rephrase a seed document, so in theory they do not need to introduce new claims, which makes BeyondWeb, NeMo Curator, Nemotron-CC, and FinePhrase like-for-like comparisons. Cosmopedia conditions on a seed extract and writes new content on its topic, so knowledge injection is the method working as designed, and hence we view its 10.2% as an upper bound on problematic claim rates rather than a defect. Our FinePhrase figure covers its FAQ, table, and tutorial configurations; we exclude its math configuration, because math and code rephrasing raises separate scientific questions that we will address in follow-up work.

Problematic claim rates run from 1.5% (NeMo Curator) to 3.7% (FinePhrase) to 10.2% (Cosmopedia), representing 2.9x, 7.0x, and 19.3x increases over standard BeyondWeb data. NeMo Curator (1.533%), which we ran using its stock Nemotron-CC SDG prompts, is the closest comparison, because we ran it on our own source documents.

We then measured the problematic claim rate on a held-out test set drawn from our current production pipeline⁴4.The test set comprised 55,097 freshly generated documents, with every one of their 1,000,011 claims labeled by the frontier judge using the three-criteria rubric.. Before any intervention, 0.53% of claims are problematic. After our interventions, zero problematic claims remain, with a one-sided 95% upper bound of 0.00051%.⁵5.n = 583,860 judged claims. Critically, the threshold that achieves this was fixed on a validation set before we scored the test set. The zero-problematic-claim corpus runs at 1.72x the generation cost of standard BeyondWeb data, with the surplus compensating for omitted documents.

The threshold above is one setting of a continuous dial, and we show the full trade-off between residual problematic claims and generation cost below:

Problematic claims fall to zero at 1.72x the generation cost. Each point is a threshold on the claim score, measured across 55,097 held-out documents whose 1,000,011 claims were all labeled by the frontier judge. Those documents are a held-out sample of a 70-million-document production corpus. The horizontal axis is generation cost, meaning generated tokens per delivered token, and the vertical axis is the residual problematic claim rate after backtest-safety interventions, in log scale. The horizontal dashed lines mark the base rates for the best public data (1.53%) and BeyondWeb without interventions (0.53%), and the shaded blue region denotes the one-sided 95% upper confidence interval. Because the vertical axis is log-scaled, settings where no problematic claim survives cannot be drawn on it and instead sit in the band at the bottom. Example points along the trade-off curve are highlighted.

Picking a few example points on the trade-off curve, we see that 1.2x the generation cost brings the rate from 0.53% in standard BeyondWeb down to 0.033%, and 1.6x brings it to one claim in 603,786. At 1.72x we hit zero flagged claims, the measurable floor of our dataset. Given the magnitude of the problematic claim reduction, we consider these costs modest.

Most of that cost is safety margin. Our own scoring is less accurate than the frontier judge's, so we tune strongly toward recall. At the 1.6x operating point we drop about 39.8% of rephrased tokens, while the documents the judge flags account for 6.9%. We would rather regenerate a safe document than keep an unsafe one. Closing that accuracy gap is active work.

Below we show the single judge-flagged claim that remains after backtest-safety interventions at the 1.6x generation cost operating point:

The single frontier judge-flagged claim out of 603,786 total claims that remain after applying backtest-safety interventions to our experimental corpus of 1,000,011 claims. For this intervention we targeted a generation cost increase of 1.6x. The panels show the relevant excerpts, not the full documents. We leave it to the reader to decide whether this rephrased claim is truly problematic for backtest integrity.

Backtest safety does not compromise data quality

Dropping 40% of the tokens in a corpus is the kind of intervention that could impact data quality, so we measured it.

Backtest safety has no measurable model quality impact, before or after post-training. We CPT'd an 8B model for 10B tokens under three data regimes: organic web data alone, a 50/50 blend of organic and standard BeyondWeb, and the same blend with BeyondWeb Ex Ante in place of the standard synthetic. The base model was trained on 160B tokens of high-quality organic web data. The left panel reports the CPT'd models' mean accuracy over ten knowledge and reasoning tasks, and the right panel reports their mean accuracy over eight post-training tasks after identical SFT and DPO. Filled markers are condition means and open circles are individual seeds. The panels use different y-ranges, because the models and task suites differ and absolute accuracies are not comparable across them.

We evaluated 8B models CPT'd for 10B tokens on a ten-task knowledge and reasoning suite⁶6.The base model benchmark suite consists of MMLU (0-shot), MMLU-Pro (0-shot), ARC-Challenge (0-shot), CommonsenseQA (0-shot), PIQA (0-shot), SocialIQA (0-shot), AGIEval-English (1-shot), BIG-Bench Hard (3-shot, CoT), DROP (5-shot), and WinoGrande (5-shot). and report the unweighted mean (see the above figure for experimental details). We also applied identical post-training to all three models using the Dolci-Instruct SFT and DPO data and then evaluated them on a suite of eight tasks⁷7.The post-trained benchmark suite consists of AGIEval-English (0-shot, CoT), MMLU-Pro (0-shot, CoT), GPQA (0-shot, CoT), IFEval (0-shot), BIG-Bench Hard (3-shot, CoT), DROP (3-shot), TruthfulQA (6-shot), and PopQA (15-shot). We took the best-performing SFT + DPO hyperparameter combination and trained an additional two seeds, for a total of three for each condition..

For the base models, BeyondWeb Ex Ante scored 0.5 pp above standard BeyondWeb, a difference inside the spread between seeds. For scale, the gain from rephrasing at all is 8.56 pp. The Ex Ante model was also 0.4 pp above the standard model after post-training, again within the spread between seeds. We find no measurable quality difference between the two, before or after post-training.

A cutoff you can measure

A source document's date is typically a matter of record. Rephrasing undermines this by injecting unsourced information, and no amount of diligence about crawl dates can mitigate this problem. So we built BeyondWeb Ex Ante: a decomposition cheap enough to run across billions of documents, three criteria that together isolate the roughly 0.5% of claims that could actually contaminate a backtest, and a threshold you set to decide how much of the corpus to regenerate. Set strictly enough, it leaves no problematic claim in 583,860 delivered claims, at 1.72x the generation cost and no measurable loss in model quality. Even the one-sided 95% upper bound on that zero, 0.00051%, is about 3,000x below the cleanest public alternative we measured.

This same approach applies beyond backtesting. Trading relevance is the axis most urgent to our fintech customers, but the structure underneath it speaks to a more general problem: once models write part of the corpus, a document stops inheriting its provenance from its source, and provenance becomes something you have to measure. Any team that needs to bound what its training data could have known is solving the same problem, whether the bound is a trading cutoff, a licensing date, or a regulatory one. Only the relevance criterion changes.

If you are training models on synthetic data and need guarantees on what that data could have known, please get in touch.

Citation

Matthew Leavitt, Parth Suresh, Siddharth Joshi, Diego Kiner, Daniel Zayas, David Schwab, Bogdan Gaza, and Ari Morcos. BeyondWeb Ex Ante: Ensuring Backtest Safety in Synthetic Data. DatologyAI Blog, Oct 2026

@article{datology2026beyondwebexante,
  author = {Matthew Leavitt and Parth Suresh and Siddharth Joshi and Diego Kiner and Daniel Zayas and David Schwab and Bogdan Gaza and Ari Morcos},
  title = {BeyondWeb Ex Ante: Ensuring Backtest Safety in Synthetic Data},
  journal = {DatologyAI Blog},
  year = {2026},
  month = {October},
  note = {https://www.datologyai.com/blog/backtest-safe-synthetic-data}
}