Two studies of trading strategies that look profitable on paper. Each is put through the checks a model-validation team runs before real money goes in: is the data right, does it survive trading costs, is it better than luck, and does it depend on an arbitrary date?
Each row compares the number a naive backtest reports with the number after the check.
We tested 1,372 distinct ways to rank nine commodities and hold the best, middle or worst group, at seven rebalance frequencies. The bars show the best one at each frequency, as the value of $100 invested at the start. The dashed line is simply holding the Bloomberg Commodity Index.
Blue: buy and hold. Green: ends above buy and hold at that cost. Red: ends below. Break-even is the cost at which the strategy falls to buy and hold.
The daily strategy is the most profitable before costs and the least profitable after them: it trades about 360 times its capital a year, so at 5 bp it turns $100 into about $50. The daily winners also trade at the same closing price they are ranked on, which nobody can do; waiting one day halves their Sharpe ratio.
For each of the 32 search grids we re-ran the same search 200 times with the rankings shuffled at random. The luck ceiling is the best Sharpe ratio a random search of that size reaches 95% of the time. Points above the diagonal beat it.
Most grids sit below the line. Six of the eight that clear it by more than 0.01 are the Worst group at daily rebalancing: a one-day reversal that exists only at prices nobody can trade at.
Momentum, the most popular idea in the literature, never clears the ceiling: buying recent winners and selling recent losers did no better than a random ranking.
An annual strategy can rebalance in January, February, or any other month. Nothing about the idea changes, yet the result does.
The best annual strategy turns $100 into $2,150 when it rebalances in December. Across all 12 possible months its median is $416.
Split each day into an overnight part (previous close to open) and a daytime part (open to close). On a free data feed the overnight part loses money steadily and the daytime part nearly doubles. On exchange futures data for the same index, neither shows anything. The difference traces to a persistent downward bias in the free feed's recorded opening prices.
Growth of $100 at each year end. A strategy built on the free-feed pattern returns 218.8%; on exchange data the same strategy returns about 1%.
If a day's opening price is recorded honestly, it should land on the day's low about as often as on the day's high. We count both and run an exact binomial test. A series fails when the imbalance is significant after allowing for the number of series tested (p below 0.01 divided by that number), or when the recorded open falls outside the day's own high-low range on more than 1% of days.
The Barchart exchange-sourced futures (dollar index and five currencies) pass, and the free dollar index feed fails with p = 3.7e-13. Tested on clean exchange data the check raised false alarms in only 3.3% of 30 three-year windows, and it catches a low price written into the open field on just 1% of days 99.8% of the time. It cannot see an open that simply repeats the previous close. The flagged Bloomberg series are the London Metal Exchange contracts, whose recorded open sits outside the daily range on 6–8% of days, and three U.S. grains (corn, soybeans, Kansas wheat) plus coffee.
Every check on this page maps to a job that asset managers, custodian banks and allocators do before they fund a strategy or trust a data feed.
Banks and asset managers must independently challenge any model that drives money (U.S. Federal Reserve guidance SR 11-7).
A wrong price feed produced a 218.8% backtest that the standard overfitting checks approved.
The most profitable strategy before costs loses half its money at 5 bp.
A committee sees one backtest. It should also see how that backtest was chosen.
The same data now feeds a tested Python risk engine: value at risk, expected shortfall, Monte Carlo and bootstrap simulation, risk budgeting and dollar exposure, for a book of 40% equities, 30% commodities and 30% currencies (Dec 2010 to Dec 2025).