The Deflated Sharpe Ratio in practice: false positives, power and correlated trials
Pick the best of N no-skill strategies by Sharpe ratio and test the winner with an ordinary Probabilistic Sharpe Ratio at 0.95: it passes 40% of the time at N = 10, 99% at N = 100, always at N = 1,000. Test it with the Deflated Sharpe Ratio instead and it passes 0.1% of the time. The closed form for the luck benchmark, the expected best Sharpe among N trials, matches Monte Carlo to within 0.005 when the dispersion term is measured across the actual trials, and keeps matching when the trials are correlated, because that dispersion shrinks with the correlation. The price: with one genuine edge among ten no-skill trials, DSR at 0.95 detects it 19% of the time where the naive PSR detects it 92%. DSR is a conservative test, and this note measures how conservative.
What is being checked
The DSR explainer gives the formula and a worked table: a strategy with Sharpe 1.50 over 120 monthly observations, skewness −0.8, kurtosis 6.0, and a dispersion of Sharpe ratios across trials of 0.50, falls from DSR 0.9997 at N = 10 to 0.2671 at N = 1,000. Four questions follow. Is the table right? Does the closed form for the expected maximum hold? What does the correction do to false positives and to power? And what happens when the N trials are not independent, as parameter sweeps never are? Everything below is synthetic.
The benchmark
DSR = PSR evaluated against SR*0 instead of 0, with the winner's own n, skewness and kurtosis
γ is the Euler–Mascheroni constant 0.5772. sd(SR) is the standard deviation of the Sharpe ratios across the N trials, which means the trials have to have been kept.
Part A: the explainer's table, recomputed
| N | Expected max Sharpe SR*0 | DSR |
|---|---|---|
| 1 | — | 1.0000 (= PSR) |
| 10 | 0.7873 | 0.9997 |
| 100 | 1.2653 | 0.8736 |
| 1,000 | 1.6276 | 0.2671 |
Denominator 2.2389, every cell to four decimals. The published table holds. One thing the table does not say: sd(SR) = 0.50 is a large dispersion. Under a pure null with 120 observations, Sharpe ratios of independent no-skill strategies scatter with a standard deviation of only √(1/n) = 0.091. A dispersion of 0.50 describes a family of genuinely different strategies, some with edge, not a parameter sweep of one; the example is illustrative of the arithmetic, not of what a sweep produces.
Part B: does the closed form hold?
N independent strategies with iid standard-normal returns over 120 periods, 5,000 repetitions. sd(SR) is taken from the trials in each repetition, as the method prescribes.
| N | sd(SR) across trials | Max SR, Monte Carlo | Max SR, closed form | Closed form with sd 0.50 |
|---|---|---|---|---|
| 10 | 0.0900 | 0.1424 | 0.1417 | 0.7873 |
| 100 | 0.0918 | 0.2322 | 0.2324 | 1.2653 |
| 1,000 | 0.0920 | 0.3039 | 0.2995 | 1.6276 |
The formula is accurate to within 0.005 at N = 10 and 100 and within 0.005 at N = 1,000, where the extreme-value approximation is at its weakest. The last column is what the explainer's sd of 0.50 implies; it is 5.5 times the null dispersion, and the benchmark scales with it linearly. Feeding DSR a dispersion that was not measured on the trials is where the number stops meaning anything.
Part C: false positives and power
Under the null (no strategy has edge), the best of N by in-sample Sharpe is tested at 0.95 both ways. With one true edge planted (mean 0.30 per period, PSR against zero about 0.94 on its own), the same is done and “power” counts the repetitions where the edge was both selected and passed.
| N | Naive PSR > 0.95, null | DSR > 0.95, null | True edge selected | Naive PSR power | DSR power |
|---|---|---|---|---|---|
| 1 | 4.7% | 4.7% | 100% | 94.4% | 94.4% |
| 10 | 40.1% | 0.0% | 93.4% | 91.5% | 18.9% |
| 100 | 99.2% | 0.1% | 75.1% | 75.1% | 13.1% |
| 1,000 | 100% | 0.1% | 49.4% | 49.4% | 5.4% |
- The naive PSR is useless on a mined winner. Its false-positive rate is 40% after ten tries and 99% after a hundred. The 4.7% at N = 1 is the nominal 5%, which is the only case where PSR is the right test.
- DSR over-corrects. Its false-positive rate is not 5% but 0.0–0.1%. The benchmark is the expected maximum, and the winner has to clear it with 95% confidence; under the null the winner sits at the expected maximum on average, so it almost never clears with margin.
- Power is the cost. A strategy with a real per-period Sharpe of 0.30 is found by the naive test 91% of the time among ten no-skill rivals and by DSR 19%; among a thousand rivals it is not even selected half the time, and DSR confirms it in 5%. Conditional on being selected, DSR's detection rate is 20%, 17% and 11%.
Read together: DSR at 0.95 is a filter that rarely lets luck through and often keeps skill out. That is a defensible trade for a research process where the cost of a false strategy is high; it is not a neutral measurement.
Part D: correlated trials
A parameter sweep produces trials that share most of their returns. Here N = 100 strategies are built as √ρ·common + √(1−ρ)·own noise, so pairwise correlation is ρ.
| ρ | sd(SR) across trials | Max SR, Monte Carlo | Closed form with nominal N = 100 | Effective N implied |
|---|---|---|---|---|
| 0.0 | 0.0920 | 0.2330 | 0.2328 | 101 |
| 0.5 | 0.0651 | 0.1656 | 0.1646 | 104 |
| 0.9 | 0.0291 | 0.0741 | 0.0735 | 106 |
Correlation does not break the formula; it shrinks sd(SR). Ninety percent common variance leaves the hundred trials with a Sharpe dispersion of 0.029 instead of 0.092, the expected maximum falls from 0.23 to 0.07, and the closed form with the nominal N = 100 still lands within 0.001 of the Monte Carlo. The trials are not independent, but the dispersion measured across them already carries that information, so the “effective number of independent trials” implied by the maximum stays near 100. The practical rule is the one the explainer already gives: measure sd(SR) on the trials you actually ran. Substituting a dispersion from elsewhere, or a nominal “independent” N smaller than the count, double-corrects.
What this does not establish
- Normal returns throughout. Fat tails widen the null dispersion of Sharpe ratios and would raise every benchmark; the explainer's skew and kurtosis inputs enter DSR's denominator, not its benchmark.
- The equicorrelated design is one structure. A sweep that produces a few clusters of near-identical strategies plus some outliers is different, and Bailey and López de Prado's own recommendation, to cluster trials first, is not tested here.
- “Power” is defined for one edge size and one n. A larger edge or a longer record changes the numbers, not the direction.
- N is self-reported everywhere in this method; no experiment can make an understated trial count honest.
Reproduce
python experiment.py # about 45 s
python ../tests/test_dsr_trials.py # expected: 5 passed
The first test recomputes the explainer table to four decimals; the others run the Monte Carlo at reduced size with matching tolerances. Code and output on GitHub.
References
- Bailey, D. H. and López de Prado, M. (2014). The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality. Journal of Portfolio Management, 40(5), 94–107.
- Bailey, D. H. and López de Prado, M. (2012). The Sharpe Ratio Efficient Frontier. Journal of Risk, 15(2).
- Harvey, C. R. and Liu, Y. (2015). Backtesting. Journal of Portfolio Management, 42(1), 13–28.
- White, H. (2000). A Reality Check for Data Snooping. Econometrica, 68(5), 1097–1126.