How uncertain is a Probabilistic Sharpe Ratio at n = 24?
A lot. At n = 24 the sample kurtosis that the Probabilistic Sharpe Ratio plugs into its denominator has a standard deviation of about 0.74 even when returns are normal, and on a fat-tailed distribution it typically reads 4 when the truth is 7. For a strong edge (per-period SR 1.5) none of this matters: the PSR is saturated near 1 whatever the inputs do. For a marginal edge (SR 0.5, true PSR 0.956) the plug-in PSR ranges from 0.70 to 1.00 across samples of the same process, and a stationary bootstrap of one typical sample spans 0.86 to 0.9996. The closed form is exact arithmetic on inputs that are not exact.
Why this experiment exists
The PSR verification report checks the closed-form arithmetic: denominator, z-statistic, probability. A reader on r/quant pointed out the second-order problem hiding in the same example: at n = 24 the kurtosis estimate itself is very noisy, so treating γ₂ = 7.00 as known treats the noisiest input as exact. The report now says so in its limits. This note measures it, with a distribution whose true moments are set to the worked example, so every estimate can be compared with a known answer. Everything below is synthetic; nothing is a market return.
What the closed form assumes
PSR = Φ(z)
SR, γ₁ (skewness) and γ₂ (Pearson kurtosis; a normal distribution has 3, not 0) enter as point estimates. The formula is an asymptotic approximation from Bailey and López de Prado (2012); it says nothing about how far those estimates sit from the population values at a given n. Two different questions follow.
Part A: how much the inputs move under normality
20,000 samples of 24 standard-normal draws. The estimators are the biased
(population) skewness and kurtosis that scipy.stats.skew and
kurtosis(fisher=False) return by default; the choice matters at
this n and is stated for that reason.
| Quantity at n = 24, normal returns | Monte Carlo | Finite-sample formula | Asymptotic |
|---|---|---|---|
| SD of sample skewness γ₁ | 0.443 | 0.442 | √(6/n) = 0.500 |
| SD of sample kurtosis γ₂ | 0.746 | 0.737 | √(24/n) = 1.000 |
| 5th–95th percentile of γ₂ (true value 3) | 1.90 – 4.16 | — | — |
| SD of sample SR when the true SR is 0 | 0.215 | — | 1/√n = 0.204 |
The finite-sample formulas are the exact variances of the biased estimators under normality, 6(n−2)/((n+1)(n+3)) and 24n(n−2)(n−3)/((n+1)²(n+3)(n+5)). The often-quoted √(24/n) overstates the n = 24 spread by about a third, and the direction of the reader's remark holds either way: a kurtosis reading anywhere between roughly 2 and 4 is consistent with a normal distribution at this sample size.
Part B: a process with the worked example's moments
To compare estimates with a known PSR the population must have SR 1.50, skewness −1.20 and kurtosis 7.00. A two-component normal mixture does it: with probability 0.10 a “bad regime” N(0.145, 1.706²), otherwise N(1.651, 0.749²), in units where the standard deviation is 1. The four parameters are solved from the four moment equations; the solver's residuals are below 10−8. A second scenario shifts both component means by −1.0, which leaves variance, skewness and kurtosis untouched and sets the true SR to 0.50: a marginal edge. Each scenario draws 20,000 samples of n = 24 and computes the plug-in PSR against SR* = 0.
| n = 24, 20,000 samples | True SR 1.50 (worked example) | True SR 0.50 (marginal edge) |
|---|---|---|
| True PSR (population moments) | 0.9981 | 0.9560 |
| Sample SR: mean, SD | 1.660, 0.528 | 0.570, 0.300 |
| Lo (2002) normal-theory SE of SR | 0.298 | 0.217 |
| Sample skewness: mean (true −1.20) | −0.67 | −0.68 |
| Sample kurtosis: mean, SD (true 7.00) | 4.22, 2.16 | 4.23, 2.15 |
| Sample kurtosis: 5th–95th percentile | 2.08 – 8.71 | 2.08 – 8.70 |
| Plug-in PSR: median | 1.0000 | 0.9806 |
| Plug-in PSR: 5th–95th percentile | 0.9850 – 1.0000 | 0.7038 – 1.0000 |
| Samples with plug-in PSR below 0.90 | 0.0% | 22.4% |
| Samples with plug-in PSR above the true PSR | 76.9% | 62.5% |
| Normal-only PSR (skew 0, kurtosis 3 assumed): median | 1.0000 | 0.9927 |
What the table says
- The kurtosis estimate cannot see the tails. With 24 observations from a process whose kurtosis is 7, the sample kurtosis averages 4.2 and lands below 3 in a fifth of samples. The 10% bad regime simply does not show up often enough in 24 draws. The closed form's correction for fat tails is therefore systematically too small, which is one reason more than half the plug-in values sit above the true PSR.
- The SR estimate is wider than normal theory says. Lo's SE, √((1 + SR²/2)/n), gives 0.298 at SR 1.5; the observed SD is 0.528. Fat tails widen the sampling distribution of SR itself, before any PSR arithmetic. The naive 1/√n = 0.204, which drops the SR²/2 term, is wrong by a factor of 2.6 here.
- At SR 1.5 the input noise is irrelevant. Every one of the 20,000 samples gives a plug-in PSR above 0.985. The verification report's example is a strong edge; its PSR of 0.9981 would survive any plausible error in the moments.
- At SR 0.5 the point estimate is mostly noise. The same process, 24 observations, gives PSR anywhere from 0.70 to 1.00. In 22% of samples the analyst reports “below 0.90”; in most of the rest, “above 0.95”. The same edge is called insignificant or near-certain depending on which 24 months happened to be drawn.
- Ignoring skew and kurtosis makes it worse, but not by much at this n. The normal-only median is 0.9927 against 0.9806 with the estimated moments. The estimated moments are so noisy that they only partially do their job.
One sample, bootstrapped
Take the marginal-edge sample whose plug-in PSR sits at the median: SR 0.538, skewness −0.64, kurtosis 3.97, PSR 0.9806. A stationary bootstrap (Politis and Romano, 1994; 5,000 resamples, mean block length 3) recomputes SR, skewness, kurtosis and PSR on each resample. The 5th, 50th and 95th percentiles are 0.859, 0.981 and 0.9996. The interval runs from “not significant at 10%” to “essentially certain”. That is the honest statement of what 24 observations of a marginal, fat-tailed strategy support, and it is not what the single number 0.9806 suggests.
The block length does not matter here because the process is independent across periods; it is included because a real return series is not, and the same code applies to one.
What this does not establish
- One synthetic process, one seed, one n. Real return series are autocorrelated, heteroskedastic and not stationary; all of that adds uncertainty this experiment does not model.
- The mixture is a device for hitting three target moments, not a model of returns. Other distributions with the same moments would give different sampling behaviour for the estimators.
- The per-period SR of 1.5 in the worked example is far above anything plausible at monthly frequency; it was kept because it is the published example's input, and the second scenario exists to show a realistic regime.
- Bias-corrected estimators of skewness and kurtosis would change the numbers in Part A and the kurtosis rows in Part B; the biased ones were used because they are what the common libraries return by default.
Reproduce
python experiment.py
python ../tests/test_psr_uncertainty.py # expected: 6 passed
The script writes outputs/summary.csv, summary.json
and the 20,000 plug-in PSR draws per scenario. Runtime about six seconds.
Because NumPy does not promise bit-identical random streams across minor
versions, the tests pin the claims above with tolerances rather than every
decimal; the committed output was produced with numpy 2.5.2 and scipy 1.18.1.
Code and output on GitHub.
References
- Bailey, D. H. and López de Prado, M. (2012). The Sharpe Ratio Efficient Frontier. Journal of Risk, 15(2).
- Lo, A. W. (2002). The Statistics of Sharpe Ratios. Financial Analysts Journal, 58(4), 36–52.
- Politis, D. N. and Romano, J. P. (1994). The Stationary Bootstrap. Journal of the American Statistical Association, 89(428), 1303–1313.
- Joanes, D. N. and Gill, C. A. (1998). Comparing measures of sample skewness and kurtosis. The Statistician, 47(1), 183–189 (finite-sample variances of the estimators).