How to verify a quantitative finance implementation against a known answer
Run the implementation on an input whose correct answer you already know, and on the ugliest inputs you can construct, before you run it on data whose answer you do not know. Every defect this site has found and fixed was found that way, none by reading the code and none by running it on real returns: a VPIN estimator that returned 0 on a perfectly one-sided tape, a distance matrix that scipy rejected as asymmetric, a worked example whose “normal” case dropped the kurtosis term, a table cell that carried 2.8955 where the arithmetic gives 2.8949. The method is eight steps; the rest of this note is what each one caught.
Why reading the code is not enough
Quantitative finance methods are usually implemented from a paper, and a paper is prose plus formulas. Prose is where conventions hide: whether kurtosis means Pearson's 3-at-normal or the excess 0-at-normal, whether a division by zero in a degenerate case should give 0.5 or should follow the sign, whether the last incomplete bucket belongs in the average. Code that reads correctly against the prose can still be wrong against the maths, and real data cannot tell you, because on real data the right answer is unknown. So the test has to be an input where it is known.
The eight steps
| Step | What it means | What it caught here |
|---|---|---|
| 1. Build a synthetic input with a planted answer | Generate data from a fixed seed with the property the method is supposed to detect placed where you know it is. | A 20,000-trade tape with informed buying in trades 8,000–11,999; VPIN rose from 0.3292 to 0.6376 across it, a 1.94× ratio the estimator had to show before anything else was believed. |
| 2. Feed it the degenerate cases | Zero variance, constant inputs, a single asset, the smallest n the formula admits, a monotone series. Decide the correct output before running. | A monotone tape: every trade higher than the last, dispersion of price changes zero. The original branch split volume 50/50 and returned VPIN 0 on the most toxic input possible. Fixed by classifying on the sign of the change; now one of 15 tests. |
| 3. Reproduce a published number | If the paper, a textbook or your own site gives a worked example, recompute it independently (scipy, a spreadsheet, by hand) and compare every intermediate, not just the final figure. | The PSR worked example: denominators of 1.000 and 2.318 as published, 1.4577 and 2.4850 when computed. Months later a reader recomputed the z-statistic on the corrected page and found 2.8949 where the table still said 2.8955. |
| 4. Run it in a clean environment with current libraries | Dependencies move. Install the pinned versions in a fresh directory, then the newest, and run both. | HRP under pandas 3: np.fill_diagonal on DataFrame.values raised because the view is read-only. Under pandas 2 the same code had run for a year. |
| 5. Check what the library rejects | Floating-point noise breaks exact properties (symmetry, positive definiteness, sums to one). Test the properties, not just the values. | The correlation-distance matrix was asymmetric at the 10−16 level and scipy.spatial.distance.squareform refused it. Fixed by symmetrising explicitly, 0.5(D + D′). |
| 6. Fix the test when the test is wrong, and say so | An over-specified assertion is a defect in the test, not in the code. The distinction has to be written down, because silently loosening tests is how real gaps get hidden. | An HRP test asserted that single and Ward linkage must produce different weights; on one seed they coincided. The methods were confirmed to differ on other seeds and the test was corrected, with the reason recorded in the report. |
| 7. Pin every intermediate the reader could check | Publish the denominator, the z, the bucket size, the seed, the version numbers. A final figure alone cannot be audited; a chain of intermediates can, and will be. | The 2.8955 catch above happened only because the intermediate was on the page. It would have been invisible behind “PSR = 99.81%”. |
| 8. Write down what the check does not establish | A synthetic test proves the implementation matches the method. It says nothing about whether the method works on markets. | Every report on this site ends with that section: VPIN's forecasting power is contested in the literature and untested here; HRP's lower drift does not mean lower realised volatility, and on 50 synthetic panels it usually does not. |
What makes a good planted answer
The input should be simple enough that the right output is obvious and strong enough that a wrong implementation cannot get it by accident. The monotone tape is the model: any correct classifier must call it entirely one-sided, so the expected VPIN is 1 to within the window's edge effects, and a test asserting VPIN > 0.9 has no false positives worth worrying about. The informed-episode tape is the weaker kind: it tests direction, not level, because the level depends on the classifier, the bucket size and the window, as the noise-floor note measures. A good suite has both: one input with an exact answer, one with a qualitative one, and the qualitative one is not allowed to be the only evidence.
For a formula rather than an algorithm, the planted answer is a second, independent computation. The PSR calculator now computes the normal CDF rather than quoting a value, and its tests pin the normal case (denominator 1.4577, z 4.935) and the skewed case (2.4850, 2.895) against scipy. Two implementations that agree can both be wrong in the same way if they share a convention; the convention itself (here, that γ₂ is non-excess kurtosis) has to be stated in the text so a reader can disagree with it.
What makes a good degenerate case
Ask what each denominator can do. Zero dispersion of price changes, zero variance of returns, a covariance matrix of rank one, n = 2 in a formula with √(n − 1), a kurtosis that makes 1 − γ₁SR + (γ₂ − 1)SR²/4 negative. Then ask what each branch that handles those cases assumes. The VPIN defect lived in the branch that handled zero dispersion; the branch existed, so the author had thought about the case, and still got its sign wrong, because “split 50/50” was the right answer for a flat tape and the wrong one for a steadily rising tape, and the branch did not distinguish them. The test that caught it constructs the rising tape explicitly. The calculator's tests likewise refuse to display a result when the variance term is non-positive rather than printing a number that means nothing.
What a verification report has to contain
- The implementation under test, at a named revision.
- The environment: language and library versions actually used.
- The input: how it was generated, its seed, what was planted.
- The expected answer and where it comes from.
- What the code returned, before and after any fix.
- Each defect: how it was detected, what the fix was, which test now guards it.
- A reproduce block: the commands that regenerate every number on the page.
- The limits: what this establishes and what it does not.
The three reports on this site (VPIN,
HRP, PSR)
follow this outline, and quantmedia-research/verify_examples.py
re-runs the first two in a clean directory and asserts that the committed
CSVs match to nine decimals. The experiment notes that followed them use
tolerance tests instead, because Monte Carlo output is not bit-stable across
NumPy versions; the note says which kind of test it is.
What this does not establish
- Passing every step proves the implementation matches the method as stated. It does not prove the method is useful, and a verification report should never be read as an endorsement of the method.
- The steps were distilled from three methods and about thirty tests. Other kinds of code (optimisers with numerical tolerances, anything stochastic by design) need additional steps this note does not cover.
- Independent recomputation catches convention and arithmetic errors; it does not catch a misreading shared by both computations. Publishing the convention is the only defence, and it depends on a reader.
Reproduce
python quantmedia-research/tests/test_hrp.py # 13
node scripts/test_psr.js # 12
python quantmedia-research/verify_examples.py
Every defect cited above is in the repository's history and the test that guards it is in the suites listed. Code on GitHub.