PineDev
Log inGet started

Methodology

How a strategy earns a verdict

A backtest answers one question: what would these rules have done on this data? The protocol asks the questions that matter before you trade it. Is the result bigger than luck, given how many things you tried? How bad could it plausibly get? Does it survive worse fills? Does it work on data it has never seen? This page documents exactly how each answer is computed.

The stages

  1. 01

    Hypothesis

    Pre-register the edge and the pass criteria before seeing any results.

  2. 02

    Build

    Implement the rules in Pine v6, with costs and a date window built in.

  3. 03

    In-sample

    Backtest on the development window only and upload the trade list.

  4. 04

    Stress test

    Monte Carlo, cost stress, outlier removal and deflated Sharpe.

  5. 05

    Out-of-sample

    One run on held-out data the strategy was never tuned on.

  6. 06

    Verdict

    Every pre-registered criterion, scored. No partial credit.

  7. 07

    Incubation

    Paper or small-size live trades, checked against the simulated band.

Why pre-registration comes first

The most common way a backtest lies is not a bug — it is the person running it. Look at enough variants and you will find one that fits the past. So the hypothesis, the date split and every pass criterion are written down and locked before any result is uploaded. Unlocking is possible; it is recorded, and each amendment counts as another trial against the deflated Sharpe. The out-of-sample window stays sealed until the in-sample work is finished, and is only meaningful the first time it is opened.

The tests

Thresholds shown are the defaults. Each study can set its own before locking.

Sample size

In-sample

Whether there are enough trades for any other statistic to mean something.

Closed trades in the latest in-sample run. Open positions are ignored: their P&L is not final.

≥ 100 trades · critical

Profit factor

In-sample

Gross profit divided by gross loss, after the costs the backtest charged.

Σ winning P&L ÷ |Σ losing P&L|

≥ 1.3

Deflated Sharpe ratio

Stress test

The probability the strategy's Sharpe ratio is real, after accounting for how many variants were tried, how many trades it rests on, and how skewed and fat-tailed its returns are.

Per-trade returns are P&L ÷ equity before the trade. The Sharpe ratio SR is their mean ÷ standard deviation. The benchmark SR₀ is the expected maximum Sharpe of N trials with no edge: SR₀ = √V · ((1−γ)·Φ⁻¹(1−1/N) + γ·Φ⁻¹(1−1/(N·e))), with γ the Euler–Mascheroni constant. N counts prior trials you declared, every in-sample upload (deleted ones included) and every hypothesis amendment. V is the variance of Sharpe estimates across trials: 1/(n−1) when fewer than three runs exist, otherwise the observed variance, never below that floor. The deflated Sharpe is Φ((SR − SR₀)·√(n−1) ÷ √(1 − γ₃·SR + (γ₄−1)/4·SR²)), with γ₃ the skewness and γ₄ the kurtosis of returns.

≥ 95% · critical

Monte Carlo drawdown

Stress test

How deep the drawdown could plausibly get, not just how deep it happened to get.

2,000 bootstrap paths: each resamples the per-trade returns with replacement to the same number of trades and compounds them. The 95th percentile of the paths' maximum drawdowns is compared with your stated tolerance. The generator is seeded from the trades, so the same upload always gives the same numbers.

95th percentile ≤ 25%

Cost stress

Stress test

Whether the edge survives worse execution than the backtest assumed.

Each trade is charged your stated round-trip cost again (doubled costs) and twice again (tripled), as basis points of the position's value when the export includes it, otherwise of the equity at entry.

Net P&L > 0 at doubled costs

Outlier dependence

Stress test

Whether the result rests on a handful of trades that may never repeat.

The best 5% of trades by P&L (at least one) are removed and net P&L is recomputed.

Net P&L > 0

Consistency

Stress test

Whether the strategy works across market conditions, or averages one good regime with several bad ones.

Net P&L per calendar year when the run spans three years or more, per quarter otherwise.

≥ 60% of periods profitable

Out-of-sample profit factor

Out-of-sample

Performance on data that played no part in developing the strategy.

The same profit factor, on the out-of-sample run. The first upload is the test; later uploads are recorded and lower the verdict by one grade.

≥ 1.1 · critical

Out-of-sample efficiency

Out-of-sample

How much of the in-sample risk-adjusted performance survives on new data. Some decay is normal.

per-trade Sharpe (out-of-sample) ÷ per-trade Sharpe (in-sample)

≥ 50%

Out-of-sample drawdown

Out-of-sample

Whether the held-out drawdown is something the in-sample trades predicted.

The out-of-sample maximum drawdown against the in-sample Monte Carlo 95th percentile.

Inside the band

Grades

Validated
Every criterion passed. Incubate at small size.
Promising
Critical tests passed, one other failed. Incubate at reduced size.
Fragile
Two or three non-critical tests failed. The edge may be real, but it is not robust.
Rejected
A critical test failed, or four or more tests did. Do not trade it.

Reusing out-of-sample data lowers the grade by one level.

Incubation

A verdict is about the past. Forward trades — paper or small size — are uploaded in batches and plotted against a fresh bootstrap of the in-sample trades of the same length. A study is on track inside the 5–95% band, below band when cumulative return falls under the 5th percentile, and broken when its drawdown exceeds the in-sample 95th-percentile drawdown.

What this does not do

  • It does not run Pine Script. Backtests run in TradingView; PineDev analyses the trades they produce.
  • It cannot detect every form of lookahead. The Build stage lints for the common ones; the rest is on the script.
  • Bootstrapping assumes trades are independent. Strategies with strongly clustered outcomes will have wider real-world ranges than shown.
  • It is not financial advice, and a validated strategy can still lose money.

References

  • Bailey, D. H. & López de Prado, M. (2012). The Sharpe Ratio Efficient Frontier. Journal of Risk, 15(2).
  • Bailey, D. H. & López de Prado, M. (2014). The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality. Journal of Portfolio Management, 40(5).
  • Harvey, C. R. & Liu, Y. (2015). Backtesting. Journal of Portfolio Management, 42(1).
  • Pardo, R. (2008). The Evaluation and Optimization of Trading Strategies, 2nd ed. Wiley.
  • Efron, B. & Tibshirani, R. (1993). An Introduction to the Bootstrap. Chapman & Hall.
Connect ClaudePricing