Test Without Looking Ahead

A backtest is a simulation using historical information. It can be useful while still being misleading. Future data, selected assets and perfect fills can create results that a real learner could never have achieved. This lesson teaches how to inspect those errors and continue with paper observation rather than treating a historical fit as proof.

Education only. Crypto assets can lose substantial value or become worthless. No lesson requires buying an asset, depositing money or sharing real passwords, private keys or recovery words.

Development, holdout and forward observation

A development period is used to design or tune a rule. An out-of-sample period is reserved for evaluating the resulting rule without adjusting it to that period's outcomes. Forward observation applies the recorded rule as new information arrives. These distinctions help identify which evidence has influenced the design.

Repeatedly inspecting and changing a rule to improve the reserved period weakens its independence. Testing many variants can also overfit a small history. Research on backtest overfitting explains why apparently attractive historical performance can arise through selection. A single holdout is useful discipline but not a universal solution, especially when many choices have already been made. Record the full search process and uncertainty.

A cook in Naples testing a recipe who changes the ingredients after tasting can never tell whether the original worked. Rules must be written before the test.

To test a crypto idea safely, use a paper-tracking test, an idea you met in Level 0. Write the rule first, with a start date, an end date, an amount and what counts as success. Then record each decision the rule would make, at the time, without real money. At the end, compare the results with a simple benchmark, as the bias and psychology lesson in Level 9 suggested.

If you change a rule, start a new version with a new test period. Applying a better-looking rule to old dates is hindsight, not evidence.

Tests and journals need regular attention, so the plan needs a calendar.

Open diagram at original size ↗

Read diagram as text

Write the rule and information-availability timing → separate development data → evaluate the holdout without retuning → observe forward and record limits.

Include feasibility, full costs, failed assets and every rule variation. A reused holdout no longer provides the same independent check.

Do not put future information into the past

Look-ahead bias uses information that was unavailable at the decision time. Examples include the day's final close before the day ended, a later corrected price, a future token classification or an entity label discovered months afterward. A spreadsheet can contain all those values in one row, making the timing mistake easy to miss.

Write an availability timestamp for each signal input. If a report is published after a period ends, it cannot be used as though it was known during that period. On-chain data may be visible promptly while a provider's interpretation arrives later or is revised. The existence of the underlying transaction does not prove that a particular labelled metric was available. Preserve point-in-time information where the test depends on it.

The same issue appears with updated token lists, revised address labels and later corrected market data. Record when each input became available, rather than only the timestamp of the event it describes.

Include failures and avoid hindsight selection

Survivorship bias occurs when the sample contains only assets or venues that survived until today. Failed, delisted or illiquid assets may be omitted, making a historical universe look safer or more profitable than it was. Choosing a start date after a major loss can similarly improve the displayed result without changing the rule.

Define the universe as it existed under the test's selection method at each time. Record delistings, unavailable exits and missing records. Do not silently substitute today's popular assets into the past. If complete data is unavailable, state the limitation and narrow the conclusion. A result on one supplied surviving asset is a demonstration on that sample, not evidence of broad performance across all crypto markets.

Model feasible fills and all relevant costs

Check spread, depth, queue uncertainty, partial fills, fees and product availability. A price touching a limit within a candle does not prove a fill for your size. A stop in a fast market may execute worse than its trigger. Funding and collateral rules affect derivative results, while withdrawal restrictions affect cash access.

Test cost sensitivity rather than assuming one favourable estimate. A gross result that disappears under modest additional costs is fragile. Record whether the venue existed and the account would have been eligible during the sample. Simulation should also handle missed actions and unavailable data. A rule that requires continuous flawless attention may be incompatible with a beginner's actual routine even when its spreadsheet result is positive.

If a candle's low touched a limit price, there may have been only a tiny quantity available at that level. A paper order larger than the available size cannot assume a full fill. If the same candle touched an entry, a stop and a target, the four summary prices may not reveal the sequence. Use finer data where appropriate or state a conservative treatment of ambiguous intervals.

Disclose limits and keep observing

A useful test report includes the rule version, all datasets, assumptions, variations, gross and net results, benchmark and limitations. Add adverse scenarios and an explanation of what the sample cannot establish. A small sample gives limited information about rare events and changing market conditions. Do not turn one favourable window into a forecast.

Continue with forward paper observation if the question remains useful. Predetermine the review interval and reasons to stop or revise. A failed test can save time by exposing an unsupported idea. A successful test can justify further research without requiring a real deposit. The next lesson explains metrics and process review so that results are assessed with the same honesty as the inputs.

A useful sensitivity check changes a stated assumption, such as adding a small extra cost or delaying entry by one interval, without pretending that the new outcome was the original test. Report the original result and the sensitivity result together. A rule that works only under one unusually favourable assumption needs further investigation.

Forward paper observation can test whether the written procedure is usable as information arrives. It still does not reproduce real queue position, emotional pressure, account restrictions or every rare event. Success supports a bounded next research step; it does not require starting real-money trading.

Try the following questions aloud or on paper before opening the worked answers. Compare your reasoning as well as your final answer.

References and further reading