Missing Data

Quick Answer

Missing data comes from halts, holidays, illiquidity and feed faults, and the danger is the fill, not the gap itself. Forward-filling is safe because it uses only past prices; back-filling or interpolating leaks future information. Mark circuit-locked days untradeable rather than inventing a fill, and never let a NaN become zero, which a backtest reads as a total crash.

Definition: Missing Data

Missing Data is any absent value in a market-data series — arising from halts, holidays, illiquidity or feed faults — whose real risk lies not in the gap but in how carelessly it is filled.

Key takeaways: Missing Data

• Missing data comes from holidays, halts, illiquidity and faults, and each needs different handling
• Forward-filling is generally safe; back-filling and interpolation leak future information
• Halted or circuit-locked days should be marked untradeable, not filled with invented prices
• A NaN coerced to zero is read as a total crash, so NaN policy must be explicit
• Dropping non-random missing rows is a quiet form of survivorship bias

Missing Data at a glance

Missing Data in simple words

Missing data is the holes in a price history: days the market was shut, a stock that did not trade, a feed that dropped out, or a cell that reads NaN. Every backtest has to decide what to do with these holes, and that decision matters more than it looks. Fill them the wrong way and you either invent prices that never existed or, worse, quietly leak future information into the past.

What Missing Data is for

Handling missing data exists because no real dataset is complete, and the method you use to patch the holes is itself a modelling assumption that can add or remove bias without you noticing.

Missing Data — professional explanation

Why data goes missing

Gaps arise for very different reasons, and the reason dictates the correct treatment. Exchange holidays and weekends are expected absences that a calendar should handle, not a fill. A trading halt or a lower-circuit day means the instrument genuinely could not be traded, so a filled price would misrepresent a period when no exit was possible. Illiquid instruments simply do not print on some days. And feed outages or corrupt records are true data faults where the value existed but was lost. Conflating these — treating a halt like a holiday, or an outage like an illiquid day — produces subtly wrong backtests.

The forward-fill versus back-fill distinction

The single most important rule is directional. Forward-filling — carrying the last known value forward until a new one arrives — is generally safe because it only ever uses information already available at that time. Back-filling — carrying a future value backwards to patch an earlier gap — is dangerous because it inserts a price that was not yet known, a direct form of look-ahead. Interpolating between a past and a future point is a subtler version of the same sin, because the interpolated value depends on data from after the gap. As a default, a backtest should forward-fill or leave gaps explicit, never back-fill prices used for decisions.

Halts and untradeable periods

When an instrument is halted or locked at a circuit limit, the honest model is not to fill a plausible price but to record that no trade was possible. A backtest that fills a halted day with an interpolated price may show an exit or entry that could never have executed, understating the risk of being trapped in a position. This is especially relevant for Indian small and mid caps that hit upper or lower circuits, where a strategy might in reality be unable to sell for several days while the backtest happily books an exit at a made-up price.

NaNs and how they propagate

Explicit missing markers such as NaN are actually safer than silent gaps because they are visible, but they must be handled deliberately before indicators are computed. A single NaN feeding into a moving average or a standard-deviation window can poison the whole window, producing a NaN signal or, if the library silently drops it, a statistic computed over the wrong sample. Worse is a NaN that gets coerced to zero, which a price series reads as a 100 percent crash and a return series reads as a total loss. Every pipeline needs an explicit, logged policy for NaNs rather than relying on a library default.

How the fill choice biases results

Each fill method leaves a fingerprint. Forward-filling a stale price flattens volatility and can suppress or delay signals, making a mean-reversion system look calmer than reality. Interpolation smooths the series and understates true gaps and drawdowns. Filling untradeable halts creates fills that inflate performance by pretending liquidity existed. Even dropping missing rows biases results if the missingness is not random — illiquid or stressed names go missing precisely when they matter most, so deleting them is a quiet cousin of survivorship bias. The correct method is the one whose assumption matches why the data is missing, and it should be stated, not defaulted.

Worked example: Missing Data

Forward-fill vs Back-fill

Limitations of Missing Data

• The correct fill depends on why data is missing, and that reason is often not recorded in the feed
• Forward-filling stale prices flattens volatility and can hide real risk
• Any back-fill or interpolation across a gap introduces look-ahead by using post-gap information
• Dropping missing rows biases results when missingness is non-random, as with stressed or illiquid names
• Circuit halts mean no trade was possible, which most fill methods silently ignore

How professionals treat Missing Data

Serious researchers first classify why a value is missing — holiday, halt, illiquidity or fault — and treat each differently, using a trading calendar for expected absences and marking halts as untradeable rather than filling them. For decision data they forward-fill or leave gaps explicit and never back-fill, and they log every fill so the pipeline is reproducible. They also test sensitivity to the fill choice, because a strategy whose result changes materially with the missing-data policy is resting on an artefact rather than an edge.

Common mistakes with Missing Data

• Back-filling or interpolating prices that are then used to make trading decisions
• Filling a halted or circuit-locked day with a plausible price and booking a fill that could not have happened
• Letting a NaN be coerced to zero, which reads as a total crash in a price or return series
• Deleting rows with missing values when the missingness is correlated with stress or illiquidity
• Using a library default fill without knowing or logging which method it applied
• Aligning multiple markets without reconciling different holiday calendars, creating false gaps or signals

Missing Data: frequently asked questions

Voice search: how people ask about Missing Data

Natural-language questions people ask about Missing Data.

Sources & references

• NSE — Trading Holidays calendar.
• NSE — Circuit Filters / Price Bands (daily price bands and market-wide circuit breakers).