Home / Data Quality / Data Cleaning
ProcessIntermediate

Data Cleaning

Data cleaning is the disciplined pipeline that turns a raw market feed into a series fit to backtest on — correcting bad ticks, handling missing values, applying corporate-action adjustments and aligning timestamps in a deliberate order — and its reliability rests on doing those steps reproducibly and without introducing look-ahead.

Quick answer: Data cleaning is the disciplined pipeline that turns a raw market feed into a series fit to backtest on — correcting bad ticks, handling missing values, applying corporate-action adjustments and aligning timestamps in a deliberate order — and its reliability rests on doing those steps reproducibly and without introducing look-ahead.

In simple words

Data cleaning is everything you do to raw market data before you trust a backtest run on it: removing erroneous prices, patching gaps sensibly, adjusting for splits and dividends, and fixing timestamps. It is best thought of as a pipeline — a fixed sequence of steps applied the same way every time. Getting the order right matters, because cleaning one problem before another can either fix the data or quietly corrupt it further.

Purpose

A data-cleaning pipeline exists because every other data-quality problem — bad ticks, missing values, corporate actions, timestamps — has to be resolved before signals are computed, and doing it ad hoc produces backtests nobody can reproduce or trust.

Visual explanation

Data Cleaning

A raw feed passing through timestamp normalisation, outlier correction, missing-value handling, corporate-action adjustment and alignment into a clean, reproducible dataset.

Data PipelineSource / FeedIngestNormaliseCorp-actionAdjustValidateStoreServe

Professional explanation

Cleaning is a pipeline, not a one-off

Raw data carries several distinct faults at once, and each has its own remedy, so cleaning is a sequence of well-defined stages rather than a single scrub. Treating it as a pipeline — ingest, validate, correct, adjust, align, store — makes the process repeatable and auditable, so the same raw input always yields the same clean output. This matters because a backtest is only as trustworthy as the data beneath it, and an undocumented, hand-edited dataset cannot be reproduced or debugged. The pipeline should be code, versioned and rerunnable, not a series of manual spreadsheet fixes.

Order of operations

The sequence of cleaning steps is not arbitrary; doing them in the wrong order corrupts the result. Outlier detection should generally run on raw prices before corporate-action adjustment, because adjustment rescales the series and can turn a genuine split into an apparent outlier or mask a real bad tick. Timestamp normalisation should precede any cross-series alignment, so events are ordered correctly before they are joined. Missing-value handling interacts with both: you must know whether a gap is a halt, a holiday or a fault before deciding to forward-fill, mark untradeable or leave it. A defensible order is: normalise timestamps, detect and correct outliers on raw data, classify and handle missing values, apply corporate-action adjustments, then align and resample.

Cleaning must not introduce look-ahead

The cardinal rule is that no cleaning step may use information from after the point being cleaned in a way a live system could not. Back-filling a gap, interpolating across it, standardising with full-sample statistics, or applying back-adjustment factors that depend on future actions all inject future knowledge. A pipeline built for backtesting must either operate point-in-time — cleaning each date using only data available by then — or be restricted to transformations that are provably causal. This is the difference between a dataset that is merely tidy and one that is honestly usable for simulation.

Reproducibility and provenance

A clean dataset is only valuable if you can say exactly how it was produced. That means recording the raw source and its version, the cleaning code and its version, every parameter used, and a log of every correction applied, so any figure in a backtest can be traced back to a raw print. Storing both the raw and the cleaned series lets you re-derive alternatives, reverse a decision, or audit a suspicious result. Without this provenance, two researchers running the same strategy on nominally the same data can get different numbers and never discover why, which quietly undermines every conclusion drawn from the backtest.

Cleaning as a boundary, not a cure

Cleaning improves data but cannot manufacture information that was never captured, and over-cleaning is its own hazard. Smoothing away real gaps, deleting inconvenient outliers, or filling untradeable halts makes the series prettier while making the backtest less honest. The goal is fidelity to the tradeable market, not visual smoothness, so a good pipeline is conservative: it corrects clear errors, flags the ambiguous cases for review, and preserves genuine risk events intact. The output should be documented with its known limitations, because a backtest inherits every assumption baked into the cleaning, and pretending the data is perfect is itself a modelling error.

Ad hoc cleaning vs Pipeline cleaning

AspectAd hoc / manualPipeline
ReproducibleNoYes, same input to same output
Order of stepsInconsistentFixed and justified
Look-ahead controlEasily violatedEnforced by design
AuditabilityCorrections untrackedEvery change logged
Debugging a resultHard or impossibleTraceable to raw source

Practical example

Illustrative example (Indian market)

You assemble three years of NSE data for a multi-stock backtest on capital of Rs 5,00,000, and you clean it as a pipeline. First you normalise all timestamps to IST and confirm each bar lands in the 9:15 to 15:30 session. Next you run outlier detection on the raw prices, flagging a one-tick spike to 49,150 on a Bank Nifty series and a stale zero on a mid-cap, correcting both with logged reasons. Then you classify gaps, marking three circuit-locked days as untradeable rather than filling them. Only then do you apply split and bonus factors from a dated corporate-action master. Had you adjusted for the split first, the pre-split price of Rs 2,000 rescaled to Rs 1,000 might have escaped your outlier filter or masked the real spike, and your bad-tick detection would have run on a distorted series. The order preserved both corrections and made the whole run reproducible from the raw files.

For an Indian equity universe the pipeline must reconcile NSE trading holidays, apply frequent bonus and split adjustments from a corporate-action master, mark small and mid-cap circuit days as untradeable, and normalise IST timestamps, all before any signal runs. Skipping any stage — for example adjusting prices before filtering bad ticks — produces a dataset that looks clean but silently corrupts the backtest.

Limitations

  • Cleaning cannot recover information that the raw feed never captured
  • Over-cleaning smooths away genuine gaps and outliers that are real risk
  • The correct order of operations depends on the data and must be reasoned, not assumed
  • A point-in-time pipeline is more expensive than a single precomputed clean series
  • Every cleaning choice is an assumption the backtest silently inherits

Common mistakes

  • Applying corporate-action adjustment before outlier detection, distorting the bad-tick filter
  • Cleaning data by hand in a spreadsheet so the process cannot be reproduced or audited
  • Using back-fill, interpolation or full-sample statistics in a step, injecting look-ahead
  • Discarding the raw data after cleaning, leaving no way to reverse or re-derive decisions
  • Over-cleaning until genuine news gaps and circuit moves vanish from the series
  • Failing to log corrections, so a suspicious backtest result cannot be traced to its cause

Professional usage

Institutional data engineering treats cleaning as versioned, testable code with a fixed and justified order of operations, run identically every time so the clean output is fully reproducible from the raw source. They enforce causality in every step to prevent look-ahead, store raw alongside cleaned data with full provenance, and log every correction for audit. Crucially they treat cleaning conservatively — correcting clear errors, flagging ambiguous ones, and preserving genuine risk events — and they document the residual limitations so downstream backtests know exactly what assumptions they inherit.

Key takeaways

  • Data cleaning is a fixed, reproducible pipeline, not a one-off manual scrub
  • Order matters: normalise timestamps, filter outliers on raw data, handle gaps, then adjust and align
  • No step may use future information; keep the pipeline causal or point-in-time
  • Store raw and cleaned data with full provenance so any result is traceable and reversible
  • Clean conservatively — preserve real risk events; over-cleaning is as harmful as under-cleaning

Frequently asked questions

What is data cleaning in backtesting?
Data cleaning is the pipeline that turns a raw market feed into a series fit to backtest on, by correcting bad ticks, handling missing values, applying corporate-action adjustments and aligning timestamps. Its purpose is a dataset that faithfully represents the tradeable market.
Why should cleaning be a pipeline rather than manual fixes?
Because a pipeline is repeatable and auditable: the same raw input always yields the same clean output. Manual, hand-edited fixes cannot be reproduced or debugged, so a backtest built on them cannot be trusted or replicated by anyone else.
Does the order of cleaning steps matter?
Very much. Outlier detection should run on raw prices before corporate-action adjustment, and timestamps should be normalised before any cross-series alignment. Doing steps in the wrong order can mask real errors or turn a split into an apparent outlier.
What is a good order of operations for cleaning?
A defensible sequence is: normalise timestamps, detect and correct outliers on raw data, classify and handle missing values, apply corporate-action adjustments, then align and resample. The exact order should be reasoned for the specific data, not copied blindly.
How can data cleaning introduce look-ahead bias?
By using information from after the point being cleaned — back-filling a gap, interpolating across it, standardising with full-sample statistics, or applying back-adjustment factors that depend on future actions. A backtesting pipeline must keep every step causal or point-in-time.
Why store both raw and cleaned data?
So you can re-derive alternatives, reverse a decision and audit a suspicious result. Keeping the raw source with full provenance means any figure in a backtest can be traced back to a real print, which is essential for debugging and trust.
What is data provenance?
Provenance is the recorded history of how a dataset was produced — the raw source and version, the cleaning code and version, the parameters used, and a log of every correction. It lets two researchers reproduce identical numbers and diagnose differences.
Can data cleaning fix any data problem?
No. Cleaning corrects errors and handles known issues, but it cannot recover information the raw feed never captured. Over-cleaning is also harmful, because smoothing away real gaps and outliers removes genuine risk the strategy must face.
What is over-cleaning?
Over-cleaning is removing or smoothing data that is actually real — deleting legitimate news gaps, filling untradeable halts, or erasing genuine outliers — to make the series look tidy. It makes the backtest prettier but less honest and less predictive of live behaviour.
Should cleaning be code or manual?
Code. Versioned, testable, rerunnable code makes the pipeline reproducible and auditable, whereas manual spreadsheet edits cannot be replicated or traced. Treating cleaning as software is what makes a backtest defensible.
How does cleaning relate to look-ahead bias?
Cleaning is a common hidden source of look-ahead, because several intuitive fixes — interpolation, full-sample scaling, back-adjustment — quietly use future data. A disciplined pipeline restricts itself to causal transformations to avoid leaking the future into the past.
Why is reproducibility so important for cleaning?
Because without it, two people running the same strategy on nominally the same data can get different results and never learn why. Reproducible cleaning ties every backtest number to a specific raw source and process, which underpins every conclusion.
Do professional teams clean data differently from retail?
Mainly in discipline. Professionals treat cleaning as versioned code with fixed order, enforced causality, full provenance and logged corrections, and they clean conservatively. The individual fixes are similar, but the reproducibility and auditability are what separate the two.
What should a cleaning pipeline output?
A clean series plus documentation of exactly how it was produced and its known limitations. Because a backtest inherits every assumption baked into cleaning, the output must state those assumptions rather than pretend the data is perfect.

Voice search & related questions

Natural-language questions people ask about Data Cleaning.

What is data cleaning?
It is everything you do to raw market data before trusting a backtest on it — removing bad prices, patching gaps, adjusting for splits and dividends, and fixing timestamps, all in a set order.
Why does the order of cleaning steps matter?
Because fixing one thing before another can either repair the data or wreck it. For example, you should filter out bad prices before adjusting for splits, or the adjustment can hide the errors.
Can cleaning data accidentally cheat?
Yes. Some common fixes, like filling a gap using a later price or scaling with the whole history, sneak future information into the past. A good pipeline only uses data that was available at the time.
Why keep the raw data after cleaning?
So you can trace any result back to a real price and undo a decision if it was wrong. If you throw the raw data away, you can never check or reproduce what you did.
Is cleaner data always better?
No. If you clean too hard you erase real events like news gaps, which are exactly the risks your strategy has to survive. The aim is honest data, not smooth-looking data.
Should I clean data by hand or with code?
With code. Hand edits in a spreadsheet cannot be repeated or checked later, whereas a coded pipeline runs the same way every time and can be traced back to the raw data.

Sources & references

    Last reviewed 12 July 2026. Educational content only — not investment advice. Markets and rules change; verify current conventions with SEBI, NSE/BSE and your broker.

    ← Previous
    Bad Ticks & Outliers
    Educational content only — not investment advice. Examples use illustrative numbers and simplified models. Backtested results are hypothetical and trading derivatives involves substantial risk. See our Risk Disclosure and SEBI Disclaimer.