Home / Research Process / Idea Generation
ResearchBeginner

Idea Generation

Idea generation is the first stage of the quant research process, in which a testable trading idea is sourced from an economic or behavioural reason a pattern should exist, rather than discovered by searching historical data until something profitable appears.

Quick answer: Idea generation is the first stage of the quant research process, in which a testable trading idea is sourced from an economic or behavioural reason a pattern should exist, rather than discovered by searching historical data until something profitable appears.

In simple words

Idea generation is deciding what to test before you test it. A good idea starts with a plausible reason the market should behave a certain way, such as a known behavioural bias or a structural feature of how an exchange works. A bad idea is one you found only because it happened to make money in the past, with no reason behind it. Starting from a reason is what separates genuine discovery from fooling yourself with noise.

Purpose

This stage exists because the order in which you think matters: forming a hypothesis first and then testing it is science, while searching data first and inventing a story afterwards is data mining dressed up as research.

Visual explanation

Idea Generation

The quant research pipeline, from idea generation through validation to a keep-or-kill decision.

Research WorkflowHypothesisDataRulesBacktestValidateReview /iterateiterate

Professional explanation

Reason first, data second

The defining discipline of honest research is that the idea comes with a reason before it meets the data. A sound idea answers the question of why an edge should exist and, just as importantly, why it should persist: who is on the other side of the trade, and what structural or behavioural force keeps them there. When the reason is stated up front, the backtest becomes a test of a claim you can be wrong about. When you skip the reason and let the data suggest the idea, you have inverted the scientific method and every subsequent test is contaminated by the fact that you chose the pattern because it already worked.

The three honest sources of ideas

Genuine ideas tend to come from three places. The first is behavioural: recurring human errors such as overreaction, anchoring or disposition effects that create temporary mispricings. The second is structural: mechanical features of markets such as index reconstitution flows, expiry-day settlement mechanics, or forced rebalancing that push prices for non-informational reasons. The third is risk-based: a return that is really the premium for bearing a risk others avoid, such as carry or volatility selling. Each source gives you a story that can be checked against data and, crucially, against the future.

The danger of data-mined ideas

If you scan thousands of indicator combinations and keep the ones that were profitable, you will always find some, even in data that is pure noise. This is not a search for truth; it is a guarantee of false positives, because with enough trials random patterns clear any threshold. The resulting idea has no reason to work tomorrow because it never worked for a reason, only by chance. The whole edifice of out-of-sample testing, walk-forward analysis and multiple-testing correction exists to defend against exactly this failure mode.

Breadth of sourcing versus depth of conviction

Casting a wide net for ideas is healthy; testing every idea that crosses the net is not. Professional researchers read academic literature, watch market microstructure, talk to practitioners and observe their own trading, but they subject each candidate to a why test before it consumes any data budget. The scarce resource in research is not ideas, which are cheap, but the number of independent tests the data can support before multiple testing erodes its statistical value. Spending that budget on ideas with a prior reason to work is what keeps the process honest.

The prior belief matters

A useful mental model is Bayesian: a backtest result should update a prior belief, not create one from nothing. If you had a genuine reason to expect an edge, a positive backtest is meaningful evidence. If you had no prior at all and simply found the pattern in the data, the same backtest is weak evidence because the search process itself makes flattering results likely. Stating your prior, and how strongly you held it before seeing the numbers, is one of the most effective guards against self-deception in the entire research process.

Documenting the idea before testing

The practical habit that operationalises all of this is a written idea log. Before running a single backtest, record the hypothesis, the economic or behavioural rationale, the instruments and period you intend to use, and what result would make you abandon the idea. This timestamped record prevents the very human tendency to rewrite the rationale after seeing the results, and it turns a vague hunch into a falsifiable claim that the rest of the pipeline can honestly evaluate.

Practical example

Illustrative example (Indian market)

A trader notices that Bank Nifty often reverses sharply in the final hour on weekly expiry days. Before testing anything, they write down a rationale: option writers hedging large positions into settlement may push the index away from and then back toward the maximum-pain strike, a structural flow rather than a forecast. This is a testable, reason-first idea. Contrast it with a second trader who runs a script over five years of Nifty data trying 400 moving-average crossover pairs and keeps the 13-by-48 combination because it showed the highest return on capital of Rs 5,00,000. The first idea can be validated because it made a claim in advance; the second is almost certainly fitted noise, because with 400 trials a good-looking result was guaranteed regardless of whether any edge exists.

Structural ideas are often the most durable in Indian markets: SEBI-mandated index rebalancing, monthly F&O expiry mechanics on NSE, and STT and stamp-duty frictions all create predictable, non-informational pressures. An idea grounded in one of these has a reason to persist that a data-mined indicator pattern simply does not.

Limitations

  • A plausible-sounding rationale can still be wrong; a good story is necessary but not sufficient and must still survive out-of-sample testing
  • Human researchers are skilled at inventing a rationale after the fact, so the reason must be recorded before the data is touched
  • Structural edges can disappear when the structure changes, for example when a regulator alters expiry or settlement rules
  • Idea generation cannot tell you the magnitude of an edge, only whether one is worth testing
  • A genuine reason does not protect against the edge being too small to survive real-world costs and slippage

Common mistakes

  • Searching historical data for profitable patterns first and constructing a rationale only afterwards
  • Treating a backtest that started from no prior belief as strong evidence of an edge
  • Testing dozens of ideas on the same data without accounting for the multiple-comparisons problem
  • Confusing a compelling narrative with a validated edge and skipping straight to live trading
  • Failing to write the idea and its rationale down before testing, then rewriting the story to fit the result
  • Ignoring who is on the other side of the trade, which usually reveals whether an edge can persist

Professional usage

Professional research desks treat ideas as hypotheses with a stated economic rationale and a documented prior, logged before any data is touched. They deliberately limit how many ideas are tested on a given dataset to preserve its statistical value, prioritising candidates with a structural or behavioural reason to persist over patterns that merely fit history. The culture is adversarial toward one's own ideas: the researcher's job is to try to kill the idea cheaply, and only ideas that survive that scrutiny earn a full validation budget.

Key takeaways

  • A good trading idea starts with a reason it should work, not with a profitable backtest
  • The three honest sources are behavioural, structural and risk-based rationales
  • Searching data first and inventing a story later manufactures false positives
  • Write the hypothesis and rationale down before testing to prevent hindsight rewriting
  • A backtest should update a prior belief, not create one from nothing

Frequently asked questions

What is idea generation in quant research?
It is the first stage of the research process, where you decide what to test and why before running any backtest. An honest idea starts from an economic, behavioural or structural reason a pattern should exist, so that the later test can genuinely confirm or refute a claim rather than simply rediscover a pattern you already found in the data.
Why should a trading idea start with a reason?
Because a reason is what gives the idea a chance of persisting into the future. If a pattern worked in the past for an identifiable cause, that cause may still operate tomorrow. If it worked for no reason, there is nothing to expect it to continue, and the backtest result is likely just noise that happened to look profitable.
What are the main sources of good trading ideas?
Three sources tend to produce durable ideas: behavioural biases that create recurring mispricings, structural market mechanics such as index rebalancing or expiry flows, and risk premia where a return compensates for bearing a risk others avoid. Each provides a testable rationale you can check against both history and the future.
What is wrong with finding ideas by searching data?
Searching data for profitable patterns and keeping the winners guarantees false positives, because with enough trials random noise will clear any threshold. The idea then has no reason to work going forward, and standard defences like out-of-sample testing exist precisely to expose such data-mined results.
Is a good rationale enough to trust an idea?
No. A plausible rationale is necessary but not sufficient. It earns the idea a place in the testing queue, but the idea must still survive out-of-sample validation, robustness checks and realistic cost assumptions before it can be trusted. A convincing story with a failing backtest is still a failed idea.
What is data mining in the context of idea generation?
Data mining here means letting the historical data suggest the strategy, typically by testing many variations and selecting the best-performing one. It inverts the scientific method because the pattern is chosen after seeing that it worked, which makes the result look strong while carrying little genuine predictive value.
How does a prior belief relate to a backtest?
A backtest is best treated as evidence that updates a prior belief. If you had a genuine reason to expect an edge, a positive result meaningfully strengthens that belief. If you had no prior and found the pattern by searching, the same result is weak evidence because the search itself makes flattering outcomes likely.
Why write down an idea before testing it?
Writing down the hypothesis, rationale and abandonment criteria before testing creates a timestamped record that prevents you from rewriting the story to fit the results afterwards. It converts a vague hunch into a falsifiable claim that the rest of the pipeline can evaluate honestly.
What does it mean to ask who is on the other side of the trade?
It means identifying the counterparty whose behaviour creates your edge and why they keep acting that way. If you can name a hedger, a forced seller or a behaviourally biased participant supplying your return, the edge has a reason to persist. If no such counterparty exists, the edge is probably illusory.
Are structural ideas more reliable than indicator patterns?
Generally yes, because structural ideas rest on mechanical features of the market such as rebalancing flows or settlement mechanics that have a clear cause. Indicator patterns discovered by search often have no cause at all. Structural ideas can still fail if the underlying mechanism changes, for example after a rule change.
How many ideas should I test on one dataset?
As few as the rationale justifies. Every independent test spends some of the dataset's statistical value, and testing many ideas raises the chance that one looks good by luck. Prioritising ideas that have a prior reason to work keeps that limited testing budget from being wasted on noise.
Can idea generation tell me how profitable a strategy is?
No. This stage only decides whether an idea is worth testing and gives it a rationale. The magnitude of any edge, and whether it survives costs, is determined later by rule definition, backtesting and validation. A strong idea can still turn out to have an edge too small to trade.
How is idea generation different from strategy development?
Idea generation produces the hypothesis and its rationale, whereas strategy development turns that hypothesis into concrete, testable rules and parameters. Idea generation is about what to test and why; the later stages are about exactly how to test it and whether the result holds up.
What role does academic literature play in idea generation?
Academic and practitioner literature is a rich source of candidate ideas with pre-stated economic rationales, such as documented risk premia or behavioural effects. It is valuable because the rationale exists independently of your own data, though published effects can decay after discovery and must still be validated on your market and period.
How do I know if an idea is data-mined or genuine?
Ask whether you could have stated the idea, and a reason for it, before ever looking at the data. If yes, it is a genuine hypothesis. If the idea only exists because you searched until something worked, it is data-mined, and its backtest should be treated with deep suspicion.

Voice search & related questions

Natural-language questions people ask about Idea Generation.

What is idea generation in trading research?
It is deciding what to test and why before you run any backtest. A good idea starts with a real reason the market should behave a certain way, not with a pattern you found by searching.
Why is starting with a reason so important?
Because a reason is what gives an edge a chance of lasting. If something worked for a cause that still operates, it may keep working. If it worked for no reason, it was probably just luck.
What is data mining and why is it bad?
Data mining is letting the data pick your strategy by testing lots of variations and keeping the winner. It is bad because random patterns always look profitable if you try enough of them.
Where do good trading ideas come from?
From three honest sources: human behaviour like overreaction, market structure like expiry flows, and risk premia where a return pays you for taking a risk others avoid.
Should I write my idea down before testing it?
Yes. Writing the idea and its reason down first stops you from inventing a story afterwards to fit whatever the backtest happens to show.
Is a good story enough to trust an idea?
No. A good story only earns the idea a test. It still has to survive out-of-sample checks and real trading costs before you can trust it.

Sources & references

    Last reviewed 12 July 2026. Educational content only — not investment advice. Markets and rules change; verify current conventions with SEBI, NSE/BSE and your broker.

    ← Previous
    Continuous Improvement
    Educational content only — not investment advice. Examples use illustrative numbers and simplified models. Backtested results are hypothetical and trading derivatives involves substantial risk. See our Risk Disclosure and SEBI Disclaimer.