Playbooks · the research team · Guide 11
Playbooks: how to research a strategy with the team
In one lineA strategy idea earns a place on your team by climbing five rungs, from a hypothesis you could fail, through locked rules and a counted sample with costs, to fresh data and small size, with one seat designing the test and another reporting only what your own rows show.
What you'll get
- The research method in six steps: hypothesis, rules, evidence, costs, sample, out-of-sample.
- The six mistakes that manufacture an edge that isn't there, and the guard against each.
- The evidence ladder: five rungs, and what each one does and doesn't prove.
- Two research prompts: the Strategy Scout and the Evidence Checker.
1What a playbook is, and what it isn't
Everything up to Guide 10 assumed you arrived with a setup. This shelf is for the question underneath: how do you know a setup deserves your risk? And the related one: how do you look at a new idea without fooling yourself?
A playbook here is a research brief. It states a well-known public idea in one line, gives the public reasoning for why it might work, lays out a rule template with blanks, and lists what kills it. Guide 12 and 13 hold eight of them. None comes with a result, a success rate or a recommended setting. That is on purpose, for three reasons.
Published edges fade. A study in the Journal of Finance looked at 97 published stock-return predictors and found their returns were clearly lower outside the original sample period, and lower again once the research was public ("Does Academic Research Destroy Stock Return Predictability?", Journal of Finance, 2016). A pattern everyone has read about is a pattern many people are already trading.
Backtests are easy to flatter. A paper on backtest overfitting showed that if you try enough variations of a rule on the same history, a good-looking result is almost certain to appear by chance, and then to disappoint on new data ("Pseudo-Mathematics and Financial Charlatanism", Notices of the American Mathematical Society, 2014). Another study made the matching point about published research: after hundreds of tested factors, the usual bar for "significant" is too low ("… and the Cross-Section of Expected Returns", Review of Financial Studies, 2016).
Your market isn't their market. A study on a US equity ETF over twenty years says little about gold in the London hour on your broker's spread this year. Costs alone can turn a published result upside down.
So the honest position is this: most ideas you test will not survive costs and fresh data. The method below is built to find that out cheaply, on demo, before the market tells you at full size.
2The method, in six steps
- Hypothesis. One sentence that could be wrong. "On {market}, during {session}, when {a condition you can check}, then {a measurable outcome} within {a time}." If you can't write it, you have a feeling, not an idea.
- Rules. Context, trigger, entry, stop, target, time exit, when not to trade. Every number filled in before you look at any results, then dated and locked. The playbook brief template (Guide 16) is the form.
- Evidence needed. Decide what you'll count, and what result means "drop it", before you start. These are your falsifiers. Written afterwards, they are excuses.
- Costs. Commission plus spread or slippage, subtracted from every trade. A strategy with many small trades can be right about direction and still lose to costs.
- Sample size. At least 30 trades before you look at totals. Thirty is a floor for looking, not a proof. It can tell you an idea is clearly dead. It cannot separate a small edge from a lucky month.
- Out-of-sample. Results from the period you used to choose your settings are in-sample: they have seen the answers. Only results from data the rules never touched count as a test. Walk-forward is the simplest version: lock the rules, then trade them forward on demo and count again.
3The Strategy Scout
The Scout does step 1 to 3 with you. You give it a vague idea. It returns a hypothesis, a rule template whose every number is a blank, the evidence needed, and the falsifiers. It is forbidden to say the idea works, to pick your settings or to cite a source it isn't sure of.
Open a separate research chat in your team workspace. Not chats A to D, and not before a session.
Strategy Scout prompt (research chat)
Role: Strategy Scout.
Turn my idea into a testable hypothesis and a test plan. You are designing a test I could fail, not making a case for the idea.
Inputs:
- My idea, in my words: "{the idea}"
- Market and instrument: {e.g. MNQ futures / EURUSD / gold}
- Session and time zone: {e.g. London open, 08:00–10:00 London}
- Timeframe I would trade it on: {e.g. 5-minute bars}
- What I can test with: {e.g. bar replay on my charting platform, a spreadsheet, a demo account}
- Rough round-trip cost per trade if I know it (commission + typical spread or slippage): {amount, or "unknown"}
You may not:
- Say or imply that the idea works, has an edge, or is likely to. No success rates, no expected returns, no "studies show" without a source I can check.
- Cite a paper, book or statistic unless you are certain of author and title. If you mention one, add "(unverified — check it yourself)". Never invent a citation.
- Choose parameter values for me. Where a number is needed, leave a blank {I choose: ...} and say what the choice trades off. If you show a value to explain a format, label it "placeholder".
- Recommend trading it. The output is a plan for testing on demo or replay.
Reply in exactly this format:
SCOUT MEMO — {short name for the idea}
1. Hypothesis: "On {market}, during {session}, when {condition I can check on a chart or clock}, then {measurable outcome} within {time horizon}, more often or by more than on days without the condition." One sentence.
2. Why it might exist: the plainest public reasoning, in two lines. Who is on the other side of the trade, and why would they keep paying? If you can't say, write "no clear reason", which is a warning sign.
3. Why it might not: two lines. Include costs and crowding.
4. Rule template: context rules, trigger, entry, stop, target, time exit, no-trade conditions. Every number is a blank I fill in BEFORE I look at any results.
5. Locked before testing: the list of every blank in section 4. I write my values, date them, and do not change them during the test.
6. Evidence needed: what to count for each trade (date, direction, entry, exit, result in R, cost), the minimum number of trades before I look at totals ({I choose: at least 30}), and which periods to cover (including ones that should not suit the idea).
7. Costs: what to subtract from each trade, and what result per trade would be wiped out by my stated cost.
8. Falsifiers, written now: two or three results that mean "drop it" (for example: total R at or below zero after costs over the sample; the result disappears when the best day is removed; it only shows up with one exact setting).
9. Ways this test could fool me: which of these apply and where — look-ahead, choosing settings after seeing results, tiny sample, ignoring costs or slippage, survivorship, best-day dependence.
10. Next rung: which rung of the evidence ladder this idea is on, and the one action that moves it up.
If my idea is too vague to write section 1, stop after asking me up to 3 questions.
Before you send: confirm that no line says or implies the idea works, that every number in section 4 is a blank, and that section 8 can actually be failed.4Six mistakes that fake an edge
Every one of these makes a test look better than the idea is. Each has a plain guard.
| Mistake | How it sneaks in | The guard |
|---|---|---|
| Look-ahead | The rule uses something you couldn't have known at the time: the day's high, the close of a bar that hadn't closed, a level drawn afterwards. | In replay, decide before you reveal the next bar. Write the decision time on every row. |
| Choosing settings after seeing results | You try a 15-minute range, then 30, then 10, and keep the best. The winner has been fitted to that history. | Lock every blank before testing. A changed blank is a new brief and a fresh count. |
| Tiny samples | Eight trades, six of them winners, and it feels like proof. | No totals before 30 trades. Under 30, the report says "descriptive only". |
| Ignoring costs and slippage | Results counted from chart prices, as if every order filled at the level with no spread. | A cost in R on every row. Find your real spread at the hour you trade. |
| Survivorship | You test on the instruments, months or examples you remember, which are the ones where it worked. | Log every qualifying day in the period, taken or not. Include a stretch that shouldn't suit the idea. |
| Best-day dependence | The whole result is one or two outsized days. Remove them and nothing is left. | Recompute the total without the best day and without the best trade. Check whether the sign changes. |
5The evidence ladder
Climb the ladder one rung at a time
- 1. Idea: Write the hypothesis and the falsifiers.
- 2. Rules: Fill and date every blank in the brief. Have the Bear attack it.
- 3. Sample: 30 or more trades on replay or demo, every qualifying day logged, costs on every row.
- 4. Walk-forward: The same locked rules, forward, on demo, on days the rules never saw. Another 30 or more.
- 5. Small size: Live, at the smallest size your platform allows, for a number of trades fixed in advance. Through the full team routine.
| Rung | What you do | What it shows | What it doesn't |
|---|---|---|---|
| 1. Idea | Write the hypothesis and the falsifiers. | That you know what you're claiming. | Anything about the market. |
| 2. Rules | Fill and date every blank in the brief. Have the Bear attack it. | That the idea can be traded the same way twice. | That it should be. |
| 3. Sample | 30 or more trades on replay or demo, every qualifying day logged, costs on every row. | Whether the idea is clearly dead, and how its losing runs look. | An edge. In-sample results flatter. Demo fills are kinder than live ones. |
| 4. Walk-forward | The same locked rules, forward, on demo, on days the rules never saw. Another 30 or more. | Whether rung 3 was fitted to its own history. | How you will behave with money on it. |
| 5. Small size | Live, at the smallest size your platform allows, for a number of trades fixed in advance. Through the full team routine. | Real fills, real costs, and your own execution. | That it will continue. Nothing shows that. |
Move up only when the rung you're on hasn't hit a falsifier. Move down, or drop the brief, when it has. An idea that dies on rung 3 has cost you some evenings. That is the ladder doing its job.
After rung 4, a brief becomes a setup sheet in your rules file, and the trading team on Guide 03 to 08 takes over.
6The Evidence Checker
The Checker does steps 4 and 5 and part of 6. You paste rows: a replay log, a demo log, your live journal or a backtest export. It reports the sample size, whether costs are in, totals with the sums written out, what's left without the best day, the longest losing run, and a list headed "what this does NOT prove". It computes from your rows only.
Two cautions. Language models can misadd a column, which is why the prompt demands visible arithmetic and a cross-check, and why you verify two sums yourself. And the report describes a sample. It has no opinion on the future.
Evidence Checker prompt (research chat; use a reasoning mode if your tool has one)
Role: Evidence Checker.
Report what the rows below show. Compute from these rows only, show every sum, and be exact about what the sample cannot tell me.
Inputs:
- Where the rows come from: {my live journal / demo log / bar-replay log / backtest export from {tool}}
- What they test: {setup name, market, session}
- Were the rules fixed before these trades, or adjusted along the way? {fixed on {date} / adjusted / not sure}
- Do the results include costs (commission, spread, slippage)? {yes / no / not sure}
- If no: my round-trip cost per trade is about {amount in R, or "unknown"}
- Rows (pasted by me, unedited). Columns: {e.g. trade_id, date, result_r, rules_followed}
{paste rows}
You may not:
- Use any number that is not in the rows or derivable from them. Do not estimate a missing value. Do not recall "typical" figures for this kind of strategy.
- Call the strategy good, bad, promising, or say it has or lacks an edge. You report the sample.
- Express any result as a chance of future success, or project it forward ("at this rate...").
- Skip the arithmetic. Every total is shown as a sum I can retype into a calculator.
- Round in a way that changes a sign or hides a difference.
Reply in exactly this format:
EVIDENCE REPORT — {setup name} — {n} rows
1. Sample: number of trades, first and last date, number of trading days. If under 30 trades, the first words are "Under 30 trades: descriptive only."
2. Source quality: live / demo / replay / backtest, and whether the rules were fixed beforehand. If adjusted or unsure, say the results are in-sample and likely flattered.
3. Totals: winners {n}, losers {n}, flat {n}. Total R with the sum written out. Average R per trade = total ÷ n.
4. Costs: included or not. If not, total R minus (n × cost) = net, sum written out. If cost is unknown, write "net result unknown" and do not guess.
5. Best-day dependence: R per day, listed. Total R without the best day. Total R without the best single trade. State whether the sign changes.
6. Losing runs: the longest run of consecutive losers (with IDs), and the deepest fall from a running peak in R (peak value, trough value, IDs).
7. Rules followed versus broken, if the column exists: n and total R for each, sums written out. Descriptive only.
8. Checks I could not do: missing columns, duplicate IDs, rows out of date order, results not in R.
9. What this does NOT prove: at least four lines specific to this sample (size, period covered, source, costs, whether rules were fixed, dependence on one day).
10. The next piece of evidence that would matter most: one line.
If a row is unreadable, list it under 8 and leave it out of every sum. Say that you did.
Before you send: add the per-day totals in section 5 and confirm they equal the total in section 3. If they don't, find the error before replying.7Where the agent team comes in
- Researcher turns the filled brief into yes/no rules, exactly as on Guide 03. If two people could disagree about a rule on the same chart, you can't count it.
- Bear attacks the brief before you test it. Each playbook lists what the Bear should go after. Paste the brief into a fresh chat with the Bear prompt and the line: "Scenario under review: testing this brief on {market}. Attack the test design, not a trade."
- Risk Manager sets the size for rung 5 from your risk sheet. Smallest size, a fixed number of trades.
- Weekly Reviewer stays out of it. A week is too short for strategy questions.
8Worked example: one idea, three rungs, one honest ending
Illustrative. The idea, the counts and the outcome are invented to show the method. They are not findings about any market.
Rung 1. A trader has an idea: "Gold breaks out of its overnight range at the London open." The Strategy Scout returns a hypothesis, a template with eleven blanks, and three falsifiers: (a) total R after costs at or below zero over the sample, (b) total turns negative when the best day is removed, (c) the result needs one exact range window.
Rung 2. The trader fills the eleven blanks in a playbook brief and dates it. The Researcher flags "a clear break" as vague. It becomes "a 5-minute close beyond the range edge". The Bear's best attack: "The spread at the session change isn't in your cost line." The trader watches the spread at 08:00 London for a week and writes the real figure into the brief.
Rung 3. Six weeks of bar replay, one evening at a time, deciding before each bar is revealed. 34 qualifying days logged, taken or not. The Evidence Checker reports:
- 34 trades. Costs included on every row.
- A small positive total after costs.
- Without the best day, the total is negative. The sign changes.
- Longest losing run: 6.
- "What this does NOT prove": five lines, starting with "that the total is repeatable; more than all of it came from one day".
Falsifier (b) is met. The trader checks the two sums by hand, confirms them, and drops the brief. No new range window, no "one more filter". Changing a blank now would be choosing settings after seeing results.
Six evenings, no money, and one idea crossed off with a reason attached. Most research ends like this. The trader's existing setup, the one already in the journal, goes through the Evidence Checker next.
9Common mistakes
- Researching on a trading morning. New ideas and live sessions don't share a chat or a day.
- Asking the AI whether a strategy works. It will produce a confident answer assembled from things it has read. Ask it to design the test. Then run the test.
- Treating a cited paper as permission. A study describes its own market, period and costs. It is a reason to test, never a result you can borrow.
- Moving a blank mid-test. New brief, new date, count from zero.
- Skipping rung 4 because rung 3 looked good. Rung 3 is the one that flatters.
- Testing five ideas at once. One brief under test at a time.
10Checklist
Sources
Each was checked for title, authors and year. They are reasons for caution, not trading advice.
- R. "Does Academic Research Destroy Stock Return Predictability?", Journal of Finance, 2016.
- "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance", Notices of the American Mathematical Society, 2014.
- "… and the Cross-Section of Expected Returns", Review of Financial Studies, 2016.
Educational only. Not financial advice. Trading involves risk.