ac-backtest-integrity
Auditing whether a trading backtest can be believed: multiple-testing deflation, look-ahead and survivorship leakage, sample and regime adequacy, execution-cost honesty, placebo and null tests, pre-registration, and the decision rule that kills a strategy. Audits evidence; never recommends a strategy. Use when asking "is this backtest real", "validate my strategy", "deflated Sharpe", "look-ahead bias", "pre-register a study". Not for data collection (ac-market-data-collection), forensics (ac-cascade-forensics), or the trade record (ac-crypto-trading-discipline).
- License 许可
- MIT
- 最近更新
- 2026-11-15
- Stars
- ★ 81
- Forks
- 17
01 前言 Front Matter
- name
- ac-backtest-integrity
- license
- MIT
- metadata
- { "domain": "crypto-trading", "status": "seedling" }
- description
- Auditing whether a trading backtest can be believed: multiple-testing deflation, look-ahead and survivorship leakage, sample and regime adequacy, execution-cost honesty, placebo and null tests, pre-registration, and the decision rule that kills a strategy. Audits evidence; never recommends a strategy. Use when asking "is this backtest real", "validate my strategy", "deflated Sharpe", "look-ahead bias", "pre-register a study". Not for data collection (ac-market-data-collection), forensics (ac-cascade-forensics), or the trade record (ac-crypto-trading-discipline).
02 资源文件 10
- 许可证LICENSE1.0 KB
- 参考文档SOURCE.md377 B
- 参考文档subskills/cost-and-execution.md4.8 KB
- 参考文档subskills/look-ahead-and-leakage.md4.7 KB
- 参考文档subskills/multiple-testing.md4.7 KB
- 参考文档subskills/placebo-and-null.md4.7 KB
- 参考文档subskills/pre-registration.md5.1 KB
- 参考文档subskills/sample-and-regime.md5.0 KB
- 参考文档subskills/verdict-and-kill.md4.7 KB
- 参考文档TRUST.auto.yaml2.2 KB
03 SKILL.md 正文
Backtest Integrity and Validation Desk
Overview
The Backtest Integrity skill provides a rigorous framework for auditing, validating, and stress-testing quantitative trading strategies. In quantitative finance and digital asset markets, apparent predictive power is frequently an artifact of data snooping, multiple testing, look-ahead bias, and unrealistic cost assumptions. This skill establishes an epistemological defense against false discoveries, ensuring that trading strategies deployed for operational consideration possess genuine statistical edge rather than phantom alpha generated by flawed backtest engineering.
Rooted in academic consensus—including Bailey and López de Prado on the Deflated Sharpe Ratio, Harvey and Liu on multiple testing haircuts, White's Reality Check, and empirical findings on alpha decay (arXiv 2605.16895)—the skill enforces strict pre-registration protocols, robust null hypothesis testing, realistic transaction cost modeling, and definitive binary kill verdicts.
Primary sources for the claims in this section: https://doi.org/10.3905/jpm.2014.40.5.094 and https://doi.org/10.3905/jpm.2015.42.1.013.
When to use
- Pre-registering hypothesis specifications, parameter grids, and decision rules prior to data execution.
- Issuing definitive PROCEED or KILL verdicts and enforcing re-registration barriers for failed strategies.
When NOT to use
- Collecting raw market data, historical price feeds, or building general data-engineering pipelines (use
ac-market-data-collection). - Performing forensic post-mortem analysis on historical liquidation crashes or market cascades (use
ac-cascade-forensics). - Recording individual live trade executions, position sizing ledgers, or discretionary trading journals (use
ac-crypto-trading-discipline). - Deploying live trading algorithms or managing real-time exchange API connections and order routing infrastructure.
Index of Subskills
| Leaf | Domain segment | Scope (one line) |
|---|---|---|
multiple-testing.md |
crypto-trading/multiple-testing |
How many trials actually happened; deflated Sharpe; Harvey-Liu haircuts; White's Reality Check; the 42-cell/41-window arithmetic as a worked example. |
look-ahead-and-leakage.md |
crypto-trading/look-ahead-and-leakage |
Point-in-time discipline (universe, restatements, feature availability), survivorship, and how a single leaked feature invalidates a backtest. |
sample-and-regime.md |
crypto-trading/sample-and-regime |
Regime heterogeneity (herd vs arbitrage regimes with identical distributions), sample size vs effect size, the one-year/five-ticker limits, and subcriticality's implication (no self-sustaining cascade to simulate). |
cost-and-execution.md |
crypto-trading/cost-and-execution |
Commission, spread, impact, funding, and the Sharpe 0.43→0.22 example; fill assumptions that flatter a backtest. |
placebo-and-null.md |
crypto-trading/placebo-and-null |
Shifted placebos, permutation and bootstrap nulls, the collider problem (sample conditioned on its own outcome), and what a null must be able to reject. |
pre-registration.md |
crypto-trading/pre-registration |
Fixing the analysis before the data: hypothesis, window, cells, test statistic, decision rule; the 42-cell/Bonferroni example; versioning the pre-registration. |
verdict-and-kill.md |
crypto-trading/verdict-and-kill |
The decision rule that kills a strategy, documenting the kill, and the conditions under which a killed strategy may be revisited (new data, new pre-registration — never a re-run on the same sample). |
Status and Honest Limits
This skill operates under strict epistemic boundaries. It evaluates quantitative strategies based on published statistical methods, econometric standards, and empirical findings. It does not predict future market returns, guarantee trading profitability, or endorse specific financial assets. All backtest validations are probabilistic assessments subject to model risk, parameter instability, and structural regime shifts.
Sources
- Bailey, D. H., & López de Prado, M. (2014). The Deflated Sharpe Ratio. Journal of Portfolio Management. https://doi.org/10.3905/jpm.2014.40.5.094
- Harvey, C. R., & Liu, Y. (2015). Backtesting. Journal of Portfolio Management. https://doi.org/10.3905/jpm.2015.42.1.013
- White, H. (2000). A Reality Check for Data Snooping. Econometrica. https://doi.org/10.1111/1468-0262.00152
- arXiv:2605.16895. Alpha Illusion. https://arxiv.org/abs/2605.16895
- Open Science Framework (OSF). Pre-Registration Standards. https://osf.io/prereg/
04 同仓库更多 Skills
troves
Deposit and withdraw from Troves DeFi yield vault strategies on Starknet. View available strategies, check positions, and manage vault deposits. Use this skill when the user mentions Troves, vault, yield vault, DeFi vault, strategy, yield farming, vault deposit, vault withdraw, earning yield through vaults, passive yield strategies, or wants to deposit into or withdraw from a yield strategy — even if they don't say "Troves" explicitly.
trade
Swap tokens on Starknet via Fibrous DEX aggregator. Supports per-provider selection (avnu, ekubo, or auto to race all). Use this skill when the user wants to swap, exchange, trade, convert, buy, or sell one token for another on Starknet. Also use when the user asks about token prices, exchange rates, DEX routing, best swap route, or wants to check how much they would receive for a given amount — even if they don't explicitly say "swap".
staking
Stake, unstake, claim, and compound rewards for STRK, WBTC, tBTC, SolvBTC, and LBTC on Starknet. View validators, pools, and staking positions. Use this skill when the user mentions staking, delegating, validators, staking pools, rewards, compounding, unstaking, earning yield, passive income on crypto, or wants to earn on their STRK or BTC tokens — even if they don't say "stake" explicitly.
send
Send STRK, ETH, or any ERC-20 token to a Starknet address. Supports simulation (dry-run) before execution. Use this skill when the user wants to send, transfer, pay, tip, or move tokens to another wallet address on Starknet. Also trigger when the user mentions sending funds to someone, making a payment, transferring assets, or moving crypto to a specific address — even if they just say "pay 10 STRK to 0x...".
portfolio
View a comprehensive DeFi portfolio dashboard and optimize portfolio allocation via automated rebalancing. Use this skill when the user wants an overview, summary, dashboard, total value, net worth, portfolio rebalancing, target allocation, or wants to see all their DeFi positions across Starknet at once. Also trigger when the user asks "what do I have", "show me everything", "how much am I worth", "my positions", "my investments", "rebalance my portfolio", "I want 50% ETH", "optimize my holdings", or any request for a holistic view of their Starknet holdings — even if they don't explicitly say "portfolio".
multi-swap
Execute multiple token swaps in a single Starknet transaction via Fibrous (default). Supports per-provider selection (avnu, ekubo, or auto to race all). Supports 2-3 swap pairs bundled into one multicall. Use this skill when the user wants to swap multiple token pairs at once, do batch swaps, execute several trades simultaneously, perform parallel swaps, or do bulk trading in one transaction. Also trigger when the user mentions "two swaps", "three swaps", "swap X and Y at the same time", or wants to convert multiple tokens in a single call.