indexing
Guide Kalshi and Polymarket market, trade, block, and legacy backfill workflows.
- License 许可
- MIT
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- name
- indexing
- license
- MIT
- metadata
- { "disco-role": "operating" }
- description
- Guide Kalshi and Polymarket market, trade, block, and legacy backfill workflows.
- disable-model-invocation
- true
02 资源文件 2
- 参考文档references/troubleshooting.md1.8 KB
- 参考文档references/workflows.md1.4 KB
03 SKILL.md 正文
indexing
Use this sub-skill for data-collection workflows that write Parquet chunks and resume from cursor files. It covers both API-based and blockchain-based collectors.
Use this route when
- The user wants to backfill markets, trades, blocks, or legacy FPMM data.
- The user wants to resume a partially completed run.
- The user wants to understand the shape of a cursor file or chunked Parquet output.
- The user wants to add a new indexer subclass.
Scope
Included
- Kalshi market and trade backfills.
- Polymarket market, trade, block, and legacy FPMM backfills.
- Cursor files, chunk naming, deduplication, and resume behavior.
KalshiClient,PolymarketClient,PolygonClient, andIndexer.load().
Excluded
- Analysis and chart output workflows, which belong to
sub-skills/analysis/. - Dataset packaging and host-tool installation, which belong to
sub-skills/data-ops/.
Read first
../../references/indexing-catalog.mdfor the full indexer list.../../references/api-reference.mdfor the client and base-class APIs.../../references/data-layout.mdfor chunk paths, cursor files, and helper files.../../references/troubleshooting.mdfor cross-cutting failures.
Core workflow
- Identify the indexer name and required inputs.
- Confirm the cursor file and output directory state.
- Provide any required env vars, especially
POLYGON_RPCfor blockchain-backed collectors. - Run the indexer through
uv run main.py indexor directly from the Python class. - Verify that the expected Parquet chunks and cursor files were updated.
Indexer families
Kalshi
kalshi_marketsbackfills market snapshots into chunked Parquet files.kalshi_tradesfetches trades for markets discovered from the market parquet tree and skips already-processed tickers.
Important details:
- Markets use
ParquetStoragefor chunking and deduplication. - Trades run concurrently and dedupe by
trade_id. kalshi_tradesonly considers markets above the built-in volume threshold.
Polymarket markets and trades
polymarket_marketspaginates the Gamma API with an offset cursor.polymarket_tradesbackfills CTF and NegRiskOrderFilledlogs by block range.
Important details:
- The trade indexer stores progress after each completed block chunk.
- The trade indexer writes chunked Parquet files under
data/polymarket/trades/. - The market indexer writes an offset cursor and chunked market files.
Polymarket blocks
polymarket_blocksbuilds a block-to-timestamp lookup.- It samples every 100 blocks and interpolates the values to fill each bucket.
- The output feeds analyses that need approximate timestamps for trade blocks.
Polymarket legacy FPMM trades
polymarket_fpmm_tradesdecodesFPMMBuyandFPMMSelllogs from the Polygon chain.- It filters to USDC-collateralized markets.
- It is the most RPC-intensive collector in the repo.
Adding or fixing an indexer
- Subclass
Indexer. - Set
nameanddescriptionin__init__. - Keep the fetch/resume/save loop explicit so interruption points are easy to reason about.
- Write Parquet chunks with predictable names.
- Save or update a cursor file only after the relevant unit of work is safely completed.
- Prefer small helper methods for decoding logs, chunking ranges, or deduplicating existing rows.
Common failure patterns
- Missing
POLYGON_RPCfor chain-backed collectors. - Invalid cursor files that point to the wrong offset or block.
- Duplicate rows when the collector is rerun without deduplication.
- Chunk files that are partially written because a run was interrupted.
- Slow or failing RPC queries when the block range is too large.
Helpful helper
../../scripts/catalog.py --indexerslists the indexer names without opening the interactive menu when run through the repo environment.
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).