AGENT SKILL

indexing

Guide Kalshi and Polymarket market, trade, block, and legacy backfill workflows.

License 许可
MIT
最近更新
2026-09-03
Stars
★ 351
Forks
27

01 前言 Front Matter

name
indexing
license
MIT
metadata
{ "disco-role": "operating" }
description
Guide Kalshi and Polymarket market, trade, block, and legacy backfill workflows.
disable-model-invocation
true

02 资源文件 2

03 SKILL.md 正文

indexing

Use this sub-skill for data-collection workflows that write Parquet chunks and resume from cursor files. It covers both API-based and blockchain-based collectors.

Use this route when

  • The user wants to backfill markets, trades, blocks, or legacy FPMM data.
  • The user wants to resume a partially completed run.
  • The user wants to understand the shape of a cursor file or chunked Parquet output.
  • The user wants to add a new indexer subclass.

Scope

Included

  • Kalshi market and trade backfills.
  • Polymarket market, trade, block, and legacy FPMM backfills.
  • Cursor files, chunk naming, deduplication, and resume behavior.
  • KalshiClient, PolymarketClient, PolygonClient, and Indexer.load().

Excluded

  • Analysis and chart output workflows, which belong to sub-skills/analysis/.
  • Dataset packaging and host-tool installation, which belong to sub-skills/data-ops/.

Read first

  • ../../references/indexing-catalog.md for the full indexer list.
  • ../../references/api-reference.md for the client and base-class APIs.
  • ../../references/data-layout.md for chunk paths, cursor files, and helper files.
  • ../../references/troubleshooting.md for cross-cutting failures.

Core workflow

  1. Identify the indexer name and required inputs.
  2. Confirm the cursor file and output directory state.
  3. Provide any required env vars, especially POLYGON_RPC for blockchain-backed collectors.
  4. Run the indexer through uv run main.py index or directly from the Python class.
  5. Verify that the expected Parquet chunks and cursor files were updated.

Indexer families

Kalshi

  • kalshi_markets backfills market snapshots into chunked Parquet files.
  • kalshi_trades fetches trades for markets discovered from the market parquet tree and skips already-processed tickers.

Important details:

  • Markets use ParquetStorage for chunking and deduplication.
  • Trades run concurrently and dedupe by trade_id.
  • kalshi_trades only considers markets above the built-in volume threshold.

Polymarket markets and trades

  • polymarket_markets paginates the Gamma API with an offset cursor.
  • polymarket_trades backfills CTF and NegRisk OrderFilled logs by block range.

Important details:

  • The trade indexer stores progress after each completed block chunk.
  • The trade indexer writes chunked Parquet files under data/polymarket/trades/.
  • The market indexer writes an offset cursor and chunked market files.

Polymarket blocks

  • polymarket_blocks builds a block-to-timestamp lookup.
  • It samples every 100 blocks and interpolates the values to fill each bucket.
  • The output feeds analyses that need approximate timestamps for trade blocks.

Polymarket legacy FPMM trades

  • polymarket_fpmm_trades decodes FPMMBuy and FPMMSell logs from the Polygon chain.
  • It filters to USDC-collateralized markets.
  • It is the most RPC-intensive collector in the repo.

Adding or fixing an indexer

  • Subclass Indexer.
  • Set name and description in __init__.
  • Keep the fetch/resume/save loop explicit so interruption points are easy to reason about.
  • Write Parquet chunks with predictable names.
  • Save or update a cursor file only after the relevant unit of work is safely completed.
  • Prefer small helper methods for decoding logs, chunking ranges, or deduplicating existing rows.

Common failure patterns

  • Missing POLYGON_RPC for chain-backed collectors.
  • Invalid cursor files that point to the wrong offset or block.
  • Duplicate rows when the collector is rerun without deduplication.
  • Chunk files that are partially written because a run was interrupted.
  • Slow or failing RPC queries when the block range is too large.

Helpful helper

  • ../../scripts/catalog.py --indexers lists the indexer names without opening the interactive menu when run through the repo environment.

04 同仓库更多 Skills

VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-triton-opt

Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-solve

END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-skill

PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-pattern-fusion

Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-orchestrate

Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-feedback

Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).

★ 351