indexing-manager
This skill should be used when pushing, syncing, or indexing files from .ai-memory/ directories to external indexing services such as the Qdrant MCP server, local handoff service, or vector stores. Use when a user says "push to indexer", "sync ai-memory", "index session files", "trigger handoff", or "push documentation to external brain". Handles endpoint discovery, batch upload with concurrency, retry logic with exponential backoff, and comprehensive progress reporting.
- License 许可
- GPL-3.0
- 最近更新
- 2026-10-21
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- indexing-manager
- description
- This skill should be used when pushing, syncing, or indexing files from .ai-memory/ directories to external indexing services such as the Qdrant MCP server, local handoff service, or vector stores. Use when a user says "push to indexer", "sync ai-memory", "index session files", "trigger handoff", or "push documentation to external brain". Handles endpoint discovery, batch upload with concurrency, retry logic with exponential backoff, and comprehensive progress reporting.
02 资源文件 5
- 资产文件assets/config_template.yaml1.2 KB
- 参考文档references/api_specifications.md10.5 KB
- 参考文档references/error_recovery.md7.3 KB
- 参考文档references/indexing_patterns.md6.7 KB
- 脚本scripts/batch_indexer.py9.9 KB
03 SKILL.md 正文
Indexing Manager Skill
Purpose
Manage batch document indexing to external services (handoff endpoints, vector stores, search engines) with automatic endpoint discovery, concurrent uploads, retry logic, and error recovery.
When to Use This Skill
Invoke this skill when:
- Pushing
.ai-memory/session documentation to indexing endpoints - Syncing project documentation to external knowledge bases
- Batch uploading files to vector stores or search services
- Migrating content between indexing systems
- Troubleshooting failed indexing operations
How to Use This Skill
Step 1: Verify Configuration
Before running any indexing operation, ensure a indexing-manager.config.yaml file exists in the project root. If not, copy the template from assets/config_template.yaml and customize for the project:
cp indexing-manager/assets/config_template.yaml ./indexing-manager.config.yaml
# Edit indexing-manager.config.yaml with project-specific values
Step 2: Run the Batch Indexer
Execute the primary indexing script:
python3 indexing-manager/scripts/batch_indexer.py
The script will:
- Load configuration from
indexing-manager.config.yaml - Validate the endpoint using
endpoint_validator.py(probe-and-discover pattern) - Discover files recursively from
memory_dirmatchingfile_extensions - Upload concurrently using
ThreadPoolExecutor(default: 3 workers) - Retry failed uploads using exponential backoff via
retry_handler.py - Report progress in real-time with success/failure counts
- Generate summary with total files, successes, failures, and execution time
Step 3: Handle Errors
If uploads fail, consult indexing-manager/references/error_recovery.md for troubleshooting steps. Common issues:
- Endpoint not found: The validator probes known endpoints. If all fail, check that the indexing service is running.
- File permission denied: Ensure the script has read access to all files in
memory_dir. - Timeout errors: Increase
retry_maxorretry_backoff_basein the config file. - Partial failures: The script logs failed files. Retry with
retry_handler.pyor manually re-upload.
Bundled Resources
Scripts
scripts/batch_indexer.py
Main execution engine. Orchestrates file discovery, concurrent uploads, and progress reporting.
Usage:
python3 indexing-manager/scripts/batch_indexer.py
Configuration: Reads from indexing-manager.config.yaml in the project root.
Output: Real-time progress + summary report.
scripts/endpoint_validator.py
Pre-flight endpoint discovery. Probes known endpoints and returns the first responding one.
Known endpoints:
/mcp(Qdrant MCP server - recommended)/api/handoff(legacy local handoff service)/api/index(alternative indexing endpoint)/api/upload(generic upload endpoint)/api/documents(document management endpoint)
Usage: Called automatically by batch_indexer.py. Can also be invoked standalone:
python3 indexing-manager/scripts/endpoint_validator.py --base-url http://127.0.0.1:5000
scripts/retry_handler.py
Exponential backoff retry logic. Implements configurable retry attempts with increasing delays.
Formula: wait_time = retry_backoff_base ^ attempt_number
Example: With retry_backoff_base=2 and retry_max=3:
- Attempt 1: Immediate
- Attempt 2: Wait 2 seconds
- Attempt 3: Wait 4 seconds
- Attempt 4: Wait 8 seconds (then fail)
Usage: Called automatically by batch_indexer.py for failed uploads.
References
references/indexing_patterns.md
Catalog of common indexing endpoint patterns and their specifications:
- Qdrant MCP Server (
http://localhost:3000/mcp) — Native Qdrant integration via Model Context Protocol - Handoff Service (
http://127.0.0.1:5000/api/handoff) — Legacy local Docker service - Elasticsearch — Bulk indexing API patterns
- Milvus — Vector database upload patterns
- Pinecone — Cloud vector store patterns
references/error_recovery.md
Comprehensive troubleshooting runbook covering:
- Endpoint discovery failures
- File permission errors
- Timeout and connection errors
- Partial upload failures
- Configuration validation errors
- Retry exhaustion scenarios
references/api_specifications.md
Technical reference for multipart form-data uploads:
- HTTP multipart form-data structure
- File encoding and MIME types
- Metadata attachment patterns
- Timeout handling and connection pooling
- Rate limiting and backoff strategies
Assets
assets/config_template.yaml
Project-scoped configuration template. Copy to project root and customize:
# indexing-manager.config.yaml
project: example-project
memory_dir: .ai-memory
endpoint: http://127.0.0.1:5000/api/handoff
workspace_slug: example-project
file_extensions:
- .md
- .sh
- .py
- .json
- .yaml
concurrency: 3
retry_max: 3
retry_backoff_base: 2
Configuration Fields:
| Field | Type | Default | Description |
|---|---|---|---|
project |
string | — | Project name (used for metadata) |
memory_dir |
string | .ai-memory |
Directory containing files to index |
endpoint |
string | — | Full URL of indexing endpoint |
collection_name |
string | agent-memory |
Qdrant collection name |
file_extensions |
list | [.md, .sh, .py, .json, .yaml] |
File types to include |
concurrency |
int | 3 |
Number of concurrent upload workers |
retry_max |
int | 3 |
Maximum retry attempts per file |
retry_backoff_base |
int | 2 |
Base for exponential backoff calculation |
Workflow Example
Scenario: Push .ai-memory/ to Local Handoff Service
# 1. Verify config exists
ls indexing-manager.config.yaml
# 2. Run indexer
python3 indexing-manager/scripts/batch_indexer.py
# Output:
# ╔════════════════════════════════════════════════════════════════╗
# ║ Indexing Manager - Batch Upload ║
# ╚════════════════════════════════════════════════════════════════╝
#
# 📁 Source Directory: .ai-memory
# 🌐 Endpoint: http://127.0.0.1:5000/api/handoff
# 🧠 Project: example-project
#
# ✓ Endpoint validated: /api/handoff responding
# ✓ Found 82 files
#
# Uploading (3 concurrent workers):
# [1/82] INCIDENT_DOCUMENTATION_INDEX.md ✓
# [2/82] INCIDENT_EXECUTIVE_SUMMARY.md ✓
# [3/82] INCIDENT_RCA_INVESTIGATION_PLAN.md ✓
# ...
# [82/82] waf-extraction-script.sh ✓
#
# ╔════════════════════════════════════════════════════════════════╗
# ║ Summary ║
# ╚════════════════════════════════════════════════════════════════╝
#
# Total Files: 82
# Successful: 82
# Failed: 0
# Execution Time: 2m 14s
# Success Rate: 100%
Migration from Legacy Scripts
This skill consolidates overlapping legacy scripts:
sync-roo-memory-handoff.sh→ DEPRECATEDpush_ai_memory_to_indexer.py→ DEPRECATED
Migration Steps:
- Copy
indexing-manager/assets/config_template.yamlto project root asindexing-manager.config.yaml - Customize configuration for your project
- Run
python3 indexing-manager/scripts/batch_indexer.py - Delete legacy scripts after verifying successful indexing
- Update CI/CD pipelines to use the new skill
Troubleshooting
Endpoint Not Found
Error: EndpointNotFoundError: No responding endpoint found
Solution: Verify the indexing service is running:
curl -v http://127.0.0.1:5000/api/handoff
If the service is not running, start it:
docker-compose up -d handoff-service
File Permission Denied
Error: PermissionError: [Errno 13] Permission denied: '.ai-memory/file.md'
Solution: Ensure the script has read access:
chmod -R 644 .ai-memory/
Timeout Errors
Error: requests.exceptions.Timeout: Connection timeout
Solution: Increase retry backoff or reduce concurrency in indexing-manager.config.yaml:
concurrency: 1 # Reduce from 3 to 1
retry_backoff_base: 3 # Increase from 2 to 3
Partial Failures
Error: Some files uploaded, others failed
Solution: Check indexing-manager/references/error_recovery.md for specific error codes. Retry failed files:
python3 indexing-manager/scripts/retry_handler.py --failed-files failed_list.txt
Performance Characteristics
| Metric | Sequential | Concurrent (3 workers) | Improvement |
|---|---|---|---|
| 82 files | ~7 minutes | ~2-3 minutes | 65-70% faster |
| Avg file size | 50KB | 50KB | — |
| Network utilization | 1 connection | 3 connections | 3x parallelism |
| CPU usage | Low | Low | Minimal overhead |
Related Documentation
ADR-010-INDEXING-MANAGER-SKILL.md— Architectural decision recordindexing-manager/references/indexing_patterns.md— Endpoint catalogindexing-manager/references/error_recovery.md— Troubleshooting guideindexing-manager/references/api_specifications.md— Technical reference
04 同仓库更多 Skills
ugip
Unified Global Initialization Protocol. Run at the start of every session to initialize GitNexus, flush .ai-memory/ to Qdrant, read ROO_CODE_CONTEXT.md, check the git branch, and run role-specific recall_context queries. Invoke with /ugip.
template-skill
Replace with description of the skill and when Claude should use it.
skill-creator
Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Claude's capabilities with specialized knowledge, workflows, or tool integrations.
puppeteer-automation
Comprehensive Puppeteer automation skill for web scraping, browser testing, screenshot generation, form automation, and performance monitoring. Use when tasks involve browser automation, web testing, data extraction, or UI validation.
model-enhancement-mcp
Guide for creating MCP servers that enhance LLM reasoning through structured processes, persistence, and workflow guidance. Use when building MCP servers for structured thinking, journaling, memory systems, or other cognitive enhancement patterns.
mcp-builder
Guide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK).