docarray
Use DocArray for multimodal Pydantic-style documents, typed DocList and DocVec batches, serialization and local storage, FastAPI payloads, and vector retrieval indexes.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- name
- docarray
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Use DocArray for multimodal Pydantic-style documents, typed DocList and DocVec batches, serialization and local storage, FastAPI payloads, and vector retrieval indexes.
- disable-model-invocation
- true
02 资源文件 4
- 参考文档references/repo-provenance.md3.0 KB
- 参考文档references/repo-routing-metadata.json398 B
- 参考文档references/troubleshooting.md2.8 KB
- 脚本scripts/check_env.py2.2 KB
03 SKILL.md 正文
DocArray
Use this repo skill when a task names DocArray or asks for a Python data model for multimodal records, typed document batches, document serialization, local document storage, FastAPI document payloads, or vector-index retrieval.
Verified baseline
The verified default is a CPU workflow using DocArray base dependencies plus the proto, pandas, and web extras. Install the public package for normal use:
pip install -U docarray
# add only the surfaces you need:
pip install -U "docarray[proto,pandas,web]"
python -c "import docarray; print(docarray.__version__)"
For the current verified DocVec path, prefer numpy<2 and run the bundled smoke helpers before trusting a new environment. Optional tensor frameworks, media loaders, cloud stores, and external vector databases are not implied by the base install.
Route by task
- Model a record or multimodal schema: read
document-modelingforBaseDoc, predefined modality docs, dynamic schemas,DocList,DocVec, nested fields, and typed tensor shapes. - Serialize, store, stream, or serve documents: read
serialization-storagefor JSON, protobuf, bytes, base64, binary files, CSV/DataFrame exchange,file://, S3 boundaries, andDocArrayResponse. - Index, search, filter, or persist vectors: read
vector-indexingforInMemoryExactNNIndex, query builders, subindexes, persistence, and optional backend selection.
Tasks that span multiple surfaces should start here, choose the schema in document-modeling, then hand the resulting typed documents to serialization-storage or vector-indexing.
Fast decisions
| Need | First choice | Watch for |
|---|---|---|
| One validated data point | BaseDoc subclass |
Required fields, nested docs, and per-document tensor shapes. |
| Mutable/reorderable/streaming collection | DocList[MyDoc] |
Typed lists are homogeneous; bare DocList may be heterogeneous. |
| Contiguous ML batch | DocVec[MyDoc] |
Homogeneous fields; optional doc/tensor columns must be all present or all missing. |
| Human-readable transport | JSON | Rich tensor/list unions may need explicit schema handling. |
| Compact trusted transport | protobuf or protobuf-array | Install docarray[proto]; document unions are not protobuf-safe. |
| Local retrieval prototype | InMemoryExactNNIndex[MyDoc] |
Dimensioned vector field and backend-specific filter/query behavior. |
| Production/vector service | Optional backend | Install and verify the chosen client, service, credentials, schema, and metric separately. |
Common failure boundaries
- Missing
google.protobuf,pandas,fastapi, or backend client: install the matching narrow extra; do not installfullby default. DocVecfails with a NumPydeviceerror: check NumPy compatibility and trynumpy<2for the current verified CPU path.- File-store push raises
FileNotFoundError: create the explicit parent namespace directory first. - CSV cannot rebuild tensor fields: use JSON, protobuf-array, binary, or DataFrame instead; CSV is safest for scalar rows.
- In-memory query builder does not support text search, and equal-score result ties can expose a comparison edge case; see the vector troubleshooting reference.
Read references/troubleshooting.md for cross-cutting recovery guidance and references/repo-provenance.md before deciding whether this skill matches a changed checkout.
Routing metadata path convention
In references/repo-routing-metadata.json, every useful_entry_points value is a path relative to this DocArray skill root (the directory containing this SKILL.md), not a repository- or bundle-prefixed path.
Safe bundled checks
These helpers are self-contained and do not require the original repository checkout:
python sub-skills/document-modeling/scripts/schema_smoke.py --help
python sub-skills/serialization-storage/scripts/roundtrip_smoke.py --help
python sub-skills/vector-indexing/scripts/inmemory_index_smoke.py --help
python scripts/check_env.py --help
Run the relevant helper after installing the package and selected extras. Helpers use tiny in-memory or temporary-file fixtures; they do not start databases, use credentials, or download models.
Public capability boundaries
DocArray also exposes Torch, TensorFlow, JAX, image/audio/video/mesh loaders, S3, HNSWLib, Qdrant, Weaviate, Elasticsearch, Redis, Milvus, MongoDB Atlas, Epsilla, and Jina/FastAPI integrations. Those are routed by the sub-skills but remain optional until their exact dependency variant, service/network/credential plan, and native smoke have been verified.
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).