AGENT SKILL

vector-databases

Choose and operate vector databases — indexing algorithms, filtering, scaling, and production concerns. Use when storing and querying embeddings at scale.

License 许可
MIT
所属仓库
TanNhatCMS/jarvis
最近更新
2026-11-07
Stars
★ 0
Forks
1

01 前言 Front Matter

name
vector-databases
category
ai-research
description
Choose and operate vector databases — indexing algorithms, filtering, scaling, and production concerns. Use when storing and querying embeddings at scale.

03 SKILL.md 正文

Vector Databases

A vector database stores embeddings and finds nearest neighbors fast at scale. The decisions: which index algorithm, how to filter, how to scale, and how to keep results good as data grows.

Overview

Core operation: given a query vector, return the k most similar stored vectors — approximately (ANN) for speed at scale. HNSW graphs dominate for the quality/speed balance; IVF partitions for huge datasets. On top: metadata filtering (so "similar" also means "in the right category/date"), hybrid search hooks, and operational concerns — indexing throughput, recall tuning, and multi-tenancy.

When to use

  • RAG retrieval over large document collections.
  • Semantic search features in products.
  • Recommendation or deduplication at scale.
  • Anytime brute-force similarity search gets too slow.

Core concepts

  • ANN indices: HNSW (graph-based, excellent recall/latency, memory-hungry), IVF (partitioned, scales to billions, needs tuning), and others. HNSW is the default; IVF for the largest scales.
  • Recall vs. speed: ANN parameters trade accuracy for speed (HNSW's ef_search, IVF's nprobe). Tune on your data: measure recall@k against brute force on a sample.
  • Metadata filtering: pre-filter (filter then search — precise, can be slow), post-filter (search then filter — fast, may return too few), or index-aware hybrid. Your filter selectivity decides.
  • Namespaces/tenancy: partition by tenant, project, or corpus — for isolation, deletion, and per-tenant tuning. Design tenancy before data grows.
  • Distance metrics: cosine, dot product, Euclidean — must match how embeddings were produced. Set once, consistently.
  • Freshness: upserts, deletes, and index rebuilds. Know your system's update latency and whether deletes are real or tombstoned.

Practical workflow

  1. Define the workload: corpus size, query rate, filter patterns, latency budget, recall target.
  2. Choose the index: HNSW for most; IVF-family for 100M+ scale. Start managed unless you have ops capacity.
  3. Set the distance metric to match your embeddings; normalize consistently.
  4. Design metadata schema and tenancy up front — retrofitting partitions is painful.
  5. Tune ANN parameters on your data: sweep ef_search/nprobe, plot recall vs. latency, pick the knee.
  6. Load-test with production-like queries and filters; monitor recall drift as the corpus grows.
Selection checklist:
[ ] Corpus size now + 12-month projection
[ ] p95 latency budget defined
[ ] Recall@k target (measured vs brute force)
[ ] Filter patterns identified (selectivity known)
[ ] Tenancy/isolation requirements set
[ ] Update/delete latency acceptable

Common pitfalls

  • Untuned ANN params: default ef_search giving 80% recall when you need 95%. Tune on your data.
  • Metric mismatch: cosine index with dot-product-trained embeddings. Consistent end-to-end or broken ranking.
  • Post-filter starvation: heavy filtering after ANN search returns fewer than k results. Use pre-filtering for selective filters.
  • No tenancy plan: one flat index for all customers, then a deletion/isolation requirement arrives. Partition early.
  • Ignoring write path: great search, terrible ingestion — or deletes that don't delete. Test the full lifecycle.
  • Recall measured once: index quality degrades as data distribution shifts. Re-measure periodically.

04 同仓库更多 Skills

TanNhatCMS
TanNhatCMS/jarvis

web-scrape

Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open/fetch fails on JS-heavy pages, or when you need clean markdown/structured extraction from a site. Requires FIRECRAWL_API_KEY.

★ 0
TanNhatCMS
TanNhatCMS/jarvis

design-taste-frontend

tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML/CSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, "AI-made", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.

★ 0
TanNhatCMS
TanNhatCMS/jarvis

test-driven-development

Use when implementing any feature or bugfix, before writing implementation code

★ 0
TanNhatCMS
TanNhatCMS/jarvis

systematic-debugging

Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes

★ 0
TanNhatCMS
TanNhatCMS/jarvis

requesting-code-review

Use when completing tasks, implementing major features, or before merging to verify work meets requirements

★ 0
TanNhatCMS
TanNhatCMS/jarvis

brainstorming

You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).

★ 0