[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M71GQXKFHT6ZF35QANYF6BCF":3,"skill-same:01M71GQXKFHT6ZF35QANYF6BCF":80},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":57,"commitSha":58,"skillDirectory":59,"forksCount":60,"repositoryHtmlUrl":61,"repositoryHomepageUrl":6,"ownerHtmlUrl":62,"ownerBlogUrl":63,"ownerDescription":64,"updatedAt":65,"install":66,"resources":79},"01M71GQXKFHT6ZF35QANYF6BCF","kb-review",null,"Review a Bedrock Knowledge Base or RAG pipeline: chunking, retrieval quality, recall@5 and cost per query.","timurgaleev\u002Fvibestack","timurgaleev","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F32696429?v=4","01M71GMS53Z9B4JQTCQEFXAE9E","Timur Galeev","01M71GMS5S1Q7BEYB3Z7GY3E85",8,"MIT","https:\u002F\u002Fgithub.com\u002Ftimurgaleev\u002Fvibestack\u002Fblob\u002Fmain\u002Fskills\u002Fkb-review\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"kb review","kb","review","cost","at","rag pipeline","chunking strategy","bedrock","store","chunking","knowledge base","query","strategy","write","read","knowledge","Bash","pipeline","vector","rag","2026-11-10 10:12:30",false,"---\nname: kb-review\ndescription: |\n  Review a Bedrock Knowledge Base or RAG pipeline: chunking, retrieval quality, recall@5 and cost per query.\ntriggers:\n  - knowledge base review\n  - rag review\n  - bedrock knowledge base\n  - chunking strategy\n  - retrieval quality\n  - recall at k\n  - vector store review\nallowed-tools:\n  - Bash\n  - Read\n  - Grep\n  - Write\n  - AskUserQuestion\n---\n\nIf this session was compacted, the copy of this skill in context may be cut off.\nRead `${CLAUDE_SKILL_DIR}\u002FSKILL.md` from disk and resume at the step you are on.\n\n## When to invoke\n\nUse when: \"knowledge base review\", \"rag review\", \"bedrock knowledge base\", \"chunking strategy\", \"retrieval quality\", \"recall at k\", \"vector store review\", \"why does the bot not find the document\".\n\n## Preamble\n\n```bash\neval \"$(\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fbin\u002Fvibe-slug\" 2>\u002Fdev\u002Fnull)\" 2>\u002Fdev\u002Fnull || SLUG=\"unknown\"\n_LEARN_FILE=\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fprojects\u002F${SLUG:-unknown}\u002Flearnings.jsonl\"\nif [ -f \"$_LEARN_FILE\" ]; then\n  _LEARN_COUNT=$(wc -l \u003C \"$_LEARN_FILE\" 2>\u002Fdev\u002Fnull | tr -d ' ')\n  echo \"LEARNINGS: $_LEARN_COUNT entries loaded\"\n  if [ \"$_LEARN_COUNT\" -gt 5 ] 2>\u002Fdev\u002Fnull; then\n    \"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fbin\u002Fvibe-learnings-search\" --limit 5 2>\u002Fdev\u002Fnull || true\n  fi\nelse\n  echo \"LEARNINGS: none yet\"\nfi\n```\n\n{{include lib\u002Fsnippets\u002Fsession-host.md}}\n\n{{include lib\u002Fsnippets\u002Fdecision-brief.md}}\n\n{{include lib\u002Fsnippets\u002Fworking-protocols.md}}\n\n{{include lib\u002Fsnippets\u002Fstate-protocols.md}}\n\n## User-invocable\nWhen the user types `\u002Fkb-review`, run this skill.\n\n---\n\n## Step 1: Inventory\n\nFind out what the knowledge base is made of before judging any of it. Two places to\nlook: infrastructure code in the repo, and the live account.\n\n**Repo first.** Grep for the Terraform resources and for the SDK calls a hand-built\npipeline would use:\n\n```bash\ngrep -rnE 'aws_bedrockagent_(knowledge_base|data_source)|aws_opensearchserverless_collection|pgvector|s3_vectors|S3_VECTORS' --include='*.tf' --include='*.py' --include='*.ts' --include='*.yaml' --include='*.yml' . 2>\u002Fdev\u002Fnull | head -40\ngrep -rnE 'RecursiveCharacterTextSplitter|SentenceSplitter|chunk_size|chunk_overlap|embed_documents|Pinecone|Qdrant|Weaviate|pgvector' --include='*.py' --include='*.ts' . 2>\u002Fdev\u002Fnull | head -40\n```\n\nIf the first grep hits, this is a Bedrock Knowledge Base and the Terraform is the\nsource of truth for configuration. If only the second hits, it is a hand-built\npipeline: read the ingestion and query code and map each of the following steps\nonto it. If neither hits, ask where the knowledge base lives.\n\n**Live account, when the CLI works.**\n\n{{include lib\u002Fsnippets\u002Faws-context.md}}\n\nThen list what exists:\n\n```bash\naws bedrock-agent list-knowledge-bases --output json\naws bedrock-agent get-knowledge-base --knowledge-base-id \u003Ckb-id> --output json\naws bedrock-agent list-data-sources --knowledge-base-id \u003Ckb-id> --output json\naws bedrock-agent get-data-source --knowledge-base-id \u003Ckb-id> --data-source-id \u003Cdata-source-id> --output json\n```\n\nEvery angle-bracket value in this skill — `\u003Ckb-id>`, `\u003Cdata-source-id>`, `\u003Cbucket>`,\n`\u003Cprefix>` and the rest — is substituted by hand from the output of the preceding\ncommand before the line is run. They are placeholders, not shell variables.\n\n**Read the knowledge base type first.** `knowledgeBaseConfiguration.type` is one of\n`VECTOR`, `MANAGED`, `KENDRA` or `SQL`, and it decides which fields exist and which\nretrieval request shape is valid in Steps 3 and 6:\n\n- `VECTOR` — a knowledge base backed by a vector store you own. Embedding model ARN\n  and dimensions come from `knowledgeBaseConfiguration.vectorKnowledgeBaseConfiguration`;\n  the store comes from `storageConfiguration.type` (OpenSearch Serverless, Aurora\n  with pgvector, S3 Vectors, Pinecone, or another). Query with\n  `retrievalConfiguration.vectorSearchConfiguration`.\n- `MANAGED` — Amazon Bedrock owns the vector store. Read\n  `knowledgeBaseConfiguration.managedKnowledgeBaseConfiguration`: `embeddingModelType`\n  is `MANAGED` (service-chosen model) or `CUSTOM` (with `embeddingModelArn`).\n  `storageConfiguration` is optional on the KnowledgeBase object and is absent here —\n  record the store as service-managed rather than reporting a missing field. Query\n  with `retrievalConfiguration.managedSearchConfiguration`.\n- `KENDRA` — an Amazon Kendra GenAI index holds the content, so there is no chunking,\n  embedding or vector-store configuration to review. Query it with `Retrieve` as you\n  would a standard knowledge base, `vectorSearchConfiguration` and all; the Kendra\n  fields come back as metadata attributes and metadata filtering works on them. Say so,\n  skip Steps 2, 4 and 7's store item, and run Steps 3, 5 and 6 unchanged.\n- `SQL` — a structured data store answers by generated SQL, not by embeddings, so there\n  is nothing chunked, embedded or stored as vectors to review. `Retrieve` returns the\n  rows the generated query produced, and none of `retrievalConfiguration` applies:\n  `numberOfResults`, search type, metadata filtering and reranking are all documented as\n  applying to unstructured data sources only. Send the query on its own, with no\n  `--retrieval-configuration` at all. Skip Steps 2, 4 and 7's store item. Step 3 keeps\n  its question but changes its evidence — there is no metadata filter to check, so read\n  the query engine's table and column inclusions and exclusions and the database grants\n  on the role the knowledge base queries with. Steps 5 and 6 run with the row-based\n  reading noted in each.\n\nSending a `vectorSearchConfiguration` to a managed knowledge base, or a\n`managedSearchConfiguration` to a `VECTOR` one, is an invalid request — a failed call\nthere is a mistake in this review, not a finding about the knowledge base. So is sending\neither one to a `SQL` knowledge base. Set the key once and reuse it:\n\n```bash\nSEARCH_KEY=vectorSearchConfiguration    # managedSearchConfiguration on a MANAGED KB\n                                        # SQL: send no retrieval configuration at all\n```\n\nThen fill in the KB summary table: knowledge base id, name and type; the store and\nembedding model from whichever branch above applies; and per data source its type\n(S3, Confluence, SharePoint, OneDrive, Google Drive, Salesforce, Box, web crawler,\ncustom — record whatever `dataSourceConfiguration.type` reports, the connector set\ngrows), location, chunking configuration, parsing configuration and\n`dataDeletionPolicy`.\n\n**Sources.** For every S3 data source, size it without downloading it:\n\n```bash\naws s3 ls s3:\u002F\u002F\u003Cbucket>\u002F\u003Cprefix> --recursive --summarize | tail -2\naws s3 ls s3:\u002F\u002F\u003Cbucket>\u002F\u003Cprefix> --recursive | awk '{print $(4)}' | sed -E 's\u002F.*\\.([A-Za-z0-9]+)$\u002F\\1\u002F' | sort | uniq -c | sort -rn\n```\n\nRecord object count, total size, and the format mix (PDF, DOCX, HTML, Markdown, CSV,\nJSON). Count the image objects (`.jpeg`, `.png`) on their own line: Step 7 prices a\nconfigured parser per page and per image, and neither number falls out of the object\ncount. When any data source sets a `parsingConfiguration`, carry a page estimate into\nStep 7 as well — read the page count of each PDF sampled in Step 2 (`pdfinfo`, or\n`mdls -name kMDItemNumberOfPages` on macOS), take the mean, multiply by the PDF object\ncount, and record the sample size beside it so the number is read as the estimate it\nis. Note the update cadence from the newest and oldest `LastModified` values and\nfrom whatever writes to the prefix (a pipeline, a person, a Confluence export) — Step 3\ntreats that writer as a trust boundary, so name it here. For\nConfluence and web sources, record the space keys or seed URLs and the crawl scope.\n\nFindings at this step: **INFO** for every data source, **MEDIUM** if a data source\npoints at a bucket prefix that also holds unrelated files (logs, exports, backups)\nthat will be embedded along with the documents.\n\n---\n\n## Step 2: Chunking\n\nRead `vectorIngestionConfiguration.chunkingConfiguration` from each data source (or\nthe splitter call in a hand-built pipeline) and compare it against the shape of the\ndocuments. Sample three to five documents per format into the scratchpad with\n`aws s3 cp`; look at them, do not paste them into the report.\n\nCheck:\n\n- **Strategy versus document shape.** Fixed-size chunks suit short, flat documents. Hierarchical chunking (parent and child levels) suits long manuals and policies where a paragraph only makes sense with its section heading. Semantic chunking suits prose with topic shifts. `NONE` means each document is treated as a single source chunk; the service applies its own chunk handling from there, and a document that violates a limit surfaces in the ingestion job, not as a quiet truncation. Do not infer truncation from the embedding model's input limit — check it. Read `numberOfDocumentsFailed` and `failureReasons` on the last job (Step 5), then run one retrieve for a phrase that appears only near the end of the largest sampled document and read `content.text` on the results. **HIGH** if documents failed ingestion, or if that tail phrase is in no returned chunk while the same document answers other questions. **MEDIUM** if the strategy is `NONE` on long documents and neither check could be run — the risk is real, the evidence is not there yet. Note also that `NONE` gives up page-number citations and the `x-amz-bedrock-kb-document-page-number` filter attribute.\n- **Chunk size and overlap.** Read `maxTokens` and `overlapPercentage` (fixed) or the level `maxTokens` and `overlapTokens` (hierarchical). **MEDIUM** if chunks are under 200 tokens on documents with long paragraphs (answers get split across chunks) or over 1000 tokens on FAQ-style content (one chunk carries several unrelated questions). **LOW** if overlap is 0 on fixed-size chunking.\n- **Tables and code.** Open a sampled PDF or DOCX that contains a table and find the corresponding chunk text with a `retrieve` call for a value from the table. **MEDIUM** if table rows come back as a run of numbers with no header context and no parsing configuration is set (`parsingConfiguration` with a foundation model or Bedrock Data Automation keeps table structure). Same check for code blocks in Markdown.\n- **Metadata files.** Each document may carry a sidecar `\u003Cname>.metadata.json` with a `metadataAttributes` object. Count them: `aws s3 ls s3:\u002F\u002F\u003Cbucket>\u002F\u003Cprefix> --recursive | grep -c '\\.metadata\\.json$'`. Compare with the document count. **MEDIUM** if fewer than 90% of documents have one and Step 3 relies on filtering.\n\n---\n\n## Step 3: Metadata and isolation\n\nThis is the step that decides whether one customer can read another customer's\ndocuments. Treat every gap here as at least **HIGH**.\n\n- **What is in the metadata.** Read three sampled `.metadata.json` files. The attributes that matter are tenant (or customer, account), department, and classification (public, internal, confidential). Record which exist and whether values are consistent (`tenant` versus `tenantId` versus `customer` across files is a finding).\n- **Which mechanism is in play.** Two are supported and they are checked differently, so decide before judging anything. *Metadata filtering*: documents carry a tenant attribute and every query passes a `filter`. *ACL awareness*, on a managed knowledge base: the connector crawls document permissions (allowed and denied users and groups) at ingestion, and the query passes a `userContext` naming the user, and the crawled permissions filter the candidates before retrieval. Some connectors add a second stage, a live call to the source per query that catches permission changes made since the last sync. Which ones do is a per-connector property and the connector set grows, so read it from that connector's own ACL page rather than from a list held in your head. Today: SharePoint, OneDrive, Google Drive, Confluence and Box document both stages; S3 and custom document pre-retrieval filtering only, because their ACLs come from a file the customer writes and there is nothing live to re-check; the web crawler supports no ACLs at all. For any connector you have not checked, record its real-time support as unknown and go read the page before a severity leans on it. Read each data source's configuration for ACL awareness, and read the call sites for which of `filter` and `userContext` they send. A knowledge base may use one, both, or neither. An ACL-enabled data source returning nothing when `userContext` is absent is documented behaviour, not a retrieval bug; non-ACL data sources in the same knowledge base still return to everyone.\n- **Filter applied on every retrieve.** Find every call site of `retrieve`, `retrieve_and_generate`, `RetrieveCommand`, or the agent's knowledge base configuration: `grep -rnE 'retrieve(_and_generate|AndGenerate)?\\(|RetrieveCommand|RetrieveAndGenerateCommand' --include='*.py' --include='*.ts' --include='*.js' .` On a metadata-filtered knowledge base, check that `retrievalConfiguration.$SEARCH_KEY.filter` is set from the caller's authenticated session, not from a request parameter the caller controls. **CRITICAL** if a call site queries a multi-tenant knowledge base with neither a filter nor a user context. **HIGH** if the filter value comes from the request body, a header or a query parameter.\n- **User context derived from a verified identity.** On an ACL-aware knowledge base the same question applies to `userContext.userId`. The service does not authenticate end users and cannot verify the identity it is handed — it filters on whatever you send — so that email must come from the verified session (the token subject or an ID-token claim). **CRITICAL** if `userId` is caller-supplied: one edited request reads another tenant's documents. **HIGH** if the address passed is not the one the connected source knows the user by; matching is on the email with no alias resolution, so a mismatch returns nothing and invites someone to \"fix\" it by dropping the context. **MEDIUM** if group membership is the only thing standing between tenants *and* the connector behind that data source has no real-time check — memberships are resolved from the last crawl, so a removal takes effect only at the next sync. On a connector that re-verifies per query the live check corrects that at query time: record it **INFO**. If which of the two the connector is was never established, say the real-time support is unknown and leave the severity off rather than guessing at one.\n- **Prove the leakage path is closed.** Run one retrieve as tenant A with a question only tenant B's documents answer. Substitute `\u003Ckb-id>` and use the branch that matches the mechanism:\n\n```bash\neval \"$(\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fbin\u002Fvibe-slug\" 2>\u002Fdev\u002Fnull)\" 2>\u002Fdev\u002Fnull || SLUG=\"unknown\"\nEVAL_DIR=\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fprojects\u002F${SLUG:-unknown}\"\nmkdir -p \"$EVAL_DIR\"\nOUT=\"$EVAL_DIR\u002Fkb-isolation-$(date +%Y-%m-%d).json\"\nSEARCH_KEY=vectorSearchConfiguration    # managedSearchConfiguration on a MANAGED KB\nCFG=$(jq -nc --arg k \"$SEARCH_KEY\" '{($k):{numberOfResults:5,filter:{equals:{key:\"tenant\",value:\"tenant-a\"}}}}')\nERR=$(aws bedrock-agent-runtime retrieve --knowledge-base-id \u003Ckb-id> \\\n  --retrieval-query \"$(jq -nc --arg text '\u003Cquestion only tenant B can answer>' '{text:$text}')\" \\\n  --retrieval-configuration \"$CFG\" \\\n  --output json 2>&1 > \"$OUT\")\nrc=$?\nif [ \"$rc\" -ne 0 ]; then\n  printf 'ISOLATION: N-A (exit %s)\\n%s\\n' \"$rc\" \"$ERR\"\nelse\n  jq -r '.retrievalResults[] | [(.location.type \u002F\u002F \"?\"), (.documentId \u002F\u002F \"-\"), (.score|tostring)] | @tsv' \"$OUT\"\n  printf 'ISOLATION: %s result(s)\\n' \"$(jq '.retrievalResults | length' \"$OUT\")\"\nfi\n```\n\nOn an ACL-aware knowledge base, drop the `filter` from `CFG` and add\n`--user-context '{\"userId\":\"user-a@example.com\"}'` with an address that belongs to\ntenant A.\n\nOn a `SQL` knowledge base there is no filter and no user context to send: drop\n`--retrieval-configuration` and the `CFG` line with it, run the probe as the caller the\napplication would use for tenant A, and read whether the returned rows belong to tenant\nA. What holds tenants apart there is the query engine's table and column inclusions and\nexclusions plus the grants on the database role, so review those alongside the probe —\na generated query that can reach another tenant's rows is **CRITICAL** on the same\nterms as a missing filter.\n\nPass the query as JSON, not as the `text=...` shorthand: the shorthand parser splits\non commas, so any question containing one is read as a list and the call fails with\n`Invalid type for parameter retrievalQuery.text`. That kills German questions in\nparticular. Without `jq`, write the object out by hand:\n`--retrieval-query '{\"text\":\"Wie lange ist die Frist, wenn ich kuendige?\"}'`.\n\nRead the exit status before the results. An empty list is a pass only when `rc` is 0;\na denied `bedrock:Retrieve`, a throttle and a genuinely empty result all print\nnothing, and only the status tells them apart, so report N-A on a non-zero exit\nrather than a clean bill. On a zero exit, every returned document must belong to\ntenant A. A tenant B document is **CRITICAL**, and the fix is at ingestion (missing or\nwrong metadata, or permissions the connector did not crawl), not in the query.\n\n- **Who can write to the corpus.** Everything a writer puts in a data-source prefix is retrieved later as context, so the write path is a trust boundary. List the principals that can write there: the `s3:PutObject` grants in the bucket policy (`aws s3api get-bucket-policy --bucket \u003Cbucket>`) and in the repo's IAM (`grep -rnE 's3:(PutObject|\\*)' --include='*.tf' --include='*.json' --include='*.yaml' --include='*.ts' .` near the bucket name). Then look for paths that put user content there: upload handlers, presigned PUT URLs, or a sync from a source end users edit, landing in a prefix a data source reads. Untrusted content that is retrievable by other tenants — the upload lands where the knowledge base embeds it and no per-tenant filter or user context keeps it with its author — is **HIGH**: one customer writes text another customer's assistant reads as context. Chain it with framing: if the answer path pastes retrieved passages into the prompt without marking them as data (`\u002Fbedrock-guardrails` Step 7), the same write is an indirect prompt injection into every tenant's session, and the finding says both.\n- **Source access control mirrored.** If the S3 bucket, Confluence space, or SharePoint site restricts who can read what, that restriction has to be reproduced as metadata plus a filter, because the knowledge base reads everything with its service role. **HIGH** if a restricted source feeds the knowledge base and no metadata carries the restriction.\n\n---\n\n## Step 4: Embeddings\n\n- **Model and dimension.** From Step 1, record the embedding model id (for example `amazon.titan-embed-text-v2:0` or `cohere.embed-multilingual-v3`) and the configured dimensions. Confirm the vector index in the store was created with the same dimension: for OpenSearch Serverless read the index mapping, for pgvector read the column type (`vector(1024)`). A mismatch fails at ingestion and shows up as every job failing; **HIGH**.\n- **Language coverage.** Ask which languages the documents and the users are in. The usual case is German plus English. **HIGH** if the corpus or the questions include German and the model is English-only (`cohere.embed-english-v3`). Titan Text Embeddings v2 and Cohere Embed Multilingual cover both. Verify with the golden set in Step 6: include at least five German questions against English passages and five the other way round, and report their recall separately.\n- **Re-embedding plan.** An embedding model cannot be changed on an existing Bedrock knowledge base; a model change means a new knowledge base, a full ingestion, and a cutover. **MEDIUM** if nothing in the repo or runbooks describes how that is done, and note the full ingestion cost from Step 7 next to the finding.\n\n---\n\n## Step 5: Sync\n\nThe knowledge base only knows what the last ingestion job saw.\n\n```bash\naws bedrock-agent list-ingestion-jobs --knowledge-base-id \u003Ckb-id> --data-source-id \u003Cdata-source-id> --max-results 20 --output json\n```\n\nFor each data source, record the last job's status, start time, and its\n`statistics` (documents scanned, new, modified, deleted, failed) plus\n`failureReasons` when present. Then check:\n\n- **Schedule.** Find what starts the job: an EventBridge rule, a Step Functions state machine, a CI step, or a person. `grep -rnE 'start_ingestion_job|StartIngestionJob|start-ingestion-job' .` and `aws events list-rules --output json | grep -i ingest`. **MEDIUM** if the only trigger is manual and the source changes weekly or faster. **HIGH** if the last successful job is older than two update cycles of the source.\n- **Failure alerts.** **MEDIUM** if a job can fail with nobody told; look for a CloudWatch alarm or a notification on the job status. `numberOfDocumentsFailed` above zero on the last job with no alert is the same finding.\n- **Deletions propagate.** `dataDeletionPolicy` on the data source: `DELETE` removes vectors when a document leaves the source, `RETAIN` keeps them. **HIGH** if `RETAIN` is set and documents are ever removed for legal or contractual reasons; the knowledge base will keep answering from them.\n- **Source versioning.** **LOW** if the S3 bucket has versioning off and nobody can say what the knowledge base was built from a month ago. **INFO** if it is on.\n\nFor a hand-built pipeline, the same four questions apply to whatever loop upserts\ninto the vector store.\n\n---\n\n## Step 6: Retrieval quality\n\nNumbers, not opinions. Build a golden set, run retrieve-only against it, then check a\nsample of generated answers.\n\n**Golden set.** Write 20 to 40 question and expected-passage pairs. Draw questions\nfrom the sampled documents in Step 2, from support tickets or chat logs if the user\ncan share them, and from the user directly. Each line names the source document that\nanswers it and a short quote of the passage. Include the language split from Step 4\nand at least three questions that the corpus does not answer (the right result is\nnothing above the score threshold). Write it here:\n\n```bash\neval \"$(\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fbin\u002Fvibe-slug\" 2>\u002Fdev\u002Fnull)\" 2>\u002Fdev\u002Fnull || SLUG=\"unknown\"\nEVAL_DIR=\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fprojects\u002F${SLUG:-unknown}\"\nmkdir -p \"$EVAL_DIR\"\nEVAL_SET=\"$EVAL_DIR\u002Fkb-eval-$(date +%Y-%m-%d).jsonl\"\necho \"EVAL_SET: $EVAL_SET\"\n```\n\nOne JSON object per line:\n\n```json\n{\"id\":\"q01\",\"question\":\"Wie lange ist die Kündigungsfrist?\",\"lang\":\"de\",\"tenant\":\"tenant-a\",\"expected_uri\":\"s3:\u002F\u002Fdocs\u002Ftenant-a\u002Fcontract-terms.pdf\",\"expected_passage\":\"Die Kündigungsfrist beträgt drei Monate zum Quartalsende.\",\"answerable\":true}\n```\n\nThe `tenant` field belongs there only when Step 3 found a tenant attribute in the\ndocument metadata. On a knowledge base with no such attribute, leave it out — every\nline then carries the same fields minus `tenant`.\n\n**Retrieve-only run.** For every line, call `retrieve` with `numberOfResults` 5,\nfiltered by tenant when the line has one, and record the ranked hits: the document\nthe hit came from, its `documentId`, its `content.type`, its `content.text`, its\n`content.row` and its score. Keep the text — the metrics below cannot be computed\nwithout it — and keep the row with it: a structured result carries its payload in\n`content.row`, an array of `columnName`, `columnValue` and `type`, with `content.type`\nset to `ROW` and `content.text` empty. `documentId` identifies the\ndocument, not the chunk, and a retrieval result carries no per-chunk identifier, so\nthe passage judgement below is made from the text and nothing else can stand in for\nit. The filter clause is dropped for a\nline with no `tenant` field: filtering on a metadata key the documents do not carry\nreturns nothing for every question, which scores 0.00 and manufactures a HIGH finding\nagainst a knowledge base that is fine.\n\nResult locations come in ten shapes, one per source type, and only one is populated\nper hit; `location` can also be absent. Reading three of them and calling the rest\nnull turns every Salesforce, SharePoint, OneDrive, Google Drive, Kendra, custom or\nstructured hit into a miss. Normalise all of them, and fall back to `documentId`.\n\nWith `jq` present the loop is this, with `\u003Ckb-id>` substituted before it runs (it\nre-derives the eval-set path, since each block runs in its own shell):\n\n```bash\nset -o pipefail\neval \"$(\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fbin\u002Fvibe-slug\" 2>\u002Fdev\u002Fnull)\" 2>\u002Fdev\u002Fnull || SLUG=\"unknown\"\nEVAL_DIR=\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fprojects\u002F${SLUG:-unknown}\"\nEVAL_SET=\"$EVAL_DIR\u002Fkb-eval-$(date +%Y-%m-%d).jsonl\"\nRESULTS=\"$EVAL_DIR\u002Fkb-eval-results-$(date +%Y-%m-%d).jsonl\"\nSEARCH_KEY=vectorSearchConfiguration    # managedSearchConfiguration on a MANAGED KB\n: > \"$RESULTS\"\nwhile IFS= read -r line; do\n  id=$(printf '%s' \"$line\" | jq -r .id)\n  q=$(printf '%s' \"$line\" | jq -r .question)\n  t=$(printf '%s' \"$line\" | jq -r '.tenant \u002F\u002F empty')\n  if [ -n \"$t\" ]; then\n    cfg=$(jq -nc --arg k \"$SEARCH_KEY\" --arg t \"$t\" '{($k):{numberOfResults:5,filter:{equals:{key:\"tenant\",value:$t}}}}')\n  else\n    cfg=$(jq -nc --arg k \"$SEARCH_KEY\" '{($k):{numberOfResults:5}}')\n  fi\n  raw=$(aws bedrock-agent-runtime retrieve --knowledge-base-id \u003Ckb-id> \\\n    --retrieval-query \"$(jq -nc --arg text \"$q\" '{text:$text}')\" \\\n    --retrieval-configuration \"$cfg\" \\\n    --output json 2>&1)\n  rc=$?\n  if [ \"$rc\" -ne 0 ]; then\n    jq -nc --arg id \"$id\" --argjson rc \"$rc\" --arg err \"$raw\" \\\n      '{id:$id, status:\"error\", exit:$rc, error:$err, hits:[]}' >> \"$RESULTS\"\n    continue\n  fi\n  printf '%s' \"$raw\" | jq -c --argjson g \"$line\" '\n    {id:$g.id, status:\"ok\", expected:$g.expected_uri,\n     hits:[.retrievalResults[] | {\n       doc: (.location.s3Location.uri \u002F\u002F .location.webLocation.url\n             \u002F\u002F .location.confluenceLocation.url \u002F\u002F .location.salesforceLocation.url\n             \u002F\u002F .location.sharePointLocation.url \u002F\u002F .location.oneDriveLocation.url\n             \u002F\u002F .location.googleDriveLocation.url \u002F\u002F .location.kendraDocumentLocation.uri\n             \u002F\u002F .location.customDocumentLocation.id \u002F\u002F .location.sqlLocation.query\n             \u002F\u002F .documentId \u002F\u002F null),\n       loc_type: (.location.type \u002F\u002F \"UNKNOWN\"),\n       doc_id: (.documentId \u002F\u002F null),\n       content_type: (.content.type \u002F\u002F \"UNKNOWN\"),\n       text: (.content.text \u002F\u002F \"\"),\n       row: (.content.row \u002F\u002F []),\n       score: .score}]}' >> \"$RESULTS\"\ndone \u003C \"$EVAL_SET\"\n```\n\nSame reason for the query object as in Step 3: `text=$q` is shorthand and splits on\nthe first comma, so a question with a comma in it fails the call and silently drops\nthat line from the score.\n\nOn a `SQL` knowledge base drop `--retrieval-configuration` and both `cfg` branches —\nthe settings they carry apply to unstructured sources only — and judge each answer\nagainst the expected value the golden line names rather than against a passage. The\njudgement reads `row`, not `text` — `text` is empty on every structured hit, and the\nreturned value is the `columnValue` of the `row` entry whose `columnName` is the one\nthe golden line expects. There are no chunks to rank, so report the share of questions\nanswered with the right value and leave the passage and MRR columns blank.\n\n**Check the denominator before scoring anything.** A throttle or a denied\n`bedrock:Retrieve` costs one row, and a metric computed over the survivors reads\nbetter than the truth:\n\nThis block runs in its own shell, so it re-derives the paths the same way Steps 3, 6\nand 7 do. It exits non-zero on anything it cannot establish — an unwritable id file\nprints an all-clear otherwise, which is the failure it exists to catch:\n\n```bash\nset -euo pipefail\neval \"$(\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fbin\u002Fvibe-slug\" 2>\u002Fdev\u002Fnull)\" 2>\u002Fdev\u002Fnull || SLUG=\"unknown\"\nEVAL_DIR=\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fprojects\u002F${SLUG:-unknown}\"\nEVAL_SET=\"$EVAL_DIR\u002Fkb-eval-$(date +%Y-%m-%d).jsonl\"\nRESULTS=\"$EVAL_DIR\u002Fkb-eval-results-$(date +%Y-%m-%d).jsonl\"\nASKED=\"$EVAL_DIR\u002F.ids-asked\"\nGOT=\"$EVAL_DIR\u002F.ids-got\"\nfor f in \"$EVAL_SET\" \"$RESULTS\"; do\n  [ -s \"$f\" ] || { printf 'DENOMINATOR: N-A (missing or empty %s)\\n' \"$f\"; exit 1; }\ndone\njq -r .id \"$EVAL_SET\" | sort > \"$ASKED\" || { printf 'DENOMINATOR: N-A (cannot write %s)\\n' \"$ASKED\"; exit 1; }\njq -r .id \"$RESULTS\"  | sort > \"$GOT\"   || { printf 'DENOMINATOR: N-A (cannot write %s)\\n' \"$GOT\"; exit 1; }\nmissing=$(comm -23 \"$ASKED\" \"$GOT\")\ndupes=$(uniq -d \u003C \"$GOT\")\nerrs=$(jq -r 'select(.status==\"error\") | .id' \"$RESULTS\" | tr '\\n' ' ')\nprintf 'asked=%s got=%s\\n' \"$(wc -l \u003C \"$ASKED\" | tr -d ' ')\" \"$(wc -l \u003C \"$GOT\" | tr -d ' ')\"\nprintf 'missing=[%s] duplicate=[%s] errors=[%s]\\n' \"$missing\" \"$dupes\" \"$errs\"\n[ -z \"${missing}${dupes}${errs}\" ] || { printf 'DENOMINATOR: FAIL — do not score this run\\n'; exit 1; }\nprintf 'DENOMINATOR: OK\\n'\n```\n\nA non-zero exit, or anything but `DENOMINATOR: OK` on the last line, stops the\nscoring. Fix the cause and re-run the affected\nquestions — pause and retry on a throttle, get the permission for an authorization\nerror — or, if it cannot be fixed, report the metrics as N-A for that subset and\nstate how many questions were dropped and why. Do not score a partial set as if it\nwere whole.\n\nWithout `jq`, run the calls one at a time and record the ranks by hand; the set is\nsmall enough.\n\n**Metrics.** Two different measurements, and the difference decides what a number is\nallowed to claim:\n\n- **Document hit rate@5** — the share of answerable questions whose `expected_uri`\n  matches the `doc` of any of the five hits. It says the right file came back. It\n  says nothing about the passage: on a long manual, a chunk about an unrelated\n  section scores exactly as a hit, so this number alone cannot support a claim about\n  retrieval quality and cannot feed a rank-based metric.\n- **Passage hit rate@5 and MRR** — read the `text` of the hits in rank order and mark\n  the first one that actually answers the question. `expected_passage` is the\n  reference; an equally correct passage elsewhere counts, a chunk from the right\n  document that does not answer it does not. Passage hit rate@5 is the share with\n  such a hit among the five; MRR is the mean of 1\u002Frank of that hit, 0 when there is\n  none. Record the rank you assigned per question alongside its row so the next\n  review can see the judgement.\n\nReport both, for the whole set and per language. If the passages were not judged,\nprint the document number under its own name, leave the passage and MRR columns\nblank, and write \"passage relevance not judged\" under the table — do not carry the\ndocument-level rank into the MRR column.\n\nFor the unanswerable questions, report how many returned a top score above the\nthreshold the application uses (a false positive).\n\nFindings, all read off the passage numbers: **HIGH** if passage hit rate@5 is under\n0.7, **MEDIUM** under 0.85, **LOW** if MRR is under 0.5 while passage hit rate@5 is\nfine (the right chunk is there but ranked low; a reranker or smaller chunks are the\nusual fix). **MEDIUM** if a language subset trails the other by more than 0.15. When\nonly the document number was measured, cap every one of these at **MEDIUM** and say\nso — that evidence cannot tell a right passage from a wrong one in the right file.\n\n**Generation and faithfulness.** Pick 10 answerable questions and run\n`aws bedrock-agent-runtime retrieve-and-generate` (or the application's own answer\nendpoint) with citations on. For each answer check two things by reading the cited\npassages: every factual claim in the answer appears in a cited passage, and the\ncitation points at the expected document. Score each answer `faithful`, `partly`\n(a claim with no support), or `wrong`. **HIGH** if two or more of ten are `wrong`.\nRecord the model id and the prompt template used, since both move the result.\n\nThese calls cost tokens. Say the count before running (40 retrieves plus 10\ngenerations is the default) and do not loop beyond the golden set.\n\n---\n\n## Step 7: Cost\n\nThe bill has more parts than the embeddings. Walk the list, mark the items this\nknowledge base actually uses, and drop the rest with a line saying it is not\nconfigured. Pull current unit prices from the billing MCP pricing tool when the\nsession has it, otherwise from the public pricing page for each service, and write\nthe date the price was read next to it.\n\n1. **Vector store.** Read the live configuration; there is no fixed floor to assume.\n   OpenSearch Serverless capacity is a minimum and a maximum OCU count for indexing\n   and for search, set per collection group — or at account level for a Classic\n   collection that belongs to no group. The minimum can be 0, in which case an idle\n   group needs no OCUs at all, and collections inside a group share OCUs. So the idle\n   cost is whatever the configured minimums are, divided across every collection\n   sharing them, not two or four per collection. Read it and measure it:\n\n   ```bash\n   aws opensearchserverless batch-get-collection --names \u003Ccollection-name> --output json\n   aws opensearchserverless list-collection-groups --output json\n   aws opensearchserverless batch-get-collection-group --ids \u003Ccollection-group-id> --output json\n   aws opensearchserverless get-account-settings --output json\n   aws cloudwatch get-metric-statistics --namespace AWS\u002FAOSS --metric-name SearchOCU \\\n     --start-time \u003C30 days ago> --end-time \u003Cnow> --period 3600 \\\n     --statistics Average Maximum --output json\n   ```\n\n   Record the configured minimums, the measured `SearchOCU` and `IndexingOCU`\n   averages, and how many collections share the group; repeat the metric call for\n   `IndexingOCU`. Aurora Serverless v2 bills ACU-hours plus storage — read the\n   cluster's minimum capacity and the `ServerlessDatabaseCapacity` metric the same\n   way. S3 Vectors bills storage plus requests, so it has no idle floor and its cost\n   tracks query and put volume.\n2. **Embedding tokens at ingestion.** Corpus tokens are roughly the bytes of\n   extracted text divided by 4. Overlap repeats text: with fixed-size chunking at\n   `maxTokens` M and `overlapPercentage` p as a fraction, consecutive chunks advance\n   by M(1-p), so the corpus is embedded about 1\u002F(1-p) times over — 1.25x at 20%, 2x\n   at 50%. It is not 1+p, and the gap widens exactly where it costs most.\n   Hierarchical chunking embeds both levels, so count parent and child separately.\n   Multiply by the embedding price per 1,000 tokens. This is the cost of one full\n   sync and of any re-embedding.\n3. **Semantic chunking, when configured.** A `chunkingStrategy` of `SEMANTIC` calls a\n   foundation model during ingestion and is charged for that on top of the\n   embeddings, scaling with corpus size. Zero for every other strategy.\n4. **Parsing, when configured.** Neither parser is charged per document, so the object\n   count from Step 1 cannot price this line. `BEDROCK_DATA_AUTOMATION` is priced per\n   page of the document and per image processed: multiply the page estimate and the\n   image count recorded in Step 1 by the per-page and per-image rates.\n   `BEDROCK_FOUNDATION_MODEL` is priced on the input and output tokens the parser\n   model processes: estimate input tokens from the extracted text of the pages it\n   parses, output tokens from the extraction it returns, and multiply by that model's\n   input and output prices. On a large scanned-PDF corpus this can be the biggest\n   single ingestion line, and per-document pricing understates a 200-page PDF by two\n   orders of magnitude. Either parser, once set, is applied to every `.pdf` in the\n   data source, text-only ones included, so the whole PDF count is in scope. The\n   default parser adds nothing.\n5. **Embedding tokens per query.** One query embedding per retrieve; small but\n   nonzero.\n6. **Reranking, when configured.** A `rerankingConfiguration` on the search\n   configuration sends the candidate chunks through a reranker model on every query,\n   which is charged per query and can rival the generation cost on a chatty\n   application. Zero when `rerankingModelType` is `NONE` or no reranking is set.\n7. **Vector store request and data-processing charges.** Separate from item 1's\n   capacity or storage: S3 Vectors charges per query and per put, and the store's own\n   data-processing charges scale with query volume. On a high-volume, small-corpus\n   knowledge base these can exceed the embedding cost, which is why they cannot be\n   left out of a per-query figure.\n8. **Generation tokens per query.** Prompt tokens are the template plus the five\n   chunks (chunk `maxTokens` times 5) plus the question; output tokens from the Step 6\n   answers. Multiply by the model's input and output prices.\n\n**Getting the store's real monthly figure.** Filter Cost Explorer down to this store.\nA service-only filter returns the whole account's spend for that service, which on any\nshared account is not this knowledge base's bill:\n\n```bash\neval \"$(\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fbin\u002Fvibe-slug\" 2>\u002Fdev\u002Fnull)\" 2>\u002Fdev\u002Fnull || SLUG=\"unknown\"\nEVAL_DIR=\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fprojects\u002F${SLUG:-unknown}\"\nmkdir -p \"$EVAL_DIR\"\nCE_OUT=\"$EVAL_DIR\u002Fkb-cost-$(date +%Y-%m-%d).json\"\nCE_ERR=$(aws ce get-cost-and-usage \\\n  --time-period Start=\u003Cfirst of last month>,End=\u003Cfirst of this month> \\\n  --granularity MONTHLY --metrics UnblendedCost \\\n  --filter '{\"And\":[{\"Dimensions\":{\"Key\":\"SERVICE\",\"Values\":[\"Amazon OpenSearch Service\"]}},{\"Tags\":{\"Key\":\"\u003Ccost-allocation-tag>\",\"Values\":[\"\u003Cvalue>\"]}}]}' \\\n  --group-by Type=DIMENSION,Key=USAGE_TYPE \\\n  --output json 2>&1 > \"$CE_OUT\")\nrc=$?\n[ \"$rc\" -ne 0 ] && printf 'COST: N-A (exit %s)\\n%s\\n' \"$rc\" \"$CE_ERR\"\n```\n\n`--filter` takes `Dimensions`, `Tags` and `CostCategories` combined with\n`And`, `Or` and `Not`, so swap the tag clause for a cost category when the account\nuses one. The tag has to be activated as a cost allocation tag and only covers usage\nfrom its activation date. Cost Explorer is a separate permission from the rest of this\nreview: on a non-zero exit, or when the store carries no tag or category that isolates\nit, report the store cost as N-A and price it instead from the measured usage in item\n1 against published unit rates. Say in the report which of the two routes produced the\nnumber. Never print an unfiltered service total as this store's cost.\n\nReport a per-1000-queries figure: fixed monthly cost of the store divided by the\nobserved or expected monthly query volume, plus the per-query items. Findings:\n**MEDIUM** if the store's idle floor is more than half of the monthly bill at the\ncurrent query volume (S3 Vectors or pgvector on an existing database would cost less)\n— this one needs both the configured minimum capacity and a month of OCU or ACU\nmetrics showing usage sitting at that minimum; with only one of the two it is **LOW**\nand written as a question for the user, not a verdict. **LOW** if prompt tokens per\nquery are over 4000 because of large chunks times `numberOfResults`.\n\n---\n\n## Output\n\n```bash\neval \"$(\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fbin\u002Fvibe-slug\" 2>\u002Fdev\u002Fnull)\" 2>\u002Fdev\u002Fnull || SLUG=\"unknown\"\nEVAL_DIR=\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fprojects\u002F${SLUG:-unknown}\"\nmkdir -p \"$EVAL_DIR\"\nREPORT=\"$EVAL_DIR\u002Fkb-review-$(date +%Y-%m-%d).md\"\necho \"REPORT: $REPORT\"\n```\n\nWrite the report to `$REPORT` (overwrite today's if it exists) and print it in full.\n\n```\nKNOWLEDGE BASE REVIEW\n=====================\n\nAccount:   \u003Calias or last 4 digits> (profile: \u003Cname>)     Region: \u003Cregion>\nDate:      2026-08-14\nMode:      Bedrock knowledge base (type VECTOR) | hand-built pipeline\n\nKB SUMMARY\nField              Value\n-----------------  ----------------------------------------------\nKnowledge base     kb-docs-prod (ABCD1234), type VECTOR\nVector store       OpenSearch Serverless, group min 2 index \u002F 2 search OCU shared\n                   with 3 collections, 1024-dim index\nEmbedding model    amazon.titan-embed-text-v2:0, 1024 dims\nData sources       s3:\u002F\u002Fdocs\u002Ftenant-*\u002F (1,240 PDF, 380 DOCX, 4.1 GB), confluence ENG space\nChunking           hierarchical, parent 1500 \u002F child 300 tokens, overlap 60\nParsing            none (default text extraction)\nIsolation          metadata filter on tenant (no ACL awareness configured)\nMetadata           tenant, classification on 1,580 of 1,620 documents\nDeletion policy    RETAIN\nLast sync          2026-08-12 03:10 UTC, 0 failed, EventBridge nightly\nLanguages          de 70% \u002F en 30%\n\nFINDINGS\n#  Severity  Area        Evidence                                              Fix\n-  --------  ----------  ----------------------------------------------------  --------------------------------------------\n1  CRITICAL  isolation   api\u002Fsearch.py:88 retrieve() has no filter             derive filter from the session tenant\n2  HIGH      sync        dataDeletionPolicy RETAIN; legal deletes monthly      set DELETE, re-run sync, verify vector count\n3  MEDIUM    chunking    table values in pricing.pdf split across 3 chunks     parsingConfiguration with a foundation model\n4  MEDIUM    metadata    40 documents without .metadata.json                   generate sidecars in the export job\n5  MEDIUM    cost        min 2 OCU idle = 68% of tagged spend; SearchOCU avg 2.0  evaluate S3 Vectors for this volume\n\nRETRIEVAL METRICS (golden set: 32 questions, 5 unanswerable, 0 retrieval errors)\nSubset      n    doc hit@5   passage hit@5   MRR    false positives\n----------  ---  ---------   -------------   ----   ---------------\nall         27   0.81        0.70            0.55   1 of 5\nde          19   0.79        0.68            0.52   -\nen           8   0.88        0.75            0.63   -\nFaithfulness (10 answers, \u003Cmodel id>): 7 faithful, 2 partly, 1 wrong\n\nCOST (USD; prices read 2026-08-14; store cost from Cost Explorer, tag kb=docs-prod)\nItem                            Monthly      Per 1,000 queries\n------------------------------  ----------   -----------------\nVector store (min 2 OCU + 6 GB) 351.00       39.00\nSemantic chunking               not configured\nParsing                         default parser, no charge\nFull re-embed (one-off)         18.40        -\nQuery embeddings                0.20         0.02\nReranking                       not configured\nStore requests                  included in OCU capacity above\nGeneration (\u003Cmodel id>)         61.00        6.80\nTotal at 9,000 queries\u002Fmonth    412.20       45.82\n\nEval set:    ${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fprojects\u002F\u003Cslug>\u002Fkb-eval-2026-08-14.jsonl\nResults:     ${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fprojects\u002F\u003Cslug>\u002Fkb-eval-results-2026-08-14.jsonl\nReport:      ${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fprojects\u002F\u003Cslug>\u002Fkb-review-2026-08-14.md\n```\n\nSeverity labels: `CRITICAL`, `HIGH`, `MEDIUM`, `LOW`, `INFO`. Every finding names a\nfile, a resource id, or a command output as evidence.\n\n---\n\n## Important Rules\n\n1. **Read-only against AWS.** This skill calls get, list, describe, retrieve and retrieve-and-generate. It never starts an ingestion job, never creates, updates or deletes a knowledge base or data source, never writes to S3, and never changes a vector index. The fixes in the findings are for the user to apply.\n2. **Never start a sync.** Even when the last job is stale. Report it; a sync costs embedding tokens and can surface documents the user did not mean to publish yet.\n3. **Sample, do not dump.** Copy at most five documents per format into the scratchpad, read them there, and quote at most one sentence per finding. Document contents never go into the report or into learnings.\n4. **The eval set is the deliverable.** Write it before scoring anything, keep it in the project directory under `~\u002F.vibestack`, and print its path. The next review reuses it so the numbers are comparable.\n5. **Say what the runtime calls cost.** Retrieve and generate calls are billed. State the count before running the golden set and stay within it.\n6. **Isolation findings are never downgraded.** A missing tenant filter, or a user context the caller can set, is CRITICAL regardless of how few tenants exist today. A retrieve that exits non-zero proves nothing either way: report it N-A, never as a pass.\n7. **Prices carry a date.** Unit prices change; every cost figure names the day the price was read and the volume it assumes.\n8. **No account ids or document text in learnings.** Refer to the account by alias or slug and to documents by their role, never by content.\n\n{{include lib\u002Fsnippets\u002Fcapture-learnings.md}}\n","If this session was compacted, the copy of this skill in context may be cut off.\nRead `${CLAUDE_SKILL_DIR}\u002FSKILL.md` from disk and resume at the step you are on.\n\n## When to invoke\n\nUse when: \"knowledge base review\", \"rag review\", \"bedrock knowledge base\", \"chunking strategy\", \"retrieval quality\", \"recall at k\", \"vector store review\", \"why does the bot not find the document\".\n\n## Preamble\n\n```bash\neval \"$(\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fbin\u002Fvibe-slug\" 2>\u002Fdev\u002Fnull)\" 2>\u002Fdev\u002Fnull || SLUG=\"unknown\"\n_LEARN_FILE=\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fprojects\u002F${SLUG:-unknown}\u002Flearnings.jsonl\"\nif [ -f \"$_LEARN_FILE\" ]; then\n  _LEARN_COUNT=$(wc -l \u003C \"$_LEARN_FILE\" 2>\u002Fdev\u002Fnull | tr -d ' ')\n  echo \"LEARNINGS: $_LEARN_COUNT entries loaded\"\n  if [ \"$_LEARN_COUNT\" -gt 5 ] 2>\u002Fdev\u002Fnull; then\n    \"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fbin\u002Fvibe-learnings-search\" --limit 5 2>\u002Fdev\u002Fnull || true\n  fi\nelse\n  echo \"LEARNINGS: none yet\"\nfi\n```\n\n{{include lib\u002Fsnippets\u002Fsession-host.md}}\n\n{{include lib\u002Fsnippets\u002Fdecision-brief.md}}\n\n{{include lib\u002Fsnippets\u002Fworking-protocols.md}}\n\n{{include lib\u002Fsnippets\u002Fstate-protocols.md}}\n\n## User-invocable\nWhen the user types `\u002Fkb-review`, run this skill.\n\n---\n\n## Step 1: Inventory\n\nFind out what the knowledge base is made of before judging any of it. Two places to\nlook: infrastructure code in the repo, and the live account.\n\n**Repo first.** Grep for the Terraform resources and for the SDK calls a hand-built\npipeline would use:\n\n```bash\ngrep -rnE 'aws_bedrockagent_(knowledge_base|data_source)|aws_opensearchserverless_collection|pgvector|s3_vectors|S3_VECTORS' --include='*.tf' --include='*.py' --include='*.ts' --include='*.yaml' --include='*.yml' . 2>\u002Fdev\u002Fnull | head -40\ngrep -rnE 'RecursiveCharacterTextSplitter|SentenceSplitter|chunk_size|chunk_overlap|embed_documents|Pinecone|Qdrant|Weaviate|pgvector' --include='*.py' --include='*.ts' . 2>\u002Fdev\u002Fnull | head -40\n```\n\nIf the first grep hits, this is a Bedrock Knowledge Base and the Terraform is the\nsource of truth for configuration. If only the second hits, it is a hand-built\npipeline: read the ingestion and query code and map each of the following steps\nonto it. If neither hits, ask where the knowledge base lives.\n\n**Live account, when the CLI works.**\n\n{{include lib\u002Fsnippets\u002Faws-context.md}}\n\nThen list what exists:\n\n```bash\naws bedrock-agent list-knowledge-bases --output json\naws bedrock-agent get-knowledge-base --knowledge-base-id \u003Ckb-id> --output json\naws bedrock-agent list-data-sources --knowledge-base-id \u003Ckb-id> --output json\naws bedrock-agent get-data-source --knowledge-base-id \u003Ckb-id> --data-source-id \u003Cdata-source-id> --output json\n```\n\nEvery angle-bracket value in this skill — `\u003Ckb-id>`, `\u003Cdata-source-id>`, `\u003Cbucket>`,\n`\u003Cprefix>` and the rest — is substituted by hand from the output of the preceding\ncommand before the line is run. They are placeholders, not shell variables.\n\n**Read the knowledge base type first.** `knowledgeBaseConfiguration.type` is one of\n`VECTOR`, `MANAGED`, `KENDRA` or `SQL`, and it decides which fields exist and which\nretrieval request shape is valid in Steps 3 and 6:\n\n- `VECTOR` — a knowledge base backed by a vector store you own. Embedding model ARN\n  and dimensions come from `knowledgeBaseConfiguration.vectorKnowledgeBaseConfiguration`;\n  the store comes from `storageConfiguration.type` (OpenSearch Serverless, Aurora\n  with pgvector, S3 Vectors, Pinecone, or another). Query with\n  `retrievalConfiguration.vectorSearchConfiguration`.\n- `MANAGED` — Amazon Bedrock owns the vector store. Read\n  `knowledgeBaseConfiguration.managedKnowledgeBaseConfiguration`: `embeddingModelType`\n  is `MANAGED` (service-chosen model) or `CUSTOM` (with `embeddingModelArn`).\n  `storageConfiguration` is optional on the KnowledgeBase object and is absent here —\n  record the store as service-managed rather than reporting a missing field. Query\n  with `retrievalConfiguration.managedSearchConfiguration`.\n- `KENDRA` — an Amazon Kendra GenAI index holds the content, so there is no chunking,\n  embedding or vector-store configuration to review. Query it with `Retrieve` as you\n  would a standard knowledge base, `vectorSearchConfiguration` and all; the Kendra\n  fields come back as metadata attributes and metadata filtering works on them. Say so,\n  skip Steps 2, 4 and 7's store item, and run Steps 3, 5 and 6 unchanged.\n- `SQL` — a structured data store answers by generated SQL, not by embeddings, so there\n  is nothing chunked, embedded or stored as vectors to review. `Retrieve` returns the\n  rows the generated query produced, and none of `retrievalConfiguration` applies:\n  `numberOfResults`, search type, metadata filtering and reranking are all documented as\n  applying to unstructured data sources only. Send the query on its own, with no\n  `--retrieval-configuration` at all. Skip Steps 2, 4 and 7's store item. Step 3 keeps\n  its question but changes its evidence — there is no metadata filter to check, so read\n  the query engine's table and column inclusions and exclusions and the database grants\n  on the role the knowledge base queries with. Steps 5 and 6 run with the row-based\n  reading noted in each.\n\nSending a `vectorSearchConfiguration` to a managed knowledge base, or a\n`managedSearchConfiguration` to a `VECTOR` one, is an invalid request — a failed call\nthere is a mistake in this review, not a finding about the knowledge base. So is sending\neither one to a `SQL` knowledge base. Set the key once and reuse it:\n\n```bash\nSEARCH_KEY=vectorSearchConfiguration    # managedSearchConfiguration on a MANAGED KB\n                                        # SQL: send no retrieval configuration at all\n```\n\nThen fill in the KB summary table: knowledge base id, name and type; the store and\nembedding model from whichever branch above applies; and per data source its type\n(S3, Confluence, SharePoint, OneDrive, Google Drive, Salesforce, Box, web crawler,\ncustom — record whatever `dataSourceConfiguration.type` reports, the connector set\ngrows), location, chunking configuration, parsing configuration and\n`dataDeletionPolicy`.\n\n**Sources.** For every S3 data source, size it without downloading it:\n\n```bash\naws s3 ls s3:\u002F\u002F\u003Cbucket>\u002F\u003Cprefix> --recursive --summarize | tail -2\naws s3 ls s3:\u002F\u002F\u003Cbucket>\u002F\u003Cprefix> --recursive | awk '{print $(4)}' | sed -E 's\u002F.*\\.([A-Za-z0-9]+)$\u002F\\1\u002F' | sort | uniq -c | sort -rn\n```\n\nRecord object count, total size, and the format mix (PDF, DOCX, HTML, Markdown, CSV,\nJSON). Count the image objects (`.jpeg`, `.png`) on their own line: Step 7 prices a\nconfigured parser per page and per image, and neither number falls out of the object\ncount. When any data source sets a `parsingConfiguration`, carry a page estimate into\nStep 7 as well — read the page count of each PDF sampled in Step 2 (`pdfinfo`, or\n`mdls -name kMDItemNumberOfPages` on macOS), take the mean, multiply by the PDF object\ncount, and record the sample size beside it so the number is read as the estimate it\nis. Note the update cadence from the newest and oldest `LastModified` values and\nfrom whatever writes to the prefix (a pipeline, a person, a Confluence export) — Step 3\ntreats that writer as a trust boundary, so name it here. For\nConfluence and web sources, record the space keys or seed URLs and the crawl scope.\n\nFindings at this step: **INFO** for every data source, **MEDIUM** if a data source\npoints at a bucket prefix that also holds unrelated files (logs, exports, backups)\nthat will be embedded along with the documents.\n\n---\n\n## Step 2: Chunking\n\nRead `vectorIngestionConfiguration.chunkingConfiguration` from each data source (or\nthe splitter call in a hand-built pipeline) and compare it against the shape of the\ndocuments. Sample three to five documents per format into the scratchpad with\n`aws s3 cp`; look at them, do not paste them into the report.\n\nCheck:\n\n- **Strategy versus document shape.** Fixed-size chunks suit short, flat documents. Hierarchical chunking (parent and child levels) suits long manuals and policies where a paragraph only makes sense with its section heading. Semantic chunking suits prose with topic shifts. `NONE` means each document is treated as a single source chunk; the service applies its own chunk handling from there, and a document that violates a limit surfaces in the ingestion job, not as a quiet truncation. Do not infer truncation from the embedding model's input limit — check it. Read `numberOfDocumentsFailed` and `failureReasons` on the last job (Step 5), then run one retrieve for a phrase that appears only near the end of the largest sampled document and read `content.text` on the results. **HIGH** if documents failed ingestion, or if that tail phrase is in no returned chunk while the same document answers other questions. **MEDIUM** if the strategy is `NONE` on long documents and neither check could be run — the risk is real, the evidence is not there yet. Note also that `NONE` gives up page-number citations and the `x-amz-bedrock-kb-document-page-number` filter attribute.\n- **Chunk size and overlap.** Read `maxTokens` and `overlapPercentage` (fixed) or the level `maxTokens` and `overlapTokens` (hierarchical). **MEDIUM** if chunks are under 200 tokens on documents with long paragraphs (answers get split across chunks) or over 1000 tokens on FAQ-style content (one chunk carries several unrelated questions). **LOW** if overlap is 0 on fixed-size chunking.\n- **Tables and code.** Open a sampled PDF or DOCX that contains a table and find the corresponding chunk text with a `retrieve` call for a value from the table. **MEDIUM** if table rows come back as a run of numbers with no header context and no parsing configuration is set (`parsingConfiguration` with a foundation model or Bedrock Data Automation keeps table structure). Same check for code blocks in Markdown.\n- **Metadata files.** Each document may carry a sidecar `\u003Cname>.metadata.json` with a `metadataAttributes` object. Count them: `aws s3 ls s3:\u002F\u002F\u003Cbucket>\u002F\u003Cprefix> --recursive | grep -c '\\.metadata\\.json$'`. Compare with the document count. **MEDIUM** if fewer than 90% of documents have one and Step 3 relies on filtering.\n\n---\n\n## Step 3: Metadata and isolation\n\nThis is the step that decides whether one customer can read another customer's\ndocuments. Treat every gap here as at least **HIGH**.\n\n- **What is in the metadata.** Read three sampled `.metadata.json` files. The attributes that matter are tenant (or customer, account), department, and classification (public, internal, confidential). Record which exist and whether values are consistent (`tenant` versus `tenantId` versus `customer` across files is a finding).\n- **Which mechanism is in play.** Two are supported and they are checked differently, so decide before judging anything. *Metadata filtering*: documents carry a tenant attribute and every query passes a `filter`. *ACL awareness*, on a managed knowledge base: the connector crawls document permissions (allowed and denied users and groups) at ingestion, and the query passes a `userContext` naming the user, and the crawled permissions filter the candidates before retrieval. Some connectors add a second stage, a live call to the source per query that catches permission changes made since the last sync. Which ones do is a per-connector property and the connector set grows, so read it from that connector's own ACL page rather than from a list held in your head. Today: SharePoint, OneDrive, Google Drive, Confluence and Box document both stages; S3 and custom document pre-retrieval filtering only, because their ACLs come from a file the customer writes and there is nothing live to re-check; the web crawler supports no ACLs at all. For any connector you have not checked, record its real-time support as unknown and go read the page before a severity leans on it. Read each data source's configuration for ACL awareness, and read the call sites for which of `filter` and `userContext` they send. A knowledge base may use one, both, or neither. An ACL-enabled data source returning nothing when `userContext` is absent is documented behaviour, not a retrieval bug; non-ACL data sources in the same knowledge base still return to everyone.\n- **Filter applied on every retrieve.** Find every call site of `retrieve`, `retrieve_and_generate`, `RetrieveCommand`, or the agent's knowledge base configuration: `grep -rnE 'retrieve(_and_generate|AndGenerate)?\\(|RetrieveCommand|RetrieveAndGenerateCommand' --include='*.py' --include='*.ts' --include='*.js' .` On a metadata-filtered knowledge base, check that `retrievalConfiguration.$SEARCH_KEY.filter` is set from the caller's authenticated session, not from a request parameter the caller controls. **CRITICAL** if a call site queries a multi-tenant knowledge base with neither a filter nor a user context. **HIGH** if the filter value comes from the request body, a header or a query parameter.\n- **User context derived from a verified identity.** On an ACL-aware knowledge base the same question applies to `userContext.userId`. The service does not authenticate end users and cannot verify the identity it is handed — it filters on whatever you send — so that email must come from the verified session (the token subject or an ID-token claim). **CRITICAL** if `userId` is caller-supplied: one edited request reads another tenant's documents. **HIGH** if the address passed is not the one the connected source knows the user by; matching is on the email with no alias resolution, so a mismatch returns nothing and invites someone to \"fix\" it by dropping the context. **MEDIUM** if group membership is the only thing standing between tenants *and* the connector behind that data source has no real-time check — memberships are resolved from the last crawl, so a removal takes effect only at the next sync. On a connector that re-verifies per query the live check corrects that at query time: record it **INFO**. If which of the two the connector is was never established, say the real-time support is unknown and leave the severity off rather than guessing at one.\n- **Prove the leakage path is closed.** Run one retrieve as tenant A with a question only tenant B's documents answer. Substitute `\u003Ckb-id>` and use the branch that matches the mechanism:\n\n```bash\neval \"$(\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fbin\u002Fvibe-slug\" 2>\u002Fdev\u002Fnull)\" 2>\u002Fdev\u002Fnull || SLUG=\"unknown\"\nEVAL_DIR=\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fprojects\u002F${SLUG:-unknown}\"\nmkdir -p \"$EVAL_DIR\"\nOUT=\"$EVAL_DIR\u002Fkb-isolation-$(date +%Y-%m-%d).json\"\nSEARCH_KEY=vectorSearchConfiguration    # managedSearchConfiguration on a MANAGED KB\nCFG=$(jq -nc --arg k \"$SEARCH_KEY\" '{($k):{numberOfResults:5,filter:{equals:{key:\"tenant\",value:\"tenant-a\"}}}}')\nERR=$(aws bedrock-agent-runtime retrieve --knowledge-base-id \u003Ckb-id> \\\n  --retrieval-query \"$(jq -nc --arg text '\u003Cquestion only tenant B can answer>' '{text:$text}')\" \\\n  --retrieval-configuration \"$CFG\" \\\n  --output json 2>&1 > \"$OUT\")\nrc=$?\nif [ \"$rc\" -ne 0 ]; then\n  printf 'ISOLATION: N-A (exit %s)\\n%s\\n' \"$rc\" \"$ERR\"\nelse\n  jq -r '.retrievalResults[] | [(.location.type \u002F\u002F \"?\"), (.documentId \u002F\u002F \"-\"), (.score|tostring)] | @tsv' \"$OUT\"\n  printf 'ISOLATION: %s result(s)\\n' \"$(jq '.retrievalResults | length' \"$OUT\")\"\nfi\n```\n\nOn an ACL-aware knowledge base, drop the `filter` from `CFG` and add\n`--user-context '{\"userId\":\"user-a@example.com\"}'` with an address that belongs to\ntenant A.\n\nOn a `SQL` knowledge base there is no filter and no user context to send: drop\n`--retrieval-configuration` and the `CFG` line with it, run the probe as the caller the\napplication would use for tenant A, and read whether the returned rows belong to tenant\nA. What holds tenants apart there is the query engine's table and column inclusions and\nexclusions plus the grants on the database role, so review those alongside the probe —\na generated query that can reach another tenant's rows is **CRITICAL** on the same\nterms as a missing filter.\n\nPass the query as JSON, not as the `text=...` shorthand: the shorthand parser splits\non commas, so any question containing one is read as a list and the call fails with\n`Invalid type for parameter retrievalQuery.text`. That kills German questions in\nparticular. Without `jq`, write the object out by hand:\n`--retrieval-query '{\"text\":\"Wie lange ist die Frist, wenn ich kuendige?\"}'`.\n\nRead the exit status before the results. An empty list is a pass only when `rc` is 0;\na denied `bedrock:Retrieve`, a throttle and a genuinely empty result all print\nnothing, and only the status tells them apart, so report N-A on a non-zero exit\nrather than a clean bill. On a zero exit, every returned document must belong to\ntenant A. A tenant B document is **CRITICAL**, and the fix is at ingestion (missing or\nwrong metadata, or permissions the connector did not crawl), not in the query.\n\n- **Who can write to the corpus.** Everything a writer puts in a data-source prefix is retrieved later as context, so the write path is a trust boundary. List the principals that can write there: the `s3:PutObject` grants in the bucket policy (`aws s3api get-bucket-policy --bucket \u003Cbucket>`) and in the repo's IAM (`grep -rnE 's3:(PutObject|\\*)' --include='*.tf' --include='*.json' --include='*.yaml' --include='*.ts' .` near the bucket name). Then look for paths that put user content there: upload handlers, presigned PUT URLs, or a sync from a source end users edit, landing in a prefix a data source reads. Untrusted content that is retrievable by other tenants — the upload lands where the knowledge base embeds it and no per-tenant filter or user context keeps it with its author — is **HIGH**: one customer writes text another customer's assistant reads as context. Chain it with framing: if the answer path pastes retrieved passages into the prompt without marking them as data (`\u002Fbedrock-guardrails` Step 7), the same write is an indirect prompt injection into every tenant's session, and the finding says both.\n- **Source access control mirrored.** If the S3 bucket, Confluence space, or SharePoint site restricts who can read what, that restriction has to be reproduced as metadata plus a filter, because the knowledge base reads everything with its service role. **HIGH** if a restricted source feeds the knowledge base and no metadata carries the restriction.\n\n---\n\n## Step 4: Embeddings\n\n- **Model and dimension.** From Step 1, record the embedding model id (for example `amazon.titan-embed-text-v2:0` or `cohere.embed-multilingual-v3`) and the configured dimensions. Confirm the vector index in the store was created with the same dimension: for OpenSearch Serverless read the index mapping, for pgvector read the column type (`vector(1024)`). A mismatch fails at ingestion and shows up as every job failing; **HIGH**.\n- **Language coverage.** Ask which languages the documents and the users are in. The usual case is German plus English. **HIGH** if the corpus or the questions include German and the model is English-only (`cohere.embed-english-v3`). Titan Text Embeddings v2 and Cohere Embed Multilingual cover both. Verify with the golden set in Step 6: include at least five German questions against English passages and five the other way round, and report their recall separately.\n- **Re-embedding plan.** An embedding model cannot be changed on an existing Bedrock knowledge base; a model change means a new knowledge base, a full ingestion, and a cutover. **MEDIUM** if nothing in the repo or runbooks describes how that is done, and note the full ingestion cost from Step 7 next to the finding.\n\n---\n\n## Step 5: Sync\n\nThe knowledge base only knows what the last ingestion job saw.\n\n```bash\naws bedrock-agent list-ingestion-jobs --knowledge-base-id \u003Ckb-id> --data-source-id \u003Cdata-source-id> --max-results 20 --output json\n```\n\nFor each data source, record the last job's status, start time, and its\n`statistics` (documents scanned, new, modified, deleted, failed) plus\n`failureReasons` when present. Then check:\n\n- **Schedule.** Find what starts the job: an EventBridge rule, a Step Functions state machine, a CI step, or a person. `grep -rnE 'start_ingestion_job|StartIngestionJob|start-ingestion-job' .` and `aws events list-rules --output json | grep -i ingest`. **MEDIUM** if the only trigger is manual and the source changes weekly or faster. **HIGH** if the last successful job is older than two update cycles of the source.\n- **Failure alerts.** **MEDIUM** if a job can fail with nobody told; look for a CloudWatch alarm or a notification on the job status. `numberOfDocumentsFailed` above zero on the last job with no alert is the same finding.\n- **Deletions propagate.** `dataDeletionPolicy` on the data source: `DELETE` removes vectors when a document leaves the source, `RETAIN` keeps them. **HIGH** if `RETAIN` is set and documents are ever removed for legal or contractual reasons; the knowledge base will keep answering from them.\n- **Source versioning.** **LOW** if the S3 bucket has versioning off and nobody can say what the knowledge base was built from a month ago. **INFO** if it is on.\n\nFor a hand-built pipeline, the same four questions apply to whatever loop upserts\ninto the vector store.\n\n---\n\n## Step 6: Retrieval quality\n\nNumbers, not opinions. Build a golden set, run retrieve-only against it, then check a\nsample of generated answers.\n\n**Golden set.** Write 20 to 40 question and expected-passage pairs. Draw questions\nfrom the sampled documents in Step 2, from support tickets or chat logs if the user\ncan share them, and from the user directly. Each line names the source document that\nanswers it and a short quote of the passage. Include the language split from Step 4\nand at least three questions that the corpus does not answer (the right result is\nnothing above the score threshold). Write it here:\n\n```bash\neval \"$(\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fbin\u002Fvibe-slug\" 2>\u002Fdev\u002Fnull)\" 2>\u002Fdev\u002Fnull || SLUG=\"unknown\"\nEVAL_DIR=\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fprojects\u002F${SLUG:-unknown}\"\nmkdir -p \"$EVAL_DIR\"\nEVAL_SET=\"$EVAL_DIR\u002Fkb-eval-$(date +%Y-%m-%d).jsonl\"\necho \"EVAL_SET: $EVAL_SET\"\n```\n\nOne JSON object per line:\n\n```json\n{\"id\":\"q01\",\"question\":\"Wie lange ist die Kündigungsfrist?\",\"lang\":\"de\",\"tenant\":\"tenant-a\",\"expected_uri\":\"s3:\u002F\u002Fdocs\u002Ftenant-a\u002Fcontract-terms.pdf\",\"expected_passage\":\"Die Kündigungsfrist beträgt drei Monate zum Quartalsende.\",\"answerable\":true}\n```\n\nThe `tenant` field belongs there only when Step 3 found a tenant attribute in the\ndocument metadata. On a knowledge base with no such attribute, leave it out — every\nline then carries the same fields minus `tenant`.\n\n**Retrieve-only run.** For every line, call `retrieve` with `numberOfResults` 5,\nfiltered by tenant when the line has one, and record the ranked hits: the document\nthe hit came from, its `documentId`, its `content.type`, its `content.text`, its\n`content.row` and its score. Keep the text — the metrics below cannot be computed\nwithout it — and keep the row with it: a structured result carries its payload in\n`content.row`, an array of `columnName`, `columnValue` and `type`, with `content.type`\nset to `ROW` and `content.text` empty. `documentId` identifies the\ndocument, not the chunk, and a retrieval result carries no per-chunk identifier, so\nthe passage judgement below is made from the text and nothing else can stand in for\nit. The filter clause is dropped for a\nline with no `tenant` field: filtering on a metadata key the documents do not carry\nreturns nothing for every question, which scores 0.00 and manufactures a HIGH finding\nagainst a knowledge base that is fine.\n\nResult locations come in ten shapes, one per source type, and only one is populated\nper hit; `location` can also be absent. Reading three of them and calling the rest\nnull turns every Salesforce, SharePoint, OneDrive, Google Drive, Kendra, custom or\nstructured hit into a miss. Normalise all of them, and fall back to `documentId`.\n\nWith `jq` present the loop is this, with `\u003Ckb-id>` substituted before it runs (it\nre-derives the eval-set path, since each block runs in its own shell):\n\n```bash\nset -o pipefail\neval \"$(\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fbin\u002Fvibe-slug\" 2>\u002Fdev\u002Fnull)\" 2>\u002Fdev\u002Fnull || SLUG=\"unknown\"\nEVAL_DIR=\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fprojects\u002F${SLUG:-unknown}\"\nEVAL_SET=\"$EVAL_DIR\u002Fkb-eval-$(date +%Y-%m-%d).jsonl\"\nRESULTS=\"$EVAL_DIR\u002Fkb-eval-results-$(date +%Y-%m-%d).jsonl\"\nSEARCH_KEY=vectorSearchConfiguration    # managedSearchConfiguration on a MANAGED KB\n: > \"$RESULTS\"\nwhile IFS= read -r line; do\n  id=$(printf '%s' \"$line\" | jq -r .id)\n  q=$(printf '%s' \"$line\" | jq -r .question)\n  t=$(printf '%s' \"$line\" | jq -r '.tenant \u002F\u002F empty')\n  if [ -n \"$t\" ]; then\n    cfg=$(jq -nc --arg k \"$SEARCH_KEY\" --arg t \"$t\" '{($k):{numberOfResults:5,filter:{equals:{key:\"tenant\",value:$t}}}}')\n  else\n    cfg=$(jq -nc --arg k \"$SEARCH_KEY\" '{($k):{numberOfResults:5}}')\n  fi\n  raw=$(aws bedrock-agent-runtime retrieve --knowledge-base-id \u003Ckb-id> \\\n    --retrieval-query \"$(jq -nc --arg text \"$q\" '{text:$text}')\" \\\n    --retrieval-configuration \"$cfg\" \\\n    --output json 2>&1)\n  rc=$?\n  if [ \"$rc\" -ne 0 ]; then\n    jq -nc --arg id \"$id\" --argjson rc \"$rc\" --arg err \"$raw\" \\\n      '{id:$id, status:\"error\", exit:$rc, error:$err, hits:[]}' >> \"$RESULTS\"\n    continue\n  fi\n  printf '%s' \"$raw\" | jq -c --argjson g \"$line\" '\n    {id:$g.id, status:\"ok\", expected:$g.expected_uri,\n     hits:[.retrievalResults[] | {\n       doc: (.location.s3Location.uri \u002F\u002F .location.webLocation.url\n             \u002F\u002F .location.confluenceLocation.url \u002F\u002F .location.salesforceLocation.url\n             \u002F\u002F .location.sharePointLocation.url \u002F\u002F .location.oneDriveLocation.url\n             \u002F\u002F .location.googleDriveLocation.url \u002F\u002F .location.kendraDocumentLocation.uri\n             \u002F\u002F .location.customDocumentLocation.id \u002F\u002F .location.sqlLocation.query\n             \u002F\u002F .documentId \u002F\u002F null),\n       loc_type: (.location.type \u002F\u002F \"UNKNOWN\"),\n       doc_id: (.documentId \u002F\u002F null),\n       content_type: (.content.type \u002F\u002F \"UNKNOWN\"),\n       text: (.content.text \u002F\u002F \"\"),\n       row: (.content.row \u002F\u002F []),\n       score: .score}]}' >> \"$RESULTS\"\ndone \u003C \"$EVAL_SET\"\n```\n\nSame reason for the query object as in Step 3: `text=$q` is shorthand and splits on\nthe first comma, so a question with a comma in it fails the call and silently drops\nthat line from the score.\n\nOn a `SQL` knowledge base drop `--retrieval-configuration` and both `cfg` branches —\nthe settings they carry apply to unstructured sources only — and judge each answer\nagainst the expected value the golden line names rather than against a passage. The\njudgement reads `row`, not `text` — `text` is empty on every structured hit, and the\nreturned value is the `columnValue` of the `row` entry whose `columnName` is the one\nthe golden line expects. There are no chunks to rank, so report the share of questions\nanswered with the right value and leave the passage and MRR columns blank.\n\n**Check the denominator before scoring anything.** A throttle or a denied\n`bedrock:Retrieve` costs one row, and a metric computed over the survivors reads\nbetter than the truth:\n\nThis block runs in its own shell, so it re-derives the paths the same way Steps 3, 6\nand 7 do. It exits non-zero on anything it cannot establish — an unwritable id file\nprints an all-clear otherwise, which is the failure it exists to catch:\n\n```bash\nset -euo pipefail\neval \"$(\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fbin\u002Fvibe-slug\" 2>\u002Fdev\u002Fnull)\" 2>\u002Fdev\u002Fnull || SLUG=\"unknown\"\nEVAL_DIR=\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fprojects\u002F${SLUG:-unknown}\"\nEVAL_SET=\"$EVAL_DIR\u002Fkb-eval-$(date +%Y-%m-%d).jsonl\"\nRESULTS=\"$EVAL_DIR\u002Fkb-eval-results-$(date +%Y-%m-%d).jsonl\"\nASKED=\"$EVAL_DIR\u002F.ids-asked\"\nGOT=\"$EVAL_DIR\u002F.ids-got\"\nfor f in \"$EVAL_SET\" \"$RESULTS\"; do\n  [ -s \"$f\" ] || { printf 'DENOMINATOR: N-A (missing or empty %s)\\n' \"$f\"; exit 1; }\ndone\njq -r .id \"$EVAL_SET\" | sort > \"$ASKED\" || { printf 'DENOMINATOR: N-A (cannot write %s)\\n' \"$ASKED\"; exit 1; }\njq -r .id \"$RESULTS\"  | sort > \"$GOT\"   || { printf 'DENOMINATOR: N-A (cannot write %s)\\n' \"$GOT\"; exit 1; }\nmissing=$(comm -23 \"$ASKED\" \"$GOT\")\ndupes=$(uniq -d \u003C \"$GOT\")\nerrs=$(jq -r 'select(.status==\"error\") | .id' \"$RESULTS\" | tr '\\n' ' ')\nprintf 'asked=%s got=%s\\n' \"$(wc -l \u003C \"$ASKED\" | tr -d ' ')\" \"$(wc -l \u003C \"$GOT\" | tr -d ' ')\"\nprintf 'missing=[%s] duplicate=[%s] errors=[%s]\\n' \"$missing\" \"$dupes\" \"$errs\"\n[ -z \"${missing}${dupes}${errs}\" ] || { printf 'DENOMINATOR: FAIL — do not score this run\\n'; exit 1; }\nprintf 'DENOMINATOR: OK\\n'\n```\n\nA non-zero exit, or anything but `DENOMINATOR: OK` on the last line, stops the\nscoring. Fix the cause and re-run the affected\nquestions — pause and retry on a throttle, get the permission for an authorization\nerror — or, if it cannot be fixed, report the metrics as N-A for that subset and\nstate how many questions were dropped and why. Do not score a partial set as if it\nwere whole.\n\nWithout `jq`, run the calls one at a time and record the ranks by hand; the set is\nsmall enough.\n\n**Metrics.** Two different measurements, and the difference decides what a number is\nallowed to claim:\n\n- **Document hit rate@5** — the share of answerable questions whose `expected_uri`\n  matches the `doc` of any of the five hits. It says the right file came back. It\n  says nothing about the passage: on a long manual, a chunk about an unrelated\n  section scores exactly as a hit, so this number alone cannot support a claim about\n  retrieval quality and cannot feed a rank-based metric.\n- **Passage hit rate@5 and MRR** — read the `text` of the hits in rank order and mark\n  the first one that actually answers the question. `expected_passage` is the\n  reference; an equally correct passage elsewhere counts, a chunk from the right\n  document that does not answer it does not. Passage hit rate@5 is the share with\n  such a hit among the five; MRR is the mean of 1\u002Frank of that hit, 0 when there is\n  none. Record the rank you assigned per question alongside its row so the next\n  review can see the judgement.\n\nReport both, for the whole set and per language. If the passages were not judged,\nprint the document number under its own name, leave the passage and MRR columns\nblank, and write \"passage relevance not judged\" under the table — do not carry the\ndocument-level rank into the MRR column.\n\nFor the unanswerable questions, report how many returned a top score above the\nthreshold the application uses (a false positive).\n\nFindings, all read off the passage numbers: **HIGH** if passage hit rate@5 is under\n0.7, **MEDIUM** under 0.85, **LOW** if MRR is under 0.5 while passage hit rate@5 is\nfine (the right chunk is there but ranked low; a reranker or smaller chunks are the\nusual fix). **MEDIUM** if a language subset trails the other by more than 0.15. When\nonly the document number was measured, cap every one of these at **MEDIUM** and say\nso — that evidence cannot tell a right passage from a wrong one in the right file.\n\n**Generation and faithfulness.** Pick 10 answerable questions and run\n`aws bedrock-agent-runtime retrieve-and-generate` (or the application's own answer\nendpoint) with citations on. For each answer check two things by reading the cited\npassages: every factual claim in the answer appears in a cited passage, and the\ncitation points at the expected document. Score each answer `faithful`, `partly`\n(a claim with no support), or `wrong`. **HIGH** if two or more of ten are `wrong`.\nRecord the model id and the prompt template used, since both move the result.\n\nThese calls cost tokens. Say the count before running (40 retrieves plus 10\ngenerations is the default) and do not loop beyond the golden set.\n\n---\n\n## Step 7: Cost\n\nThe bill has more parts than the embeddings. Walk the list, mark the items this\nknowledge base actually uses, and drop the rest with a line saying it is not\nconfigured. Pull current unit prices from the billing MCP pricing tool when the\nsession has it, otherwise from the public pricing page for each service, and write\nthe date the price was read next to it.\n\n1. **Vector store.** Read the live configuration; there is no fixed floor to assume.\n   OpenSearch Serverless capacity is a minimum and a maximum OCU count for indexing\n   and for search, set per collection group — or at account level for a Classic\n   collection that belongs to no group. The minimum can be 0, in which case an idle\n   group needs no OCUs at all, and collections inside a group share OCUs. So the idle\n   cost is whatever the configured minimums are, divided across every collection\n   sharing them, not two or four per collection. Read it and measure it:\n\n   ```bash\n   aws opensearchserverless batch-get-collection --names \u003Ccollection-name> --output json\n   aws opensearchserverless list-collection-groups --output json\n   aws opensearchserverless batch-get-collection-group --ids \u003Ccollection-group-id> --output json\n   aws opensearchserverless get-account-settings --output json\n   aws cloudwatch get-metric-statistics --namespace AWS\u002FAOSS --metric-name SearchOCU \\\n     --start-time \u003C30 days ago> --end-time \u003Cnow> --period 3600 \\\n     --statistics Average Maximum --output json\n   ```\n\n   Record the configured minimums, the measured `SearchOCU` and `IndexingOCU`\n   averages, and how many collections share the group; repeat the metric call for\n   `IndexingOCU`. Aurora Serverless v2 bills ACU-hours plus storage — read the\n   cluster's minimum capacity and the `ServerlessDatabaseCapacity` metric the same\n   way. S3 Vectors bills storage plus requests, so it has no idle floor and its cost\n   tracks query and put volume.\n2. **Embedding tokens at ingestion.** Corpus tokens are roughly the bytes of\n   extracted text divided by 4. Overlap repeats text: with fixed-size chunking at\n   `maxTokens` M and `overlapPercentage` p as a fraction, consecutive chunks advance\n   by M(1-p), so the corpus is embedded about 1\u002F(1-p) times over — 1.25x at 20%, 2x\n   at 50%. It is not 1+p, and the gap widens exactly where it costs most.\n   Hierarchical chunking embeds both levels, so count parent and child separately.\n   Multiply by the embedding price per 1,000 tokens. This is the cost of one full\n   sync and of any re-embedding.\n3. **Semantic chunking, when configured.** A `chunkingStrategy` of `SEMANTIC` calls a\n   foundation model during ingestion and is charged for that on top of the\n   embeddings, scaling with corpus size. Zero for every other strategy.\n4. **Parsing, when configured.** Neither parser is charged per document, so the object\n   count from Step 1 cannot price this line. `BEDROCK_DATA_AUTOMATION` is priced per\n   page of the document and per image processed: multiply the page estimate and the\n   image count recorded in Step 1 by the per-page and per-image rates.\n   `BEDROCK_FOUNDATION_MODEL` is priced on the input and output tokens the parser\n   model processes: estimate input tokens from the extracted text of the pages it\n   parses, output tokens from the extraction it returns, and multiply by that model's\n   input and output prices. On a large scanned-PDF corpus this can be the biggest\n   single ingestion line, and per-document pricing understates a 200-page PDF by two\n   orders of magnitude. Either parser, once set, is applied to every `.pdf` in the\n   data source, text-only ones included, so the whole PDF count is in scope. The\n   default parser adds nothing.\n5. **Embedding tokens per query.** One query embedding per retrieve; small but\n   nonzero.\n6. **Reranking, when configured.** A `rerankingConfiguration` on the search\n   configuration sends the candidate chunks through a reranker model on every query,\n   which is charged per query and can rival the generation cost on a chatty\n   application. Zero when `rerankingModelType` is `NONE` or no reranking is set.\n7. **Vector store request and data-processing charges.** Separate from item 1's\n   capacity or storage: S3 Vectors charges per query and per put, and the store's own\n   data-processing charges scale with query volume. On a high-volume, small-corpus\n   knowledge base these can exceed the embedding cost, which is why they cannot be\n   left out of a per-query figure.\n8. **Generation tokens per query.** Prompt tokens are the template plus the five\n   chunks (chunk `maxTokens` times 5) plus the question; output tokens from the Step 6\n   answers. Multiply by the model's input and output prices.\n\n**Getting the store's real monthly figure.** Filter Cost Explorer down to this store.\nA service-only filter returns the whole account's spend for that service, which on any\nshared account is not this knowledge base's bill:\n\n```bash\neval \"$(\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fbin\u002Fvibe-slug\" 2>\u002Fdev\u002Fnull)\" 2>\u002Fdev\u002Fnull || SLUG=\"unknown\"\nEVAL_DIR=\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fprojects\u002F${SLUG:-unknown}\"\nmkdir -p \"$EVAL_DIR\"\nCE_OUT=\"$EVAL_DIR\u002Fkb-cost-$(date +%Y-%m-%d).json\"\nCE_ERR=$(aws ce get-cost-and-usage \\\n  --time-period Start=\u003Cfirst of last month>,End=\u003Cfirst of this month> \\\n  --granularity MONTHLY --metrics UnblendedCost \\\n  --filter '{\"And\":[{\"Dimensions\":{\"Key\":\"SERVICE\",\"Values\":[\"Amazon OpenSearch Service\"]}},{\"Tags\":{\"Key\":\"\u003Ccost-allocation-tag>\",\"Values\":[\"\u003Cvalue>\"]}}]}' \\\n  --group-by Type=DIMENSION,Key=USAGE_TYPE \\\n  --output json 2>&1 > \"$CE_OUT\")\nrc=$?\n[ \"$rc\" -ne 0 ] && printf 'COST: N-A (exit %s)\\n%s\\n' \"$rc\" \"$CE_ERR\"\n```\n\n`--filter` takes `Dimensions`, `Tags` and `CostCategories` combined with\n`And`, `Or` and `Not`, so swap the tag clause for a cost category when the account\nuses one. The tag has to be activated as a cost allocation tag and only covers usage\nfrom its activation date. Cost Explorer is a separate permission from the rest of this\nreview: on a non-zero exit, or when the store carries no tag or category that isolates\nit, report the store cost as N-A and price it instead from the measured usage in item\n1 against published unit rates. Say in the report which of the two routes produced the\nnumber. Never print an unfiltered service total as this store's cost.\n\nReport a per-1000-queries figure: fixed monthly cost of the store divided by the\nobserved or expected monthly query volume, plus the per-query items. Findings:\n**MEDIUM** if the store's idle floor is more than half of the monthly bill at the\ncurrent query volume (S3 Vectors or pgvector on an existing database would cost less)\n— this one needs both the configured minimum capacity and a month of OCU or ACU\nmetrics showing usage sitting at that minimum; with only one of the two it is **LOW**\nand written as a question for the user, not a verdict. **LOW** if prompt tokens per\nquery are over 4000 because of large chunks times `numberOfResults`.\n\n---\n\n## Output\n\n```bash\neval \"$(\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fbin\u002Fvibe-slug\" 2>\u002Fdev\u002Fnull)\" 2>\u002Fdev\u002Fnull || SLUG=\"unknown\"\nEVAL_DIR=\"${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fprojects\u002F${SLUG:-unknown}\"\nmkdir -p \"$EVAL_DIR\"\nREPORT=\"$EVAL_DIR\u002Fkb-review-$(date +%Y-%m-%d).md\"\necho \"REPORT: $REPORT\"\n```\n\nWrite the report to `$REPORT` (overwrite today's if it exists) and print it in full.\n\n```\nKNOWLEDGE BASE REVIEW\n=====================\n\nAccount:   \u003Calias or last 4 digits> (profile: \u003Cname>)     Region: \u003Cregion>\nDate:      2026-08-14\nMode:      Bedrock knowledge base (type VECTOR) | hand-built pipeline\n\nKB SUMMARY\nField              Value\n-----------------  ----------------------------------------------\nKnowledge base     kb-docs-prod (ABCD1234), type VECTOR\nVector store       OpenSearch Serverless, group min 2 index \u002F 2 search OCU shared\n                   with 3 collections, 1024-dim index\nEmbedding model    amazon.titan-embed-text-v2:0, 1024 dims\nData sources       s3:\u002F\u002Fdocs\u002Ftenant-*\u002F (1,240 PDF, 380 DOCX, 4.1 GB), confluence ENG space\nChunking           hierarchical, parent 1500 \u002F child 300 tokens, overlap 60\nParsing            none (default text extraction)\nIsolation          metadata filter on tenant (no ACL awareness configured)\nMetadata           tenant, classification on 1,580 of 1,620 documents\nDeletion policy    RETAIN\nLast sync          2026-08-12 03:10 UTC, 0 failed, EventBridge nightly\nLanguages          de 70% \u002F en 30%\n\nFINDINGS\n#  Severity  Area        Evidence                                              Fix\n-  --------  ----------  ----------------------------------------------------  --------------------------------------------\n1  CRITICAL  isolation   api\u002Fsearch.py:88 retrieve() has no filter             derive filter from the session tenant\n2  HIGH      sync        dataDeletionPolicy RETAIN; legal deletes monthly      set DELETE, re-run sync, verify vector count\n3  MEDIUM    chunking    table values in pricing.pdf split across 3 chunks     parsingConfiguration with a foundation model\n4  MEDIUM    metadata    40 documents without .metadata.json                   generate sidecars in the export job\n5  MEDIUM    cost        min 2 OCU idle = 68% of tagged spend; SearchOCU avg 2.0  evaluate S3 Vectors for this volume\n\nRETRIEVAL METRICS (golden set: 32 questions, 5 unanswerable, 0 retrieval errors)\nSubset      n    doc hit@5   passage hit@5   MRR    false positives\n----------  ---  ---------   -------------   ----   ---------------\nall         27   0.81        0.70            0.55   1 of 5\nde          19   0.79        0.68            0.52   -\nen           8   0.88        0.75            0.63   -\nFaithfulness (10 answers, \u003Cmodel id>): 7 faithful, 2 partly, 1 wrong\n\nCOST (USD; prices read 2026-08-14; store cost from Cost Explorer, tag kb=docs-prod)\nItem                            Monthly      Per 1,000 queries\n------------------------------  ----------   -----------------\nVector store (min 2 OCU + 6 GB) 351.00       39.00\nSemantic chunking               not configured\nParsing                         default parser, no charge\nFull re-embed (one-off)         18.40        -\nQuery embeddings                0.20         0.02\nReranking                       not configured\nStore requests                  included in OCU capacity above\nGeneration (\u003Cmodel id>)         61.00        6.80\nTotal at 9,000 queries\u002Fmonth    412.20       45.82\n\nEval set:    ${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fprojects\u002F\u003Cslug>\u002Fkb-eval-2026-08-14.jsonl\nResults:     ${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fprojects\u002F\u003Cslug>\u002Fkb-eval-results-2026-08-14.jsonl\nReport:      ${VIBESTACK_HOME:-$HOME\u002F.vibestack}\u002Fprojects\u002F\u003Cslug>\u002Fkb-review-2026-08-14.md\n```\n\nSeverity labels: `CRITICAL`, `HIGH`, `MEDIUM`, `LOW`, `INFO`. Every finding names a\nfile, a resource id, or a command output as evidence.\n\n---\n\n## Important Rules\n\n1. **Read-only against AWS.** This skill calls get, list, describe, retrieve and retrieve-and-generate. It never starts an ingestion job, never creates, updates or deletes a knowledge base or data source, never writes to S3, and never changes a vector index. The fixes in the findings are for the user to apply.\n2. **Never start a sync.** Even when the last job is stale. Report it; a sync costs embedding tokens and can surface documents the user did not mean to publish yet.\n3. **Sample, do not dump.** Copy at most five documents per format into the scratchpad, read them there, and quote at most one sentence per finding. Document contents never go into the report or into learnings.\n4. **The eval set is the deliverable.** Write it before scoring anything, keep it in the project directory under `~\u002F.vibestack`, and print its path. The next review reuses it so the numbers are comparable.\n5. **Say what the runtime calls cost.** Retrieve and generate calls are billed. State the count before running the golden set and stay within it.\n6. **Isolation findings are never downgraded.** A missing tenant filter, or a user context the caller can set, is CRITICAL regardless of how few tenants exist today. A retrieve that exits non-zero proves nothing either way: report it N-A, never as a pass.\n7. **Prices carry a date.** Unit prices change; every cost figure names the day the price was read and the volume it assumes.\n8. **No account ids or document text in learnings.** Refer to the account by alias or slug and to documents by their role, never by content.\n\n{{include lib\u002Fsnippets\u002Fcapture-learnings.md}}\n",{"name":5,"triggers":44,"description":51,"allowed-tools":52},[45,46,47,25,48,49,50],"knowledge base review","rag review","bedrock knowledge base","retrieval quality","recall at k","vector store review","Review a Bedrock Knowledge Base or RAG pipeline: chunking, retrieval quality, recall@5 and cost per query.\n",[35,53,54,55,56],"Read","Grep","Write","AskUserQuestion","main","777ec643a83c17a0955856b2eb7f596601709fda","skills\u002Fkb-review",0,"https:\u002F\u002Fgithub.com\u002Ftimurgaleev\u002Fvibestack","https:\u002F\u002Fgithub.com\u002Ftimurgaleev","https:\u002F\u002Ftgaleev.com\u002F","Forward Deployed Engineer \u002F\r\nAWS Community Builder \u002F Cloud Architect ","2026-11-11 04:50:32",{"source":8,"skillName":5,"commands":67,"prompt":77,"zipAvailable":78},[68,71,74],{"tool":69,"command":70},"npx","npx skills add timurgaleev\u002Fvibestack --skill kb-review",{"tool":72,"command":73},"bunx","bunx skills add timurgaleev\u002Fvibestack --skill kb-review",{"tool":75,"command":76},"pnpm","pnpm dlx skills add timurgaleev\u002Fvibestack --skill kb-review","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Ftimurgaleev\u002Fvibestack\u002Fblob\u002Fmain\u002Fskills\u002Fkb-review\u002FSKILL.md",true,[],[81,87,93,99,105,111],{"skillNo":82,"name":83,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":39},"01M71GVHCCSQCCPVM8XPSPTA73","docs","https:\u002F\u002Fgithub.com\u002Ftimurgaleev\u002Fvibestack\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Frender\u002F15-backtick-info-string\u002Fskills\u002Fdocs\u002FSKILL.md",[],[83],{"skillNo":88,"name":89,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":39},"01M71GVFT1S5MRNM0AYRT2E6VQ","fences","https:\u002F\u002Fgithub.com\u002Ftimurgaleev\u002Fvibestack\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Frender\u002F14-indented-tilde-fence\u002Fskills\u002Ffences\u002FSKILL.md",[],[89],{"skillNo":94,"name":95,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":39},"01M71GVEYB54K1S83FGRNKN8CF","dirty","https:\u002F\u002Fgithub.com\u002Ftimurgaleev\u002Fvibestack\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Frender\u002F10-check-with-drift\u002Fskills\u002Fdirty\u002FSKILL.md",[],[95],{"skillNo":100,"name":101,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":102,"categories":103,"keywords":104,"publishedAt":39},"01M71GVDX9PMND1337C4388DK4","clean","https:\u002F\u002Fgithub.com\u002Ftimurgaleev\u002Fvibestack\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Frender\u002F09-check-no-drift\u002Fskills\u002Fclean\u002FSKILL.md",[],[101],{"skillNo":106,"name":107,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":108,"categories":109,"keywords":110,"publishedAt":39},"01M71GVC3VM66F1ZQT2G8JXDR0","bare","https:\u002F\u002Fgithub.com\u002Ftimurgaleev\u002Fvibestack\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Frender\u002F08-no-frontmatter\u002Fskills\u002Fbare\u002FSKILL.md",[],[107],{"skillNo":112,"name":113,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":114,"categories":115,"keywords":116,"publishedAt":39},"01M71GV9T8DAYZT0PX52QX0TE1","idem","https:\u002F\u002Fgithub.com\u002Ftimurgaleev\u002Fvibestack\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Frender\u002F07-idempotent-rerender\u002Fskills\u002Fidem\u002FSKILL.md",[],[113]]