[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M786HS8DACMKFS3159RJ1V1B":3,"skill-same:01M786HS8DACMKFS3159RJ1V1B":57},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":31,"commitSha":32,"skillDirectory":33,"forksCount":34,"repositoryHtmlUrl":35,"repositoryHomepageUrl":6,"ownerHtmlUrl":36,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":37,"install":38,"resources":50},"01M786HS8DACMKFS3159RJ1V1B","loki-skill",null,"Query logs and service state in Grafana Loki via the logcli terminal client. Use for investigating service problems, exploring labels and series, and incident response. The agent is read-only: no mutations, no writes, no deletes.","kalaverin\u002Fskills","kalaverin","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F6841312?v=4","01M786EMXK5R6A138KWFAS33T1","Alex Kalaverin","01M786EMYBQEDP3548REZHRX2Q",1,"BSD-3-Clause","https:\u002F\u002Fgithub.com\u002Fkalaverin\u002Fskills\u002Fblob\u002Fmain\u002F_on_demand\u002Floki-skill\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: loki-skill\ndescription: \"Query logs and service state in Grafana Loki via the logcli terminal client. Use for investigating service problems, exploring labels and series, and incident response. The agent is read-only: no mutations, no writes, no deletes.\"\ntriggers:\n  request: \"logcli, loki, grafana logs, посмотри логи, найди в логах, посмотри логи на стейдже, посмотри логи на проде, проверь в локи, посмотри в графане, расследование инцидентов, разбор инцидента, разбор проблем, sentry, логи sentry, incident\"\n  reason: \"User needs to query logs or service state through Grafana Loki.\"\nruntime: true\nrequires:\n  - mandatory-tools\nversion: 0.3.0\n---\n\n# SKILL: Grafana Loki via logcli\n\n[ref: #loki-skill-intro]\n\nThis skill governs how the agent uses the `logcli` command-line client to query Grafana Loki. The agent acts as a read-only investigator: it may query logs, labels, series, and read-only metadata, but it must never mutate Loki state, write data, or run destructive commands. The skill is atomic: everything — rules, workflows, and the LogQL\u002Flogcli reference — lives in this file.\n\n## 1. Allowed commands\n\n[ref: #loki-allowed-commands]\n\nThe agent may use only these read-only `logcli` commands:\n\n- `logcli query` — main command for log lines.\n- `logcli instant-query` — evaluate a metric query at a single point in time.\n- `logcli labels` — list available label names or values.\n- `logcli series` — list series matching a selector.\n- `logcli stats` — index statistics for a selector.\n- `logcli volume` — aggregate log volume for a selector.\n- `logcli detected-fields` — fields detected by parsers.\n\nAny other subcommand is forbidden. The agent must never use `delete`, `push`, or any write-oriented command.\n\n## 2. Connection and auth\n\n[ref: #loki-connection-auth]\n\n- `LOKI_ADDR` is read from the environment. If it is not set, the agent asks the user.\n- The agent does not pass `--addr` on the command line.\n- `--tls-skip-verify` is used by default (§16 documents the internal-CA symptom it solves).\n- **Project default org (HARD):** `LOKI_ORG_ID` MUST hold the project's default tenant. If `LOKI_ORG_ID` is empty, the agent STOPS before the first query and asks the user to set the project's default org. One Loki endpoint commonly serves several tenants per business scope, so an unset org silently queries the wrong tenant or fails with 401\u002F403.\n- **Non-default scope (HARD):** when the investigation targets a scope outside the project's default org, the agent passes `--org-id` explicitly on those queries (flags override the environment) and states which org was used in the summary.\n- The agent does not handle basic auth, bearer tokens, or any other credentials. If Loki rejects the connection for authentication reasons, escalate to the user.\n- No credentials, hostnames, org ids, or tenant names are hardcoded in examples or transcripts.\n\n## 3. Hard parameter rules\n\n[ref: #loki-hard-parameter-rules]\n\n| Parameter | Rule |\n|---|---|\n| `--limit` | Always set. Never `0`. Default is `30`. Exploration: `10–50`. Deep investigation: `100–200`. Need more — ask the user. |\n| `--since` | Prefer narrow windows: `5m`, `15m`, `1h`. |\n| `--from` \u002F `--to` | RFC3339Nano with explicit timezone suffix, always `Z` for UTC, e.g. `2026-08-18T10:00:00Z`. Combine with `--timezone=UTC`. |\n| `--output` | Always `jsonl`, always paired with `--output-timestamp-format=rfc3339nano`, and always piped through `unjsonl` before the output reaches a file or the context. `raw` is forbidden. |\n| `--parallel-*` | Forbidden. Removed from the skill. |\n| `--interval` | Forbidden. Removed from the skill. |\n| `--tail` | Forbidden. Removed from the skill. |\n\n## 4. Workflows\n\n[ref: #loki-workflows]\n\n### 4.1 Query recent logs\n\n[ref: #loki-workflow-query-recent]\n\n1. Create the output directory: `mkdir -p .tmp\u002Floki-skill`.\n2. Start narrow (the default org comes from `LOKI_ORG_ID`; pass `--org-id` only for a non-default scope):\n   ```bash\n   timeout 30s logcli query \\\n     --tls-skip-verify --timezone=UTC --since=15m --output=jsonl \\\n     --output-timestamp-format=rfc3339nano --quiet --limit=30 \\\n     '{app=\"example\", env=\"dev\"}' \\\n     | unjsonl > \".tmp\u002Floki-skill\u002F$(date -u +%Y%m%dT%H%M%S)_$$.log\"\n   ```\n3. Adjust `--since` and `--limit` based on results.\n4. Add a LogQL filter: `... '{app=\"example\", env=\"dev\"} |= \"error\"'`.\n5. Summarize findings; do not paste raw output into the response.\n\n### 4.2 Explore labels and series\n\n[ref: #loki-workflow-explore]\n\n1. List labels: `logcli labels`.\n2. List values for a label: `logcli labels \u003Clabel_name>`.\n3. List series for a selector: `logcli series '{app=\"example\"}'`.\n4. Use tight selectors and narrow time windows; ask before broad selectors.\n\n### 4.3 Deep investigation\n\n[ref: #loki-workflow-deep]\n\n1. Start from a known anchor: trace id, request id, error message, or Sentry event id.\n2. Query with a tight filter on that anchor.\n3. Widen time window only if the anchor is not found.\n4. Cross-reference with related services only after the first hit.\n\n## 5. Context protection\n\n[ref: #loki-context-protection]\n\n1. **Create the output directory.** Run `mkdir -p .tmp\u002Floki-skill` before the first query.\n2. **Every query goes to a file.** The agent always pipes `logcli query` output through `unjsonl` into a temporary file under `.tmp\u002Floki-skill\u002F\u003Cutc-timestamp>_\u003Cshort-query-id>.log`. Stdout is never read into context directly.\n3. **Size gate: 50 KiB.** Check the file size with `rtk wc -c \u003Cpath>`. If the file is 50 KiB or less, read and summarize it. If it is larger, either re-fetch with a narrower query, or slice the file with `rg`, `jq`, or `rtk` equivalents.\n4. **Start narrow.** First query uses `--since=15m --limit=30`, or an exact event-based query such as a trace id or Sentry event id.\n5. **Filter first.** Always apply a LogQL filter (`|=`, `|~`, `!=`, `!~`) before broadening the query.\n6. **Output format.** Always `--output=jsonl` with `--output-timestamp-format=rfc3339nano`, piped through `unjsonl`. Raw jsonl never enters the context unprocessed.\n7. **Abort on hang.** Run `timeout 30s logcli ...`. On timeout, suggest narrowing the query.\n8. **All bulk-producing commands go to a file.** Apply the temporary-file, timeout, and size-gate rules to `logcli query`, `instant-query`, `stats`, `volume`, and `detected-fields`.\n9. **No credentials in examples.** Mask hosts, org ids, and tenant names in transcripts and summaries.\n\n## 6. Pipeline\n\n[ref: #loki-pipeline]\n\nForward flow:\n\n1. User mentions `logcli`, Loki, Grafana logs, looking at logs on stage\u002Fprod, checking Loki, investigating an incident, or Sentry problem analysis.\n2. Runtime trigger activates `loki-skill`; the agent reads `SKILL.md`.\n3. Agent detects intent: query recent logs, explore labels\u002Fseries, deep investigation, or event-based query.\n4. Agent checks environment: `LOKI_ADDR`, `LOKI_ORG_ID` (project default org).\n5. Agent builds command from allowed commands, connection flags, hard parameter rules, and context protection rules.\n6. Agent executes `timeout 30s logcli ... --output=jsonl --output-timestamp-format=rfc3339nano ... | unjsonl > .tmp\u002Floki-skill\u002F...`.\n7. Agent applies the 50 KiB size gate.\n8. Agent summarizes and responds.\n\nError routes:\n\n- Missing `LOKI_ADDR` → ask the user.\n- Missing `LOKI_ORG_ID` → ask the user to set the project's default org before any query.\n- Investigation targets a non-default scope → pass `--org-id` explicitly and name the org in the summary.\n- Loki requires authentication (basic auth, bearer token, etc.) → escalate to the user.\n- `logcli: command not found` → hard stop; ask the user to install `logcli` (prefer the Homebrew build on macOS).\n- `unjsonl: command not found` → hard stop; ask the user to install `unjsonl`.\n- Timeout → suggest narrowing.\n- Output > 50 KiB → re-fetch or slice with `rg`, `jq`, or `rtk` equivalents.\n- No results → suggest widening the window or changing the filter.\n\n## 7. Violation protocol\n\n[ref: #loki-violation-protocol]\n\nIf the agent is about to run a forbidden command, omit `--limit`, use `--limit=0`, pull raw multi-line logs into the response, or bypass the temporary-file rule: stop, discard the command, and rebuild it according to this skill.\n\n## 8. Loki architecture\n\n[ref: #loki-ref-architecture]\n\nLoki is a horizontally scalable log aggregation system inspired by Prometheus. It groups incoming log lines into streams identified by a unique set of labels. Loki indexes only the label set and the timestamp; the full log line is compressed and stored in chunks inside an object store such as S3, GCS, or Azure Blob Storage.\n\nKey components:\n\n- **Distributor** receives push requests and hashes streams to ingesters.\n- **Ingester** buffers streams in memory and writes chunks to storage.\n- **Query frontend** splits queries and merges results.\n- **Querier** fetches data from ingesters and object storage.\n- **Ruler** evaluates alerting and recording rules.\n\nStorage format:\n\n- **Index** is a table of contents mapping label sets to chunks. TSDB is the recommended index format.\n- **Chunk** contains log entries for one stream over a time range, compressed.\n\nMulti-tenancy:\n\n- Loki partitions data by tenant id.\n- The `X-Scope-OrgID` header carries the tenant.\n- Single-tenant deployments ignore the header and use tenant id `fake`.\n\nBecause only labels are indexed, query performance depends heavily on choosing low-cardinality, high-selectivity labels in the stream selector.\n\n## 9. Labels and streams\n\n[ref: #loki-ref-labels-streams]\n\nA **log stream** is the set of log lines that share exactly the same label set. Loki stores one chunk per stream per time window, so the number of streams directly impacts storage efficiency and query speed.\n\nGood labels are static and bounded:\n\n- `namespace`, `app`, `service_name`, `env`, `cluster`, `host`.\n\nBad labels are dynamic or unbounded:\n\n- `trace_id`, `request_id`, `user_id`, `order_id`.\n\nCardinality guidance:\n\n- Keep a single tenant under roughly 100,000 active streams.\n- Prefer fewer streams with larger chunks over many streams with tiny chunks.\n- Use line filters such as `|= \"trace_id=abc\"` for high-cardinality values only when the raw line literally contains that form (for example logfmt). For JSON logs parse first: `{app=\"api\"} | json | trace_id=\"abc\"`.\n\nStructured metadata allows attaching non-indexed key-value pairs to log lines at ingestion time. These values can be extracted at query time with parsers but do not explode the index.\n\n## 10. LogQL syntax overview\n\n[ref: #loki-ref-logql-syntax]\n\nEvery LogQL query has the form:\n\n```logql\n{ log stream selector } | pipeline stage | pipeline stage | ...\n```\n\nThe log stream selector is mandatory. The pipeline is optional.\n\nPipeline stages are evaluated left to right:\n\n1. **Line filters** narrow by substring or regex in the raw line.\n2. **Parsers** extract structured fields as labels.\n3. **Label filters** filter on extracted or existing labels.\n4. **Format expressions** reshape output for display.\n\nA query can end as a log query, returning lines, or as a metric query, wrapping the pipeline in a range vector and aggregation function.\n\nExamples:\n\n```logql\n{app=\"api\", env=\"prod\"} |= \"error\"\n{app=\"api\"} | json | status >= 500\n{app=\"api\"} | json | line_format \"{{.method}} {{.path}} -> {{.status}}\"\n```\n\n## 11. Stream selectors\n\n[ref: #loki-ref-stream-selectors]\n\nThe stream selector chooses which log streams to process. It uses label matchers inside curly braces.\n\nOperators:\n\n- `=` exact equality.\n- `!=` inequality.\n- `=~` regex match.\n- `!~` negative regex match.\n\nMultiple matchers inside one selector are AND-ed together.\n\nExamples:\n\n```logql\n{app=\"billing\", env=\"prod\"}\n{namespace=~\"team-.*\", service_name=\"wallet\"}\n{app!=\"debug-app\"}\n```\n\nBest practice: use the most specific label first. If `app_name` selects a smaller subset than `namespace`, prefer `{app_name=\"wallet\"}` alone.\n\n## 12. Line filters\n\n[ref: #loki-ref-line-filters]\n\nLine filters operate on the raw log line text. They are the cheapest filter type and should be placed as early as possible.\n\nOperators:\n\n- `|=` line contains string.\n- `!=` line does not contain string.\n- `|~` line matches regex.\n- `!~` line does not match regex.\n\nPerformance order:\n\n1. `|=` and `!=` are fastest.\n2. `|~` and `!~` are slower.\n3. Parsers are slower than line filters.\n\nExamples:\n\n```logql\n{app=\"api\"} |= \"timeout\"\n{app=\"api\"} != \"healthcheck\"\n{app=\"api\"} |~ \"(error|fatal)\"\n```\n\nPattern match filters `|>` and `!>` (Loki 3.x+) match the whole line against a pattern where `\u003C_>` matches arbitrary text. They are faster than regex for lines with a predictable shape. Do not confuse them with the `| pattern` parser.\n\n```logql\n{app=\"api\"} |> \"\u003C_> level=error \u003C_>\"\n```\n\n## 13. Label filters\n\n[ref: #loki-ref-label-filters]\n\nLabel filters operate on labels that already exist or were extracted by a parser. They are written after parsers or directly after the selector.\n\nOperators:\n\n- `==` or `=` equality.\n- `!=` inequality.\n- `>`, `>=`, `\u003C`, `\u003C=` numeric comparison.\n- `and` and `or` boolean chaining.\n\nExamples:\n\n```logql\n{app=\"api\"} | json | status >= 500\n{app=\"api\"} | json | method=\"POST\" and status >= 400\n{app=\"api\"} | logfmt | duration > 1.5\n```\n\nPipeline errors from parsers produce a system label `__error__`. Filter it explicitly:\n\n```logql\n{app=\"api\"} | json | __error__=\"\"\n```\n\n## 14. Parsers\n\n[ref: #loki-ref-parsers]\n\nParsers extract structured data from log lines and create temporary labels for the rest of the pipeline.\n\nAvailable parsers:\n\n- `json` parses JSON log lines.\n- `logfmt` parses `key=value` or `key=\"value\"` pairs.\n- `pattern` extracts fields via a lightweight wildcard pattern.\n- `regexp` extracts fields via a regular expression with named capture groups.\n- `unpack` extracts labels from structured metadata in the log line.\n\nJSON parser examples:\n\n```logql\n{app=\"api\"} | json\n{app=\"api\"} | json | status=\"500\"\n{app=\"api\"} | json method, status\n```\n\nThe last example extracts only `method` and `status` as labels. The parser also supports explicit mappings such as `json method=\"http_method\", status=\"http_status\"`.\n\nLogfmt parser example:\n\n```logql\n{app=\"api\"} | logfmt | level=\"error\"\n```\n\nPattern parser example:\n\n```logql\n{app=\"api\"} | pattern \"\u003C_> caller=\u003Ccaller> level=\u003Clevel> msg=\u003Cmsg>\"\n```\n\nRegexp parser example:\n\n```logql\n{app=\"api\"} | regexp \"(?P\u003Cmethod>\\w+) (?P\u003Cpath>\\S+) HTTP\u002F(?P\u003Cversion>\\d\\.\\d)\"\n```\n\nUse a single backslash inside the regex; LogQL passes it directly to the Go regex engine.\n\nParser errors populate `__error__`. Always handle errors when parser reliability is uncertain.\n\n## 15. Format expressions\n\n[ref: #loki-ref-format-expressions]\n\nFormat expressions change how results are displayed. They do not modify stored data.\n\n`line_format` rewrites the log line using Go templates:\n\n```logql\n{app=\"api\"} | json | line_format \"{{.method}} {{.path}} -> {{.status}}\"\n```\n\n`label_format` creates or rewrites labels:\n\n```logql\n{app=\"api\"} | json | label_format severity=\"{{.level | upper}}\"\n```\n\nCommon template helpers include `upper`, `lower`, and `trim`. Keep formatting late in the pipeline, after all filtering and parsing.\n\n## 16. Metric queries\n\n[ref: #loki-ref-metric-queries]\n\nMetric queries convert log streams into numeric time series. They wrap a log pipeline in a range vector `[duration]` and apply an aggregation function.\n\nCommon functions:\n\n- `rate({selector} |= \"error\" [1m])` entries per second.\n- `count_over_time({selector} [5m])` total entries in range.\n- `sum by (status) (rate({selector} | json | status >= 500 [5m]))` grouped error rate.\n- `quantile_over_time(0.99, {selector} | json | unwrap latency_ms [5m])` percentile.\n\nRange vector durations:\n\n- Use `[1m]`, `[5m]`, `[1h]` depending on granularity.\n- The range must be longer than or equal to the query step.\n- Always pass `--step` explicitly when running metric queries with `logcli` to control sampling density and server load.\n\nAggregation modifiers:\n\n- `by (label1, label2)` keeps listed labels in result.\n- `without (label1)` drops listed labels from result.\n\nBinary operators (`+`, `-`, `*`, `\u002F`, `%`, `^`) and comparison operators (`==`, `!=`, `>`, `\u003C`) work on vectors and scalars. Use `bool` after comparison to keep zeros and ones instead of filtering.\n\n## 17. logcli commands reference\n\n[ref: #loki-ref-logcli-commands]\n\n`logcli` is the command-line client for Loki. Read-only commands useful for agent work:\n\n- `logcli query \u003Clogql>` returns log lines or metric results.\n- `logcli instant-query \u003Clogql>` evaluates a metric query at a single point in time.\n- `logcli labels [label]` lists label names or values.\n- `logcli series \u003Cmatcher>` lists streams matching a selector.\n- `logcli stats \u003Cquery>` returns index statistics for a selector.\n- `logcli volume \u003Cquery>` returns aggregate log volume for a selector.\n- `logcli detected-fields \u003Cquery>` reports fields detected by parsers.\n\nForbidden commands for read-only agent use:\n\n- `delete` and all its subcommands.\n- Any write, push, or ingest operation.\n\n## 18. Connection reference\n\n[ref: #loki-ref-logcli-connection]\n\nConnection settings can come from environment variables or command-line flags. Command-line flags override environment variables; environment variables provide defaults.\n\nImportant environment variables and flags:\n\n- `LOKI_ADDR` \u002F `--addr` — Loki server URL.\n- `LOKI_ORG_ID` \u002F `--org-id` — tenant id added as `X-Scope-OrgID`.\n- `LOKI_USERNAME` \u002F `--username` and `LOKI_PASSWORD` \u002F `--password` — HTTP basic auth.\n- `LOKI_BEARER_TOKEN` \u002F `--bearer-token` — bearer token for Authorization header.\n- `LOKI_TLS_SKIP_VERIFY` \u002F `--tls-skip-verify` — skip TLS verification.\n- `LOKI_CA_CERT_PATH` \u002F `--ca-cert` — custom CA certificate.\n- `LOKI_CLIENT_CERT_PATH` \u002F `--cert` and `LOKI_CLIENT_KEY_PATH` \u002F `--key` — mutual TLS.\n\nOne endpoint frequently serves several business scopes as different tenants (for example `loki-gw.prod.internal.example-corp.net` with org id `10` for custodial wallet logs and `30` for payment-core logs, the dev gateway carrying its own id). The prescriptive rules live in §2.\n\n## 19. Output and time flags reference\n\n[ref: #loki-ref-logcli-output-time]\n\nOutput flags:\n\n- `-o default` \u002F `--output=default` — timestamp, labels, and line.\n- `-o raw` \u002F `--output=raw` — line content only (forbidden for agent use, see §3).\n- `-o jsonl` \u002F `--output=jsonl` — one JSON object per returned log line with `labels`, `line`, and `timestamp` fields.\n- `--output-timestamp-format=rfc3339nano` — nanosecond RFC3339 timestamps.\n- `--timezone=UTC` — format timestamps in UTC.\n- `-q` \u002F `--quiet` — suppress query metadata.\n\nTime flags:\n\n- `--since=15m` — relative lookback window.\n- `--from=2026-08-18T10:00:00Z` — absolute start, RFC3339Nano with explicit `Z` UTC suffix.\n- `--to=2026-08-18T11:00:00Z` — absolute end, RFC3339Nano with explicit `Z` UTC suffix.\n- `--timezone=UTC` — format output timestamps in UTC and interpret bounds without an offset as UTC.\n\nLimit flag:\n\n- `--limit=30` — cap returned entries. Never set to `0`.\n\nAgent defaults:\n\n- `--output=jsonl --output-timestamp-format=rfc3339nano --quiet --timezone=UTC --limit=30 --since=15m`, piped through `unjsonl` before the output reaches a file or the context.\n\n## 20. Query patterns\n\n[ref: #loki-ref-query-patterns]\n\nAll bash examples below are simplified. In agent use, wrap every `logcli query` in `timeout 30s`, pipe output through `unjsonl` into `.tmp\u002Floki-skill\u002F\u003Ctimestamp>_\u003Cid>.log`, and apply the 50 KiB size gate.\n\nInternal label convention: selectors are typically `{namespace=..., service_name=...}` where `service_name` is the workload name.\n\n### Recent errors for a service\n\n```bash\nmkdir -p .tmp\u002Floki-skill\ntimeout 30s logcli query \\\n  --tls-skip-verify --timezone=UTC --since=15m --output=jsonl \\\n  --output-timestamp-format=rfc3339nano --quiet --limit=30 \\\n  '{app=\"billing\", env=\"prod\"} |= \"error\"' \\\n  | unjsonl > \".tmp\u002Floki-skill\u002F$(date -u +%Y%m%dT%H%M%S)_$$.log\"\n```\n\n### Trace id search\n\nFor logfmt or literal `trace_id=...` lines:\n\n```bash\ntimeout 30s logcli query --tls-skip-verify --timezone=UTC --since=1h --output=jsonl \\\n  --output-timestamp-format=rfc3339nano --quiet --limit=100 \\\n  '{app=~\"api|worker\"} |= \"trace_id=7a98025445386b1feef1402bb13362e5\"' | unjsonl\n```\n\nFor JSON logs use a parser:\n\n```logql\n{app=~\"api|worker\"} | json | trace_id=\"7a98025445386b1feef1402bb13362e5\"\n```\n\n### HTTP 5xx with JSON logs\n\n```logql\n{app=\"api\"} | json | status >= 500\n```\n\n### Error rate per service\n\n```logql\nsum by (app) (rate({app=~\"api|worker\"} |= \"error\" [5m]))\n```\n\nFor `logcli`, add `--step=1m` (or smaller than the range) when running metric queries.\n\n### Sentry event correlation\n\nGiven a Sentry event id, search raw logs for that literal token only if the line contains it literally:\n\n```logql\n{app=~\"api|worker\"} |= \"event_id=6d9ab75c00ac479f98a629f07ad253e7\"\n```\n\nFor JSON logs parse the field first:\n\n```logql\n{app=~\"api|worker\"} | json | event_id=\"6d9ab75c00ac479f98a629f07ad253e7\"\n```\n\n### Business identifier search\n\nThree canonical anchors inside one service: URL path, workflow\u002Ftrace UUID, business `key=value` parameter (identifiers below are fictional):\n\n```logql\n{namespace=\"payments\", service_name=\"gateway-api\"} |= \"\u002Fapi\u002Fv2\u002Fwebhook\"\n{namespace=\"payments\", service_name=\"gateway-api\"} |= \"a1b2c3d4-1111-4222-8333-abcdef012345\"\n{namespace=\"payments\", service_name=\"gateway-api\"} |= \"client_id=42\"\n```\n\n## 21. Troubleshooting\n\n[ref: #loki-ref-troubleshooting]\n\n| Symptom | Likely cause | Remedy |\n|---|---|---|\n| Empty results | Time range too narrow or selector too strict | Widen `--since`, relax selectors, check labels with `logcli labels`. |\n| Timeout or slow query | Time range too wide, low-selectivity selector, or heavy parser | Narrow time, add \u003Ccode>&#124;=\u003C\u002Fcode> filter first, avoid regex. |\n| 401\u002F403 | `LOKI_ORG_ID` unset or wrong tenant for the scope | Set `LOKI_ORG_ID` to the project default org; pass explicit `--org-id` for non-default scopes. |\n| macOS cannot resolve an internal host (`*.internal.*`) | `mise`-installed `logcli` is built with the `netgo` tag and bypasses the macOS system resolver | Use the Homebrew build (`\u002Fopt\u002Fhomebrew\u002Fbin\u002Flogcli --version`) or another cgo-resolver build. |\n| `tls: failed to verify certificate: ... not standards compliant` | Go rejects some internal CA certificates as non-compliant | Keep `--tls-skip-verify`; it is the intended default, not a misconfiguration. |\n| `unjsonl: command not found` | `unjsonl` missing from PATH | Hard stop; ask the user to install `unjsonl`. |\n| `__error__` labels | Parser failed on some lines | Add \u003Ccode>&#124; __error__=\"\"\u003C\u002Fcode> or \u003Ccode>&#124; __error__!=\"\"\u003C\u002Fcode> depending on intent. |\n| Huge output file | No filter or `--limit=0` | Refetch with filter and limit; slice with `rg`, `jq`, or `rtk` equivalents. |\n| Out-of-order lines with parallel flags | `--parallel-*` splits range into parts | Avoid parallel flags; use narrow sequential queries. |\n","# SKILL: Grafana Loki via logcli\n\n[ref: #loki-skill-intro]\n\nThis skill governs how the agent uses the `logcli` command-line client to query Grafana Loki. The agent acts as a read-only investigator: it may query logs, labels, series, and read-only metadata, but it must never mutate Loki state, write data, or run destructive commands. The skill is atomic: everything — rules, workflows, and the LogQL\u002Flogcli reference — lives in this file.\n\n## 1. Allowed commands\n\n[ref: #loki-allowed-commands]\n\nThe agent may use only these read-only `logcli` commands:\n\n- `logcli query` — main command for log lines.\n- `logcli instant-query` — evaluate a metric query at a single point in time.\n- `logcli labels` — list available label names or values.\n- `logcli series` — list series matching a selector.\n- `logcli stats` — index statistics for a selector.\n- `logcli volume` — aggregate log volume for a selector.\n- `logcli detected-fields` — fields detected by parsers.\n\nAny other subcommand is forbidden. The agent must never use `delete`, `push`, or any write-oriented command.\n\n## 2. Connection and auth\n\n[ref: #loki-connection-auth]\n\n- `LOKI_ADDR` is read from the environment. If it is not set, the agent asks the user.\n- The agent does not pass `--addr` on the command line.\n- `--tls-skip-verify` is used by default (§16 documents the internal-CA symptom it solves).\n- **Project default org (HARD):** `LOKI_ORG_ID` MUST hold the project's default tenant. If `LOKI_ORG_ID` is empty, the agent STOPS before the first query and asks the user to set the project's default org. One Loki endpoint commonly serves several tenants per business scope, so an unset org silently queries the wrong tenant or fails with 401\u002F403.\n- **Non-default scope (HARD):** when the investigation targets a scope outside the project's default org, the agent passes `--org-id` explicitly on those queries (flags override the environment) and states which org was used in the summary.\n- The agent does not handle basic auth, bearer tokens, or any other credentials. If Loki rejects the connection for authentication reasons, escalate to the user.\n- No credentials, hostnames, org ids, or tenant names are hardcoded in examples or transcripts.\n\n## 3. Hard parameter rules\n\n[ref: #loki-hard-parameter-rules]\n\n| Parameter | Rule |\n|---|---|\n| `--limit` | Always set. Never `0`. Default is `30`. Exploration: `10–50`. Deep investigation: `100–200`. Need more — ask the user. |\n| `--since` | Prefer narrow windows: `5m`, `15m`, `1h`. |\n| `--from` \u002F `--to` | RFC3339Nano with explicit timezone suffix, always `Z` for UTC, e.g. `2026-08-18T10:00:00Z`. Combine with `--timezone=UTC`. |\n| `--output` | Always `jsonl`, always paired with `--output-timestamp-format=rfc3339nano`, and always piped through `unjsonl` before the output reaches a file or the context. `raw` is forbidden. |\n| `--parallel-*` | Forbidden. Removed from the skill. |\n| `--interval` | Forbidden. Removed from the skill. |\n| `--tail` | Forbidden. Removed from the skill. |\n\n## 4. Workflows\n\n[ref: #loki-workflows]\n\n### 4.1 Query recent logs\n\n[ref: #loki-workflow-query-recent]\n\n1. Create the output directory: `mkdir -p .tmp\u002Floki-skill`.\n2. Start narrow (the default org comes from `LOKI_ORG_ID`; pass `--org-id` only for a non-default scope):\n   ```bash\n   timeout 30s logcli query \\\n     --tls-skip-verify --timezone=UTC --since=15m --output=jsonl \\\n     --output-timestamp-format=rfc3339nano --quiet --limit=30 \\\n     '{app=\"example\", env=\"dev\"}' \\\n     | unjsonl > \".tmp\u002Floki-skill\u002F$(date -u +%Y%m%dT%H%M%S)_$$.log\"\n   ```\n3. Adjust `--since` and `--limit` based on results.\n4. Add a LogQL filter: `... '{app=\"example\", env=\"dev\"} |= \"error\"'`.\n5. Summarize findings; do not paste raw output into the response.\n\n### 4.2 Explore labels and series\n\n[ref: #loki-workflow-explore]\n\n1. List labels: `logcli labels`.\n2. List values for a label: `logcli labels \u003Clabel_name>`.\n3. List series for a selector: `logcli series '{app=\"example\"}'`.\n4. Use tight selectors and narrow time windows; ask before broad selectors.\n\n### 4.3 Deep investigation\n\n[ref: #loki-workflow-deep]\n\n1. Start from a known anchor: trace id, request id, error message, or Sentry event id.\n2. Query with a tight filter on that anchor.\n3. Widen time window only if the anchor is not found.\n4. Cross-reference with related services only after the first hit.\n\n## 5. Context protection\n\n[ref: #loki-context-protection]\n\n1. **Create the output directory.** Run `mkdir -p .tmp\u002Floki-skill` before the first query.\n2. **Every query goes to a file.** The agent always pipes `logcli query` output through `unjsonl` into a temporary file under `.tmp\u002Floki-skill\u002F\u003Cutc-timestamp>_\u003Cshort-query-id>.log`. Stdout is never read into context directly.\n3. **Size gate: 50 KiB.** Check the file size with `rtk wc -c \u003Cpath>`. If the file is 50 KiB or less, read and summarize it. If it is larger, either re-fetch with a narrower query, or slice the file with `rg`, `jq`, or `rtk` equivalents.\n4. **Start narrow.** First query uses `--since=15m --limit=30`, or an exact event-based query such as a trace id or Sentry event id.\n5. **Filter first.** Always apply a LogQL filter (`|=`, `|~`, `!=`, `!~`) before broadening the query.\n6. **Output format.** Always `--output=jsonl` with `--output-timestamp-format=rfc3339nano`, piped through `unjsonl`. Raw jsonl never enters the context unprocessed.\n7. **Abort on hang.** Run `timeout 30s logcli ...`. On timeout, suggest narrowing the query.\n8. **All bulk-producing commands go to a file.** Apply the temporary-file, timeout, and size-gate rules to `logcli query`, `instant-query`, `stats`, `volume`, and `detected-fields`.\n9. **No credentials in examples.** Mask hosts, org ids, and tenant names in transcripts and summaries.\n\n## 6. Pipeline\n\n[ref: #loki-pipeline]\n\nForward flow:\n\n1. User mentions `logcli`, Loki, Grafana logs, looking at logs on stage\u002Fprod, checking Loki, investigating an incident, or Sentry problem analysis.\n2. Runtime trigger activates `loki-skill`; the agent reads `SKILL.md`.\n3. Agent detects intent: query recent logs, explore labels\u002Fseries, deep investigation, or event-based query.\n4. Agent checks environment: `LOKI_ADDR`, `LOKI_ORG_ID` (project default org).\n5. Agent builds command from allowed commands, connection flags, hard parameter rules, and context protection rules.\n6. Agent executes `timeout 30s logcli ... --output=jsonl --output-timestamp-format=rfc3339nano ... | unjsonl > .tmp\u002Floki-skill\u002F...`.\n7. Agent applies the 50 KiB size gate.\n8. Agent summarizes and responds.\n\nError routes:\n\n- Missing `LOKI_ADDR` → ask the user.\n- Missing `LOKI_ORG_ID` → ask the user to set the project's default org before any query.\n- Investigation targets a non-default scope → pass `--org-id` explicitly and name the org in the summary.\n- Loki requires authentication (basic auth, bearer token, etc.) → escalate to the user.\n- `logcli: command not found` → hard stop; ask the user to install `logcli` (prefer the Homebrew build on macOS).\n- `unjsonl: command not found` → hard stop; ask the user to install `unjsonl`.\n- Timeout → suggest narrowing.\n- Output > 50 KiB → re-fetch or slice with `rg`, `jq`, or `rtk` equivalents.\n- No results → suggest widening the window or changing the filter.\n\n## 7. Violation protocol\n\n[ref: #loki-violation-protocol]\n\nIf the agent is about to run a forbidden command, omit `--limit`, use `--limit=0`, pull raw multi-line logs into the response, or bypass the temporary-file rule: stop, discard the command, and rebuild it according to this skill.\n\n## 8. Loki architecture\n\n[ref: #loki-ref-architecture]\n\nLoki is a horizontally scalable log aggregation system inspired by Prometheus. It groups incoming log lines into streams identified by a unique set of labels. Loki indexes only the label set and the timestamp; the full log line is compressed and stored in chunks inside an object store such as S3, GCS, or Azure Blob Storage.\n\nKey components:\n\n- **Distributor** receives push requests and hashes streams to ingesters.\n- **Ingester** buffers streams in memory and writes chunks to storage.\n- **Query frontend** splits queries and merges results.\n- **Querier** fetches data from ingesters and object storage.\n- **Ruler** evaluates alerting and recording rules.\n\nStorage format:\n\n- **Index** is a table of contents mapping label sets to chunks. TSDB is the recommended index format.\n- **Chunk** contains log entries for one stream over a time range, compressed.\n\nMulti-tenancy:\n\n- Loki partitions data by tenant id.\n- The `X-Scope-OrgID` header carries the tenant.\n- Single-tenant deployments ignore the header and use tenant id `fake`.\n\nBecause only labels are indexed, query performance depends heavily on choosing low-cardinality, high-selectivity labels in the stream selector.\n\n## 9. Labels and streams\n\n[ref: #loki-ref-labels-streams]\n\nA **log stream** is the set of log lines that share exactly the same label set. Loki stores one chunk per stream per time window, so the number of streams directly impacts storage efficiency and query speed.\n\nGood labels are static and bounded:\n\n- `namespace`, `app`, `service_name`, `env`, `cluster`, `host`.\n\nBad labels are dynamic or unbounded:\n\n- `trace_id`, `request_id`, `user_id`, `order_id`.\n\nCardinality guidance:\n\n- Keep a single tenant under roughly 100,000 active streams.\n- Prefer fewer streams with larger chunks over many streams with tiny chunks.\n- Use line filters such as `|= \"trace_id=abc\"` for high-cardinality values only when the raw line literally contains that form (for example logfmt). For JSON logs parse first: `{app=\"api\"} | json | trace_id=\"abc\"`.\n\nStructured metadata allows attaching non-indexed key-value pairs to log lines at ingestion time. These values can be extracted at query time with parsers but do not explode the index.\n\n## 10. LogQL syntax overview\n\n[ref: #loki-ref-logql-syntax]\n\nEvery LogQL query has the form:\n\n```logql\n{ log stream selector } | pipeline stage | pipeline stage | ...\n```\n\nThe log stream selector is mandatory. The pipeline is optional.\n\nPipeline stages are evaluated left to right:\n\n1. **Line filters** narrow by substring or regex in the raw line.\n2. **Parsers** extract structured fields as labels.\n3. **Label filters** filter on extracted or existing labels.\n4. **Format expressions** reshape output for display.\n\nA query can end as a log query, returning lines, or as a metric query, wrapping the pipeline in a range vector and aggregation function.\n\nExamples:\n\n```logql\n{app=\"api\", env=\"prod\"} |= \"error\"\n{app=\"api\"} | json | status >= 500\n{app=\"api\"} | json | line_format \"{{.method}} {{.path}} -> {{.status}}\"\n```\n\n## 11. Stream selectors\n\n[ref: #loki-ref-stream-selectors]\n\nThe stream selector chooses which log streams to process. It uses label matchers inside curly braces.\n\nOperators:\n\n- `=` exact equality.\n- `!=` inequality.\n- `=~` regex match.\n- `!~` negative regex match.\n\nMultiple matchers inside one selector are AND-ed together.\n\nExamples:\n\n```logql\n{app=\"billing\", env=\"prod\"}\n{namespace=~\"team-.*\", service_name=\"wallet\"}\n{app!=\"debug-app\"}\n```\n\nBest practice: use the most specific label first. If `app_name` selects a smaller subset than `namespace`, prefer `{app_name=\"wallet\"}` alone.\n\n## 12. Line filters\n\n[ref: #loki-ref-line-filters]\n\nLine filters operate on the raw log line text. They are the cheapest filter type and should be placed as early as possible.\n\nOperators:\n\n- `|=` line contains string.\n- `!=` line does not contain string.\n- `|~` line matches regex.\n- `!~` line does not match regex.\n\nPerformance order:\n\n1. `|=` and `!=` are fastest.\n2. `|~` and `!~` are slower.\n3. Parsers are slower than line filters.\n\nExamples:\n\n```logql\n{app=\"api\"} |= \"timeout\"\n{app=\"api\"} != \"healthcheck\"\n{app=\"api\"} |~ \"(error|fatal)\"\n```\n\nPattern match filters `|>` and `!>` (Loki 3.x+) match the whole line against a pattern where `\u003C_>` matches arbitrary text. They are faster than regex for lines with a predictable shape. Do not confuse them with the `| pattern` parser.\n\n```logql\n{app=\"api\"} |> \"\u003C_> level=error \u003C_>\"\n```\n\n## 13. Label filters\n\n[ref: #loki-ref-label-filters]\n\nLabel filters operate on labels that already exist or were extracted by a parser. They are written after parsers or directly after the selector.\n\nOperators:\n\n- `==` or `=` equality.\n- `!=` inequality.\n- `>`, `>=`, `\u003C`, `\u003C=` numeric comparison.\n- `and` and `or` boolean chaining.\n\nExamples:\n\n```logql\n{app=\"api\"} | json | status >= 500\n{app=\"api\"} | json | method=\"POST\" and status >= 400\n{app=\"api\"} | logfmt | duration > 1.5\n```\n\nPipeline errors from parsers produce a system label `__error__`. Filter it explicitly:\n\n```logql\n{app=\"api\"} | json | __error__=\"\"\n```\n\n## 14. Parsers\n\n[ref: #loki-ref-parsers]\n\nParsers extract structured data from log lines and create temporary labels for the rest of the pipeline.\n\nAvailable parsers:\n\n- `json` parses JSON log lines.\n- `logfmt` parses `key=value` or `key=\"value\"` pairs.\n- `pattern` extracts fields via a lightweight wildcard pattern.\n- `regexp` extracts fields via a regular expression with named capture groups.\n- `unpack` extracts labels from structured metadata in the log line.\n\nJSON parser examples:\n\n```logql\n{app=\"api\"} | json\n{app=\"api\"} | json | status=\"500\"\n{app=\"api\"} | json method, status\n```\n\nThe last example extracts only `method` and `status` as labels. The parser also supports explicit mappings such as `json method=\"http_method\", status=\"http_status\"`.\n\nLogfmt parser example:\n\n```logql\n{app=\"api\"} | logfmt | level=\"error\"\n```\n\nPattern parser example:\n\n```logql\n{app=\"api\"} | pattern \"\u003C_> caller=\u003Ccaller> level=\u003Clevel> msg=\u003Cmsg>\"\n```\n\nRegexp parser example:\n\n```logql\n{app=\"api\"} | regexp \"(?P\u003Cmethod>\\w+) (?P\u003Cpath>\\S+) HTTP\u002F(?P\u003Cversion>\\d\\.\\d)\"\n```\n\nUse a single backslash inside the regex; LogQL passes it directly to the Go regex engine.\n\nParser errors populate `__error__`. Always handle errors when parser reliability is uncertain.\n\n## 15. Format expressions\n\n[ref: #loki-ref-format-expressions]\n\nFormat expressions change how results are displayed. They do not modify stored data.\n\n`line_format` rewrites the log line using Go templates:\n\n```logql\n{app=\"api\"} | json | line_format \"{{.method}} {{.path}} -> {{.status}}\"\n```\n\n`label_format` creates or rewrites labels:\n\n```logql\n{app=\"api\"} | json | label_format severity=\"{{.level | upper}}\"\n```\n\nCommon template helpers include `upper`, `lower`, and `trim`. Keep formatting late in the pipeline, after all filtering and parsing.\n\n## 16. Metric queries\n\n[ref: #loki-ref-metric-queries]\n\nMetric queries convert log streams into numeric time series. They wrap a log pipeline in a range vector `[duration]` and apply an aggregation function.\n\nCommon functions:\n\n- `rate({selector} |= \"error\" [1m])` entries per second.\n- `count_over_time({selector} [5m])` total entries in range.\n- `sum by (status) (rate({selector} | json | status >= 500 [5m]))` grouped error rate.\n- `quantile_over_time(0.99, {selector} | json | unwrap latency_ms [5m])` percentile.\n\nRange vector durations:\n\n- Use `[1m]`, `[5m]`, `[1h]` depending on granularity.\n- The range must be longer than or equal to the query step.\n- Always pass `--step` explicitly when running metric queries with `logcli` to control sampling density and server load.\n\nAggregation modifiers:\n\n- `by (label1, label2)` keeps listed labels in result.\n- `without (label1)` drops listed labels from result.\n\nBinary operators (`+`, `-`, `*`, `\u002F`, `%`, `^`) and comparison operators (`==`, `!=`, `>`, `\u003C`) work on vectors and scalars. Use `bool` after comparison to keep zeros and ones instead of filtering.\n\n## 17. logcli commands reference\n\n[ref: #loki-ref-logcli-commands]\n\n`logcli` is the command-line client for Loki. Read-only commands useful for agent work:\n\n- `logcli query \u003Clogql>` returns log lines or metric results.\n- `logcli instant-query \u003Clogql>` evaluates a metric query at a single point in time.\n- `logcli labels [label]` lists label names or values.\n- `logcli series \u003Cmatcher>` lists streams matching a selector.\n- `logcli stats \u003Cquery>` returns index statistics for a selector.\n- `logcli volume \u003Cquery>` returns aggregate log volume for a selector.\n- `logcli detected-fields \u003Cquery>` reports fields detected by parsers.\n\nForbidden commands for read-only agent use:\n\n- `delete` and all its subcommands.\n- Any write, push, or ingest operation.\n\n## 18. Connection reference\n\n[ref: #loki-ref-logcli-connection]\n\nConnection settings can come from environment variables or command-line flags. Command-line flags override environment variables; environment variables provide defaults.\n\nImportant environment variables and flags:\n\n- `LOKI_ADDR` \u002F `--addr` — Loki server URL.\n- `LOKI_ORG_ID` \u002F `--org-id` — tenant id added as `X-Scope-OrgID`.\n- `LOKI_USERNAME` \u002F `--username` and `LOKI_PASSWORD` \u002F `--password` — HTTP basic auth.\n- `LOKI_BEARER_TOKEN` \u002F `--bearer-token` — bearer token for Authorization header.\n- `LOKI_TLS_SKIP_VERIFY` \u002F `--tls-skip-verify` — skip TLS verification.\n- `LOKI_CA_CERT_PATH` \u002F `--ca-cert` — custom CA certificate.\n- `LOKI_CLIENT_CERT_PATH` \u002F `--cert` and `LOKI_CLIENT_KEY_PATH` \u002F `--key` — mutual TLS.\n\nOne endpoint frequently serves several business scopes as different tenants (for example `loki-gw.prod.internal.example-corp.net` with org id `10` for custodial wallet logs and `30` for payment-core logs, the dev gateway carrying its own id). The prescriptive rules live in §2.\n\n## 19. Output and time flags reference\n\n[ref: #loki-ref-logcli-output-time]\n\nOutput flags:\n\n- `-o default` \u002F `--output=default` — timestamp, labels, and line.\n- `-o raw` \u002F `--output=raw` — line content only (forbidden for agent use, see §3).\n- `-o jsonl` \u002F `--output=jsonl` — one JSON object per returned log line with `labels`, `line`, and `timestamp` fields.\n- `--output-timestamp-format=rfc3339nano` — nanosecond RFC3339 timestamps.\n- `--timezone=UTC` — format timestamps in UTC.\n- `-q` \u002F `--quiet` — suppress query metadata.\n\nTime flags:\n\n- `--since=15m` — relative lookback window.\n- `--from=2026-08-18T10:00:00Z` — absolute start, RFC3339Nano with explicit `Z` UTC suffix.\n- `--to=2026-08-18T11:00:00Z` — absolute end, RFC3339Nano with explicit `Z` UTC suffix.\n- `--timezone=UTC` — format output timestamps in UTC and interpret bounds without an offset as UTC.\n\nLimit flag:\n\n- `--limit=30` — cap returned entries. Never set to `0`.\n\nAgent defaults:\n\n- `--output=jsonl --output-timestamp-format=rfc3339nano --quiet --timezone=UTC --limit=30 --since=15m`, piped through `unjsonl` before the output reaches a file or the context.\n\n## 20. Query patterns\n\n[ref: #loki-ref-query-patterns]\n\nAll bash examples below are simplified. In agent use, wrap every `logcli query` in `timeout 30s`, pipe output through `unjsonl` into `.tmp\u002Floki-skill\u002F\u003Ctimestamp>_\u003Cid>.log`, and apply the 50 KiB size gate.\n\nInternal label convention: selectors are typically `{namespace=..., service_name=...}` where `service_name` is the workload name.\n\n### Recent errors for a service\n\n```bash\nmkdir -p .tmp\u002Floki-skill\ntimeout 30s logcli query \\\n  --tls-skip-verify --timezone=UTC --since=15m --output=jsonl \\\n  --output-timestamp-format=rfc3339nano --quiet --limit=30 \\\n  '{app=\"billing\", env=\"prod\"} |= \"error\"' \\\n  | unjsonl > \".tmp\u002Floki-skill\u002F$(date -u +%Y%m%dT%H%M%S)_$$.log\"\n```\n\n### Trace id search\n\nFor logfmt or literal `trace_id=...` lines:\n\n```bash\ntimeout 30s logcli query --tls-skip-verify --timezone=UTC --since=1h --output=jsonl \\\n  --output-timestamp-format=rfc3339nano --quiet --limit=100 \\\n  '{app=~\"api|worker\"} |= \"trace_id=7a98025445386b1feef1402bb13362e5\"' | unjsonl\n```\n\nFor JSON logs use a parser:\n\n```logql\n{app=~\"api|worker\"} | json | trace_id=\"7a98025445386b1feef1402bb13362e5\"\n```\n\n### HTTP 5xx with JSON logs\n\n```logql\n{app=\"api\"} | json | status >= 500\n```\n\n### Error rate per service\n\n```logql\nsum by (app) (rate({app=~\"api|worker\"} |= \"error\" [5m]))\n```\n\nFor `logcli`, add `--step=1m` (or smaller than the range) when running metric queries.\n\n### Sentry event correlation\n\nGiven a Sentry event id, search raw logs for that literal token only if the line contains it literally:\n\n```logql\n{app=~\"api|worker\"} |= \"event_id=6d9ab75c00ac479f98a629f07ad253e7\"\n```\n\nFor JSON logs parse the field first:\n\n```logql\n{app=~\"api|worker\"} | json | event_id=\"6d9ab75c00ac479f98a629f07ad253e7\"\n```\n\n### Business identifier search\n\nThree canonical anchors inside one service: URL path, workflow\u002Ftrace UUID, business `key=value` parameter (identifiers below are fictional):\n\n```logql\n{namespace=\"payments\", service_name=\"gateway-api\"} |= \"\u002Fapi\u002Fv2\u002Fwebhook\"\n{namespace=\"payments\", service_name=\"gateway-api\"} |= \"a1b2c3d4-1111-4222-8333-abcdef012345\"\n{namespace=\"payments\", service_name=\"gateway-api\"} |= \"client_id=42\"\n```\n\n## 21. Troubleshooting\n\n[ref: #loki-ref-troubleshooting]\n\n| Symptom | Likely cause | Remedy |\n|---|---|---|\n| Empty results | Time range too narrow or selector too strict | Widen `--since`, relax selectors, check labels with `logcli labels`. |\n| Timeout or slow query | Time range too wide, low-selectivity selector, or heavy parser | Narrow time, add \u003Ccode>&#124;=\u003C\u002Fcode> filter first, avoid regex. |\n| 401\u002F403 | `LOKI_ORG_ID` unset or wrong tenant for the scope | Set `LOKI_ORG_ID` to the project default org; pass explicit `--org-id` for non-default scopes. |\n| macOS cannot resolve an internal host (`*.internal.*`) | `mise`-installed `logcli` is built with the `netgo` tag and bypasses the macOS system resolver | Use the Homebrew build (`\u002Fopt\u002Fhomebrew\u002Fbin\u002Flogcli --version`) or another cgo-resolver build. |\n| `tls: failed to verify certificate: ... not standards compliant` | Go rejects some internal CA certificates as non-compliant | Keep `--tls-skip-verify`; it is the intended default, not a misconfiguration. |\n| `unjsonl: command not found` | `unjsonl` missing from PATH | Hard stop; ask the user to install `unjsonl`. |\n| `__error__` labels | Parser failed on some lines | Add \u003Ccode>&#124; __error__=\"\"\u003C\u002Fcode> or \u003Ccode>&#124; __error__!=\"\"\u003C\u002Fcode> depending on intent. |\n| Huge output file | No filter or `--limit=0` | Refetch with filter and limit; slice with `rg`, `jq`, or `rtk` equivalents. |\n| Out-of-order lines with parallel flags | `--parallel-*` splits range into parts | Avoid parallel flags; use narrow sequential queries. |\n",{"name":5,"runtime":24,"version":25,"requires":26,"triggers":28,"description":7},true,"0.3.0",[27],"mandatory-tools",{"reason":29,"request":30},"User needs to query logs or service state through Grafana Loki.","logcli, loki, grafana logs, посмотри логи, найди в логах, посмотри логи на стейдже, посмотри логи на проде, проверь в локи, посмотри в графане, расследование инцидентов, разбор инцидента, разбор проблем, sentry, логи sentry, incident","main","7da8c355f6e99dcb88d8bb9f6ff27b3cb35bae47","_on_demand\u002Floki-skill",0,"https:\u002F\u002Fgithub.com\u002Fkalaverin\u002Fskills","https:\u002F\u002Fgithub.com\u002Fkalaverin","2026-11-15 01:37:57",{"source":8,"skillName":5,"commands":39,"prompt":49,"zipAvailable":24},[40,43,46],{"tool":41,"command":42},"npx","npx skills add kalaverin\u002Fskills --skill loki-skill",{"tool":44,"command":45},"bunx","bunx skills add kalaverin\u002Fskills --skill loki-skill",{"tool":47,"command":48},"pnpm","pnpm dlx skills add kalaverin\u002Fskills --skill loki-skill","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fkalaverin\u002Fskills\u002Fblob\u002Fmain\u002F_on_demand\u002Floki-skill\u002FSKILL.md",[51],{"path":52,"resourceType":53,"resourceTypeDesc":54,"sizeBytes":55,"sourceHtmlUrl":56},"README.md","REFERENCE","参考文档",4404,"https:\u002F\u002Fgithub.com\u002Fkalaverin\u002Fskills\u002Fblob\u002Fmain\u002F_on_demand\u002Floki-skill\u002FREADME.md",[58,65,72,79,86,93],{"skillNo":59,"name":60,"nameZh":6,"description":61,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":62,"categories":63,"keywords":64,"publishedAt":19},"01M786R9KK7RP1J8EB7X9TC9KR","todo-protocol","Mandatory protocol for using the SetTodoList tool. Governs when to create a todo list, how to update it without violating immutability rules, how to synchronize item status after every tool call, and the forbidden mutations. Always active.","https:\u002F\u002Fgithub.com\u002Fkalaverin\u002Fskills\u002Fblob\u002Fmain\u002Ftodo-protocol\u002FSKILL.md",[],[],{"skillNo":66,"name":67,"nameZh":6,"description":68,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":69,"categories":70,"keywords":71,"publishedAt":19},"01M786QZTC43FBK3A251NTY62Y","subagents-protocol","Mandatory protocol for delegating work to built-in subagents (coder, explore, plan), also called our bots by users. Always active. Owns the context-hygiene delegation test (material vs answer, cost balance, warm-instance reuse), launch parameters (mandatory explicit `model`), the single-source model-selection table, answer budgets, degradation handling, the launch checklist, and the role library for delegated review\u002Faudit with intent routing (`references\u002Froles.md`).","https:\u002F\u002Fgithub.com\u002Fkalaverin\u002Fskills\u002Fblob\u002Fmain\u002Fsubagents-protocol\u002FSKILL.md",[],[],{"skillNo":73,"name":74,"nameZh":6,"description":75,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":76,"categories":77,"keywords":78,"publishedAt":19},"01M786QSXFDM550NCC4PQXN26X","serena-protocol","MANDATORY skill for Serena (MCP) memory, knowledge base, and symbolic code exploration operations. Use whenever the agent needs to: read or write Serena memories, mutate or persist memory, explore code via symbolic MCP tools, resolve contradictions in stored knowledge, or follow memory metadata, naming, and routing conventions.","https:\u002F\u002Fgithub.com\u002Fkalaverin\u002Fskills\u002Fblob\u002Fmain\u002Fserena-protocol\u002FSKILL.md",[],[],{"skillNo":80,"name":81,"nameZh":6,"description":82,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":19},"01M786QMXA58XD5DJZ2AZER5ZP","read-for-comments","Mandatory local standards library and archive. Before fetching any standard from the internet, the agent MUST check the Serena `standard\u002F` archive and this skill's `references\u002F` seed; after ANY standard-body download, the agent MUST archive it per the event-bound Archival Rule. Governs standard lookup, fetch, archival, and the family and full-body rules.","https:\u002F\u002Fgithub.com\u002Fkalaverin\u002Fskills\u002Fblob\u002Fmain\u002Fread-for-comments\u002FSKILL.md",[],[],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":19},"01M786QD59A2R2JK1NJTK7MADP","python-lang","MANDATORY skill for Python code. Use when writing, editing, refactoring, or reviewing Python files, modules, packages, classes, functions, type annotations, imports, exceptions, comprehensions, decorators, or docstrings. Enforces Google Python Style Guide and a mandatory, unconditional Ruff format+check pipeline scoped strictly to the agent's own changes.","https:\u002F\u002Fgithub.com\u002Fkalaverin\u002Fskills\u002Fblob\u002Fmain\u002Fpython-lang\u002FSKILL.md",[],[],{"skillNo":94,"name":95,"nameZh":6,"description":96,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":97,"categories":98,"keywords":99,"publishedAt":19},"01M786Q683RHDC93NWABY7XKQN","preflight-checklist","Mandatory session-initialization closer and Startup Gate audit. Always loaded. Runs EXACTLY ONCE at session start, after all skill loading completes: verifies Startup Gate completion with evidence (mirror sync, forced import, discovery, activation, memory priming, MCP declaration, think block), runs dynamic session checks derived from the active skill set (mode states, delegation readiness), and reports a compact evidence-backed verdict table to the user before the first task output. Any FAIL is fixed before proceeding.","https:\u002F\u002Fgithub.com\u002Fkalaverin\u002Fskills\u002Fblob\u002Fmain\u002Fpreflight-checklist\u002FSKILL.md",[],[]]