loki-skill
Query logs and service state in Grafana Loki via the logcli terminal client. Use for investigating service problems, exploring labels and series, and incident response. The agent is read-only: no mutations, no writes, no deletes.
- License 许可
- BSD-3-Clause
- 所属仓库
- kalaverin/skills
- 最近更新
- 2026-11-15
- Stars
- ★ 1
- Forks
- 0
01 前言 Front Matter
- name
- loki-skill
- runtime
- true
- version
- 0.3.0
- requires
- [ "mandatory-tools" ]
- triggers
- { "reason": "User needs to query logs or service state through Grafana Loki.", "request": "logcli, loki, grafana logs, посмотри логи, найди в логах, посмотри логи на стейдже, посмотри логи на проде, проверь в локи, посмотри в графане, расследование инцидентов, разбор инцидента, разбор проблем, sentry, логи sentry, incident" }
- description
- Query logs and service state in Grafana Loki via the logcli terminal client. Use for investigating service problems, exploring labels and series, and incident response. The agent is read-only: no mutations, no writes, no deletes.
02 资源文件 1
- 参考文档README.md4.3 KB
03 SKILL.md 正文
SKILL: Grafana Loki via logcli
[ref: #loki-skill-intro]
This skill governs how the agent uses the logcli command-line client to query Grafana Loki. The agent acts as a read-only investigator: it may query logs, labels, series, and read-only metadata, but it must never mutate Loki state, write data, or run destructive commands. The skill is atomic: everything — rules, workflows, and the LogQL/logcli reference — lives in this file.
1. Allowed commands
[ref: #loki-allowed-commands]
The agent may use only these read-only logcli commands:
logcli query— main command for log lines.logcli instant-query— evaluate a metric query at a single point in time.logcli labels— list available label names or values.logcli series— list series matching a selector.logcli stats— index statistics for a selector.logcli volume— aggregate log volume for a selector.logcli detected-fields— fields detected by parsers.
Any other subcommand is forbidden. The agent must never use delete, push, or any write-oriented command.
2. Connection and auth
[ref: #loki-connection-auth]
LOKI_ADDRis read from the environment. If it is not set, the agent asks the user.- The agent does not pass
--addron the command line. --tls-skip-verifyis used by default (§16 documents the internal-CA symptom it solves).- Project default org (HARD):
LOKI_ORG_IDMUST hold the project's default tenant. IfLOKI_ORG_IDis empty, the agent STOPS before the first query and asks the user to set the project's default org. One Loki endpoint commonly serves several tenants per business scope, so an unset org silently queries the wrong tenant or fails with 401/403. - Non-default scope (HARD): when the investigation targets a scope outside the project's default org, the agent passes
--org-idexplicitly on those queries (flags override the environment) and states which org was used in the summary. - The agent does not handle basic auth, bearer tokens, or any other credentials. If Loki rejects the connection for authentication reasons, escalate to the user.
- No credentials, hostnames, org ids, or tenant names are hardcoded in examples or transcripts.
3. Hard parameter rules
[ref: #loki-hard-parameter-rules]
| Parameter | Rule |
|---|---|
--limit |
Always set. Never 0. Default is 30. Exploration: 10–50. Deep investigation: 100–200. Need more — ask the user. |
--since |
Prefer narrow windows: 5m, 15m, 1h. |
--from / --to |
RFC3339Nano with explicit timezone suffix, always Z for UTC, e.g. 2026-08-18T10:00:00Z. Combine with --timezone=UTC. |
--output |
Always jsonl, always paired with --output-timestamp-format=rfc3339nano, and always piped through unjsonl before the output reaches a file or the context. raw is forbidden. |
--parallel-* |
Forbidden. Removed from the skill. |
--interval |
Forbidden. Removed from the skill. |
--tail |
Forbidden. Removed from the skill. |
4. Workflows
[ref: #loki-workflows]
4.1 Query recent logs
[ref: #loki-workflow-query-recent]
- Create the output directory:
mkdir -p .tmp/loki-skill. - Start narrow (the default org comes from
LOKI_ORG_ID; pass--org-idonly for a non-default scope):timeout 30s logcli query \ --tls-skip-verify --timezone=UTC --since=15m --output=jsonl \ --output-timestamp-format=rfc3339nano --quiet --limit=30 \ '{app="example", env="dev"}' \ | unjsonl > ".tmp/loki-skill/$(date -u +%Y%m%dT%H%M%S)_$$.log" - Adjust
--sinceand--limitbased on results. - Add a LogQL filter:
... '{app="example", env="dev"} |= "error"'. - Summarize findings; do not paste raw output into the response.
4.2 Explore labels and series
[ref: #loki-workflow-explore]
- List labels:
logcli labels. - List values for a label:
logcli labels <label_name>. - List series for a selector:
logcli series '{app="example"}'. - Use tight selectors and narrow time windows; ask before broad selectors.
4.3 Deep investigation
[ref: #loki-workflow-deep]
- Start from a known anchor: trace id, request id, error message, or Sentry event id.
- Query with a tight filter on that anchor.
- Widen time window only if the anchor is not found.
- Cross-reference with related services only after the first hit.
5. Context protection
[ref: #loki-context-protection]
- Create the output directory. Run
mkdir -p .tmp/loki-skillbefore the first query. - Every query goes to a file. The agent always pipes
logcli queryoutput throughunjsonlinto a temporary file under.tmp/loki-skill/<utc-timestamp>_<short-query-id>.log. Stdout is never read into context directly. - Size gate: 50 KiB. Check the file size with
rtk wc -c <path>. If the file is 50 KiB or less, read and summarize it. If it is larger, either re-fetch with a narrower query, or slice the file withrg,jq, orrtkequivalents. - Start narrow. First query uses
--since=15m --limit=30, or an exact event-based query such as a trace id or Sentry event id. - Filter first. Always apply a LogQL filter (
|=,|~,!=,!~) before broadening the query. - Output format. Always
--output=jsonlwith--output-timestamp-format=rfc3339nano, piped throughunjsonl. Raw jsonl never enters the context unprocessed. - Abort on hang. Run
timeout 30s logcli .... On timeout, suggest narrowing the query. - All bulk-producing commands go to a file. Apply the temporary-file, timeout, and size-gate rules to
logcli query,instant-query,stats,volume, anddetected-fields. - No credentials in examples. Mask hosts, org ids, and tenant names in transcripts and summaries.
6. Pipeline
[ref: #loki-pipeline]
Forward flow:
- User mentions
logcli, Loki, Grafana logs, looking at logs on stage/prod, checking Loki, investigating an incident, or Sentry problem analysis. - Runtime trigger activates
loki-skill; the agent readsSKILL.md. - Agent detects intent: query recent logs, explore labels/series, deep investigation, or event-based query.
- Agent checks environment:
LOKI_ADDR,LOKI_ORG_ID(project default org). - Agent builds command from allowed commands, connection flags, hard parameter rules, and context protection rules.
- Agent executes
timeout 30s logcli ... --output=jsonl --output-timestamp-format=rfc3339nano ... | unjsonl > .tmp/loki-skill/.... - Agent applies the 50 KiB size gate.
- Agent summarizes and responds.
Error routes:
- Missing
LOKI_ADDR→ ask the user. - Missing
LOKI_ORG_ID→ ask the user to set the project's default org before any query. - Investigation targets a non-default scope → pass
--org-idexplicitly and name the org in the summary. - Loki requires authentication (basic auth, bearer token, etc.) → escalate to the user.
logcli: command not found→ hard stop; ask the user to installlogcli(prefer the Homebrew build on macOS).unjsonl: command not found→ hard stop; ask the user to installunjsonl.- Timeout → suggest narrowing.
- Output > 50 KiB → re-fetch or slice with
rg,jq, orrtkequivalents. - No results → suggest widening the window or changing the filter.
7. Violation protocol
[ref: #loki-violation-protocol]
If the agent is about to run a forbidden command, omit --limit, use --limit=0, pull raw multi-line logs into the response, or bypass the temporary-file rule: stop, discard the command, and rebuild it according to this skill.
8. Loki architecture
[ref: #loki-ref-architecture]
Loki is a horizontally scalable log aggregation system inspired by Prometheus. It groups incoming log lines into streams identified by a unique set of labels. Loki indexes only the label set and the timestamp; the full log line is compressed and stored in chunks inside an object store such as S3, GCS, or Azure Blob Storage.
Key components:
- Distributor receives push requests and hashes streams to ingesters.
- Ingester buffers streams in memory and writes chunks to storage.
- Query frontend splits queries and merges results.
- Querier fetches data from ingesters and object storage.
- Ruler evaluates alerting and recording rules.
Storage format:
- Index is a table of contents mapping label sets to chunks. TSDB is the recommended index format.
- Chunk contains log entries for one stream over a time range, compressed.
Multi-tenancy:
- Loki partitions data by tenant id.
- The
X-Scope-OrgIDheader carries the tenant. - Single-tenant deployments ignore the header and use tenant id
fake.
Because only labels are indexed, query performance depends heavily on choosing low-cardinality, high-selectivity labels in the stream selector.
9. Labels and streams
[ref: #loki-ref-labels-streams]
A log stream is the set of log lines that share exactly the same label set. Loki stores one chunk per stream per time window, so the number of streams directly impacts storage efficiency and query speed.
Good labels are static and bounded:
namespace,app,service_name,env,cluster,host.
Bad labels are dynamic or unbounded:
trace_id,request_id,user_id,order_id.
Cardinality guidance:
- Keep a single tenant under roughly 100,000 active streams.
- Prefer fewer streams with larger chunks over many streams with tiny chunks.
- Use line filters such as
|= "trace_id=abc"for high-cardinality values only when the raw line literally contains that form (for example logfmt). For JSON logs parse first:{app="api"} | json | trace_id="abc".
Structured metadata allows attaching non-indexed key-value pairs to log lines at ingestion time. These values can be extracted at query time with parsers but do not explode the index.
10. LogQL syntax overview
[ref: #loki-ref-logql-syntax]
Every LogQL query has the form:
{ log stream selector } | pipeline stage | pipeline stage | ...
The log stream selector is mandatory. The pipeline is optional.
Pipeline stages are evaluated left to right:
- Line filters narrow by substring or regex in the raw line.
- Parsers extract structured fields as labels.
- Label filters filter on extracted or existing labels.
- Format expressions reshape output for display.
A query can end as a log query, returning lines, or as a metric query, wrapping the pipeline in a range vector and aggregation function.
Examples:
{app="api", env="prod"} |= "error"
{app="api"} | json | status >= 500
{app="api"} | json | line_format "{{.method}} {{.path}} -> {{.status}}"
11. Stream selectors
[ref: #loki-ref-stream-selectors]
The stream selector chooses which log streams to process. It uses label matchers inside curly braces.
Operators:
=exact equality.!=inequality.=~regex match.!~negative regex match.
Multiple matchers inside one selector are AND-ed together.
Examples:
{app="billing", env="prod"}
{namespace=~"team-.*", service_name="wallet"}
{app!="debug-app"}
Best practice: use the most specific label first. If app_name selects a smaller subset than namespace, prefer {app_name="wallet"} alone.
12. Line filters
[ref: #loki-ref-line-filters]
Line filters operate on the raw log line text. They are the cheapest filter type and should be placed as early as possible.
Operators:
|=line contains string.!=line does not contain string.|~line matches regex.!~line does not match regex.
Performance order:
|=and!=are fastest.|~and!~are slower.- Parsers are slower than line filters.
Examples:
{app="api"} |= "timeout"
{app="api"} != "healthcheck"
{app="api"} |~ "(error|fatal)"
Pattern match filters |> and !> (Loki 3.x+) match the whole line against a pattern where <_> matches arbitrary text. They are faster than regex for lines with a predictable shape. Do not confuse them with the | pattern parser.
{app="api"} |> "<_> level=error <_>"
13. Label filters
[ref: #loki-ref-label-filters]
Label filters operate on labels that already exist or were extracted by a parser. They are written after parsers or directly after the selector.
Operators:
==or=equality.!=inequality.>,>=,<,<=numeric comparison.andandorboolean chaining.
Examples:
{app="api"} | json | status >= 500
{app="api"} | json | method="POST" and status >= 400
{app="api"} | logfmt | duration > 1.5
Pipeline errors from parsers produce a system label __error__. Filter it explicitly:
{app="api"} | json | __error__=""
14. Parsers
[ref: #loki-ref-parsers]
Parsers extract structured data from log lines and create temporary labels for the rest of the pipeline.
Available parsers:
jsonparses JSON log lines.logfmtparseskey=valueorkey="value"pairs.patternextracts fields via a lightweight wildcard pattern.regexpextracts fields via a regular expression with named capture groups.unpackextracts labels from structured metadata in the log line.
JSON parser examples:
{app="api"} | json
{app="api"} | json | status="500"
{app="api"} | json method, status
The last example extracts only method and status as labels. The parser also supports explicit mappings such as json method="http_method", status="http_status".
Logfmt parser example:
{app="api"} | logfmt | level="error"
Pattern parser example:
{app="api"} | pattern "<_> caller=<caller> level=<level> msg=<msg>"
Regexp parser example:
{app="api"} | regexp "(?P<method>\w+) (?P<path>\S+) HTTP/(?P<version>\d\.\d)"
Use a single backslash inside the regex; LogQL passes it directly to the Go regex engine.
Parser errors populate __error__. Always handle errors when parser reliability is uncertain.
15. Format expressions
[ref: #loki-ref-format-expressions]
Format expressions change how results are displayed. They do not modify stored data.
line_format rewrites the log line using Go templates:
{app="api"} | json | line_format "{{.method}} {{.path}} -> {{.status}}"
label_format creates or rewrites labels:
{app="api"} | json | label_format severity="{{.level | upper}}"
Common template helpers include upper, lower, and trim. Keep formatting late in the pipeline, after all filtering and parsing.
16. Metric queries
[ref: #loki-ref-metric-queries]
Metric queries convert log streams into numeric time series. They wrap a log pipeline in a range vector [duration] and apply an aggregation function.
Common functions:
rate({selector} |= "error" [1m])entries per second.count_over_time({selector} [5m])total entries in range.sum by (status) (rate({selector} | json | status >= 500 [5m]))grouped error rate.quantile_over_time(0.99, {selector} | json | unwrap latency_ms [5m])percentile.
Range vector durations:
- Use
[1m],[5m],[1h]depending on granularity. - The range must be longer than or equal to the query step.
- Always pass
--stepexplicitly when running metric queries withlogclito control sampling density and server load.
Aggregation modifiers:
by (label1, label2)keeps listed labels in result.without (label1)drops listed labels from result.
Binary operators (+, -, *, /, %, ^) and comparison operators (==, !=, >, <) work on vectors and scalars. Use bool after comparison to keep zeros and ones instead of filtering.
17. logcli commands reference
[ref: #loki-ref-logcli-commands]
logcli is the command-line client for Loki. Read-only commands useful for agent work:
logcli query <logql>returns log lines or metric results.logcli instant-query <logql>evaluates a metric query at a single point in time.logcli labels [label]lists label names or values.logcli series <matcher>lists streams matching a selector.logcli stats <query>returns index statistics for a selector.logcli volume <query>returns aggregate log volume for a selector.logcli detected-fields <query>reports fields detected by parsers.
Forbidden commands for read-only agent use:
deleteand all its subcommands.- Any write, push, or ingest operation.
18. Connection reference
[ref: #loki-ref-logcli-connection]
Connection settings can come from environment variables or command-line flags. Command-line flags override environment variables; environment variables provide defaults.
Important environment variables and flags:
LOKI_ADDR/--addr— Loki server URL.LOKI_ORG_ID/--org-id— tenant id added asX-Scope-OrgID.LOKI_USERNAME/--usernameandLOKI_PASSWORD/--password— HTTP basic auth.LOKI_BEARER_TOKEN/--bearer-token— bearer token for Authorization header.LOKI_TLS_SKIP_VERIFY/--tls-skip-verify— skip TLS verification.LOKI_CA_CERT_PATH/--ca-cert— custom CA certificate.LOKI_CLIENT_CERT_PATH/--certandLOKI_CLIENT_KEY_PATH/--key— mutual TLS.
One endpoint frequently serves several business scopes as different tenants (for example loki-gw.prod.internal.example-corp.net with org id 10 for custodial wallet logs and 30 for payment-core logs, the dev gateway carrying its own id). The prescriptive rules live in §2.
19. Output and time flags reference
[ref: #loki-ref-logcli-output-time]
Output flags:
-o default/--output=default— timestamp, labels, and line.-o raw/--output=raw— line content only (forbidden for agent use, see §3).-o jsonl/--output=jsonl— one JSON object per returned log line withlabels,line, andtimestampfields.--output-timestamp-format=rfc3339nano— nanosecond RFC3339 timestamps.--timezone=UTC— format timestamps in UTC.-q/--quiet— suppress query metadata.
Time flags:
--since=15m— relative lookback window.--from=2026-08-18T10:00:00Z— absolute start, RFC3339Nano with explicitZUTC suffix.--to=2026-08-18T11:00:00Z— absolute end, RFC3339Nano with explicitZUTC suffix.--timezone=UTC— format output timestamps in UTC and interpret bounds without an offset as UTC.
Limit flag:
--limit=30— cap returned entries. Never set to0.
Agent defaults:
--output=jsonl --output-timestamp-format=rfc3339nano --quiet --timezone=UTC --limit=30 --since=15m, piped throughunjsonlbefore the output reaches a file or the context.
20. Query patterns
[ref: #loki-ref-query-patterns]
All bash examples below are simplified. In agent use, wrap every logcli query in timeout 30s, pipe output through unjsonl into .tmp/loki-skill/<timestamp>_<id>.log, and apply the 50 KiB size gate.
Internal label convention: selectors are typically {namespace=..., service_name=...} where service_name is the workload name.
Recent errors for a service
mkdir -p .tmp/loki-skill
timeout 30s logcli query \
--tls-skip-verify --timezone=UTC --since=15m --output=jsonl \
--output-timestamp-format=rfc3339nano --quiet --limit=30 \
'{app="billing", env="prod"} |= "error"' \
| unjsonl > ".tmp/loki-skill/$(date -u +%Y%m%dT%H%M%S)_$$.log"
Trace id search
For logfmt or literal trace_id=... lines:
timeout 30s logcli query --tls-skip-verify --timezone=UTC --since=1h --output=jsonl \
--output-timestamp-format=rfc3339nano --quiet --limit=100 \
'{app=~"api|worker"} |= "trace_id=7a98025445386b1feef1402bb13362e5"' | unjsonl
For JSON logs use a parser:
{app=~"api|worker"} | json | trace_id="7a98025445386b1feef1402bb13362e5"
HTTP 5xx with JSON logs
{app="api"} | json | status >= 500
Error rate per service
sum by (app) (rate({app=~"api|worker"} |= "error" [5m]))
For logcli, add --step=1m (or smaller than the range) when running metric queries.
Sentry event correlation
Given a Sentry event id, search raw logs for that literal token only if the line contains it literally:
{app=~"api|worker"} |= "event_id=6d9ab75c00ac479f98a629f07ad253e7"
For JSON logs parse the field first:
{app=~"api|worker"} | json | event_id="6d9ab75c00ac479f98a629f07ad253e7"
Business identifier search
Three canonical anchors inside one service: URL path, workflow/trace UUID, business key=value parameter (identifiers below are fictional):
{namespace="payments", service_name="gateway-api"} |= "/api/v2/webhook"
{namespace="payments", service_name="gateway-api"} |= "a1b2c3d4-1111-4222-8333-abcdef012345"
{namespace="payments", service_name="gateway-api"} |= "client_id=42"
21. Troubleshooting
[ref: #loki-ref-troubleshooting]
| Symptom | Likely cause | Remedy |
|---|---|---|
| Empty results | Time range too narrow or selector too strict | Widen --since, relax selectors, check labels with logcli labels. |
| Timeout or slow query | Time range too wide, low-selectivity selector, or heavy parser | Narrow time, add <code>|=</code> filter first, avoid regex. |
| 401/403 | LOKI_ORG_ID unset or wrong tenant for the scope |
Set LOKI_ORG_ID to the project default org; pass explicit --org-id for non-default scopes. |
macOS cannot resolve an internal host (*.internal.*) |
mise-installed logcli is built with the netgo tag and bypasses the macOS system resolver |
Use the Homebrew build (/opt/homebrew/bin/logcli --version) or another cgo-resolver build. |
tls: failed to verify certificate: ... not standards compliant |
Go rejects some internal CA certificates as non-compliant | Keep --tls-skip-verify; it is the intended default, not a misconfiguration. |
unjsonl: command not found |
unjsonl missing from PATH |
Hard stop; ask the user to install unjsonl. |
__error__ labels |
Parser failed on some lines | Add <code>| error=""</code> or <code>| error!=""</code> depending on intent. |
| Huge output file | No filter or --limit=0 |
Refetch with filter and limit; slice with rg, jq, or rtk equivalents. |
| Out-of-order lines with parallel flags | --parallel-* splits range into parts |
Avoid parallel flags; use narrow sequential queries. |
04 同仓库更多 Skills
todo-protocol
Mandatory protocol for using the SetTodoList tool. Governs when to create a todo list, how to update it without violating immutability rules, how to synchronize item status after every tool call, and the forbidden mutations. Always active.
subagents-protocol
Mandatory protocol for delegating work to built-in subagents (coder, explore, plan), also called our bots by users. Always active. Owns the context-hygiene delegation test (material vs answer, cost balance, warm-instance reuse), launch parameters (mandatory explicit `model`), the single-source model-selection table, answer budgets, degradation handling, the launch checklist, and the role library for delegated review/audit with intent routing (`references/roles.md`).
serena-protocol
MANDATORY skill for Serena (MCP) memory, knowledge base, and symbolic code exploration operations. Use whenever the agent needs to: read or write Serena memories, mutate or persist memory, explore code via symbolic MCP tools, resolve contradictions in stored knowledge, or follow memory metadata, naming, and routing conventions.
read-for-comments
Mandatory local standards library and archive. Before fetching any standard from the internet, the agent MUST check the Serena `standard/` archive and this skill's `references/` seed; after ANY standard-body download, the agent MUST archive it per the event-bound Archival Rule. Governs standard lookup, fetch, archival, and the family and full-body rules.
python-lang
MANDATORY skill for Python code. Use when writing, editing, refactoring, or reviewing Python files, modules, packages, classes, functions, type annotations, imports, exceptions, comprehensions, decorators, or docstrings. Enforces Google Python Style Guide and a mandatory, unconditional Ruff format+check pipeline scoped strictly to the agent's own changes.
preflight-checklist
Mandatory session-initialization closer and Startup Gate audit. Always loaded. Runs EXACTLY ONCE at session start, after all skill loading completes: verifies Startup Gate completion with evidence (mirror sync, forced import, discovery, activation, memory priming, MCP declaration, think block), runs dynamic session checks derived from the active skill set (mode states, delegation readiness), and reports a compact evidence-backed verdict table to the user before the first task output. Any FAIL is fixed before proceeding.