AGENT SKILL

sysadmin-observability-troubleshooting-and-operations

Use when a server is slow, broken, or needs operating at scale: performance methodology rather than tool trivia, reading memory correctly, CPU and profiling, eBPF as the modern observability layer, metrics and logs, a troubleshooting sequence and the four-places-state problem, containers and the namespaces and cgroups underneath them, configuration management and automation, and backup and recovery that actually restores.

License 许可
MIT
最近更新
2026-11-15
Stars
★ 5
Forks
3

01 前言 Front Matter

name
sysadmin-observability-troubleshooting-and-operations
description
Use when a server is slow, broken, or needs operating at scale: performance methodology rather than tool trivia, reading memory correctly, CPU and profiling, eBPF as the modern observability layer, metrics and logs, a troubleshooting sequence and the four-places-state problem, containers and the namespaces and cgroups underneath them, configuration management and automation, and backup and recovery that actually restores.

03 SKILL.md 正文

Linux Server Administration: Observability, Troubleshooting, Containers, Automation, and Backup

Part 4 of 5 of the Linux Server Administration reference (plugin linux-server-admin), covering §10–§14. Sibling skills: sysadmin-filesystem-permissions-and-processes (§0–§3), sysadmin-systemd-storage-and-networking (§4–§6), sysadmin-packages-boot-and-hardening (§7–§9), sysadmin-reference (§15–§20). Section numbers are shared across the set; a reference written as §N → skill points into that sibling skill.

Currency: The core — POSIX semantics, kernel subsystems, systemd, networking — is stable; distribution lifecycle dates and kernel LTS windows move. See §17 → sysadmin-reference for the dated table.

How to read this. Command-dense and operational. Assumes you can already use a shell; this is about understanding the machine well enough to fix it under pressure.

⚠️ GOTCHA boxes mark what causes outages or silent data loss.

The three mental models that make everything else make sense:

  1. ⚠️ Everything is a file descriptor, and the kernel is the only thing that can actually do anything. Userspace asks via syscalls. When something "doesn't work," the question is always: which syscall failed, and what errno? — and strace will tell you (§12).
  2. ⚠️ State lives in four places and they disagree: what's on disk, what's in the kernel's memory, what the running process loaded at start, and what your config management thinks. Most confusing bugs are a disagreement between these (§11.2).
  3. ⚠️ A server is a cache of your configuration management, not a pet. If you cannot rebuild it from source control, you don't have a server — you have an artifact nobody understands (§15 → sysadmin-reference).

§10. Observability and Performance

10.1 The method, not the tools

⚠️ Brendan Gregg's USE method, applied to every resource (CPU, memory, disk, network):

Utilization   — how busy
Saturation    — ⚠️ queued work; usually the number that actually matters
Errors        — ⚠️ check these FIRST; they're cheap and often decisive

And the 60-second triage:

uptime              load trend
dmesg -T | tail     ⚠️ OOM kills, hardware errors, resets
vmstat 1            r, b, si/so (⚠️ swap!), us/sy/id/wa
mpstat -P ALL 1     ⚠️ per-CPU — one hot core is a different problem
pidstat 1           per-process CPU
iostat -xz 1        await, aqu-sz, %util
free -m             ⚠️ read "available", not "free"
sar -n DEV 1        network throughput
sar -n TCP,ETCP 1   retransmits, errors
top / htop

10.2 Memory, read correctly

⚠️ GOTCHA — "free" memory is not the number you want. Linux uses free RAM for page cache deliberately. A healthy busy server shows very little free memory and that is correct. Read available — it accounts for reclaimable cache. ⚠️ Panicking about low "free" and adding RAM is the most common false diagnosis in Linux administration.

Real pressure signals: swap in/out activity (si/so in vmstat — ⚠️ not swap used, which can be stale and harmless), OOM kills in dmesg, and PSI — cat /proc/pressure/{cpu,memory,io} — ⚠️ which is the modern, direct measure of contention and is far better than inferring from utilization.

10.3 CPU and profiling

perf top, perf record -F 99 -g -p PID -- sleep 30 then perf report. Flame graphs for stack aggregation. ⚠️ %steal in a VM means the hypervisor is taking your CPU — no amount of guest tuning fixes it.

10.4 eBPF — the modern layer

⚠️ Why it displaced the older tools: strace uses ptrace and stops the process on every syscall — profiling a high-throughput service with it is an incident waiting to happen. eBPF runs verified, sandboxed programs inside the kernel with near-zero overhead, no module loading, no reboots. SystemTap needed debug symbols and compiled kernel modules; eBPF with CO-RE (Compile Once, Run Everywhere) is portable.

# BCC tools (apt install bpfcc-tools / dnf install bcc-tools) — 80+ ready scripts
execsnoop      ⚠️ every process exec — brilliant for "what keeps spawning?"
opensnoop      file opens — "which config is it ACTUALLY reading?"
biosnoop       block I/O with latency per operation
biolatency     I/O latency histogram
tcplife        TCP sessions with duration and bytes
tcpconnect / tcpaccept / tcpretrans
runqlat        ⚠️ scheduler run-queue latency — CPU saturation without high utilization
cachestat      page cache hit ratio
memleak        outstanding allocations
profile        CPU stack sampling
# bpftrace — awk for the kernel
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'
bpftrace -e 'tracepoint:syscalls:sys_enter_read /comm=="nginx"/ { @ = hist(args->count); }'
bpftrace -l 'tracepoint:syscalls:*'     ⚠️ list available probes

⚠️ Two cautions: overhead is low but not zero — tracing every scheduler context switch on a busy box will hurt; filter or sample. And kprobes attach to kernel internals that can change between versions — ⚠️ prefer tracepoints (stable API) over kprobes where one exists.

Adjacent: Cilium/Hubble (Kubernetes networking and observability), Falco (runtime security), Pixie, Inspektor Gadget.

10.5 Metrics and logs

Prometheus + node_exporter + Grafana is the default stack; Loki or the ELK/OpenSearch stack for logs; OpenTelemetry for traces. ⚠️ Alert on symptoms users feel (latency, error rate, saturation), not on causes (CPU 80%). Every alert should have a runbook, and an alert nobody acts on should be deleted.


§11. Troubleshooting Methodology

11.1 The sequence

1. ⚠️ What CHANGED?  deploys, package updates, config management runs, certs, DNS,
                      upstream, traffic pattern. Check this BEFORE theorizing.
2. Define the symptom precisely — who, what, since when, how often, all or some?
3. Bisect the stack:  client → DNS → network → LB → host → service → dependency → storage
4. Read the errors:   journalctl -p err -b · dmesg -T · the app's own log
5. Form ONE hypothesis, test it cheaply, record the result
6. Fix, verify, and write down what it was

⚠️ The single highest-yield question is "what changed?" and it is the one people skip while forming theories.

11.2 The four-places-state problem

⚠️ When behaviour makes no sense, check whether these agree:

On disk           cat the actual file — ⚠️ and check for .rpmnew (§7)
Kernel/live       sysctl -a · ip a · nft list ruleset · systemctl show
Process memory    ⚠️ the process loaded config at START — it may be running old config
                  ls -l /proc/PID/exe · /proc/PID/environ · restart to be sure
Config management ⚠️ is Ansible/Puppet about to revert your manual fix?

⚠️ "I fixed it and it broke again 30 minutes later" is config management reverting you.

11.3 Specific scenarios

Disk full but du disagrees → ⚠️ deleted-but-open files: lsof +L1 (§1.2 → sysadmin-filesystem-permissions-and-processes). Also check for a filesystem mounted over a populated directory hiding its contents. Out of inodes → df -i (§1.2 → sysadmin-filesystem-permissions-and-processes). Service won't start → systemctl status -l, journalctl -xeu NAME, systemd-analyze verify unit, ⚠️ and check SELinux with ausearch -m avc -ts recent. Port already in use → ss -tulpn | grep :PORT. Slow but idle CPU → ⚠️ runqlat, PSI, %steal, I/O await — saturation without utilization. Intermittent DNS → ⚠️ resolvectl status, check search domains and ndots, compare dig @server against dig. Clock drift → timedatectl, chronyc tracking — ⚠️ breaks TLS, Kerberos, and log correlation in ways that look like other problems. Certificate expiry → openssl s_client -connect host:443 </dev/null 2>/dev/null | openssl x509 -noout -dates. ⚠️ Monitor this; it is a fully preventable, recurring outage cause.


§12. Containers and Namespaces

⚠️ A container is not a VM. It is a normal process with restricted visibility, built from kernel primitives:

Namespaces  mnt, pid, net, ipc, uts, user, cgroup, time   ⚠️ WHAT IT CAN SEE
cgroups v2  cpu, memory, io, pids                          ⚠️ WHAT IT CAN USE
capabilities, seccomp, LSM (SELinux/AppArmor)              ⚠️ WHAT IT CAN DO
overlayfs                                                  layered images
lsns · nsenter -t PID -n ip a          ⚠️ enter a container's netns from the host
systemd-cgls · systemd-cgtop           cgroup tree and live resource use
cat /proc/PID/cgroup                   which cgroup is this process in
podman / docker / nerdctl · crictl     runtimes

⚠️ The security consequence: the kernel is shared. A kernel vulnerability crosses the container boundary. Run rootless (podman, or userns-remap), drop capabilities, apply seccomp, and never --privileged in production.

⚠️ Systemd and containers overlap deliberately — systemd-nspawn, and Delegate=yes for cgroup delegation. A hardened systemd unit (§4.3 → sysadmin-systemd-storage-and-networking) gives you most of a container's isolation without the image.


§13. Automation and Configuration Management

⚠️ The principle: the box is disposable, the source of truth is the repo. Ansible (agentless over SSH, ⚠️ the low-friction default), Puppet/Chef/Salt (agent-based, stronger for continuous enforcement), Terraform/OpenTofu for infrastructure, cloud-init for first boot, Packer for images.

⚠️ Idempotence is the property that matters — running twice must equal running once. Test with --check --diff before applying.

Immutable infrastructure — build an image, deploy it, ⚠️ never patch in place; replace. Reduces drift to zero, at the cost of build pipeline complexity.

⚠️ Practical discipline that pays for itself: everything in git including /etc (consider etckeeper), change one thing at a time, stage before production, and a documented rollback for anything you can't undo in five minutes.


§14. Backup and Recovery

⚠️ 3-2-1: three copies, two media, one offsite — and now "one offline or immutable," because ransomware targets the backup server first.

Tools: restic and borg (⚠️ deduplicating, encrypted, and the sane defaults for most people), rsync (⚠️ --link-dest for hardlinked incrementals), zfs send/btrfs send for filesystem-native, tar for archives.

⚠️ GOTCHA — an untested backup is not a backup. Schedule restores. The recurring failure modes are: backups silently stopped weeks ago, the encryption key was only on the machine that died, the restore takes 40 hours and RTO is 4, and the database backup is a file copy of a running database and is therefore corrupt. ⚠️ Databases need their own consistent dump or snapshot mechanism — pg_dump, mysqldump --single-transaction, or a filesystem snapshot with the DB quiesced.

Document and test: RPO (how much data can you lose), RTO (how fast must you be back), and ⚠️ a written recovery procedure someone who isn't you can follow at 3am.

04 同仓库更多 Skills

the-vibey-project
the-vibey-project/vibey

white-paper-writing

Use when writing white papers, technical reports, policy documents, thought leadership pieces, or any long-form authoritative document. Triggers on requests to write, structure, outline, or review white papers, technical reports, position papers, policy briefs, or evidence-driven arguments. Also use for B2B content marketing strategy, technical documentation requiring authority, academic-style reports, or any document that must educate while persuading. Use when someone asks "how should I structure this" for a long-form document.

★ 5
the-vibey-project
the-vibey-project/vibey

prose-technical-mechanics

Use when editing or writing prose at a technical/mechanical level: choosing between Anglo-Saxon and Latinate diction, placing information in topic vs stress positions, writing periodic vs cumulative sentences, handling sentence rhythm, punctuation as logic, transitions, paragraph structure, or phonesthetics. Use when someone wants to understand WHY prose works or doesn't, not just surface-level style advice. Triggers on word choice, sentence structure, emphasis, rhythm, diction, or any deep writing mechanics question.

★ 5
the-vibey-project
the-vibey-project/vibey

prose-mechanics

Use when writing, editing, or analyzing prose style; when trying to avoid generic AI-sounding writing; when applying specific prose styles such as journalistic, academic, narrative, KJV biblical, Hemingway plain, Latinate/Jamesian, Orwellian, Augustan, Victorian, Early Modern (Bacon, Browne, Milton), or free indirect discourse; or when coaching someone on writing voice and mechanics. Triggers on any request to improve, analyze, or write in a specific prose style, to fix "AI-sounding" writing, to make prose more vivid or concrete, to vary sentence rhythm, or to understand how a particular writer achieves their effects.

★ 5
the-vibey-project
the-vibey-project/vibey

narrative-structure

Use when structuring stories, pitches, presentations, white papers, or any persuasive narrative; analyzing the hero's journey in fiction or non-fiction; applying StoryBrand for business communication; designing narrative arcs for thought leadership; or understanding Vogler, Snyder, Propp, or Campbell frameworks. Triggers on any narrative structure, storytelling, pitch structure, StoryBrand, hero's journey, three-act structure, Freytag, Save the Cat, narrative arc, or "how should I structure this story/pitch/presentation" request.

★ 5
the-vibey-project
the-vibey-project/vibey

legal-brief-writing

Use when writing legal briefs, motions, or any court filing; when analyzing judicial cognition and persuasion; when applying elite brief-writing techniques from Roberts, Clement, Kagan, Garner; or when structuring legal arguments for maximum persuasion. Also use for autistic/neurodivergent advocates who need to understand how their natural communication style maps to elite legal advocacy. Triggers on any legal writing, brief drafting, motion preparation, or legal argument structuring request.

★ 5
the-vibey-project
the-vibey-project/vibey

journal-publishability-criteria

Use whenever a research paper, article or preprint is being written, revised, reviewed or published, and specifically before every publication of docs/paper.md. Covers what legitimate journals actually gate on, in order of gating — the editor's desk screen (scope, novelty, package problems), what reviewers judge (methodological soundness, claims the evidence can carry, statistical and qualitative reporting), EQUATOR reporting guidelines, the ICMJE authorship criteria, COPE's view of text recycling, the near-universal generative-AI policy (no AI author, disclosure, author responsibility), TOP 2025 transparency standards, how to verify a journal is legitimate, submitting without an institution, and a pre-submission checklist. Triggers on requests to evaluate, review, prepare or submit a paper, to judge publishability, to write a declarations or data-availability section, to disclose AI use, or to check a journal.

★ 5