sysadmin-observability-troubleshooting-and-operations
Use when a server is slow, broken, or needs operating at scale: performance methodology rather than tool trivia, reading memory correctly, CPU and profiling, eBPF as the modern observability layer, metrics and logs, a troubleshooting sequence and the four-places-state problem, containers and the namespaces and cgroups underneath them, configuration management and automation, and backup and recovery that actually restores.
- License 许可
- MIT
- 最近更新
- 2026-11-15
- Stars
- ★ 5
- Forks
- 3
01 前言 Front Matter
- name
- sysadmin-observability-troubleshooting-and-operations
- description
- Use when a server is slow, broken, or needs operating at scale: performance methodology rather than tool trivia, reading memory correctly, CPU and profiling, eBPF as the modern observability layer, metrics and logs, a troubleshooting sequence and the four-places-state problem, containers and the namespaces and cgroups underneath them, configuration management and automation, and backup and recovery that actually restores.
03 SKILL.md 正文
Linux Server Administration: Observability, Troubleshooting, Containers, Automation, and Backup
Part 4 of 5 of the Linux Server Administration reference (plugin
linux-server-admin), covering §10–§14. Sibling skills:sysadmin-filesystem-permissions-and-processes(§0–§3),sysadmin-systemd-storage-and-networking(§4–§6),sysadmin-packages-boot-and-hardening(§7–§9),sysadmin-reference(§15–§20). Section numbers are shared across the set; a reference written as §N →skillpoints into that sibling skill.Currency: The core — POSIX semantics, kernel subsystems, systemd, networking — is stable; distribution lifecycle dates and kernel LTS windows move. See §17 →
sysadmin-referencefor the dated table.
How to read this. Command-dense and operational. Assumes you can already use a shell; this is about understanding the machine well enough to fix it under pressure.
⚠️ GOTCHA boxes mark what causes outages or silent data loss.
The three mental models that make everything else make sense:
- ⚠️ Everything is a file descriptor, and the kernel is the only thing that can actually do anything. Userspace asks via syscalls. When something "doesn't work," the question is always: which syscall failed, and what errno? — and
stracewill tell you (§12).- ⚠️ State lives in four places and they disagree: what's on disk, what's in the kernel's memory, what the running process loaded at start, and what your config management thinks. Most confusing bugs are a disagreement between these (§11.2).
- ⚠️ A server is a cache of your configuration management, not a pet. If you cannot rebuild it from source control, you don't have a server — you have an artifact nobody understands (§15 →
sysadmin-reference).
§10. Observability and Performance
10.1 The method, not the tools
⚠️ Brendan Gregg's USE method, applied to every resource (CPU, memory, disk, network):
Utilization — how busy
Saturation — ⚠️ queued work; usually the number that actually matters
Errors — ⚠️ check these FIRST; they're cheap and often decisive
And the 60-second triage:
uptime load trend
dmesg -T | tail ⚠️ OOM kills, hardware errors, resets
vmstat 1 r, b, si/so (⚠️ swap!), us/sy/id/wa
mpstat -P ALL 1 ⚠️ per-CPU — one hot core is a different problem
pidstat 1 per-process CPU
iostat -xz 1 await, aqu-sz, %util
free -m ⚠️ read "available", not "free"
sar -n DEV 1 network throughput
sar -n TCP,ETCP 1 retransmits, errors
top / htop
10.2 Memory, read correctly
⚠️ GOTCHA — "free" memory is not the number you want. Linux uses free RAM for page cache deliberately. A healthy busy server shows very little free memory and that is correct. Read
available— it accounts for reclaimable cache. ⚠️ Panicking about low "free" and adding RAM is the most common false diagnosis in Linux administration.
Real pressure signals: swap in/out activity (si/so in vmstat — ⚠️ not swap
used, which can be stale and harmless), OOM kills in dmesg, and
PSI — cat /proc/pressure/{cpu,memory,io} — ⚠️ which is the modern, direct measure
of contention and is far better than inferring from utilization.
10.3 CPU and profiling
perf top, perf record -F 99 -g -p PID -- sleep 30 then perf report.
Flame graphs for stack aggregation. ⚠️ %steal in a VM means the hypervisor is
taking your CPU — no amount of guest tuning fixes it.
10.4 eBPF — the modern layer
⚠️ Why it displaced the older tools: strace uses ptrace and stops the process on
every syscall — profiling a high-throughput service with it is an incident waiting to
happen. eBPF runs verified, sandboxed programs inside the kernel with near-zero
overhead, no module loading, no reboots. SystemTap needed debug symbols and compiled
kernel modules; eBPF with CO-RE (Compile Once, Run Everywhere) is portable.
# BCC tools (apt install bpfcc-tools / dnf install bcc-tools) — 80+ ready scripts
execsnoop ⚠️ every process exec — brilliant for "what keeps spawning?"
opensnoop file opens — "which config is it ACTUALLY reading?"
biosnoop block I/O with latency per operation
biolatency I/O latency histogram
tcplife TCP sessions with duration and bytes
tcpconnect / tcpaccept / tcpretrans
runqlat ⚠️ scheduler run-queue latency — CPU saturation without high utilization
cachestat page cache hit ratio
memleak outstanding allocations
profile CPU stack sampling
# bpftrace — awk for the kernel
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'
bpftrace -e 'tracepoint:syscalls:sys_enter_read /comm=="nginx"/ { @ = hist(args->count); }'
bpftrace -l 'tracepoint:syscalls:*' ⚠️ list available probes
⚠️ Two cautions: overhead is low but not zero — tracing every scheduler context switch on a busy box will hurt; filter or sample. And kprobes attach to kernel internals that can change between versions — ⚠️ prefer tracepoints (stable API) over kprobes where one exists.
Adjacent: Cilium/Hubble (Kubernetes networking and observability), Falco (runtime security), Pixie, Inspektor Gadget.
10.5 Metrics and logs
Prometheus + node_exporter + Grafana is the default stack; Loki or the ELK/OpenSearch stack for logs; OpenTelemetry for traces. ⚠️ Alert on symptoms users feel (latency, error rate, saturation), not on causes (CPU 80%). Every alert should have a runbook, and an alert nobody acts on should be deleted.
§11. Troubleshooting Methodology
11.1 The sequence
1. ⚠️ What CHANGED? deploys, package updates, config management runs, certs, DNS,
upstream, traffic pattern. Check this BEFORE theorizing.
2. Define the symptom precisely — who, what, since when, how often, all or some?
3. Bisect the stack: client → DNS → network → LB → host → service → dependency → storage
4. Read the errors: journalctl -p err -b · dmesg -T · the app's own log
5. Form ONE hypothesis, test it cheaply, record the result
6. Fix, verify, and write down what it was
⚠️ The single highest-yield question is "what changed?" and it is the one people skip while forming theories.
11.2 The four-places-state problem
⚠️ When behaviour makes no sense, check whether these agree:
On disk cat the actual file — ⚠️ and check for .rpmnew (§7)
Kernel/live sysctl -a · ip a · nft list ruleset · systemctl show
Process memory ⚠️ the process loaded config at START — it may be running old config
ls -l /proc/PID/exe · /proc/PID/environ · restart to be sure
Config management ⚠️ is Ansible/Puppet about to revert your manual fix?
⚠️ "I fixed it and it broke again 30 minutes later" is config management reverting you.
11.3 Specific scenarios
Disk full but du disagrees → ⚠️ deleted-but-open files: lsof +L1 (§1.2 → sysadmin-filesystem-permissions-and-processes).
Also check for a filesystem mounted over a populated directory hiding its contents.
Out of inodes → df -i (§1.2 → sysadmin-filesystem-permissions-and-processes).
Service won't start → systemctl status -l, journalctl -xeu NAME,
systemd-analyze verify unit, ⚠️ and check SELinux with ausearch -m avc -ts recent.
Port already in use → ss -tulpn | grep :PORT.
Slow but idle CPU → ⚠️ runqlat, PSI, %steal, I/O await — saturation without
utilization.
Intermittent DNS → ⚠️ resolvectl status, check search domains and ndots, compare
dig @server against dig.
Clock drift → timedatectl, chronyc tracking — ⚠️ breaks TLS, Kerberos, and
log correlation in ways that look like other problems.
Certificate expiry → openssl s_client -connect host:443 </dev/null 2>/dev/null | openssl x509 -noout -dates. ⚠️ Monitor this; it is a fully preventable, recurring
outage cause.
§12. Containers and Namespaces
⚠️ A container is not a VM. It is a normal process with restricted visibility, built from kernel primitives:
Namespaces mnt, pid, net, ipc, uts, user, cgroup, time ⚠️ WHAT IT CAN SEE
cgroups v2 cpu, memory, io, pids ⚠️ WHAT IT CAN USE
capabilities, seccomp, LSM (SELinux/AppArmor) ⚠️ WHAT IT CAN DO
overlayfs layered images
lsns · nsenter -t PID -n ip a ⚠️ enter a container's netns from the host
systemd-cgls · systemd-cgtop cgroup tree and live resource use
cat /proc/PID/cgroup which cgroup is this process in
podman / docker / nerdctl · crictl runtimes
⚠️ The security consequence: the kernel is shared. A kernel vulnerability crosses
the container boundary. Run rootless (podman, or userns-remap), drop capabilities, apply
seccomp, and never --privileged in production.
⚠️ Systemd and containers overlap deliberately — systemd-nspawn, and Delegate=yes
for cgroup delegation. A hardened systemd unit (§4.3 → sysadmin-systemd-storage-and-networking) gives you most of a container's
isolation without the image.
§13. Automation and Configuration Management
⚠️ The principle: the box is disposable, the source of truth is the repo. Ansible (agentless over SSH, ⚠️ the low-friction default), Puppet/Chef/Salt (agent-based, stronger for continuous enforcement), Terraform/OpenTofu for infrastructure, cloud-init for first boot, Packer for images.
⚠️ Idempotence is the property that matters — running twice must equal running once.
Test with --check --diff before applying.
Immutable infrastructure — build an image, deploy it, ⚠️ never patch in place; replace. Reduces drift to zero, at the cost of build pipeline complexity.
⚠️ Practical discipline that pays for itself: everything in git including /etc
(consider etckeeper), change one thing at a time, stage before production, and
a documented rollback for anything you can't undo in five minutes.
§14. Backup and Recovery
⚠️ 3-2-1: three copies, two media, one offsite — and now "one offline or immutable," because ransomware targets the backup server first.
Tools: restic and borg (⚠️ deduplicating, encrypted, and the sane defaults for
most people), rsync (⚠️ --link-dest for hardlinked incrementals), zfs send/btrfs send for filesystem-native, tar for archives.
⚠️ GOTCHA — an untested backup is not a backup. Schedule restores. The recurring failure modes are: backups silently stopped weeks ago, the encryption key was only on the machine that died, the restore takes 40 hours and RTO is 4, and the database backup is a file copy of a running database and is therefore corrupt. ⚠️ Databases need their own consistent dump or snapshot mechanism —
pg_dump,mysqldump --single-transaction, or a filesystem snapshot with the DB quiesced.
Document and test: RPO (how much data can you lose), RTO (how fast must you be back), and ⚠️ a written recovery procedure someone who isn't you can follow at 3am.
04 同仓库更多 Skills
white-paper-writing
Use when writing white papers, technical reports, policy documents, thought leadership pieces, or any long-form authoritative document. Triggers on requests to write, structure, outline, or review white papers, technical reports, position papers, policy briefs, or evidence-driven arguments. Also use for B2B content marketing strategy, technical documentation requiring authority, academic-style reports, or any document that must educate while persuading. Use when someone asks "how should I structure this" for a long-form document.
prose-technical-mechanics
Use when editing or writing prose at a technical/mechanical level: choosing between Anglo-Saxon and Latinate diction, placing information in topic vs stress positions, writing periodic vs cumulative sentences, handling sentence rhythm, punctuation as logic, transitions, paragraph structure, or phonesthetics. Use when someone wants to understand WHY prose works or doesn't, not just surface-level style advice. Triggers on word choice, sentence structure, emphasis, rhythm, diction, or any deep writing mechanics question.
prose-mechanics
Use when writing, editing, or analyzing prose style; when trying to avoid generic AI-sounding writing; when applying specific prose styles such as journalistic, academic, narrative, KJV biblical, Hemingway plain, Latinate/Jamesian, Orwellian, Augustan, Victorian, Early Modern (Bacon, Browne, Milton), or free indirect discourse; or when coaching someone on writing voice and mechanics. Triggers on any request to improve, analyze, or write in a specific prose style, to fix "AI-sounding" writing, to make prose more vivid or concrete, to vary sentence rhythm, or to understand how a particular writer achieves their effects.
narrative-structure
Use when structuring stories, pitches, presentations, white papers, or any persuasive narrative; analyzing the hero's journey in fiction or non-fiction; applying StoryBrand for business communication; designing narrative arcs for thought leadership; or understanding Vogler, Snyder, Propp, or Campbell frameworks. Triggers on any narrative structure, storytelling, pitch structure, StoryBrand, hero's journey, three-act structure, Freytag, Save the Cat, narrative arc, or "how should I structure this story/pitch/presentation" request.
legal-brief-writing
Use when writing legal briefs, motions, or any court filing; when analyzing judicial cognition and persuasion; when applying elite brief-writing techniques from Roberts, Clement, Kagan, Garner; or when structuring legal arguments for maximum persuasion. Also use for autistic/neurodivergent advocates who need to understand how their natural communication style maps to elite legal advocacy. Triggers on any legal writing, brief drafting, motion preparation, or legal argument structuring request.
journal-publishability-criteria
Use whenever a research paper, article or preprint is being written, revised, reviewed or published, and specifically before every publication of docs/paper.md. Covers what legitimate journals actually gate on, in order of gating — the editor's desk screen (scope, novelty, package problems), what reviewers judge (methodological soundness, claims the evidence can carry, statistical and qualitative reporting), EQUATOR reporting guidelines, the ICMJE authorship criteria, COPE's view of text recycling, the near-universal generative-AI policy (no AI author, disclosure, author responsibility), TOP 2025 transparency standards, how to verify a journal is legitimate, submitting without an institution, and a pre-submission checklist. Triggers on requests to evaluate, review, prepare or submit a paper, to judge publishability, to write a declarations or data-availability section, to disclose AI use, or to check a journal.