[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7APAWTVK7KNPC0YW3DSA3P7":3,"skill-same:01M7APAWTVK7KNPC0YW3DSA3P7":48},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":29,"ownerHtmlUrl":30,"ownerBlogUrl":31,"ownerDescription":32,"updatedAt":33,"install":34,"resources":47},"01M7APAWTVK7KNPC0YW3DSA3P7","sysadmin-observability-troubleshooting-and-operations",null,"Use when a server is slow, broken, or needs operating at scale: performance methodology rather than tool trivia, reading memory correctly, CPU and profiling, eBPF as the modern observability layer, metrics and logs, a troubleshooting sequence and the four-places-state problem, containers and the namespaces and cgroups underneath them, configuration management and automation, and backup and recovery that actually restores.","the-vibey-project\u002Fvibey","the-vibey-project","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F329298758?v=4","01M7AJHGZ9722CTJ4JNQZVRH0Q","the vibey project","01M7AJHGZVMG2K7TGXQ6XFHVG3",5,"MIT","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Flinux-server-admin\u002Fskills\u002Fsysadmin-observability-troubleshooting-and-operations\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: sysadmin-observability-troubleshooting-and-operations\ndescription: \"Use when a server is slow, broken, or needs operating at scale: performance methodology rather than tool trivia, reading memory correctly, CPU and profiling, eBPF as the modern observability layer, metrics and logs, a troubleshooting sequence and the four-places-state problem, containers and the namespaces and cgroups underneath them, configuration management and automation, and backup and recovery that actually restores.\"\n---\n\n# Linux Server Administration: Observability, Troubleshooting, Containers, Automation, and Backup\n\n> **Part 4 of 5** of the *Linux Server Administration* reference (plugin `linux-server-admin`), covering §10–§14. Sibling skills: `sysadmin-filesystem-permissions-and-processes` (§0–§3), `sysadmin-systemd-storage-and-networking` (§4–§6), `sysadmin-packages-boot-and-hardening` (§7–§9), `sysadmin-reference` (§15–§20). Section numbers are shared across the set; a reference written as §N → `skill` points into that sibling skill.\n>\n> **Currency:** The core — POSIX semantics, kernel subsystems, systemd, networking — is stable; distribution lifecycle dates and kernel LTS windows move. See §17 → `sysadmin-reference` for the dated table.\n\n> **How to read this.** Command-dense and operational. Assumes you can already use a\n> shell; this is about understanding the machine well enough to fix it under pressure.\n>\n> **⚠️ GOTCHA** boxes mark what causes outages or silent data loss.\n>\n> **The three mental models that make everything else make sense:**\n> 1. **⚠️ Everything is a file descriptor, and the kernel is the only thing that can\n>    actually do anything.** Userspace asks via syscalls. **When something \"doesn't work,\"\n>    the question is always: which syscall failed, and what errno?** — and `strace` will\n>    tell you (§12).\n> 2. **⚠️ State lives in four places and they disagree**: what's on disk, what's in the\n>    kernel's memory, what the running process loaded at start, and what your config\n>    management thinks. **Most confusing bugs are a disagreement between these** (§11.2).\n> 3. **⚠️ A server is a cache of your configuration management, not a pet.** If you cannot\n>    rebuild it from source control, you don't have a server — you have an artifact\n>    nobody understands (§15 → `sysadmin-reference`).\n\n---\n\n## §10. Observability and Performance\n\n### 10.1 The method, not the tools\n**⚠️ Brendan Gregg's USE method**, applied to every resource (CPU, memory, disk, network):\n```\nUtilization   — how busy\nSaturation    — ⚠️ queued work; usually the number that actually matters\nErrors        — ⚠️ check these FIRST; they're cheap and often decisive\n```\n**And the 60-second triage:**\n```\nuptime              load trend\ndmesg -T | tail     ⚠️ OOM kills, hardware errors, resets\nvmstat 1            r, b, si\u002Fso (⚠️ swap!), us\u002Fsy\u002Fid\u002Fwa\nmpstat -P ALL 1     ⚠️ per-CPU — one hot core is a different problem\npidstat 1           per-process CPU\niostat -xz 1        await, aqu-sz, %util\nfree -m             ⚠️ read \"available\", not \"free\"\nsar -n DEV 1        network throughput\nsar -n TCP,ETCP 1   retransmits, errors\ntop \u002F htop\n```\n\n### 10.2 Memory, read correctly\n> **⚠️ GOTCHA — \"free\" memory is not the number you want.** Linux uses free RAM for page\n> cache deliberately. **A healthy busy server shows very little free memory and that is\n> correct.** **Read `available`** — it accounts for reclaimable cache.\n> ⚠️ **Panicking about low \"free\" and adding RAM is the most common false diagnosis in\n> Linux administration.**\n\n**Real pressure signals**: **swap in\u002Fout activity** (`si`\u002F`so` in vmstat — ⚠️ **not swap\n*used*, which can be stale and harmless**), **OOM kills in dmesg**, and\n**PSI** — `cat \u002Fproc\u002Fpressure\u002F{cpu,memory,io}` — ⚠️ **which is the modern, direct measure\nof contention and is far better than inferring from utilization.**\n\n### 10.3 CPU and profiling\n`perf top`, `perf record -F 99 -g -p PID -- sleep 30` then `perf report`.\n**Flame graphs** for stack aggregation. ⚠️ **`%steal` in a VM means the hypervisor is\ntaking your CPU — no amount of guest tuning fixes it.**\n\n### 10.4 eBPF — the modern layer\n**⚠️ Why it displaced the older tools**: `strace` uses ptrace and **stops the process on\nevery syscall** — profiling a high-throughput service with it is an incident waiting to\nhappen. **eBPF runs verified, sandboxed programs inside the kernel with near-zero\noverhead, no module loading, no reboots.** SystemTap needed debug symbols and compiled\nkernel modules; **eBPF with CO-RE (Compile Once, Run Everywhere) is portable.**\n\n```\n# BCC tools (apt install bpfcc-tools \u002F dnf install bcc-tools) — 80+ ready scripts\nexecsnoop      ⚠️ every process exec — brilliant for \"what keeps spawning?\"\nopensnoop      file opens — \"which config is it ACTUALLY reading?\"\nbiosnoop       block I\u002FO with latency per operation\nbiolatency     I\u002FO latency histogram\ntcplife        TCP sessions with duration and bytes\ntcpconnect \u002F tcpaccept \u002F tcpretrans\nrunqlat        ⚠️ scheduler run-queue latency — CPU saturation without high utilization\ncachestat      page cache hit ratio\nmemleak        outstanding allocations\nprofile        CPU stack sampling\n```\n```\n# bpftrace — awk for the kernel\nbpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf(\"%s %s\\n\", comm, str(args->filename)); }'\nbpftrace -e 'tracepoint:syscalls:sys_enter_read \u002Fcomm==\"nginx\"\u002F { @ = hist(args->count); }'\nbpftrace -l 'tracepoint:syscalls:*'     ⚠️ list available probes\n```\n**⚠️ Two cautions**: overhead is low but **not zero — tracing every scheduler context\nswitch on a busy box will hurt**; filter or sample. And **kprobes attach to kernel\ninternals that can change between versions** — ⚠️ **prefer tracepoints (stable API) over\nkprobes where one exists.**\n\n**Adjacent**: **Cilium\u002FHubble** (Kubernetes networking and observability), **Falco**\n(runtime security), **Pixie**, **Inspektor Gadget**.\n\n### 10.5 Metrics and logs\n**Prometheus + node_exporter + Grafana** is the default stack; **Loki** or the\n**ELK\u002FOpenSearch** stack for logs; **OpenTelemetry** for traces.\n**⚠️ Alert on symptoms users feel** (latency, error rate, saturation), **not on causes**\n(CPU 80%). **Every alert should have a runbook, and an alert nobody acts on should be\ndeleted.**\n\n---\n\n## §11. Troubleshooting Methodology\n\n### 11.1 The sequence\n```\n1. ⚠️ What CHANGED?  deploys, package updates, config management runs, certs, DNS,\n                      upstream, traffic pattern. Check this BEFORE theorizing.\n2. Define the symptom precisely — who, what, since when, how often, all or some?\n3. Bisect the stack:  client → DNS → network → LB → host → service → dependency → storage\n4. Read the errors:   journalctl -p err -b · dmesg -T · the app's own log\n5. Form ONE hypothesis, test it cheaply, record the result\n6. Fix, verify, and write down what it was\n```\n**⚠️ The single highest-yield question is \"what changed?\" and it is the one people skip\nwhile forming theories.**\n\n### 11.2 The four-places-state problem\n**⚠️ When behaviour makes no sense, check whether these agree:**\n```\nOn disk           cat the actual file — ⚠️ and check for .rpmnew (§7)\nKernel\u002Flive       sysctl -a · ip a · nft list ruleset · systemctl show\nProcess memory    ⚠️ the process loaded config at START — it may be running old config\n                  ls -l \u002Fproc\u002FPID\u002Fexe · \u002Fproc\u002FPID\u002Fenviron · restart to be sure\nConfig management ⚠️ is Ansible\u002FPuppet about to revert your manual fix?\n```\n**⚠️ \"I fixed it and it broke again 30 minutes later\" is config management reverting you.**\n\n### 11.3 Specific scenarios\n**Disk full but `du` disagrees** → ⚠️ **deleted-but-open files: `lsof +L1`** (§1.2 → `sysadmin-filesystem-permissions-and-processes`).\nAlso check for a filesystem mounted *over* a populated directory hiding its contents.\n**Out of inodes** → `df -i` (§1.2 → `sysadmin-filesystem-permissions-and-processes`).\n**Service won't start** → `systemctl status -l`, `journalctl -xeu NAME`,\n`systemd-analyze verify unit`, ⚠️ **and check SELinux with `ausearch -m avc -ts recent`.**\n**Port already in use** → `ss -tulpn | grep :PORT`.\n**Slow but idle CPU** → ⚠️ **`runqlat`, PSI, `%steal`, I\u002FO `await`** — saturation without\nutilization.\n**Intermittent DNS** → ⚠️ **`resolvectl status`, check search domains and `ndots`, compare\n`dig @server` against `dig`.**\n**Clock drift** → `timedatectl`, `chronyc tracking` — ⚠️ **breaks TLS, Kerberos, and\nlog correlation in ways that look like other problems.**\n**Certificate expiry** → `openssl s_client -connect host:443 \u003C\u002Fdev\u002Fnull 2>\u002Fdev\u002Fnull |\nopenssl x509 -noout -dates`. ⚠️ **Monitor this; it is a fully preventable, recurring\noutage cause.**\n\n---\n\n## §12. Containers and Namespaces\n\n**⚠️ A container is not a VM.** It is a normal process with restricted visibility, built\nfrom kernel primitives:\n```\nNamespaces  mnt, pid, net, ipc, uts, user, cgroup, time   ⚠️ WHAT IT CAN SEE\ncgroups v2  cpu, memory, io, pids                          ⚠️ WHAT IT CAN USE\ncapabilities, seccomp, LSM (SELinux\u002FAppArmor)              ⚠️ WHAT IT CAN DO\noverlayfs                                                  layered images\n```\n```\nlsns · nsenter -t PID -n ip a          ⚠️ enter a container's netns from the host\nsystemd-cgls · systemd-cgtop           cgroup tree and live resource use\ncat \u002Fproc\u002FPID\u002Fcgroup                   which cgroup is this process in\npodman \u002F docker \u002F nerdctl · crictl     runtimes\n```\n**⚠️ The security consequence**: the kernel is **shared**. A kernel vulnerability crosses\nthe container boundary. **Run rootless (podman, or userns-remap), drop capabilities, apply\nseccomp, and never `--privileged` in production.**\n\n**⚠️ Systemd and containers overlap deliberately** — `systemd-nspawn`, and `Delegate=yes`\nfor cgroup delegation. **A hardened systemd unit (§4.3 → `sysadmin-systemd-storage-and-networking`) gives you most of a container's\nisolation without the image.**\n\n---\n\n## §13. Automation and Configuration Management\n\n**⚠️ The principle: the box is disposable, the source of truth is the repo.**\n**Ansible** (agentless over SSH, ⚠️ **the low-friction default**), **Puppet\u002FChef\u002FSalt**\n(agent-based, stronger for continuous enforcement), **Terraform\u002FOpenTofu** for\ninfrastructure, **cloud-init** for first boot, **Packer** for images.\n\n**⚠️ Idempotence is the property that matters** — running twice must equal running once.\n**Test with `--check --diff` before applying.**\n\n**Immutable infrastructure** — build an image, deploy it, ⚠️ **never patch in place;\nreplace.** **Reduces drift to zero, at the cost of build pipeline complexity.**\n\n**⚠️ Practical discipline that pays for itself**: everything in git including `\u002Fetc`\n(consider `etckeeper`), **change one thing at a time**, **stage before production**, and\n**a documented rollback for anything you can't undo in five minutes.**\n\n---\n\n## §14. Backup and Recovery\n\n**⚠️ 3-2-1: three copies, two media, one offsite — and now \"one offline or immutable,\"\nbecause ransomware targets the backup server first.**\n\n**Tools**: `restic` and `borg` (⚠️ **deduplicating, encrypted, and the sane defaults for\nmost people**), `rsync` (⚠️ **`--link-dest` for hardlinked incrementals**), `zfs\nsend`\u002F`btrfs send` for filesystem-native, `tar` for archives.\n\n> **⚠️ GOTCHA — an untested backup is not a backup.** **Schedule restores.** The\n> recurring failure modes are: backups silently stopped weeks ago, the encryption key was\n> only on the machine that died, the restore takes 40 hours and RTO is 4, and the database\n> backup is a file copy of a running database and is therefore corrupt.\n> **⚠️ Databases need their own consistent dump or snapshot mechanism** — `pg_dump`,\n> `mysqldump --single-transaction`, or a filesystem snapshot with the DB quiesced.\n\n**Document and test**: RPO (how much data can you lose), RTO (how fast must you be back),\nand ⚠️ **a written recovery procedure someone who isn't you can follow at 3am.**\n","# Linux Server Administration: Observability, Troubleshooting, Containers, Automation, and Backup\n\n> **Part 4 of 5** of the *Linux Server Administration* reference (plugin `linux-server-admin`), covering §10–§14. Sibling skills: `sysadmin-filesystem-permissions-and-processes` (§0–§3), `sysadmin-systemd-storage-and-networking` (§4–§6), `sysadmin-packages-boot-and-hardening` (§7–§9), `sysadmin-reference` (§15–§20). Section numbers are shared across the set; a reference written as §N → `skill` points into that sibling skill.\n>\n> **Currency:** The core — POSIX semantics, kernel subsystems, systemd, networking — is stable; distribution lifecycle dates and kernel LTS windows move. See §17 → `sysadmin-reference` for the dated table.\n\n> **How to read this.** Command-dense and operational. Assumes you can already use a\n> shell; this is about understanding the machine well enough to fix it under pressure.\n>\n> **⚠️ GOTCHA** boxes mark what causes outages or silent data loss.\n>\n> **The three mental models that make everything else make sense:**\n> 1. **⚠️ Everything is a file descriptor, and the kernel is the only thing that can\n>    actually do anything.** Userspace asks via syscalls. **When something \"doesn't work,\"\n>    the question is always: which syscall failed, and what errno?** — and `strace` will\n>    tell you (§12).\n> 2. **⚠️ State lives in four places and they disagree**: what's on disk, what's in the\n>    kernel's memory, what the running process loaded at start, and what your config\n>    management thinks. **Most confusing bugs are a disagreement between these** (§11.2).\n> 3. **⚠️ A server is a cache of your configuration management, not a pet.** If you cannot\n>    rebuild it from source control, you don't have a server — you have an artifact\n>    nobody understands (§15 → `sysadmin-reference`).\n\n---\n\n## §10. Observability and Performance\n\n### 10.1 The method, not the tools\n**⚠️ Brendan Gregg's USE method**, applied to every resource (CPU, memory, disk, network):\n```\nUtilization   — how busy\nSaturation    — ⚠️ queued work; usually the number that actually matters\nErrors        — ⚠️ check these FIRST; they're cheap and often decisive\n```\n**And the 60-second triage:**\n```\nuptime              load trend\ndmesg -T | tail     ⚠️ OOM kills, hardware errors, resets\nvmstat 1            r, b, si\u002Fso (⚠️ swap!), us\u002Fsy\u002Fid\u002Fwa\nmpstat -P ALL 1     ⚠️ per-CPU — one hot core is a different problem\npidstat 1           per-process CPU\niostat -xz 1        await, aqu-sz, %util\nfree -m             ⚠️ read \"available\", not \"free\"\nsar -n DEV 1        network throughput\nsar -n TCP,ETCP 1   retransmits, errors\ntop \u002F htop\n```\n\n### 10.2 Memory, read correctly\n> **⚠️ GOTCHA — \"free\" memory is not the number you want.** Linux uses free RAM for page\n> cache deliberately. **A healthy busy server shows very little free memory and that is\n> correct.** **Read `available`** — it accounts for reclaimable cache.\n> ⚠️ **Panicking about low \"free\" and adding RAM is the most common false diagnosis in\n> Linux administration.**\n\n**Real pressure signals**: **swap in\u002Fout activity** (`si`\u002F`so` in vmstat — ⚠️ **not swap\n*used*, which can be stale and harmless**), **OOM kills in dmesg**, and\n**PSI** — `cat \u002Fproc\u002Fpressure\u002F{cpu,memory,io}` — ⚠️ **which is the modern, direct measure\nof contention and is far better than inferring from utilization.**\n\n### 10.3 CPU and profiling\n`perf top`, `perf record -F 99 -g -p PID -- sleep 30` then `perf report`.\n**Flame graphs** for stack aggregation. ⚠️ **`%steal` in a VM means the hypervisor is\ntaking your CPU — no amount of guest tuning fixes it.**\n\n### 10.4 eBPF — the modern layer\n**⚠️ Why it displaced the older tools**: `strace` uses ptrace and **stops the process on\nevery syscall** — profiling a high-throughput service with it is an incident waiting to\nhappen. **eBPF runs verified, sandboxed programs inside the kernel with near-zero\noverhead, no module loading, no reboots.** SystemTap needed debug symbols and compiled\nkernel modules; **eBPF with CO-RE (Compile Once, Run Everywhere) is portable.**\n\n```\n# BCC tools (apt install bpfcc-tools \u002F dnf install bcc-tools) — 80+ ready scripts\nexecsnoop      ⚠️ every process exec — brilliant for \"what keeps spawning?\"\nopensnoop      file opens — \"which config is it ACTUALLY reading?\"\nbiosnoop       block I\u002FO with latency per operation\nbiolatency     I\u002FO latency histogram\ntcplife        TCP sessions with duration and bytes\ntcpconnect \u002F tcpaccept \u002F tcpretrans\nrunqlat        ⚠️ scheduler run-queue latency — CPU saturation without high utilization\ncachestat      page cache hit ratio\nmemleak        outstanding allocations\nprofile        CPU stack sampling\n```\n```\n# bpftrace — awk for the kernel\nbpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf(\"%s %s\\n\", comm, str(args->filename)); }'\nbpftrace -e 'tracepoint:syscalls:sys_enter_read \u002Fcomm==\"nginx\"\u002F { @ = hist(args->count); }'\nbpftrace -l 'tracepoint:syscalls:*'     ⚠️ list available probes\n```\n**⚠️ Two cautions**: overhead is low but **not zero — tracing every scheduler context\nswitch on a busy box will hurt**; filter or sample. And **kprobes attach to kernel\ninternals that can change between versions** — ⚠️ **prefer tracepoints (stable API) over\nkprobes where one exists.**\n\n**Adjacent**: **Cilium\u002FHubble** (Kubernetes networking and observability), **Falco**\n(runtime security), **Pixie**, **Inspektor Gadget**.\n\n### 10.5 Metrics and logs\n**Prometheus + node_exporter + Grafana** is the default stack; **Loki** or the\n**ELK\u002FOpenSearch** stack for logs; **OpenTelemetry** for traces.\n**⚠️ Alert on symptoms users feel** (latency, error rate, saturation), **not on causes**\n(CPU 80%). **Every alert should have a runbook, and an alert nobody acts on should be\ndeleted.**\n\n---\n\n## §11. Troubleshooting Methodology\n\n### 11.1 The sequence\n```\n1. ⚠️ What CHANGED?  deploys, package updates, config management runs, certs, DNS,\n                      upstream, traffic pattern. Check this BEFORE theorizing.\n2. Define the symptom precisely — who, what, since when, how often, all or some?\n3. Bisect the stack:  client → DNS → network → LB → host → service → dependency → storage\n4. Read the errors:   journalctl -p err -b · dmesg -T · the app's own log\n5. Form ONE hypothesis, test it cheaply, record the result\n6. Fix, verify, and write down what it was\n```\n**⚠️ The single highest-yield question is \"what changed?\" and it is the one people skip\nwhile forming theories.**\n\n### 11.2 The four-places-state problem\n**⚠️ When behaviour makes no sense, check whether these agree:**\n```\nOn disk           cat the actual file — ⚠️ and check for .rpmnew (§7)\nKernel\u002Flive       sysctl -a · ip a · nft list ruleset · systemctl show\nProcess memory    ⚠️ the process loaded config at START — it may be running old config\n                  ls -l \u002Fproc\u002FPID\u002Fexe · \u002Fproc\u002FPID\u002Fenviron · restart to be sure\nConfig management ⚠️ is Ansible\u002FPuppet about to revert your manual fix?\n```\n**⚠️ \"I fixed it and it broke again 30 minutes later\" is config management reverting you.**\n\n### 11.3 Specific scenarios\n**Disk full but `du` disagrees** → ⚠️ **deleted-but-open files: `lsof +L1`** (§1.2 → `sysadmin-filesystem-permissions-and-processes`).\nAlso check for a filesystem mounted *over* a populated directory hiding its contents.\n**Out of inodes** → `df -i` (§1.2 → `sysadmin-filesystem-permissions-and-processes`).\n**Service won't start** → `systemctl status -l`, `journalctl -xeu NAME`,\n`systemd-analyze verify unit`, ⚠️ **and check SELinux with `ausearch -m avc -ts recent`.**\n**Port already in use** → `ss -tulpn | grep :PORT`.\n**Slow but idle CPU** → ⚠️ **`runqlat`, PSI, `%steal`, I\u002FO `await`** — saturation without\nutilization.\n**Intermittent DNS** → ⚠️ **`resolvectl status`, check search domains and `ndots`, compare\n`dig @server` against `dig`.**\n**Clock drift** → `timedatectl`, `chronyc tracking` — ⚠️ **breaks TLS, Kerberos, and\nlog correlation in ways that look like other problems.**\n**Certificate expiry** → `openssl s_client -connect host:443 \u003C\u002Fdev\u002Fnull 2>\u002Fdev\u002Fnull |\nopenssl x509 -noout -dates`. ⚠️ **Monitor this; it is a fully preventable, recurring\noutage cause.**\n\n---\n\n## §12. Containers and Namespaces\n\n**⚠️ A container is not a VM.** It is a normal process with restricted visibility, built\nfrom kernel primitives:\n```\nNamespaces  mnt, pid, net, ipc, uts, user, cgroup, time   ⚠️ WHAT IT CAN SEE\ncgroups v2  cpu, memory, io, pids                          ⚠️ WHAT IT CAN USE\ncapabilities, seccomp, LSM (SELinux\u002FAppArmor)              ⚠️ WHAT IT CAN DO\noverlayfs                                                  layered images\n```\n```\nlsns · nsenter -t PID -n ip a          ⚠️ enter a container's netns from the host\nsystemd-cgls · systemd-cgtop           cgroup tree and live resource use\ncat \u002Fproc\u002FPID\u002Fcgroup                   which cgroup is this process in\npodman \u002F docker \u002F nerdctl · crictl     runtimes\n```\n**⚠️ The security consequence**: the kernel is **shared**. A kernel vulnerability crosses\nthe container boundary. **Run rootless (podman, or userns-remap), drop capabilities, apply\nseccomp, and never `--privileged` in production.**\n\n**⚠️ Systemd and containers overlap deliberately** — `systemd-nspawn`, and `Delegate=yes`\nfor cgroup delegation. **A hardened systemd unit (§4.3 → `sysadmin-systemd-storage-and-networking`) gives you most of a container's\nisolation without the image.**\n\n---\n\n## §13. Automation and Configuration Management\n\n**⚠️ The principle: the box is disposable, the source of truth is the repo.**\n**Ansible** (agentless over SSH, ⚠️ **the low-friction default**), **Puppet\u002FChef\u002FSalt**\n(agent-based, stronger for continuous enforcement), **Terraform\u002FOpenTofu** for\ninfrastructure, **cloud-init** for first boot, **Packer** for images.\n\n**⚠️ Idempotence is the property that matters** — running twice must equal running once.\n**Test with `--check --diff` before applying.**\n\n**Immutable infrastructure** — build an image, deploy it, ⚠️ **never patch in place;\nreplace.** **Reduces drift to zero, at the cost of build pipeline complexity.**\n\n**⚠️ Practical discipline that pays for itself**: everything in git including `\u002Fetc`\n(consider `etckeeper`), **change one thing at a time**, **stage before production**, and\n**a documented rollback for anything you can't undo in five minutes.**\n\n---\n\n## §14. Backup and Recovery\n\n**⚠️ 3-2-1: three copies, two media, one offsite — and now \"one offline or immutable,\"\nbecause ransomware targets the backup server first.**\n\n**Tools**: `restic` and `borg` (⚠️ **deduplicating, encrypted, and the sane defaults for\nmost people**), `rsync` (⚠️ **`--link-dest` for hardlinked incrementals**), `zfs\nsend`\u002F`btrfs send` for filesystem-native, `tar` for archives.\n\n> **⚠️ GOTCHA — an untested backup is not a backup.** **Schedule restores.** The\n> recurring failure modes are: backups silently stopped weeks ago, the encryption key was\n> only on the machine that died, the restore takes 40 hours and RTO is 4, and the database\n> backup is a file copy of a running database and is therefore corrupt.\n> **⚠️ Databases need their own consistent dump or snapshot mechanism** — `pg_dump`,\n> `mysqldump --single-transaction`, or a filesystem snapshot with the DB quiesced.\n\n**Document and test**: RPO (how much data can you lose), RTO (how fast must you be back),\nand ⚠️ **a written recovery procedure someone who isn't you can follow at 3am.**\n",{"name":5,"description":7},"develop","ae32736a1aabddabca5d2c96d665aa0ebc7d8700","src\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Flinux-server-admin\u002Fskills\u002Fsysadmin-observability-troubleshooting-and-operations",3,"https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey","https:\u002F\u002Fthe-vibey-project.github.io\u002Fvibey\u002F","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project","https:\u002F\u002Fvibewithadam.matthewsteinberger.com","freedom for software engineers","2026-11-15 05:47:42",{"source":8,"skillName":5,"commands":35,"prompt":45,"zipAvailable":46},[36,39,42],{"tool":37,"command":38},"npx","npx skills add the-vibey-project\u002Fvibey --skill sysadmin-observability-troubleshooting-and-operations",{"tool":40,"command":41},"bunx","bunx skills add the-vibey-project\u002Fvibey --skill sysadmin-observability-troubleshooting-and-operations",{"tool":43,"command":44},"pnpm","pnpm dlx skills add the-vibey-project\u002Fvibey --skill sysadmin-observability-troubleshooting-and-operations","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Flinux-server-admin\u002Fskills\u002Fsysadmin-observability-troubleshooting-and-operations\u002FSKILL.md",true,[],[49,56,63,70,77,84],{"skillNo":50,"name":51,"nameZh":6,"description":52,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":53,"categories":54,"keywords":55,"publishedAt":19},"01M7AT1FQJAE1FQYAMDTF490Y4","white-paper-writing","Use when writing white papers, technical reports, policy documents, thought leadership pieces, or any long-form authoritative document. Triggers on requests to write, structure, outline, or review white papers, technical reports, position papers, policy briefs, or evidence-driven arguments. Also use for B2B content marketing strategy, technical documentation requiring authority, academic-style reports, or any document that must educate while persuading. Use when someone asks \"how should I structure this\" for a long-form document.","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fwriting-craft\u002Fskills\u002Fwhite-paper-writing\u002FSKILL.md",[],[],{"skillNo":57,"name":58,"nameZh":6,"description":59,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":60,"categories":61,"keywords":62,"publishedAt":19},"01M7AT16BJRJJKK2MASXRZS76G","prose-technical-mechanics","Use when editing or writing prose at a technical\u002Fmechanical level: choosing between Anglo-Saxon and Latinate diction, placing information in topic vs stress positions, writing periodic vs cumulative sentences, handling sentence rhythm, punctuation as logic, transitions, paragraph structure, or phonesthetics. Use when someone wants to understand WHY prose works or doesn't, not just surface-level style advice. Triggers on word choice, sentence structure, emphasis, rhythm, diction, or any deep writing mechanics question.","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fwriting-craft\u002Fskills\u002Fprose-technical-mechanics\u002FSKILL.md",[],[],{"skillNo":64,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":67,"categories":68,"keywords":69,"publishedAt":19},"01M7AT0W9HT0ZZ9Y287NW99MR3","prose-mechanics","Use when writing, editing, or analyzing prose style; when trying to avoid generic AI-sounding writing; when applying specific prose styles such as journalistic, academic, narrative, KJV biblical, Hemingway plain, Latinate\u002FJamesian, Orwellian, Augustan, Victorian, Early Modern (Bacon, Browne, Milton), or free indirect discourse; or when coaching someone on writing voice and mechanics. Triggers on any request to improve, analyze, or write in a specific prose style, to fix \"AI-sounding\" writing, to make prose more vivid or concrete, to vary sentence rhythm, or to understand how a particular writer achieves their effects.","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fwriting-craft\u002Fskills\u002Fprose-mechanics\u002FSKILL.md",[],[],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":19},"01M7AT0KR635JX66X2X6F3E2CP","narrative-structure","Use when structuring stories, pitches, presentations, white papers, or any persuasive narrative; analyzing the hero's journey in fiction or non-fiction; applying StoryBrand for business communication; designing narrative arcs for thought leadership; or understanding Vogler, Snyder, Propp, or Campbell frameworks. Triggers on any narrative structure, storytelling, pitch structure, StoryBrand, hero's journey, three-act structure, Freytag, Save the Cat, narrative arc, or \"how should I structure this story\u002Fpitch\u002Fpresentation\" request.","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fwriting-craft\u002Fskills\u002Fnarrative-structure\u002FSKILL.md",[],[],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":19},"01M7AT0AKGW09NKKZDYBT8VRDE","legal-brief-writing","Use when writing legal briefs, motions, or any court filing; when analyzing judicial cognition and persuasion; when applying elite brief-writing techniques from Roberts, Clement, Kagan, Garner; or when structuring legal arguments for maximum persuasion. Also use for autistic\u002Fneurodivergent advocates who need to understand how their natural communication style maps to elite legal advocacy. Triggers on any legal writing, brief drafting, motion preparation, or legal argument structuring request.","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fwriting-craft\u002Fskills\u002Flegal-brief-writing\u002FSKILL.md",[],[],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":19},"01M7AT00VWPS4Q8B5VEMXX9NQT","journal-publishability-criteria","Use whenever a research paper, article or preprint is being written, revised, reviewed or published, and specifically before every publication of docs\u002Fpaper.md. Covers what legitimate journals actually gate on, in order of gating — the editor's desk screen (scope, novelty, package problems), what reviewers judge (methodological soundness, claims the evidence can carry, statistical and qualitative reporting), EQUATOR reporting guidelines, the ICMJE authorship criteria, COPE's view of text recycling, the near-universal generative-AI policy (no AI author, disclosure, author responsibility), TOP 2025 transparency standards, how to verify a journal is legitimate, submitting without an institution, and a pre-submission checklist. Triggers on requests to evaluate, review, prepare or submit a paper, to judge publishability, to write a declarations or data-availability section, to disclose AI use, or to check a journal.","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fwriting-craft\u002Fskills\u002Fjournal-publishability-criteria\u002FSKILL.md",[],[]]