sysadmin-reference
Use when checking a sysadmin anti-pattern, looking up a kernel or system limit, confirming a distribution lifecycle or kernel LTS date (verified August 2026), finding the books, or needing the fifteen commands that carry most of the load, a first-five-minutes routine for an unfamiliar sick box, and a pre-change checklist. Companion to the other linux-server-admin skills.
- License 许可
- MIT
- 最近更新
- 2026-11-15
- Stars
- ★ 5
- Forks
- 3
01 前言 Front Matter
- name
- sysadmin-reference
- description
- Use when checking a sysadmin anti-pattern, looking up a kernel or system limit, confirming a distribution lifecycle or kernel LTS date (verified August 2026), finding the books, or needing the fifteen commands that carry most of the load, a first-five-minutes routine for an unfamiliar sick box, and a pre-change checklist. Companion to the other linux-server-admin skills.
03 SKILL.md 正文
Linux Server Administration: Anti-Patterns, Limits, Lifecycle Dates, and Command Reference
Part 5 of 5 of the Linux Server Administration reference (plugin
linux-server-admin), covering §15–§20. Sibling skills:sysadmin-filesystem-permissions-and-processes(§0–§3),sysadmin-systemd-storage-and-networking(§4–§6),sysadmin-packages-boot-and-hardening(§7–§9),sysadmin-observability-troubleshooting-and-operations(§10–§14). Section numbers are shared across the set; a reference written as §N →skillpoints into that sibling skill.Currency: The core — POSIX semantics, kernel subsystems, systemd, networking — is stable; distribution lifecycle dates and kernel LTS windows move. See §17 below for the dated table.
How to read this. Command-dense and operational. Assumes you can already use a shell; this is about understanding the machine well enough to fix it under pressure.
⚠️ GOTCHA boxes mark what causes outages or silent data loss.
The three mental models that make everything else make sense:
- ⚠️ Everything is a file descriptor, and the kernel is the only thing that can actually do anything. Userspace asks via syscalls. When something "doesn't work," the question is always: which syscall failed, and what errno? — and
stracewill tell you (§12 →sysadmin-observability-troubleshooting-and-operations).- ⚠️ State lives in four places and they disagree: what's on disk, what's in the kernel's memory, what the running process loaded at start, and what your config management thinks. Most confusing bugs are a disagreement between these (§11.2 →
sysadmin-observability-troubleshooting-and-operations).- ⚠️ A server is a cache of your configuration management, not a pet. If you cannot rebuild it from source control, you don't have a server — you have an artifact nobody understands (§15).
§15. Anti-Patterns
| Anti-pattern | Why |
|---|---|
| Disabling SELinux | ⚠️ Fix the label. Two commands (§9.2 → sysadmin-packages-boot-and-hardening) |
Editing vendor unit files in /usr/lib |
⚠️ Overwritten on update. Use systemctl edit (§4.1 → sysadmin-systemd-storage-and-networking) |
Forgetting daemon-reload |
Your edit isn't loaded (§4.1 → sysadmin-systemd-storage-and-networking) |
Requires= without After= |
⚠️ Starts in parallel; fails (§4.2 → sysadmin-systemd-storage-and-networking) |
/dev/sdX in fstab |
⚠️ Names aren't stable. Use UUID (§5.3 → sysadmin-systemd-storage-and-networking) |
Rebooting without mount -a |
⚠️ Stranded at emergency shell (§5.3 → sysadmin-systemd-storage-and-networking) |
kill -9 first |
Skips flush and cleanup (§3 → sysadmin-filesystem-permissions-and-processes) |
| Panicking about low "free" memory | ⚠️ Read available. Cache is doing its job (§10.2 → sysadmin-observability-troubleshooting-and-operations) |
| Enabling a firewall before allowing SSH | ⚠️ Lockout (§6.3 → sysadmin-systemd-storage-and-networking) |
| Changing firewall rules with no rollback timer | Same (§6.3 → sysadmin-systemd-storage-and-networking) |
| Password SSH auth | ⚠️ Keys only is the biggest single win (§9.4 → sysadmin-packages-boot-and-hardening) |
Reloading sshd without sshd -t |
Console trip (§9.4 → sysadmin-packages-boot-and-hardening) |
net.ipv4.tcp_tw_recycle |
⚠️ Removed from the kernel; broke NAT clients (§6.4 → sysadmin-systemd-storage-and-networking) |
Ignoring .rpmnew/.dpkg-dist |
⚠️ Running old config plus a file you never read (§7 → sysadmin-packages-boot-and-hardening) |
Using strace on a hot production process |
⚠️ ptrace stops it on every syscall (§10.4 → sysadmin-observability-troubleshooting-and-operations) |
| Treating LVM snapshots as backups | They fill and break (§5.2 → sysadmin-systemd-storage-and-networking) |
| Untested backups | ⚠️ Not backups (§14 → sysadmin-observability-troubleshooting-and-operations) |
| File-copying a running database | ⚠️ Corrupt restore (§14 → sysadmin-observability-troubleshooting-and-operations) |
--privileged containers |
Kernel is shared (§12 → sysadmin-observability-troubleshooting-and-operations) |
| Alerting on CPU% instead of symptoms | Noise, then ignored alerts (§10.5 → sysadmin-observability-troubleshooting-and-operations) |
| Manual fixes with config management running | ⚠️ Reverted in 30 minutes (§11.2 → sysadmin-observability-troubleshooting-and-operations) |
| Theorizing before asking "what changed?" | ⚠️ The highest-yield question (§11.1 → sysadmin-observability-troubleshooting-and-operations) |
| Volatile journal on a production box | ⚠️ No logs from the crash (§4.4 → sysadmin-systemd-storage-and-networking) |
| Running as root because it's easier | Capabilities and systemd sandboxing exist (§2.2 → sysadmin-filesystem-permissions-and-processes, §4.3 → sysadmin-systemd-storage-and-networking) |
ifconfig/netstat muscle memory |
Deprecated; ip/ss show more (§6.1 → sysadmin-systemd-storage-and-networking) |
§16. Numbers and Limits
SIGNALS TERM 15 · KILL 9 · HUP 1 · INT 2 · QUIT 3 · USR1 10 · USR2 12
PERMISSIONS r4 w2 x1 · setuid 4000 · setgid 2000 · sticky 1000 · umask 022
EXIT CODES 0 ok · 1 general · 2 misuse · 126 not executable · 127 not found
⚠️ 128+N = killed by signal N (137 = SIGKILL, 143 = SIGTERM)
PORTS <1024 privileged (⚠️ or CAP_NET_BIND_SERVICE)
22 SSH · 25 SMTP · 53 DNS · 80/443 HTTP(S) · 123 NTP · 3306 MySQL
5432 Postgres · 6379 Redis · 9090 Prometheus
LIMITS Default nofile often 1024 ⚠️ (far too low for servers — set 65535)
PID max default 4194304 on 64-bit
Ephemeral ports default 32768–60999
LOAD Divide by core count; ⚠️ includes D-state (I/O), not just CPU
FS ext4 max file 16 TiB · XFS ⚠️ cannot shrink · inode exhaustion is separate
TIME ⚠️ Always UTC on servers. `timedatectl set-timezone UTC`
§17. Lifecycle Dates — verified August 2026
⚠️ Lifecycle dates are the one genuinely perishable thing in this document. Verify against the vendor before planning a migration.
| Distribution | Status as of August 2026 |
|---|---|
| RHEL 10 | Released 20 May 2025. Full support to ~May 2030, maintenance to ~May 2035. Latest minor 10.2 (May 2026) |
| RHEL 9 | Full support to 31 May 2027, maintenance to 31 May 2032 |
| RHEL 8 | ⚠️ Full support ended 31 May 2024; 8.10 is the final minor, maintenance to 31 May 2029 |
| RHEL 7 | ⚠️ Maintenance ended 30 June 2024; ELS extended to 31 May 2029 |
| Ubuntu 26.04 LTS | Current LTS, released April 2026, standard support to 30 April 2031 |
| Ubuntu 24.04 LTS | Supported to 31 May 2029 |
| Ubuntu 25.10 | ⚠️ EOL 1 July 2026 — interim releases get 9 months |
⚠️ RHEL's lifecycle is unusually complex — overlapping Full Support → Maintenance Support → Extended Life → ELS phases, each meaning something different about which patches you actually receive. "Is it EOL?" is the wrong question; "which phase, and does that phase still ship security errata?" is the right one. Ubuntu LTS is 5 years standard, extendable via Pro/ESM.
⚠️ Kernel LTS windows have been volatile: historically six years, cut to a 2-year default in 2023 as maintainers cited burnout, then partially extended again in early 2026 after industry pushback. ⚠️ But upstream kernel.org lifecycle usually isn't what governs you — your distribution's backporting is. A RHEL 9 kernel gets security fixes long after the upstream branch is dead.
⚠️ One claim I encountered and am flagging rather than repeating as fact: a source asserts a Linux 7.0 release in 2026 following the 6.x series. I could not corroborate this against kernel.org and have not built anything in this document on it. Version numbering in Linux is arbitrary by policy, so it's plausible — verify before quoting.
eBPF tooling (§10.4 → sysadmin-observability-troubleshooting-and-operations) is mature and stable in interface: BCC ships 80+ ready-made
tools, bpftrace is the one-liner language, and libbpf + CO-RE is the production
path for custom programs. ⚠️ This is now baseline knowledge rather than specialist
tooling.
§18. Books
| Author | Work | Why |
|---|---|---|
| Nemeth et al. | UNIX and Linux System Administration Handbook | ⚠️ The one book. Comprehensive, opinionated, current |
| Gregg | Systems Performance | ⚠️ §10 → sysadmin-observability-troubleshooting-and-operations's method. The definitive performance text |
| Gregg | BPF Performance Tools | §10.4 → sysadmin-observability-troubleshooting-and-operations, from the person who built most of them |
| Love | Linux System Programming | What syscalls actually do |
| Bovet & Cesati | Understanding the Linux Kernel | Deeper, older, still clarifying |
| Kerrisk | The Linux Programming Interface | ⚠️ The syscall reference. Enormous and authoritative |
| Beyer et al. | Site Reliability Engineering | ⚠️ Free online. The operational philosophy |
| Limoncelli et al. | The Practice of System and Network Administration | The process side |
| Barrett et al. | SSH: The Definitive Guide | §9.4 → sysadmin-packages-boot-and-hardening in depth |
| Ward | How Linux Works | ⚠️ The best entry point if the model isn't solid yet |
Primary and practical: man pages (⚠️ man 7 capabilities, man 5 systemd.exec
and man 7 signal in particular — better than most blog posts), systemd.io,
Arch Wiki (⚠️ the best Linux documentation on the internet regardless of your
distro), Red Hat and Ubuntu documentation, brendangregg.com, ebpf.io,
endoflife.date for §17.
§19. Command Reference
19.1 The fifteen that carry most of the load
systemctl status|list-units --failed|cat|edit service truth
journalctl -u X -f | -p err -b | -b -1 log truth
ss -tulpn what's listening
ip -br a · ip r get IP network truth
lsblk -f · df -h · df -i storage truth
du -xh --max-depth=1 / where did the space go
lsof +L1 ⚠️ deleted-but-open files
free -m ⚠️ read "available"
vmstat 1 · iostat -xz 1 · mpstat -P ALL 1 saturation
dmesg -T | tail -50 ⚠️ OOM, hardware, resets
ps auxf · pstree -p process tree
strace -f -p PID / execsnoop, opensnoop ⚠️ what is it actually doing
ausearch -m avc -ts recent SELinux denials
nft list ruleset firewall truth
find /etc -name '*.rpmnew' -o -name '*.dpkg-dist' ⚠️ config drift after updates
19.2 First five minutes on an unfamiliar sick box
uptime; w load, who's on, how long up
systemctl list-units --failed ⚠️ start here
journalctl -p err -b --no-pager | tail -50
dmesg -T | tail -50 ⚠️ OOM? disk errors? link flaps?
df -h; df -i ⚠️ both
free -m available, and swap activity
ss -tulpn expected listeners present?
vmstat 1 5; iostat -xz 1 5 where's the saturation
ip -br a; ip r network sane?
timedatectl ⚠️ clock right? NTP synced?
last -n 20; journalctl -u sshd | tail who's been on
19.3 Pre-change checklist
- [ ] Do I know what this currently does, and have I captured it? (
systemctl cat, backup the file) - [ ] Is there a rollback, and can I execute it without network access? (§6.3 →
sysadmin-systemd-storage-and-networking) - [ ] Second SSH session open before touching sshd or the firewall? (§9.4 →
sysadmin-packages-boot-and-hardening) - [ ]
sshd -t/nft -c/visudo/mount -a— syntax validated? (§5.3 →sysadmin-systemd-storage-and-networking, §9.4 →sysadmin-packages-boot-and-hardening) - [ ]
daemon-reloadafter unit changes? (§4.1 →sysadmin-systemd-storage-and-networking) - [ ] Will config management revert this? (§11.2 →
sysadmin-observability-troubleshooting-and-operations) - [ ] Is this change in source control? (§13 →
sysadmin-observability-troubleshooting-and-operations) - [ ] Do I know how to tell whether it worked, and whether anything else broke?
§20. Method
§1–§16 → sysadmin-filesystem-permissions-and-processes, sysadmin-systemd-storage-and-networking, sysadmin-packages-boot-and-hardening, sysadmin-observability-troubleshooting-and-operations and §19 rest on stable material — POSIX file semantics, kernel subsystems,
systemd, TCP/IP, and standard tooling — plus the reference works in §18, chiefly
Nemeth, Gregg's Systems Performance for the observability method, and
Kerrisk for syscall behaviour. ⚠️ None of that needed web verification; the man
pages and those books are the authority and they change slowly.
Two searches were run in August 2026, confined to what genuinely perishes: distribution lifecycle dates and the eBPF tooling landscape.
Confidence. High in §1–§16 → sysadmin-filesystem-permissions-and-processes, sysadmin-systemd-storage-and-networking, sysadmin-packages-boot-and-hardening, sysadmin-observability-troubleshooting-and-operations and §19 — these are mechanisms and commands I've stated
with their failure modes, and the failure modes are the valuable part. High in §10.4 → sysadmin-observability-troubleshooting-and-operations's
eBPF characterization, which is consistent across sources and matches the primary tooling
documentation.
⚠️ Lower confidence on §17's dates specifically, and I want to be direct about why. Most sources returned for lifecycle questions are EOL-tracking aggregator sites, several of which are commercial services with an interest in urgency, and they occasionally disagree at the margins. The RHEL dates are consistent across several of them and align with Red Hat's published phase structure; the Ubuntu dates follow the documented 5-year LTS policy. But before you plan a migration on these, check Red Hat's own lifecycle page or Canonical's release-cycle page — ⚠️ vendors extend ELS windows, and RHEL 7's ELS extension to 2029 is exactly that kind of change.
⚠️ And I have explicitly declined to assert one claim: a source stated Linux 7.0 shipped in 2026. I could not corroborate it and nothing here depends on it. Version numbers in Linux are chosen arbitrarily rather than by semantic rule, so it is plausible — but an uncorroborated version claim is not something to put in a reference document unflagged.
04 同仓库更多 Skills
white-paper-writing
Use when writing white papers, technical reports, policy documents, thought leadership pieces, or any long-form authoritative document. Triggers on requests to write, structure, outline, or review white papers, technical reports, position papers, policy briefs, or evidence-driven arguments. Also use for B2B content marketing strategy, technical documentation requiring authority, academic-style reports, or any document that must educate while persuading. Use when someone asks "how should I structure this" for a long-form document.
prose-technical-mechanics
Use when editing or writing prose at a technical/mechanical level: choosing between Anglo-Saxon and Latinate diction, placing information in topic vs stress positions, writing periodic vs cumulative sentences, handling sentence rhythm, punctuation as logic, transitions, paragraph structure, or phonesthetics. Use when someone wants to understand WHY prose works or doesn't, not just surface-level style advice. Triggers on word choice, sentence structure, emphasis, rhythm, diction, or any deep writing mechanics question.
prose-mechanics
Use when writing, editing, or analyzing prose style; when trying to avoid generic AI-sounding writing; when applying specific prose styles such as journalistic, academic, narrative, KJV biblical, Hemingway plain, Latinate/Jamesian, Orwellian, Augustan, Victorian, Early Modern (Bacon, Browne, Milton), or free indirect discourse; or when coaching someone on writing voice and mechanics. Triggers on any request to improve, analyze, or write in a specific prose style, to fix "AI-sounding" writing, to make prose more vivid or concrete, to vary sentence rhythm, or to understand how a particular writer achieves their effects.
narrative-structure
Use when structuring stories, pitches, presentations, white papers, or any persuasive narrative; analyzing the hero's journey in fiction or non-fiction; applying StoryBrand for business communication; designing narrative arcs for thought leadership; or understanding Vogler, Snyder, Propp, or Campbell frameworks. Triggers on any narrative structure, storytelling, pitch structure, StoryBrand, hero's journey, three-act structure, Freytag, Save the Cat, narrative arc, or "how should I structure this story/pitch/presentation" request.
legal-brief-writing
Use when writing legal briefs, motions, or any court filing; when analyzing judicial cognition and persuasion; when applying elite brief-writing techniques from Roberts, Clement, Kagan, Garner; or when structuring legal arguments for maximum persuasion. Also use for autistic/neurodivergent advocates who need to understand how their natural communication style maps to elite legal advocacy. Triggers on any legal writing, brief drafting, motion preparation, or legal argument structuring request.
journal-publishability-criteria
Use whenever a research paper, article or preprint is being written, revised, reviewed or published, and specifically before every publication of docs/paper.md. Covers what legitimate journals actually gate on, in order of gating — the editor's desk screen (scope, novelty, package problems), what reviewers judge (methodological soundness, claims the evidence can carry, statistical and qualitative reporting), EQUATOR reporting guidelines, the ICMJE authorship criteria, COPE's view of text recycling, the near-universal generative-AI policy (no AI author, disclosure, author responsibility), TOP 2025 transparency standards, how to verify a journal is legitimate, submitting without an institution, and a pre-submission checklist. Triggers on requests to evaluate, review, prepare or submit a paper, to judge publishability, to write a declarations or data-availability section, to disclose AI use, or to check a journal.