[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6X0PPYTT815EX5NTDFQDX5Y":3,"skill-same:01M6X0PPYTT815EX5NTDFQDX5Y":82},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":51,"commitSha":52,"skillDirectory":53,"forksCount":14,"repositoryHtmlUrl":54,"repositoryHomepageUrl":55,"ownerHtmlUrl":56,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":57,"install":58,"resources":71},"01M6X0PPYTT815EX5NTDFQDX5Y","linux-administration",null,"Linux sysadmin for developers — permissions, processes, systemd, journald, cron, disks. Use when managing Linux servers, writing systemd units, querying journalctl, debugging disk or process issues, or wiring up cron and timers.","projectious-work\u002Fprocesskit","projectious-work","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F263889726?v=4","01M6N97TJJ118WQ35CHRV8RD0E","projectious.work","01M6N97TPWJPF0BMKYQ4PBY5W8",0,"MIT","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Farchive\u002Fprocesskit-install-20261101\u002Fcontext\u002Ftemplates\u002Fprocesskit\u002Fv0.22.0\u002Fcontext\u002Fskills\u002Fdevops\u002Flinux-administration\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"linux","administration","linux administration","when","writing","servers","wiring","for","v1","sysadmin","units","up","devops","use","process","work","disk","issues","permissions","debugging","2026-11-10 10:12:30",false,"---\nname: linux-administration\ndescription: |\n  Linux sysadmin for developers — permissions, processes, systemd, journald, cron, disks. Use when managing Linux servers, writing systemd units, querying journalctl, debugging disk or process issues, or wiring up cron and timers.\nmetadata:\n  processkit:\n    apiVersion: processkit.projectious.work\u002Fv1\n    id: SKILL-linux-administration\n    version: \"1.1.0\"\n    created: 2026-04-06T00:00:00Z\n    category: devops\n---\n\n# Linux Administration\n\n## Intro\n\nDay-to-day Linux admin comes down to a small set of tools: `chmod`,\n`ps`\u002F`ss`, `systemctl`\u002F`journalctl`, `cron` or timers, and `df`\u002F`du`.\nGet comfortable with each and most production incidents resolve with\na handful of commands.\n\n## Overview\n\n### File permissions and ownership\n\nLinux uses the rwx model for owner, group, and others. Use numeric\n(`chmod 755`) or symbolic (`chmod u+x`) notation:\n\n```bash\nchmod 755 script.sh          # rwxr-xr-x\nchmod u+x,g+r,o-w file       # symbolic mode\nchown app:app \u002Fsrv\u002Fdata      # change owner and group\nchown -R app:app \u002Fsrv\u002Fdata   # recursive\n```\n\nKey values: 644 (files, rw-r--r--), 755 (executables\u002Fdirs), 600\n(secrets, owner-only), 700 (private dirs). Directories need execute\npermission for traversal. Special bits: setuid (4000), setgid (2000),\nsticky (1000). Inspect concisely with `stat -c '%a %U:%G %n' file`.\n\n### Process management\n\n```bash\nps aux                         # all processes\nps aux --sort=-%mem            # sort by memory\npgrep -af \"pattern\"            # find by name\nkill -TERM \u003Cpid>               # graceful termination\nkill -KILL \u003Cpid>               # force kill (last resort)\ntop -b -n 1 -o %MEM            # batch snapshot sorted by memory\n```\n\nSend `SIGTERM` (15) first; reach for `SIGKILL` (9) only when the\nprocess refuses to exit. Check open files with `lsof -p \u003Cpid>` and\nlistening ports with `ss -tlnp`.\n\n### User and group management\n\n```bash\nuseradd -m -s \u002Fbin\u002Fbash -G sudo newuser  # create user with home dir\nusermod -aG docker existinguser           # append to group (keep existing)\npasswd username                           # set password\nuserdel -r username                       # remove user and home dir\ngroups username                           # list user's groups\nid username                               # uid, gid, groups\n```\n\nPrefer `useradd` over `adduser` for scripts (consistent across\ndistros). Always use `-aG` (append) with `usermod` — without `-a` you\nreplace the user's entire group list.\n\n### Systemd services\n\nCreate unit files in `\u002Fetc\u002Fsystemd\u002Fsystem\u002F`. Always run\n`daemon-reload` after editing:\n\n```bash\nsystemctl start myapp\nsystemctl stop myapp\nsystemctl restart myapp\nsystemctl enable myapp            # start on boot\nsystemctl enable --now myapp      # enable and start immediately\nsystemctl status myapp            # status with recent logs\nsystemctl daemon-reload           # reload after editing unit files\n```\n\n### Journald\n\n```bash\njournalctl -u myapp -f                       # follow service logs\njournalctl -u myapp --since \"1 hour ago\"     # recent\njournalctl -u myapp -p err                   # errors and above\njournalctl -b -1                             # previous boot\njournalctl --disk-usage                      # journal size\njournalctl --vacuum-size=500M                # reclaim space\n```\n\n### Cron\n\nEdit with `crontab -e`. Format: `minute hour day month weekday command`.\n\n```cron\n0 * * * *    \u002Fusr\u002Flocal\u002Fbin\u002Fhourly-task.sh\n0 2 * * *    \u002Fusr\u002Flocal\u002Fbin\u002Fnightly-backup.sh\n*\u002F5 * * * *  \u002Fusr\u002Flocal\u002Fbin\u002Fhealth-check.sh\n0 0 * * 0    \u002Fusr\u002Flocal\u002Fbin\u002Fweekly-cleanup.sh\n```\n\nAlways redirect cron output: `command >> \u002Fvar\u002Flog\u002Ftask.log 2>&1`.\nConsider systemd timers as a modern alternative.\n\n### Disk and filesystem\n\n```bash\ndf -h                    # filesystem usage\ndu -sh \u002Fvar\u002Flog\u002F*        # directory sizes\nlsblk                    # block device tree\nmount | column -t        # current mounts\nfindmnt --target \u002Fdata   # find mount point for a path\n```\n\nFor disk-space emergencies: check `\u002Fvar\u002Flog`, `\u002Ftmp`, old kernels\n(`apt autoremove`), Docker (`docker system prune`), and journal logs\n(`journalctl --vacuum-size`).\n\n### Package management\n\n**Debian\u002FUbuntu (apt):**\n\n```bash\napt update && apt upgrade -y\napt install -y --no-install-recommends package\napt search keyword\napt autoremove -y\ndpkg -l | grep package\n```\n\n**RHEL\u002FFedora (dnf):**\n\n```bash\ndnf check-update\ndnf install -y package\ndnf autoremove\n```\n\n## Gotchas\n\nAgent-specific failure modes — provider-neutral pause-and-self-check items:\n\n- **`usermod -G` without `-a` replaces the entire group list.** `usermod -G docker username` removes the user from every group except `docker` — including `sudo`. Always use `usermod -aG docker username` (append). Losing `sudo` access on a remote server that requires sudo for SSH key management can lock you out.\n- **Editing unit files in `\u002Flib\u002Fsystemd\u002Fsystem\u002F` instead of `\u002Fetc\u002Fsystemd\u002Fsystem\u002F`.** Files in `\u002Flib\u002Fsystemd\u002Fsystem\u002F` are owned by the package manager and will be overwritten on the next package upgrade. Override units by copying them to `\u002Fetc\u002Fsystemd\u002Fsystem\u002F` or by creating a drop-in file at `\u002Fetc\u002Fsystemd\u002Fsystem\u002Funit.d\u002Foverride.conf`. Admin files in `\u002Fetc\u002F` always win.\n- **Forgetting `systemctl daemon-reload` after editing a unit file.** systemd caches unit file contents. Editing a `.service` file and then restarting the service with `systemctl restart` will restart the service using the old cached definition. Always run `daemon-reload` after any unit file change.\n- **`kill -9` (SIGKILL) as the first option.** SIGKILL cannot be caught or ignored by the process, so it terminates immediately without flushing buffers, releasing file locks, or running shutdown hooks. This can corrupt data or leave stale lock files. Always try `kill -TERM \u003Cpid>` (graceful) first; wait a few seconds; use SIGKILL only if the process refuses to exit.\n- **Not redirecting cron job output.** A cron job that produces output sends it to the local mail spool by default, which fills up silently on systems without a mail server configured. Add `>> \u002Fvar\u002Flog\u002Fjobname.log 2>&1` to every cron job so output is captured to a file and errors are included. Alternatively use `systemd` timers which capture output to the journal automatically.\n- **`rm -rf \u002Fvar\u002Flog\u002F*` to free disk space.** Deleting log files while the daemons that own them are running does not actually free disk space — the file descriptors remain open and the blocks are still allocated until the process closes them. The correct approach is log rotation (`logrotate`) or `journalctl --vacuum-size` for journal logs. Files that must be cleared immediately can be truncated with `> \u002Fvar\u002Flog\u002Ffile.log` while the daemon continues writing.\n- **Setting permissions with `chmod 777` to fix a permission problem.** `chmod 777` (world-readable and world-writable) is almost never the correct fix and makes files accessible to every user and every process on the system. Diagnose which user and group actually need access, set `chown` to the correct owner, and use `chmod 640` or `chmod 755` as appropriate. `777` is a security problem, not a solution.\n\n## Full reference\n\n### Systemd unit file reference\n\n```ini\n[Unit]\nDescription=My Application Service\nDocumentation=https:\u002F\u002Fexample.com\u002Fdocs\nAfter=network.target postgresql.service\nWants=postgresql.service\n\n[Service]\nType=simple\nUser=app\nGroup=app\nWorkingDirectory=\u002Fsrv\u002Fmyapp\nExecStart=\u002Fsrv\u002Fmyapp\u002Fbin\u002Fserver --config \u002Fetc\u002Fmyapp\u002Fconfig.toml\nExecReload=\u002Fbin\u002Fkill -HUP $MAINPID\nRestart=on-failure\nRestartSec=5\nTimeoutStartSec=30\nTimeoutStopSec=30\n\n# Security hardening\nNoNewPrivileges=yes\nProtectSystem=strict\nProtectHome=yes\nReadWritePaths=\u002Fsrv\u002Fmyapp\u002Fdata \u002Fvar\u002Flog\u002Fmyapp\nPrivateTmp=yes\n\n# Environment\nEnvironment=NODE_ENV=production\nEnvironmentFile=\u002Fetc\u002Fmyapp\u002Fenv\n\n[Install]\nWantedBy=multi-user.target\n```\n\n**Service types:**\n\n- `simple` (default) — ExecStart is the main process\n- `exec` — like simple, but systemd waits for the binary to execute\n- `forking` — process forks and parent exits (use `PIDFile=`)\n- `oneshot` — runs once and exits (combine with `RemainAfterExit=yes`)\n- `notify` — process sends `sd_notify` when ready\n- `idle` — delayed until other jobs finish\n\n**Restart policies:** `no`, `on-failure`, `on-abnormal`, `always`,\n`on-success`. Use `RestartSec=` for delay between restarts and\n`StartLimitIntervalSec=` \u002F `StartLimitBurst=` to prevent restart\nloops.\n\n**Dependencies:** `After=`\u002F`Before=` for ordering only, `Requires=`\nfor hard dependencies, `Wants=` for soft dependencies, `BindsTo=` to\nstop when the dependency stops.\n\n### Systemd commands\n\n```bash\nsystemctl status myapp             # status with recent logs\nsystemctl show myapp               # all properties\nsystemctl cat myapp                # show unit file content\nsystemctl list-dependencies myapp  # dependency tree\nsystemctl is-active myapp          # for scripts\nsystemctl is-enabled myapp\nsystemctl is-failed myapp\nsystemctl list-units --failed      # show failed units\nsystemctl list-timers              # show all timers\nsystemctl list-units --type=service --state=running\n```\n\n### Journalctl patterns\n\n```bash\njournalctl -u myapp --since \"2024-01-15 09:00\" --until \"2024-01-15 10:00\"\njournalctl -u myapp -p warning..err  # warnings through errors\njournalctl -u myapp -o json-pretty   # structured JSON\njournalctl -u myapp -o short-precise # precise timestamps\njournalctl -u myapp -o cat           # message only\njournalctl -b                        # current boot\njournalctl --list-boots              # all recorded boots\njournalctl -k                        # kernel messages\njournalctl --vacuum-time=30d         # keep only last 30 days\n```\n\n### Systemd timers (cron alternative)\n\n```ini\n# \u002Fetc\u002Fsystemd\u002Fsystem\u002Fbackup.timer\n[Unit]\nDescription=Run backup nightly\n\n[Timer]\nOnCalendar=*-*-* 02:00:00\nPersistent=true\nRandomizedDelaySec=300\n\n[Install]\nWantedBy=timers.target\n```\n\n```ini\n# \u002Fetc\u002Fsystemd\u002Fsystem\u002Fbackup.service\n[Unit]\nDescription=Backup job\n\n[Service]\nType=oneshot\nExecStart=\u002Fusr\u002Flocal\u002Fbin\u002Fbackup.sh\nUser=backup\n```\n\nCalendar syntax examples:\n\n```\n*-*-* 02:00:00          # daily at 2 AM\nMon *-*-* 09:00:00      # every Monday at 9 AM\n*-*-01 00:00:00         # first day of every month\n*-*-* *:00\u002F15:00        # every 15 minutes\nhourly \u002F daily \u002F weekly # shorthands\n```\n\nVerify expressions with `systemd-analyze calendar \"Mon *-*-* 09:00:00\"`.\nEnable with `systemctl enable --now backup.timer`; list with\n`systemctl list-timers`.\n\nMonotonic timers use relative intervals instead:\n\n```ini\n[Timer]\nOnBootSec=5min          # 5 minutes after boot\nOnUnitActiveSec=1h      # 1 hour after last activation\n```\n\n### Command cheatsheet by task\n\n**Files and directories:**\n\n```bash\nls -la\nls -lah --sort=size\nstat file\nfind \u002Fpath -name \"*.log\" -mtime +30 -delete\nfind \u002Fpath -type f -size +100M\nrsync -avz src\u002F dest\u002F\ntar czf archive.tar.gz dir\u002F\ntar xzf archive.tar.gz\n```\n\n**Text processing:**\n\n```bash\ngrep -rn \"pattern\" \u002Fpath\u002F\ngrep -rli \"pattern\" \u002Fpath\u002F\nsort file | uniq -c | sort -rn\ncut -d: -f1 \u002Fetc\u002Fpasswd\nawk '{print $1, $NF}' file\nsed -i 's\u002Fold\u002Fnew\u002Fg' file\ndiff -u file1 file2\n```\n\n**Network:**\n\n```bash\nss -tlnp                  # listening TCP with processes\nss -tunap                 # all TCP\u002FUDP connections\nip addr show              # interfaces and IPs\nip route show             # routing table\nnc -zv host port          # test TCP port reachability\ndig example.com           # DNS lookup\n```\n\n**Disk:**\n\n```bash\ndf -h                     # filesystem usage\ndf -ih                    # inode usage\ndu -sh \u002Fpath\u002F* | sort -rh # directories sorted by size\nlsblk -f                  # block devices with filesystems\nblkid                     # partition UUIDs and types\niostat -x 1 5             # disk I\u002FO statistics\n```\n\n**System info:**\n\n```bash\nuname -a\nhostnamectl\nuptime\nfree -h\nnproc\nlscpu\ndmesg -T | tail -50\n```\n\n### Worked examples\n\n**Diagnose high disk usage:**\n\n```bash\ndf -h\ndu -sh \u002F* 2>\u002Fdev\u002Fnull | sort -rh | head -20\ndu -sh \u002Fvar\u002Flog\u002F* | sort -rh | head -10\njournalctl --vacuum-size=200M\napt autoremove -y\n```\n\n**Create a systemd service for a Python app:**\n\n```bash\ncat > \u002Fetc\u002Fsystemd\u002Fsystem\u002Fmyapp.service \u003C\u003C 'EOF'\n[Unit]\nDescription=My Python Application\nAfter=network.target\n\n[Service]\nType=simple\nUser=app\nWorkingDirectory=\u002Fsrv\u002Fmyapp\nExecStart=\u002Fsrv\u002Fmyapp\u002F.venv\u002Fbin\u002Fpython -m myapp\nRestart=on-failure\nRestartSec=5\nEnvironment=PYTHONUNBUFFERED=1\n\n[Install]\nWantedBy=multi-user.target\nEOF\n\nsystemctl daemon-reload\nsystemctl enable --now myapp\nsystemctl status myapp\n```\n\n**Investigate a memory-hungry process:**\n\n```bash\nps aux --sort=-%mem | head -10\nPID=12345\nls -la \u002Fproc\u002F$PID\u002Ffd | wc -l           # open file descriptors\nss -tlnp | grep $PID                   # listening ports\ncat \u002Fproc\u002F$PID\u002Fstatus | grep -i vmrss  # resident memory\nkill -TERM $PID\nsleep 2 && ps -p $PID || echo \"Process terminated\"\n```\n\n### Anti-patterns\n\n- **Editing package-installed unit files in `\u002Flib\u002Fsystemd\u002Fsystem\u002F`** —\n  drop your override in `\u002Fetc\u002Fsystemd\u002Fsystem\u002F` instead (admin wins)\n- **Forgetting `daemon-reload` after unit edits** — systemd keeps\n  serving the old definition until you reload\n- **`usermod -G` without `-a`** — this replaces the user's group list\n  instead of appending\n- **Bare `kill -9`** — always try SIGTERM first so processes can\n  clean up\n- **Not redirecting cron output** — silent failures with no logs\n- **`rm -rf \u002Fvar\u002Flog\u002F*` to reclaim disk** — breaks running daemons\n  that have log files open; use `journalctl --vacuum-*` and rotate\n  logs instead\n","# Linux Administration\n\n## Intro\n\nDay-to-day Linux admin comes down to a small set of tools: `chmod`,\n`ps`\u002F`ss`, `systemctl`\u002F`journalctl`, `cron` or timers, and `df`\u002F`du`.\nGet comfortable with each and most production incidents resolve with\na handful of commands.\n\n## Overview\n\n### File permissions and ownership\n\nLinux uses the rwx model for owner, group, and others. Use numeric\n(`chmod 755`) or symbolic (`chmod u+x`) notation:\n\n```bash\nchmod 755 script.sh          # rwxr-xr-x\nchmod u+x,g+r,o-w file       # symbolic mode\nchown app:app \u002Fsrv\u002Fdata      # change owner and group\nchown -R app:app \u002Fsrv\u002Fdata   # recursive\n```\n\nKey values: 644 (files, rw-r--r--), 755 (executables\u002Fdirs), 600\n(secrets, owner-only), 700 (private dirs). Directories need execute\npermission for traversal. Special bits: setuid (4000), setgid (2000),\nsticky (1000). Inspect concisely with `stat -c '%a %U:%G %n' file`.\n\n### Process management\n\n```bash\nps aux                         # all processes\nps aux --sort=-%mem            # sort by memory\npgrep -af \"pattern\"            # find by name\nkill -TERM \u003Cpid>               # graceful termination\nkill -KILL \u003Cpid>               # force kill (last resort)\ntop -b -n 1 -o %MEM            # batch snapshot sorted by memory\n```\n\nSend `SIGTERM` (15) first; reach for `SIGKILL` (9) only when the\nprocess refuses to exit. Check open files with `lsof -p \u003Cpid>` and\nlistening ports with `ss -tlnp`.\n\n### User and group management\n\n```bash\nuseradd -m -s \u002Fbin\u002Fbash -G sudo newuser  # create user with home dir\nusermod -aG docker existinguser           # append to group (keep existing)\npasswd username                           # set password\nuserdel -r username                       # remove user and home dir\ngroups username                           # list user's groups\nid username                               # uid, gid, groups\n```\n\nPrefer `useradd` over `adduser` for scripts (consistent across\ndistros). Always use `-aG` (append) with `usermod` — without `-a` you\nreplace the user's entire group list.\n\n### Systemd services\n\nCreate unit files in `\u002Fetc\u002Fsystemd\u002Fsystem\u002F`. Always run\n`daemon-reload` after editing:\n\n```bash\nsystemctl start myapp\nsystemctl stop myapp\nsystemctl restart myapp\nsystemctl enable myapp            # start on boot\nsystemctl enable --now myapp      # enable and start immediately\nsystemctl status myapp            # status with recent logs\nsystemctl daemon-reload           # reload after editing unit files\n```\n\n### Journald\n\n```bash\njournalctl -u myapp -f                       # follow service logs\njournalctl -u myapp --since \"1 hour ago\"     # recent\njournalctl -u myapp -p err                   # errors and above\njournalctl -b -1                             # previous boot\njournalctl --disk-usage                      # journal size\njournalctl --vacuum-size=500M                # reclaim space\n```\n\n### Cron\n\nEdit with `crontab -e`. Format: `minute hour day month weekday command`.\n\n```cron\n0 * * * *    \u002Fusr\u002Flocal\u002Fbin\u002Fhourly-task.sh\n0 2 * * *    \u002Fusr\u002Flocal\u002Fbin\u002Fnightly-backup.sh\n*\u002F5 * * * *  \u002Fusr\u002Flocal\u002Fbin\u002Fhealth-check.sh\n0 0 * * 0    \u002Fusr\u002Flocal\u002Fbin\u002Fweekly-cleanup.sh\n```\n\nAlways redirect cron output: `command >> \u002Fvar\u002Flog\u002Ftask.log 2>&1`.\nConsider systemd timers as a modern alternative.\n\n### Disk and filesystem\n\n```bash\ndf -h                    # filesystem usage\ndu -sh \u002Fvar\u002Flog\u002F*        # directory sizes\nlsblk                    # block device tree\nmount | column -t        # current mounts\nfindmnt --target \u002Fdata   # find mount point for a path\n```\n\nFor disk-space emergencies: check `\u002Fvar\u002Flog`, `\u002Ftmp`, old kernels\n(`apt autoremove`), Docker (`docker system prune`), and journal logs\n(`journalctl --vacuum-size`).\n\n### Package management\n\n**Debian\u002FUbuntu (apt):**\n\n```bash\napt update && apt upgrade -y\napt install -y --no-install-recommends package\napt search keyword\napt autoremove -y\ndpkg -l | grep package\n```\n\n**RHEL\u002FFedora (dnf):**\n\n```bash\ndnf check-update\ndnf install -y package\ndnf autoremove\n```\n\n## Gotchas\n\nAgent-specific failure modes — provider-neutral pause-and-self-check items:\n\n- **`usermod -G` without `-a` replaces the entire group list.** `usermod -G docker username` removes the user from every group except `docker` — including `sudo`. Always use `usermod -aG docker username` (append). Losing `sudo` access on a remote server that requires sudo for SSH key management can lock you out.\n- **Editing unit files in `\u002Flib\u002Fsystemd\u002Fsystem\u002F` instead of `\u002Fetc\u002Fsystemd\u002Fsystem\u002F`.** Files in `\u002Flib\u002Fsystemd\u002Fsystem\u002F` are owned by the package manager and will be overwritten on the next package upgrade. Override units by copying them to `\u002Fetc\u002Fsystemd\u002Fsystem\u002F` or by creating a drop-in file at `\u002Fetc\u002Fsystemd\u002Fsystem\u002Funit.d\u002Foverride.conf`. Admin files in `\u002Fetc\u002F` always win.\n- **Forgetting `systemctl daemon-reload` after editing a unit file.** systemd caches unit file contents. Editing a `.service` file and then restarting the service with `systemctl restart` will restart the service using the old cached definition. Always run `daemon-reload` after any unit file change.\n- **`kill -9` (SIGKILL) as the first option.** SIGKILL cannot be caught or ignored by the process, so it terminates immediately without flushing buffers, releasing file locks, or running shutdown hooks. This can corrupt data or leave stale lock files. Always try `kill -TERM \u003Cpid>` (graceful) first; wait a few seconds; use SIGKILL only if the process refuses to exit.\n- **Not redirecting cron job output.** A cron job that produces output sends it to the local mail spool by default, which fills up silently on systems without a mail server configured. Add `>> \u002Fvar\u002Flog\u002Fjobname.log 2>&1` to every cron job so output is captured to a file and errors are included. Alternatively use `systemd` timers which capture output to the journal automatically.\n- **`rm -rf \u002Fvar\u002Flog\u002F*` to free disk space.** Deleting log files while the daemons that own them are running does not actually free disk space — the file descriptors remain open and the blocks are still allocated until the process closes them. The correct approach is log rotation (`logrotate`) or `journalctl --vacuum-size` for journal logs. Files that must be cleared immediately can be truncated with `> \u002Fvar\u002Flog\u002Ffile.log` while the daemon continues writing.\n- **Setting permissions with `chmod 777` to fix a permission problem.** `chmod 777` (world-readable and world-writable) is almost never the correct fix and makes files accessible to every user and every process on the system. Diagnose which user and group actually need access, set `chown` to the correct owner, and use `chmod 640` or `chmod 755` as appropriate. `777` is a security problem, not a solution.\n\n## Full reference\n\n### Systemd unit file reference\n\n```ini\n[Unit]\nDescription=My Application Service\nDocumentation=https:\u002F\u002Fexample.com\u002Fdocs\nAfter=network.target postgresql.service\nWants=postgresql.service\n\n[Service]\nType=simple\nUser=app\nGroup=app\nWorkingDirectory=\u002Fsrv\u002Fmyapp\nExecStart=\u002Fsrv\u002Fmyapp\u002Fbin\u002Fserver --config \u002Fetc\u002Fmyapp\u002Fconfig.toml\nExecReload=\u002Fbin\u002Fkill -HUP $MAINPID\nRestart=on-failure\nRestartSec=5\nTimeoutStartSec=30\nTimeoutStopSec=30\n\n# Security hardening\nNoNewPrivileges=yes\nProtectSystem=strict\nProtectHome=yes\nReadWritePaths=\u002Fsrv\u002Fmyapp\u002Fdata \u002Fvar\u002Flog\u002Fmyapp\nPrivateTmp=yes\n\n# Environment\nEnvironment=NODE_ENV=production\nEnvironmentFile=\u002Fetc\u002Fmyapp\u002Fenv\n\n[Install]\nWantedBy=multi-user.target\n```\n\n**Service types:**\n\n- `simple` (default) — ExecStart is the main process\n- `exec` — like simple, but systemd waits for the binary to execute\n- `forking` — process forks and parent exits (use `PIDFile=`)\n- `oneshot` — runs once and exits (combine with `RemainAfterExit=yes`)\n- `notify` — process sends `sd_notify` when ready\n- `idle` — delayed until other jobs finish\n\n**Restart policies:** `no`, `on-failure`, `on-abnormal`, `always`,\n`on-success`. Use `RestartSec=` for delay between restarts and\n`StartLimitIntervalSec=` \u002F `StartLimitBurst=` to prevent restart\nloops.\n\n**Dependencies:** `After=`\u002F`Before=` for ordering only, `Requires=`\nfor hard dependencies, `Wants=` for soft dependencies, `BindsTo=` to\nstop when the dependency stops.\n\n### Systemd commands\n\n```bash\nsystemctl status myapp             # status with recent logs\nsystemctl show myapp               # all properties\nsystemctl cat myapp                # show unit file content\nsystemctl list-dependencies myapp  # dependency tree\nsystemctl is-active myapp          # for scripts\nsystemctl is-enabled myapp\nsystemctl is-failed myapp\nsystemctl list-units --failed      # show failed units\nsystemctl list-timers              # show all timers\nsystemctl list-units --type=service --state=running\n```\n\n### Journalctl patterns\n\n```bash\njournalctl -u myapp --since \"2024-01-15 09:00\" --until \"2024-01-15 10:00\"\njournalctl -u myapp -p warning..err  # warnings through errors\njournalctl -u myapp -o json-pretty   # structured JSON\njournalctl -u myapp -o short-precise # precise timestamps\njournalctl -u myapp -o cat           # message only\njournalctl -b                        # current boot\njournalctl --list-boots              # all recorded boots\njournalctl -k                        # kernel messages\njournalctl --vacuum-time=30d         # keep only last 30 days\n```\n\n### Systemd timers (cron alternative)\n\n```ini\n# \u002Fetc\u002Fsystemd\u002Fsystem\u002Fbackup.timer\n[Unit]\nDescription=Run backup nightly\n\n[Timer]\nOnCalendar=*-*-* 02:00:00\nPersistent=true\nRandomizedDelaySec=300\n\n[Install]\nWantedBy=timers.target\n```\n\n```ini\n# \u002Fetc\u002Fsystemd\u002Fsystem\u002Fbackup.service\n[Unit]\nDescription=Backup job\n\n[Service]\nType=oneshot\nExecStart=\u002Fusr\u002Flocal\u002Fbin\u002Fbackup.sh\nUser=backup\n```\n\nCalendar syntax examples:\n\n```\n*-*-* 02:00:00          # daily at 2 AM\nMon *-*-* 09:00:00      # every Monday at 9 AM\n*-*-01 00:00:00         # first day of every month\n*-*-* *:00\u002F15:00        # every 15 minutes\nhourly \u002F daily \u002F weekly # shorthands\n```\n\nVerify expressions with `systemd-analyze calendar \"Mon *-*-* 09:00:00\"`.\nEnable with `systemctl enable --now backup.timer`; list with\n`systemctl list-timers`.\n\nMonotonic timers use relative intervals instead:\n\n```ini\n[Timer]\nOnBootSec=5min          # 5 minutes after boot\nOnUnitActiveSec=1h      # 1 hour after last activation\n```\n\n### Command cheatsheet by task\n\n**Files and directories:**\n\n```bash\nls -la\nls -lah --sort=size\nstat file\nfind \u002Fpath -name \"*.log\" -mtime +30 -delete\nfind \u002Fpath -type f -size +100M\nrsync -avz src\u002F dest\u002F\ntar czf archive.tar.gz dir\u002F\ntar xzf archive.tar.gz\n```\n\n**Text processing:**\n\n```bash\ngrep -rn \"pattern\" \u002Fpath\u002F\ngrep -rli \"pattern\" \u002Fpath\u002F\nsort file | uniq -c | sort -rn\ncut -d: -f1 \u002Fetc\u002Fpasswd\nawk '{print $1, $NF}' file\nsed -i 's\u002Fold\u002Fnew\u002Fg' file\ndiff -u file1 file2\n```\n\n**Network:**\n\n```bash\nss -tlnp                  # listening TCP with processes\nss -tunap                 # all TCP\u002FUDP connections\nip addr show              # interfaces and IPs\nip route show             # routing table\nnc -zv host port          # test TCP port reachability\ndig example.com           # DNS lookup\n```\n\n**Disk:**\n\n```bash\ndf -h                     # filesystem usage\ndf -ih                    # inode usage\ndu -sh \u002Fpath\u002F* | sort -rh # directories sorted by size\nlsblk -f                  # block devices with filesystems\nblkid                     # partition UUIDs and types\niostat -x 1 5             # disk I\u002FO statistics\n```\n\n**System info:**\n\n```bash\nuname -a\nhostnamectl\nuptime\nfree -h\nnproc\nlscpu\ndmesg -T | tail -50\n```\n\n### Worked examples\n\n**Diagnose high disk usage:**\n\n```bash\ndf -h\ndu -sh \u002F* 2>\u002Fdev\u002Fnull | sort -rh | head -20\ndu -sh \u002Fvar\u002Flog\u002F* | sort -rh | head -10\njournalctl --vacuum-size=200M\napt autoremove -y\n```\n\n**Create a systemd service for a Python app:**\n\n```bash\ncat > \u002Fetc\u002Fsystemd\u002Fsystem\u002Fmyapp.service \u003C\u003C 'EOF'\n[Unit]\nDescription=My Python Application\nAfter=network.target\n\n[Service]\nType=simple\nUser=app\nWorkingDirectory=\u002Fsrv\u002Fmyapp\nExecStart=\u002Fsrv\u002Fmyapp\u002F.venv\u002Fbin\u002Fpython -m myapp\nRestart=on-failure\nRestartSec=5\nEnvironment=PYTHONUNBUFFERED=1\n\n[Install]\nWantedBy=multi-user.target\nEOF\n\nsystemctl daemon-reload\nsystemctl enable --now myapp\nsystemctl status myapp\n```\n\n**Investigate a memory-hungry process:**\n\n```bash\nps aux --sort=-%mem | head -10\nPID=12345\nls -la \u002Fproc\u002F$PID\u002Ffd | wc -l           # open file descriptors\nss -tlnp | grep $PID                   # listening ports\ncat \u002Fproc\u002F$PID\u002Fstatus | grep -i vmrss  # resident memory\nkill -TERM $PID\nsleep 2 && ps -p $PID || echo \"Process terminated\"\n```\n\n### Anti-patterns\n\n- **Editing package-installed unit files in `\u002Flib\u002Fsystemd\u002Fsystem\u002F`** —\n  drop your override in `\u002Fetc\u002Fsystemd\u002Fsystem\u002F` instead (admin wins)\n- **Forgetting `daemon-reload` after unit edits** — systemd keeps\n  serving the old definition until you reload\n- **`usermod -G` without `-a`** — this replaces the user's group list\n  instead of appending\n- **Bare `kill -9`** — always try SIGTERM first so processes can\n  clean up\n- **Not redirecting cron output** — silent failures with no logs\n- **`rm -rf \u002Fvar\u002Flog\u002F*` to reclaim disk** — breaks running daemons\n  that have log files open; use `journalctl --vacuum-*` and rotate\n  logs instead\n",{"name":5,"metadata":44,"description":50},{"processkit":45},{"id":46,"created":47,"version":48,"category":31,"apiVersion":49},"SKILL-linux-administration",1775433600000,"1.1.0","processkit.projectious.work\u002Fv1","Linux sysadmin for developers — permissions, processes, systemd, journald, cron, disks. Use when managing Linux servers, writing systemd units, querying journalctl, debugging disk or process issues, or wiring up cron and timers.\n","main","4876a92620ee792aac2ac9a1a806322d8a572367","archive\u002Fprocesskit-install-20261101\u002Fcontext\u002Ftemplates\u002Fprocesskit\u002Fv0.22.0\u002Fcontext\u002Fskills\u002Fdevops\u002Flinux-administration","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit","https:\u002F\u002Fprojectious-work.github.io\u002Fprocesskit\u002F","https:\u002F\u002Fgithub.com\u002Fprojectious-work","2026-11-11 23:24:18",{"source":8,"skillName":5,"commands":59,"prompt":69,"zipAvailable":70},[60,63,66],{"tool":61,"command":62},"npx","npx skills add projectious-work\u002Fprocesskit --skill linux-administration",{"tool":64,"command":65},"bunx","bunx skills add projectious-work\u002Fprocesskit --skill linux-administration",{"tool":67,"command":68},"pnpm","pnpm dlx skills add projectious-work\u002Fprocesskit --skill linux-administration","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Farchive\u002Fprocesskit-install-20261101\u002Fcontext\u002Ftemplates\u002Fprocesskit\u002Fv0.22.0\u002Fcontext\u002Fskills\u002Fdevops\u002Flinux-administration\u002FSKILL.md",true,[72,78],{"path":73,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":76,"sourceHtmlUrl":77},"references\u002Fcommands-cheatsheet.md","REFERENCE","参考文档",5068,"https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Farchive\u002Fprocesskit-install-20261101\u002Fcontext\u002Ftemplates\u002Fprocesskit\u002Fv0.22.0\u002Fcontext\u002Fskills\u002Fdevops\u002Flinux-administration\u002Freferences\u002Fcommands-cheatsheet.md",{"path":79,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":80,"sourceHtmlUrl":81},"references\u002Fsystemd-reference.md",5372,"https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Farchive\u002Fprocesskit-install-20261101\u002Fcontext\u002Ftemplates\u002Fprocesskit\u002Fv0.22.0\u002Fcontext\u002Fskills\u002Fdevops\u002Flinux-administration\u002Freferences\u002Fsystemd-reference.md",[83,109,129,150,167,185],{"skillNo":84,"name":85,"nameZh":6,"description":86,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":87,"categories":88,"keywords":94,"publishedAt":39},"01M6X5MTC89B4ZGXW3JVSQCPF8","skill-management","Manage Skill package manifests as v1 ontology entities.","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Fsrc\u002Fcontext\u002Fskills\u002Fprocesskit\u002Fskill-management\u002FSKILL.md",[89],{"categoryNo":90,"categoryName":91,"taxonomyType":92,"taxonomyTypeDesc":93},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[95,96,97,98,99,100,34,101,102,103,85,104,105,106,27,107,108],"management","skill management","create","update","package","alpha","list","manage","ontology","get","as","manifests","skill package","transition",{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":113,"categories":114,"keywords":116,"publishedAt":39},"01M6X5MCV74NHW5MEDNHZRF2C8","schema-management","Manage generated processkit schema contracts through deterministic MCP rebuild and inspection workflows.","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Fsrc\u002Fcontext\u002Fskills\u002Fprocesskit\u002Fschema-management\u002FSKILL.md",[115],{"categoryNo":90,"categoryName":91,"taxonomyType":92,"taxonomyTypeDesc":93},[95,117,118,119,120,121,100,34,122,123,102,124,125,126,104,127,128],"schema","schema management","MCP","workflows","validation","generated","rebuild","contracts","wed","deterministic","contract","mode",{"skillNo":130,"name":131,"nameZh":6,"description":132,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":133,"categories":134,"keywords":135,"publishedAt":39},"01M6X5M5G6YJ96AMZGSCZJ1D13","proposition-management","Record claims, risks, beliefs, world facts, WSJF estimates, and assumptions through the Proposition interface.","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Fsrc\u002Fcontext\u002Fskills\u002Fprocesskit\u002Fproposition-management\u002FSKILL.md",[],[95,136,137,97,98,138,100,34,139,140,141,142,104,143,144,145,146,147,148,149],"proposition","proposition management","index","log","world","record","query","creation","event","interface","updates","facts","claims","WSJF",{"skillNo":151,"name":152,"nameZh":6,"description":153,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":154,"categories":155,"keywords":156,"publishedAt":39},"01M6X5KMQ83QN9M6MPWZM7344F","ontology-management","Manage generated v1 ontology entities that do not require a dedicated domain server.","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Fsrc\u002Fcontext\u002Fskills\u002Fprocesskit\u002Fontology-management\u002FSKILL.md",[],[95,103,157,97,100,158,159,122,160,139,102,161,162,104,163,164,144,165,27,166],"ontology management","lifecycle","archive","roadmap","do","domain","iteration","keep","measurement","generic",{"skillNo":168,"name":169,"nameZh":6,"description":170,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":171,"categories":172,"keywords":173,"publishedAt":39},"01M6X5KCY2DN3SM21RERSFNEBT","okf-compatibility","Import, export, and validate conformant Open Knowledge Format bundles.","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Fsrc\u002Fcontext\u002Fskills\u002Fprocesskit\u002Fokf-compatibility\u002FSKILL.md",[],[174,175,176,177,100,178,179,180,34,181,182,183,184],"okf","compatibility","okf compatibility","validate","open","knowledge","format","export","bundles","bundle","import",{"skillNo":186,"name":187,"nameZh":6,"description":188,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":189,"categories":190,"keywords":191,"publishedAt":39},"01M6X5JA8S5PNGJVFW4EQY8T83","capability-management","Manage versioned Capability entities and their governed lifecycle.","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Fsrc\u002Fcontext\u002Fskills\u002Fprocesskit\u002Fcapability-management\u002FSKILL.md",[],[95,192,193,97,98,138,100,158,34,101,139,194,102,141,104,164,144,145,195,108],"capability","capability management","queries","changes"]