pii-scrubbing
Systematically scrub Personally Identifiable Information (PII) from a codebase — inventory real domains, hostnames, credentials, and email addresses across all files; patch them to placeholders; create .example templates for gitignored configs; rewrite git history to remove PII from past commits; and verify the result is clean.
- License 许可
- MIT
- 最近更新
- 2026-11-14
- Stars
- ★ 4
- Forks
- 1
01 前言 Front Matter
- name
- pii-scrubbing
- tags
- [ "pii", "security", "git-filter-repo", "history-rewrite", "credentials", "secrets" ]
- version
- 1.0.0
- category
- security
- description
- Systematically scrub Personally Identifiable Information (PII) from a codebase — inventory real domains, hostnames, credentials, and email addresses across all files; patch them to placeholders; create .example templates for gitignored configs; rewrite git history to remove PII from past commits; and verify the result is clean.
03 SKILL.md 正文
PII Scrubbing — Systematic Codebase Cleanup
Trigger
Use this skill when:
- Asked to "remove PII from the repo" or "scrub real URLs/hostnames/credentials"
- Preparing a repo for open-source publication
- A security review finds real domains, passwords, or emails in code/docs
- Asked to create
.exampletemplates for config files with secrets - Asked to remove PII from git history
Workflow
Phase 1: Inventory
Search the repo systematically for PII patterns. Run ALL of these searches:
| Pattern | What it catches |
|---|---|
grep -r 'realdomain.com' . |
Real infrastructure domains |
grep -r 'customer-host.com' . |
Real server hostnames |
grep -r 'user:pass@' . |
Credentials embedded in URLs |
grep -rE '[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}' . |
Email addresses |
grep -r 'sk-[a-zA-Z0-9]{20,}' . |
API keys |
grep -rE '\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b' . |
IP addresses (check each — 127.x.x.x is safe) |
| `grep -r 'password | secret |
grep -rE ':1[0-9]{4}[0-9]*' . |
Port numbers (check if they're custom infrastructure) |
find . -name '*cred*' -o -name '*secret*' -o -name '*token*' -o -name '*.pwd' 2>/dev/null |
Credential files in working tree |
Gitignore audit for credential files: Every credential file found in the working tree must be gitignored. For each one:
git check-ignore <filename>
# Returns the path if gitignored (safe), nothing if NOT gitignored (CRITICAL)
Also verify the .gitignore patterns that catch them. Common patterns: *cred*, *secret*, *token*, *.pem, *.key. Run grep -n 'cred\|secret\|token\|pem\|key' .gitignore to confirm coverage. If a credential file isn't covered by any existing pattern, add one.
PII categories to flag (any one of these is actionable):
| Category | Examples | Treatment |
|---|---|---|
| Infrastructure domain | your-infra-domain.org, customer-app.com |
Replace with your-domain.com |
| Production hostname | mweb-stage.customer.or.kr, api.customer.org |
Replace with your-customer-host.com |
| Agent hostname | gisu-host, joseph-host |
Replace with your-agent-host |
| Real credentials | user:pass, sk-abc..., T1tus!nbox_2026 |
Replace with your-credential-placeholder |
| Real email addresses | admin@customer.org, audit@customer.org |
Replace with admin@client-domain.com |
| Real file paths with user | /home/luke, /Users/luke |
Replace with /path/to/app or $HOME |
| Local username / absolute home path | /home/luke, /home/joseph |
Grep the REAL /home/<operator>/, not a blanket /home/; replace shell→$HOME, docs→~/, tests→pathlib.Path.home() |
| Messaging / chat IDs | Telegram chat/channel ID 1234567890 |
Isolates a real person; use a TELEGRAM_CHAT_ID placeholder |
| Client project names | clientco-royalty, client-app |
Evaluate: is this identifying? If yes, use client-project |
| Company/organization names | CLIENTCO, AcmeCorp |
Replace with ExampleCorp or generic ClientName |
| Real person names in paths | joseph/, luke/ |
Rename to generic descriptor (operator/, user/). Scan directory listing, not just file contents |
| Real person names in content | Luke, Joseph (not biblical/historical figures) |
Replace with the user, the operator, or a role descriptor |
| Multiple TLD variants (same base domain) | client.org, client.or.kr, client.com |
Search for the base word across all TLDs, not just the one you found first |
| API endpoint URLs | https://works.customer.org/api/... |
Replace with https://api.client-domain.com/... |
Phase 2: Patch Working Tree
For each file with PII, determine the right treatment:
A) Scripts with hardcoded default URLs:
# Before
URL = os.environ.get("SOME_VAR", "https://realdomain.com:13004")
# After
URL = os.environ.get("SOME_VAR", "https://your-domain.com:13004")
Pattern: change the fallback default to your-domain.com. The env var override still works for actual deployments.
B) Config templates:
# Before
MOSES_INBOX_URL="https://realdomain.com:13004"
# After
MOSES_INBOX_URL="${DOMAIN_URL:-https://your-domain.com:13004}"
Or replace directly with placebo values and document that the user must set their actual values.
C) Example curl commands in docs/code:
# Before
curl https://realdomain.com:13004/api/send
# After
curl https://your-domain.com:13004/api/send
D) Credentials in example code:
# Before
curl -sk -u "user:password123"
# After
curl -sk -u "user:your-password"
E) Email addresses in skill references:
# Before
admin@client-domain.com
# After
admin@client-domain.com
Phase 3: Create .example Templates for Gitignored Files
For any config file that contains real secrets and is gitignored:
- Create
filename.examplewith placeholder values - Ensure the real file is in
.gitignore - Add
!*.exampleexception to.gitignoreif.exampleextension is generally gitignored - Commit both the
.gitignorefix and the.examplefile
Phase 3b: Template + Setup Script Pattern (for programmatic configs)
For JSON configs or structured data that need programmatic generation (not manual editing), use a three-file pattern instead of .example:
| File | Location | Purpose |
|---|---|---|
| Template | src/name.template.json (repo) |
Placeholder values like {{FLEET_DOMAIN}} |
| Setup script | ops/scripts/setup-name.sh (repo) |
Reads env vars or prompts, substitutes placeholders, writes real file outside repo |
| Real config | ~/.hermes/state/name.json (outside repo) |
Actual URLs/credentials, never committed |
| Consumer code | Python/sh scripts | Reads private path first, falls back to template if not found |
Structure in the repo:
Repo (public):
src/agent-registry.template.json ← {{PLACEHOLDER}} values only
ops/scripts/setup-agent-registry.sh ← creates real file at ~/.hermes/state/
Filesystem (gitignored / outside repo):
~/.hermes/state/agent-registry.json ← real URLs, NEVER committed
Consumer code pattern:
# 1. Check ~/.hermes/state/ first
# 2. Fall back to src/*.template.json if not found
# 3. This lets new installs run with defaults, production uses real config
When to use this pattern:
- Config file is JSON or other structured format (not plain key=value)
- Values need validation beyond "is it non-empty"
- Multiple URL/credential pairs need filling (e.g. agent registry with 5+ agents)
- Same setup process runs on multiple machines
When to use .example pattern instead:
- Simple key=value configs (
.env,.ini) - Files the user will manually edit once
- Self-explanatory values that don't need validation
Proven in Steadfaste Fleet:
models.env— env vars withFLEET_DOMAINoverridehermes-inbox.conf— created by setup withFLEET_INBOX_*varsagent-registry.json— multi-agent registry created bysetup-agent-registry.sh
Non-interactive deployment:
# Set env vars, then run the script — no prompts
FLEET_DOMAIN=myhost.com FLEET_HEALTH_PORT=13007 \
ESTHER_DOMAIN=otherhost.com ESTHER_HEALTH_PORT=13007 \
bash ops/scripts/setup-agent-registry.sh
For .env files specifically:
- Audit
docker-compose*.ymlfor all${VAR:?...}and${VAR:-...}references - Collect required and optional variables
- Create
deploy/.env.examplewith:- Comment header explaining purpose and where to put the file
- Quick-seed commands (e.g.,
openssl rand -hex 32for generating secrets) — embed these in the example so users can copy-paste them - Required vars listed first with empty values (the user MUST fill these)
- Optional vars commented out with defaults documented
- Hermes integration vars as a separate commented section if they go in a different file (
~/.hermes/.env)
- Ensure
!.env.exampleexception is in.gitignoreAFTER.env.*:.env .env.local .env.* !.env.example # ← the negation must come AFTER .env.* - See:
deploy/.env.examplein steadfaste-fleet for a worked example
For service-specific config files (e.g., ops/scripts/inbox.conf.example):
- Follow the same pattern: real values → placeholders, document what the user should replace
Phase 4: Rewrite Git History
Use git-filter-repo to scrub PII from ALL past commits:
-
Install git-filter-repo (if needed):
pip3 install git-filter-repo -
Create replacements file (
/tmp/pii-replacements.txt):literal:realdomain.org==>your-domain.com literal:realdomain.com==>your-domain.com literal:customer.host.name==>your-customer-host literal:RealPassword123!==>your-password-placeholder -
Clone to a temp directory (run filter-repo on a clone, not the working repo):
cd /tmp rm -rf repo-clean 2>/dev/null git clone /path/to/working-repo repo-clean cd repo-clean -
Run filter-repo:
git-filter-repo --replace-text /tmp/pii-replacements.txt --force -
Verify — search for every pattern:
git grep -i "realdomain\|password123\|other-pattern" . # Should return EXIT 1 (no matches) -
Handle branch protection (GitHub):
- If
mainis protected: push tomain-sanitizedfirst, then ask the user to temporarily disable branch protection - Then force-push the cleaned history:
git remote add origin https://github.com/user/repo.git git push --force --all origin
- If
Phase 4b: Cover Commit Messages (Separate Pass)
⚠️ --replace-text only covers file contents and filenames — it does NOT touch commit messages. Commit messages need a dedicated --replace-message pass.
After --replace-text completes, run a second filter-repo pass with the same replacements file:
cd /tmp/repo-clean
git filter-repo --force --refs HEAD --replace-message /tmp/pii-replacements.txt
This rewrites the history again, applying the same literal replacements to commit subjects and bodies. Without this, commit messages like fix: add Gisu to mirror chain for CLIENTCO images still leak the real project name.
When to use: Always. Unless you've verified there's no PII in any commit message (via git log --oneline --grep), assume there is.
Phase 4c: File Renaming via --filename-callback
When obfuscating a project name that appears in filenames (e.g. clientco-development-patterns.md → acme-development-patterns.md), use --filename-callback.
The callback receives bytes, not str — decode/encode accordingly:
git filter-repo --force --refs HEAD \
--replace-text /tmp/replacements.txt \
--filename-callback 'return filename.replace(b"clientco", b"acme").replace(b"CLIENTCO", b"ACME").replace(b"Clientco", b"Acme")'
After renaming, update the replacement rules file to include the case variants:
CLIENTCO==>ACME
Clientco==>Acme
clientco==>acme
Phase 4d: Case Variants
A project name can appear in three case forms:
- UPPERCASE:
CLIENTCO— in titles, headings, CISAC codes - Titlecase:
Clientco— in running text ("Clientco-royalty monorepo") - lowercase:
clientco— in URLs, paths, repo names (clientco-works,clientco-royalty)
--replace-text does case-sensitive exact match — clientco==>acme will NOT match CLIENTCO or Clientco. Add ALL variants to the replacements file:
CLIENTCO==>ACME
Clientco==>Acme
clientco==>acme
Same for --filename-callback — each variant needs its own .replace() call.
Phase 5: Verify — Check Both Content AND Commit Messages
Comprehensive check after all changes:
| Check | How | Expected |
|---|---|---|
| No PII in file contents | git grep -i "realdomain|original-pattern" . |
Exit 1 (no matches) |
| No PII in filenames | find . -iname "*oldproject*" |
Empty |
| No PII in commit messages | git log HEAD --oneline --grep="oldproject|other-pattern" |
Empty |
| No PII in past commits | git-filter-repo output shows 0 matches |
Clean run |
| Real configs gitignored | git check-ignore path/to/config.json |
Exit 0 (ignored) |
| Example templates tracked | git ls-files includes .example files |
Listed |
Verification gotcha: After cloning from a local repo that's already been rewritten, git log --all --grep= shows false positives from refs/remotes/origin/main (the stale remote tracking ref). Always verify against HEAD explicitly: git log HEAD --oneline --grep=... — not --all.
Ordering Rules in Replacements File
Place more specific patterns FIRST in the replacements file, more general patterns LAST. Filter-repo applies rules sequentially — a vague match can consume a URL before the specific domain rule fires.
Wrong ordering:
clientco==>acme # ← fires first on "clientco.or.kr" → "acme.or.kr" ❌
clientco.or.kr==>client-domain.com # ← never reached
Correct ordering:
clientco.or.kr==>client-domain.com # ← fires first, catches full domain ✅
mweb-stage.clientco.or.kr==>your-gisu-host # ← even more specific should come first
clientco==>acme # ← now safe: only matches non-domain uses
Pitfalls
-
Don't run
git-filter-repoon the working repo directly — it removes remotes and you lose the origin. Always clone to tmp first. -
Don't forget
.examplegitignore exceptions — if.gitignorehas.env.*,.env.examplewill be ignored too. Add!.env.exampleAFTER the pattern. -
Don't stop at the working tree — PII in past commits is still visible via
git blameand accessible to anyone who clones history. Always usegit-filter-repofor a full scrub. -
Don't miss the second domain variant — if the user has
realdomain.comandrealdomain.org, search for BOTH. Users often have one domain for infrastructure and another for email/SSL. -
Don't miss commit messages —
--replace-textis for file contents only. Commit messages need a separate--replace-messagepass. This is the #1 oversight — without it, the commit log still leaks. -
Don't use
--allin verification —git log --all --grep=includes stale remote tracking refs that predate the filter. Usegit log HEAD --grep=to verify the actual branch being pushed. -
Don't forget filename callbacks return bytes —
--filename-callbackin filter-repo receivesbytes, notstr. Usefilename.replace(b"old", b"new")notfilename.replace("old", "new"). -
Don't forget case variants —
clientcois three different strings (clientco,Clientco,CLIENTCO). Add all to the replacements file. -
Don't order replacements wrong — put specific full-domain patterns before generic project-name patterns. A
clientco==>acmerule that fires beforeclientco.or.kr==>client-domain.comproducesacme.or.krinstead ofclient-domain.com. -
Don't force-push protected branches — check with
git push --dry-run --force origin mainfirst, or push to an unprotected branch and ask for protection to be temporarily lifted. -
Don't propose a history rewrite the operator can't execute — before offering to scrub commit messages via force-push, verify the operator can actually run the required fleet resets (SSH access to every clone; a force-push needs
git reset --hard origin/mainon all hosts). Observed 2026-08-13: allow-list IPs leaked into 4 commit messages; the rewrite was blocked because Luke has no SSH. The correct response was saving a tracked task for when access exists (task de541a0a) plus a content guardrail, NOT pushing the rewrite and NOT leaving the messages as-is silently. -
Don't leave real passwords in example code — even if the example looks generic, a real password like
T1tus!nbox_2026is a production credential. Replace withyour-inbox-passwordor similar. -
Shell
echostatements with escaped quotes — shell scripts that print config vars viaecho " VAR=\"value\""are fragile underpatchtool's escape-drift. The\"sequence can cause the patch tool to double-escape backslashes. Fix: read the exact line withread_file, then use the literal file content (including escaped quotes) as both old and new strings, or bypasspatchand usewrite_filefor the entire script section. -
Don't stop after one TLD — if you found
client.org, also search forclient.com,client.or.kr,client.co.kr, etc. Users often have different TLDs for different services (infrastructure vs email vs SSL certs). After fixing one, search for just the base word (e.g.clientco) to catch all variants. -
Don't treat
127.0.0.1orlocalhostas PII — these are internal addresses and are safe to leave in public code. -
--refs HEADprevents re-processing backup refs — after a prior filter-repo run,refs/original/*backup refs exist. Without--refs HEAD, the new pass will process them again and potentially create duplicate processing issues. Always pass--refs HEAD. -
Clone from just-rewritten repo may fail checkout — after
git filter-reporewrites history in a local repo, cloning from it can givefatal: this operation must be run in a work tree. The clone succeeds (objects transferred) but checkout fails. Fix withgit restore --source=HEAD :/after clone. This happens because filter-repo invalidates the worktree index. If evengit restorefails, re-clone from the remote after force-push instead. -
Post-rewrite re-clone loses gitignored local files — after force-pushing rewritten history and
rm -rf/re-cloning the working repo, any gitignored files (agent-registry.json, .env, etc.) are gone. Always back up gitignored configs before destroying and re-cloning the working tree. -
Don't assume credential files are safe just because they exist in the working tree. A file like
agent-creds.mdmay be caught by a glob pattern (*cred*), but verify withgit check-ignore <file>. If it returns nothing, add a.gitignorepattern immediately. A credential file sitting un-ignored in the working tree is one accidentalgit add -Aaway from being committed. -
Don't treat the repo's own GitHub URL as PII —
github.com/user/repois the repo's public identity. Scrubbing the username from clone URLs, badge URLs, and script URLs breaks the repo for everyone. Distinguish: the repo's public address stays; personal infrastructure domains go. -
Grep the REAL operator home path, not a blanket
/home/— generic/home/user/,/home/pi/,/home/test/are intentional placeholders; only the actual/home/<operator>/leaks the username and machine home layout. Resolve it fromPath.home().nameand target exactly that; report hits per file without restating the path. But the PII WRITE GATE is stricter than this grep guidance: it refuses ANY literal/home/-prefixed path in a shared-surface write — the generic placeholder/home/user/xincluded, because the file is one deploy/copy from the public repo. When a test corpus or example command needs a NON-scratch absolute path, use/srv/or/var/lib/— never a/home/literal; rewrite the affected corpus entry rather than arguing with the gate. -
Replace a hardcoded absolute home path context-appropriately — shell script defaults →
$HOME/, docs/skills →~/, Python test fixtures →pathlib.Path.home()at runtime. A test that hardcodes the operator's absolute home path is non-hermetic AND leaks it; build it fromPath.home()so it passes on any machine and reveals nothing. -
Python
open()/Path()/file()do NOT expand~— turning/home/<operator>/...into~/.hermes/...inside a Python path call is not a redaction, it is aFileNotFoundErrorwaiting at first run, and the self-adversarial close review flags the broken snippet. When a home path sits inside a Python path call in a doc or code snippet, write it asopen(os.path.expanduser('~/.hermes/...'))/Path('~/.hermes/...').expanduser()and addimport os(or thePathimport) to that snippet. Keep~/as a portable literal only where it is not executed (shell, prose); reserve the<HOME>marker for rewritten captured-output text, never inside code that runs. -
Treat messaging/app chat IDs as PII — a Telegram chat/channel ID correlates a real person; scrub it to a
TELEGRAM_CHAT_IDplaceholder in the same pass as the domain/username. -
Keep PII gate checks for personal identifiers path-independent — a home-folder write (a skill write, a deploy copy,
git addof a local file) is one step from the public repo, so a gate that early-returns for non-shared paths lets the same identifiers through the back door. Scope generic email/phone checks to shared surfaces; scope uniquely-personal identifiers (surname, operator domain, WAN address) to EVERY write target. -
Preserve historical evidence records unless the operator says redact —
docs/evidence, test artifacts, and dated records are kept verbatim by fleet policy; scrubbing committed history is an operator decision. Scrub the active code/docs surface, then report the evidence-file scope and ask — do not silently rewrite records. -
Don't forget
.env.exampleis tracked in git — unlike.env(which is gitignored),.env.exampleis tracked and can contain real domains/URLs. Search it explicitly: it's the most common place for a real URL to lurk after you've scrubbed the main codebase. Same for any.example,.template, or.samplefile. -
macOS
grep -Psilently no-ops — BSD grep has no-P;grep -Pn 'pattern'exits 2, and a scanner that doesn't check the exit code silently reports "no matches" and skips the scan entirely (observed 2026-08-04: the secret-leak detector PII scan no-op'd on macOS for exactly this reason; fixed by converting everygrep -P/\s/(?:construct to portable ERE —grep -E/[[:space:]]). When writing or auditing any PII-scan script: usegrep -E, never-P; and if a scan step exits non-zero, treat it as a FAIL with a loud warning, never as a clean pass. -
Redact captured evidence as a LABELED redaction, never a disguised rewrite — when the operator authorizes scrubbing evidence/records (captured command output, transcripts, verification artefacts), do NOT change an absolute operator path to a bare
~/and present it as the raw capture: the self-adversarial close review flags a rewritten capture as FABRICATION, and a body edited under an unchanged sha256 is an unverified claim. Replace the operator path with an explicit, unambiguous marker (<HOME>/...), add a header note that the output was post-processed to redact home-directory paths (this is a redacted copy, not verbatim), and annotate or recompute any sha256 so it no longer appears to attest to the edited body. A redaction that looks like normal captured output is a lie; one that is visibly marked is honest. -
After any regex substitution pass, grep for leaked backreference artifacts — if a replacement string carries a double-escaped backreference (e.g.
re.sub(r"~/(...)", r"<HOME>/\\1", s)), Python emits the LITERAL token\1(the group is never substituted), so files end up with garbage like<HOME>/\1-cortex. Use a single backslash (r"...\1") for a real group reference, and afterward grep the result for any\Nresidue before committing. -
Don't blind-batch a uniform redaction + note header across a whole directory — applying one replacement (e.g. a markdown '<HOME> + redacted-copy' header prepend) to EVERY file matching a pattern, including serialized evidence (.json, .raw.json) and MEMORY snapshots, corrupts machine-readable files and adds spurious headers to dozens of unrelated records. Redact per file class; after each pass re-validate anything that must stay machine-readable (parse the JSON, bash -n the scripts, compile the .py) before you commit.
-
Never
.replace()a short substring that is the prefix of a longer real token —str.replaceand a per-linere.subdo naive substring matching:.replace("import json, subproc", "import json, subproc, os")chopsimport json, subprocess, osinto the corruptimport json, subproc, osess, os(the needlesubprocmatched the start ofsubprocess). For identifier-like text use a word-boundary regex or match the full token; after a mass-replace of identifiers/imports, grep the touched set for chopped tokens and for a doubled import (import json, os, osfrom a needle that ran twice) before committing.
References
deploy/.env.example— reference template for Langfuse docker-compose env varssrc/agent-registry.json.example— reference template for agent registry with placeholder URLsops/scripts/inbox.conf.example— reference template for inbox configskills/software-development/public-contribution/SKILL.md— PII→placeholder mapping table for contributions
04 同仓库更多 Skills
macbook-linux-maintenance
Legacy MacBook Linux fixes: GPU wedge, reboot hang, thermal.
governance-regression-triage
Triage a host's governance issues: what changed.
enforcement-immutability
Use when enforcement files lose their immutable flag.
steadfaste-fleet-repo-deploy
Use when committing to the steadfaste-fleet repo.
steadfaste-fleet-maintenance
Maintain an installed Steadfaste Fleet instance — update both the upstream Hermes Agent and the fleet repo layer, merge diverged histories, sync skills safely, and troubleshoot common issues. Covers the dual-update workflow (hermes update + fleet-update.sh), encrypted brain data backup, mycortex import, and recovery techniques.
fleet-preflight
DEPRECATED alias — merged into survey-before-action (2026-08-20). Load survey-before-action instead; its "Repo-Specific Pre-Flight" section covers git search, Hermes boundary, and deployment verification.