[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6FP4D1HDFNEF08D336P7CP9":3,"skill-same:01M6FP4D1HDFNEF08D336P7CP9":78},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":38,"linkOnly":39,"markdownContent":40,"markdownBody":41,"frontMatter":42,"refName":51,"commitSha":52,"skillDirectory":53,"forksCount":54,"repositoryHtmlUrl":55,"repositoryHomepageUrl":6,"ownerHtmlUrl":56,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":57,"install":58,"resources":71},"01M6FP4D1HDFNEF08D336P7CP9","encoding-formats",null,"Encode, decode, and convert between data formats. Use when working with Base64, URL encoding, hex, Unicode, JWT tokens, hashing, checksums, or converting between serialization formats like JSON, MessagePack, and protobuf wire format.","kbarbel640-del\u002Fskills","kbarbel640-del","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F255315490?v=4","01M6CJPPQGCCESXMG3K7BBGQ5J","01M6CJPPX2PB2BCZERFVM2C2B3",1,"MIT","https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fgitgoodordietrying\u002Fencoding-formats\u002FSKILL.md",[],[18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37],"encoding","encoding formats","python3","checksums","URL","with","Unicode","tokens","hex","hashing","use","linux","format","json","convert","openssl","base64","data","win32","protobuf","2026-11-10 10:12:30",false,"---\nname: encoding-formats\ndescription: Encode, decode, and convert between data formats. Use when working with Base64, URL encoding, hex, Unicode, JWT tokens, hashing, checksums, or converting between serialization formats like JSON, MessagePack, and protobuf wire format.\nmetadata: {\"clawdbot\":{\"emoji\":\"?\",\"requires\":{\"anyBins\":[\"base64\",\"python3\",\"openssl\",\"xxd\"]},\"os\":[\"linux\",\"darwin\",\"win32\"]}}\n---\n\n# Encoding & Formats\n\nEncode, decode, and inspect data in common formats. Covers Base64, URL encoding, hex, Unicode, JWTs, hashing, checksums, and serialization formats.\n\n## When to Use\n\n- Decoding a Base64 string from an API response or config\n- URL-encoding parameters for HTTP requests\n- Inspecting hex dumps of binary data\n- Decoding JWT tokens to see claims\n- Computing or verifying file checksums\n- Converting between character encodings (UTF-8, Latin-1, etc.)\n- Understanding wire formats (protobuf, MessagePack)\n\n## Base64\n\n### Encode and decode\n\n```bash\n# Encode string\necho -n \"Hello, World!\" | base64\n# SGVsbG8sIFdvcmxkIQ==\n\n# Decode string\necho \"SGVsbG8sIFdvcmxkIQ==\" | base64 -d\n# Hello, World!\n\n# Encode a file\nbase64 image.png > image.b64\ncat file.bin | base64\n\n# Decode a file\nbase64 -d image.b64 > image.png\n\n# Base64url (URL-safe variant: + → -, \u002F → _, no padding)\necho -n \"Hello\" | base64 | tr '+\u002F' '-_' | tr -d '='\n# Base64url decode\necho \"SGVsbG8\" | tr '-_' '+\u002F' | base64 -d\n```\n\n### In code\n\n```javascript\n\u002F\u002F JavaScript (browser + Node.js 16+)\nbtoa('Hello');                    \u002F\u002F \"SGVsbG8=\"\natob('SGVsbG8=');                 \u002F\u002F \"Hello\"\n\n\u002F\u002F Node.js Buffer\nBuffer.from('Hello').toString('base64');           \u002F\u002F \"SGVsbG8=\"\nBuffer.from('SGVsbG8=', 'base64').toString();      \u002F\u002F \"Hello\"\n\n\u002F\u002F Binary data\nBuffer.from(binaryData).toString('base64');\nBuffer.from(b64String, 'base64');\n```\n\n```python\n# Python\nimport base64\n\nbase64.b64encode(b\"Hello\").decode()     # \"SGVsbG8=\"\nbase64.b64decode(\"SGVsbG8=\")            # b\"Hello\"\n\n# URL-safe Base64\nbase64.urlsafe_b64encode(b\"Hello\").decode()\nbase64.urlsafe_b64decode(\"SGVsbG8=\")\n```\n\n## URL Encoding\n\n### Encode and decode\n\n```bash\n# Python one-liner\npython3 -c \"from urllib.parse import quote; print(quote('hello world & foo=bar'))\"\n# hello%20world%20%26%20foo%3Dbar\n\n# Decode\npython3 -c \"from urllib.parse import unquote; print(unquote('hello%20world%20%26%20foo%3Dbar'))\"\n# hello world & foo=bar\n\n# curl does it automatically for --data-urlencode\ncurl -G --data-urlencode \"q=hello world & more\" https:\u002F\u002Fapi.example.com\u002Fsearch\n```\n\n### In code\n\n```javascript\n\u002F\u002F JavaScript\nencodeURIComponent('hello world & foo=bar');\n\u002F\u002F \"hello%20world%20%26%20foo%3Dbar\"\n\ndecodeURIComponent('hello%20world%20%26%20foo%3Dbar');\n\u002F\u002F \"hello world & foo=bar\"\n\n\u002F\u002F encodeURI vs encodeURIComponent:\nencodeURI('https:\u002F\u002Fexample.com\u002Fpath?q=hello world');\n\u002F\u002F \"https:\u002F\u002Fexample.com\u002Fpath?q=hello%20world\" (preserves URL structure)\nencodeURIComponent('https:\u002F\u002Fexample.com\u002Fpath?q=hello world');\n\u002F\u002F \"https%3A%2F%2Fexample.com%2Fpath%3Fq%3Dhello%20world\" (encodes everything)\n```\n\n```python\nfrom urllib.parse import quote, unquote, urlencode\n\nquote('hello world')            # 'hello%20world'\nunquote('hello%20world')        # 'hello world'\nurlencode({'q': 'hello world', 'page': 1})  # 'q=hello+world&page=1'\n```\n\n## Hex\n\n### View and convert\n\n```bash\n# File hex dump\nxxd file.bin | head -20\nxxd -l 64 file.bin          # First 64 bytes only\n\n# Hex dump (compact, no ASCII)\nxxd -p file.bin\n\n# Convert hex to binary\necho \"48656c6c6f\" | xxd -r -p\n# Hello\n\n# od (alternative)\nod -A x -t x1z file.bin | head -20\n\n# hexdump\nhexdump -C file.bin | head -20\n\n# Python\npython3 -c \"print(bytes.fromhex('48656c6c6f').decode())\"  # Hello\npython3 -c \"print('Hello'.encode().hex())\"                 # 48656c6c6f\n```\n\n### In code\n\n```javascript\n\u002F\u002F JavaScript\nBuffer.from('Hello').toString('hex');           \u002F\u002F \"48656c6c6f\"\nBuffer.from('48656c6c6f', 'hex').toString();    \u002F\u002F \"Hello\"\n\n\u002F\u002F Number to hex\n(255).toString(16);     \u002F\u002F \"ff\"\nparseInt('ff', 16);     \u002F\u002F 255\n```\n\n```python\n# Python\n\"Hello\".encode().hex()                  # '48656c6c6f'\nbytes.fromhex('48656c6c6f').decode()    # 'Hello'\nhex(255)                                # '0xff'\nint('ff', 16)                           # 255\n```\n\n## Unicode\n\n### Inspect characters\n\n```bash\n# Show Unicode code points\necho -n \"Hello 世界\" | python3 -c \"\nimport sys\nfor char in sys.stdin.read():\n    print(f'U+{ord(char):04X}  {char}  {char.encode(\\\"utf-8\\\").hex()}')\"\n# U+0048  H  48\n# U+0065  e  65\n# ...\n# U+4E16  世  e4b896\n# U+754C  界  e7958c\n\n# Convert Unicode escape to character\nprintf '\\u0048\\u0065\\u006c\\u006c\\u006f'   # Hello\necho -e '\\xE4\\xB8\\x96\\xE7\\x95\\x8C'       # 世界\n\n# File encoding detection\nfile -bi document.txt\n# text\u002Fplain; charset=utf-8\n```\n\n### Encoding conversion\n\n```bash\n# Convert between encodings\niconv -f ISO-8859-1 -t UTF-8 input.txt > output.txt\niconv -f UTF-16 -t UTF-8 input.txt > output.txt\n\n# List available encodings\niconv -l\n\n# Python\npython3 -c \"\nwith open('latin1.txt', 'r', encoding='iso-8859-1') as f:\n    content = f.read()\nwith open('utf8.txt', 'w', encoding='utf-8') as f:\n    f.write(content)\n\"\n```\n\n### Common Unicode issues\n\n```\nBOM (Byte Order Mark):\n  UTF-8 BOM: EF BB BF at start of file\n  Remove: sed -i '1s\u002F^\\xEF\\xBB\\xBF\u002F\u002F' file.txt\n\nNormalization (NFC vs NFD):\n  \"é\" can be U+00E9 (one char) or U+0065 U+0301 (e + combining accent)\n  Python: import unicodedata; unicodedata.normalize('NFC', text)\n\nMojibake (wrong encoding):\n  \"café\" appears as \"café\" → file is UTF-8 but read as Latin-1\n  Fix: re-read with correct encoding\n```\n\n## JWT (JSON Web Tokens)\n\n### Decode a JWT\n\n```bash\n# JWT has 3 parts separated by dots: header.payload.signature\n# Each part is Base64url-encoded\n\n# Decode header and payload\nTOKEN=\"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c\"\n\n# Decode header\necho \"$TOKEN\" | cut -d. -f1 | tr '-_' '+\u002F' | base64 -d 2>\u002Fdev\u002Fnull | jq\n# {\"alg\":\"HS256\",\"typ\":\"JWT\"}\n\n# Decode payload\necho \"$TOKEN\" | cut -d. -f2 | tr '-_' '+\u002F' | base64 -d 2>\u002Fdev\u002Fnull | jq\n# {\"sub\":\"1234567890\",\"name\":\"John Doe\",\"iat\":1516239022}\n\n# One-liner function\njwt_decode() {\n    echo \"$1\" | cut -d. -f2 | tr '-_' '+\u002F' | base64 -d 2>\u002Fdev\u002Fnull | jq\n}\njwt_decode \"$TOKEN\"\n```\n\n### In code\n\n```javascript\n\u002F\u002F JavaScript (no library needed for decoding)\nfunction decodeJWT(token) {\n    const [header, payload] = token.split('.').slice(0, 2)\n        .map(part => JSON.parse(atob(part.replace(\u002F-\u002Fg, '+').replace(\u002F_\u002Fg, '\u002F'))));\n    return { header, payload };\n}\n\n\u002F\u002F Check expiry\nfunction isJWTExpired(token) {\n    const { payload } = decodeJWT(token);\n    return payload.exp && payload.exp \u003C Math.floor(Date.now() \u002F 1000);\n}\n```\n\n```python\n# Python\nimport json, base64\n\ndef decode_jwt(token):\n    parts = token.split('.')\n    # Add padding\n    def pad(s): return s + '=' * (4 - len(s) % 4)\n    header = json.loads(base64.urlsafe_b64decode(pad(parts[0])))\n    payload = json.loads(base64.urlsafe_b64decode(pad(parts[1])))\n    return header, payload\n\nheader, payload = decode_jwt(token)\n```\n\n## Hashing\n\n### Common hash functions\n\n```bash\n# MD5 (not for security — only for checksums\u002Fdedup)\necho -n \"Hello\" | md5sum        # Linux\necho -n \"Hello\" | md5           # macOS\n\n# SHA-256 (standard for integrity)\necho -n \"Hello\" | sha256sum\necho -n \"Hello\" | shasum -a 256\n\n# SHA-1 (deprecated for security, still used in git)\necho -n \"Hello\" | sha1sum\n\n# SHA-512\necho -n \"Hello\" | sha512sum\n\n# Hash a file\nsha256sum file.bin\nmd5sum file.bin\n\n# openssl (works everywhere)\necho -n \"Hello\" | openssl dgst -sha256\nopenssl dgst -sha256 file.bin\n```\n\n### In code\n\n```javascript\n\u002F\u002F Node.js\nconst crypto = require('crypto');\ncrypto.createHash('sha256').update('Hello').digest('hex');\n\u002F\u002F \"185f8db32271fe25f561a6fc938b2e264306ec304eda518007d1764826381969\"\n\n\u002F\u002F File hash\nconst fs = require('fs');\nconst hash = crypto.createHash('sha256');\nhash.update(fs.readFileSync('file.bin'));\nconsole.log(hash.digest('hex'));\n```\n\n```python\nimport hashlib\nhashlib.sha256(b\"Hello\").hexdigest()\n# \"185f8db32271fe25f561a6fc938b2e264306ec304eda518007d1764826381969\"\n\n# File hash\nwith open(\"file.bin\", \"rb\") as f:\n    print(hashlib.sha256(f.read()).hexdigest())\n```\n\n### Checksums for file integrity\n\n```bash\n# Generate checksum file\nsha256sum *.tar.gz > checksums.sha256\n\n# Verify checksums\nsha256sum -c checksums.sha256\n\n# Compare two files without reading content\nsha256sum file1.bin file2.bin\n# or\ncmp file1.bin file2.bin && echo \"Identical\" || echo \"Different\"\n```\n\n## Serialization Formats\n\n### JSON ↔ other formats\n\n```bash\n# JSON to YAML\npython3 -c \"import json, yaml, sys; yaml.dump(json.load(sys.stdin), sys.stdout)\" \u003C data.json\n\n# YAML to JSON\npython3 -c \"import json, yaml, sys; json.dump(yaml.safe_load(sys.stdin), sys.stdout, indent=2)\" \u003C data.yaml\n\n# JSON to CSV\njq -r '.[] | [.id, .name, .email] | @csv' data.json > data.csv\n\n# CSV to JSON\npython3 -c \"\nimport csv, json, sys\nreader = csv.DictReader(open(sys.argv[1]))\nprint(json.dumps(list(reader), indent=2))\n\" data.csv\n\n# JSON to TOML\npython3 -c \"import json, tomli_w, sys; tomli_w.dump(json.load(sys.stdin), sys.stdout.buffer)\" \u003C data.json\n\n# Pretty-print JSON\njq '.' data.json\npython3 -m json.tool data.json\n```\n\n### Binary formats (inspection)\n\n```bash\n# MessagePack → JSON\npython3 -c \"\nimport msgpack, json, sys\ndata = msgpack.unpackb(sys.stdin.buffer.read(), raw=False)\nprint(json.dumps(data, indent=2))\n\" \u003C data.msgpack\n\n# Protobuf (decode without schema — shows field numbers)\nprotoc --decode_raw \u003C data.pb\n\n# CBOR → JSON\npython3 -c \"\nimport cbor2, json, sys\ndata = cbor2.loads(sys.stdin.buffer.read())\nprint(json.dumps(data, indent=2, default=str))\n\" \u003C data.cbor\n```\n\n## Quick Decode Script\n\n```bash\n#!\u002Fbin\u002Fbash\n# decode.sh — Auto-detect and decode common encoded strings\nINPUT=\"${1:-$(cat)}\"\n\n# Try Base64\nB64_DECODED=$(echo \"$INPUT\" | base64 -d 2>\u002Fdev\u002Fnull)\nif [[ $? -eq 0 && -n \"$B64_DECODED\" ]]; then\n    echo \"Base64 → $B64_DECODED\"\nfi\n\n# Try URL encoding\nif echo \"$INPUT\" | grep -q '%[0-9A-Fa-f]\\{2\\}'; then\n    URL_DECODED=$(python3 -c \"from urllib.parse import unquote; print(unquote('$INPUT'))\" 2>\u002Fdev\u002Fnull)\n    echo \"URL   → $URL_DECODED\"\nfi\n\n# Try JWT\nif echo \"$INPUT\" | grep -qP '^eyJ[A-Za-z0-9_-]+\\.eyJ[A-Za-z0-9_-]+\\.'; then\n    echo \"JWT header:\"\n    echo \"$INPUT\" | cut -d. -f1 | tr '-_' '+\u002F' | base64 -d 2>\u002Fdev\u002Fnull | jq\n    echo \"JWT payload:\"\n    echo \"$INPUT\" | cut -d. -f2 | tr '-_' '+\u002F' | base64 -d 2>\u002Fdev\u002Fnull | jq\nfi\n\n# Try hex\nif echo \"$INPUT\" | grep -qP '^[0-9a-fA-F]+$' && [[ $((${#INPUT} % 2)) -eq 0 ]]; then\n    HEX_DECODED=$(echo \"$INPUT\" | xxd -r -p 2>\u002Fdev\u002Fnull)\n    if [[ -n \"$HEX_DECODED\" ]]; then\n        echo \"Hex   → $HEX_DECODED\"\n    fi\nfi\n```\n\n## Tips\n\n- Base64 increases data size by ~33%. Use it for embedding binary data in text formats (JSON, XML, email), not for compression or encryption.\n- Base64url (RFC 4648) uses `-` and `_` instead of `+` and `\u002F`, and omits padding `=`. JWTs and URL parameters use this variant.\n- SHA-256 is the standard for integrity checks. MD5 is fine for dedup and non-security checksums but broken for cryptographic use.\n- JWTs are signed, not encrypted. Anyone can decode the header and payload. Only the signature verifies authenticity. Never put secrets in JWT claims.\n- When files display garbled text (mojibake), the problem is almost always wrong encoding assumption. Check with `file -bi` and re-read with the correct encoding.\n- `xxd -p` (plain hex) and `xxd -r -p` (reverse) are the fastest way to convert between binary and hex on the command line.\n- URL-encode with `encodeURIComponent` (JavaScript) or `urllib.parse.quote` (Python), not by hand. Manual encoding misses edge cases.\n","# Encoding & Formats\n\nEncode, decode, and inspect data in common formats. Covers Base64, URL encoding, hex, Unicode, JWTs, hashing, checksums, and serialization formats.\n\n## When to Use\n\n- Decoding a Base64 string from an API response or config\n- URL-encoding parameters for HTTP requests\n- Inspecting hex dumps of binary data\n- Decoding JWT tokens to see claims\n- Computing or verifying file checksums\n- Converting between character encodings (UTF-8, Latin-1, etc.)\n- Understanding wire formats (protobuf, MessagePack)\n\n## Base64\n\n### Encode and decode\n\n```bash\n# Encode string\necho -n \"Hello, World!\" | base64\n# SGVsbG8sIFdvcmxkIQ==\n\n# Decode string\necho \"SGVsbG8sIFdvcmxkIQ==\" | base64 -d\n# Hello, World!\n\n# Encode a file\nbase64 image.png > image.b64\ncat file.bin | base64\n\n# Decode a file\nbase64 -d image.b64 > image.png\n\n# Base64url (URL-safe variant: + → -, \u002F → _, no padding)\necho -n \"Hello\" | base64 | tr '+\u002F' '-_' | tr -d '='\n# Base64url decode\necho \"SGVsbG8\" | tr '-_' '+\u002F' | base64 -d\n```\n\n### In code\n\n```javascript\n\u002F\u002F JavaScript (browser + Node.js 16+)\nbtoa('Hello');                    \u002F\u002F \"SGVsbG8=\"\natob('SGVsbG8=');                 \u002F\u002F \"Hello\"\n\n\u002F\u002F Node.js Buffer\nBuffer.from('Hello').toString('base64');           \u002F\u002F \"SGVsbG8=\"\nBuffer.from('SGVsbG8=', 'base64').toString();      \u002F\u002F \"Hello\"\n\n\u002F\u002F Binary data\nBuffer.from(binaryData).toString('base64');\nBuffer.from(b64String, 'base64');\n```\n\n```python\n# Python\nimport base64\n\nbase64.b64encode(b\"Hello\").decode()     # \"SGVsbG8=\"\nbase64.b64decode(\"SGVsbG8=\")            # b\"Hello\"\n\n# URL-safe Base64\nbase64.urlsafe_b64encode(b\"Hello\").decode()\nbase64.urlsafe_b64decode(\"SGVsbG8=\")\n```\n\n## URL Encoding\n\n### Encode and decode\n\n```bash\n# Python one-liner\npython3 -c \"from urllib.parse import quote; print(quote('hello world & foo=bar'))\"\n# hello%20world%20%26%20foo%3Dbar\n\n# Decode\npython3 -c \"from urllib.parse import unquote; print(unquote('hello%20world%20%26%20foo%3Dbar'))\"\n# hello world & foo=bar\n\n# curl does it automatically for --data-urlencode\ncurl -G --data-urlencode \"q=hello world & more\" https:\u002F\u002Fapi.example.com\u002Fsearch\n```\n\n### In code\n\n```javascript\n\u002F\u002F JavaScript\nencodeURIComponent('hello world & foo=bar');\n\u002F\u002F \"hello%20world%20%26%20foo%3Dbar\"\n\ndecodeURIComponent('hello%20world%20%26%20foo%3Dbar');\n\u002F\u002F \"hello world & foo=bar\"\n\n\u002F\u002F encodeURI vs encodeURIComponent:\nencodeURI('https:\u002F\u002Fexample.com\u002Fpath?q=hello world');\n\u002F\u002F \"https:\u002F\u002Fexample.com\u002Fpath?q=hello%20world\" (preserves URL structure)\nencodeURIComponent('https:\u002F\u002Fexample.com\u002Fpath?q=hello world');\n\u002F\u002F \"https%3A%2F%2Fexample.com%2Fpath%3Fq%3Dhello%20world\" (encodes everything)\n```\n\n```python\nfrom urllib.parse import quote, unquote, urlencode\n\nquote('hello world')            # 'hello%20world'\nunquote('hello%20world')        # 'hello world'\nurlencode({'q': 'hello world', 'page': 1})  # 'q=hello+world&page=1'\n```\n\n## Hex\n\n### View and convert\n\n```bash\n# File hex dump\nxxd file.bin | head -20\nxxd -l 64 file.bin          # First 64 bytes only\n\n# Hex dump (compact, no ASCII)\nxxd -p file.bin\n\n# Convert hex to binary\necho \"48656c6c6f\" | xxd -r -p\n# Hello\n\n# od (alternative)\nod -A x -t x1z file.bin | head -20\n\n# hexdump\nhexdump -C file.bin | head -20\n\n# Python\npython3 -c \"print(bytes.fromhex('48656c6c6f').decode())\"  # Hello\npython3 -c \"print('Hello'.encode().hex())\"                 # 48656c6c6f\n```\n\n### In code\n\n```javascript\n\u002F\u002F JavaScript\nBuffer.from('Hello').toString('hex');           \u002F\u002F \"48656c6c6f\"\nBuffer.from('48656c6c6f', 'hex').toString();    \u002F\u002F \"Hello\"\n\n\u002F\u002F Number to hex\n(255).toString(16);     \u002F\u002F \"ff\"\nparseInt('ff', 16);     \u002F\u002F 255\n```\n\n```python\n# Python\n\"Hello\".encode().hex()                  # '48656c6c6f'\nbytes.fromhex('48656c6c6f').decode()    # 'Hello'\nhex(255)                                # '0xff'\nint('ff', 16)                           # 255\n```\n\n## Unicode\n\n### Inspect characters\n\n```bash\n# Show Unicode code points\necho -n \"Hello 世界\" | python3 -c \"\nimport sys\nfor char in sys.stdin.read():\n    print(f'U+{ord(char):04X}  {char}  {char.encode(\\\"utf-8\\\").hex()}')\"\n# U+0048  H  48\n# U+0065  e  65\n# ...\n# U+4E16  世  e4b896\n# U+754C  界  e7958c\n\n# Convert Unicode escape to character\nprintf '\\u0048\\u0065\\u006c\\u006c\\u006f'   # Hello\necho -e '\\xE4\\xB8\\x96\\xE7\\x95\\x8C'       # 世界\n\n# File encoding detection\nfile -bi document.txt\n# text\u002Fplain; charset=utf-8\n```\n\n### Encoding conversion\n\n```bash\n# Convert between encodings\niconv -f ISO-8859-1 -t UTF-8 input.txt > output.txt\niconv -f UTF-16 -t UTF-8 input.txt > output.txt\n\n# List available encodings\niconv -l\n\n# Python\npython3 -c \"\nwith open('latin1.txt', 'r', encoding='iso-8859-1') as f:\n    content = f.read()\nwith open('utf8.txt', 'w', encoding='utf-8') as f:\n    f.write(content)\n\"\n```\n\n### Common Unicode issues\n\n```\nBOM (Byte Order Mark):\n  UTF-8 BOM: EF BB BF at start of file\n  Remove: sed -i '1s\u002F^\\xEF\\xBB\\xBF\u002F\u002F' file.txt\n\nNormalization (NFC vs NFD):\n  \"é\" can be U+00E9 (one char) or U+0065 U+0301 (e + combining accent)\n  Python: import unicodedata; unicodedata.normalize('NFC', text)\n\nMojibake (wrong encoding):\n  \"café\" appears as \"café\" → file is UTF-8 but read as Latin-1\n  Fix: re-read with correct encoding\n```\n\n## JWT (JSON Web Tokens)\n\n### Decode a JWT\n\n```bash\n# JWT has 3 parts separated by dots: header.payload.signature\n# Each part is Base64url-encoded\n\n# Decode header and payload\nTOKEN=\"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c\"\n\n# Decode header\necho \"$TOKEN\" | cut -d. -f1 | tr '-_' '+\u002F' | base64 -d 2>\u002Fdev\u002Fnull | jq\n# {\"alg\":\"HS256\",\"typ\":\"JWT\"}\n\n# Decode payload\necho \"$TOKEN\" | cut -d. -f2 | tr '-_' '+\u002F' | base64 -d 2>\u002Fdev\u002Fnull | jq\n# {\"sub\":\"1234567890\",\"name\":\"John Doe\",\"iat\":1516239022}\n\n# One-liner function\njwt_decode() {\n    echo \"$1\" | cut -d. -f2 | tr '-_' '+\u002F' | base64 -d 2>\u002Fdev\u002Fnull | jq\n}\njwt_decode \"$TOKEN\"\n```\n\n### In code\n\n```javascript\n\u002F\u002F JavaScript (no library needed for decoding)\nfunction decodeJWT(token) {\n    const [header, payload] = token.split('.').slice(0, 2)\n        .map(part => JSON.parse(atob(part.replace(\u002F-\u002Fg, '+').replace(\u002F_\u002Fg, '\u002F'))));\n    return { header, payload };\n}\n\n\u002F\u002F Check expiry\nfunction isJWTExpired(token) {\n    const { payload } = decodeJWT(token);\n    return payload.exp && payload.exp \u003C Math.floor(Date.now() \u002F 1000);\n}\n```\n\n```python\n# Python\nimport json, base64\n\ndef decode_jwt(token):\n    parts = token.split('.')\n    # Add padding\n    def pad(s): return s + '=' * (4 - len(s) % 4)\n    header = json.loads(base64.urlsafe_b64decode(pad(parts[0])))\n    payload = json.loads(base64.urlsafe_b64decode(pad(parts[1])))\n    return header, payload\n\nheader, payload = decode_jwt(token)\n```\n\n## Hashing\n\n### Common hash functions\n\n```bash\n# MD5 (not for security — only for checksums\u002Fdedup)\necho -n \"Hello\" | md5sum        # Linux\necho -n \"Hello\" | md5           # macOS\n\n# SHA-256 (standard for integrity)\necho -n \"Hello\" | sha256sum\necho -n \"Hello\" | shasum -a 256\n\n# SHA-1 (deprecated for security, still used in git)\necho -n \"Hello\" | sha1sum\n\n# SHA-512\necho -n \"Hello\" | sha512sum\n\n# Hash a file\nsha256sum file.bin\nmd5sum file.bin\n\n# openssl (works everywhere)\necho -n \"Hello\" | openssl dgst -sha256\nopenssl dgst -sha256 file.bin\n```\n\n### In code\n\n```javascript\n\u002F\u002F Node.js\nconst crypto = require('crypto');\ncrypto.createHash('sha256').update('Hello').digest('hex');\n\u002F\u002F \"185f8db32271fe25f561a6fc938b2e264306ec304eda518007d1764826381969\"\n\n\u002F\u002F File hash\nconst fs = require('fs');\nconst hash = crypto.createHash('sha256');\nhash.update(fs.readFileSync('file.bin'));\nconsole.log(hash.digest('hex'));\n```\n\n```python\nimport hashlib\nhashlib.sha256(b\"Hello\").hexdigest()\n# \"185f8db32271fe25f561a6fc938b2e264306ec304eda518007d1764826381969\"\n\n# File hash\nwith open(\"file.bin\", \"rb\") as f:\n    print(hashlib.sha256(f.read()).hexdigest())\n```\n\n### Checksums for file integrity\n\n```bash\n# Generate checksum file\nsha256sum *.tar.gz > checksums.sha256\n\n# Verify checksums\nsha256sum -c checksums.sha256\n\n# Compare two files without reading content\nsha256sum file1.bin file2.bin\n# or\ncmp file1.bin file2.bin && echo \"Identical\" || echo \"Different\"\n```\n\n## Serialization Formats\n\n### JSON ↔ other formats\n\n```bash\n# JSON to YAML\npython3 -c \"import json, yaml, sys; yaml.dump(json.load(sys.stdin), sys.stdout)\" \u003C data.json\n\n# YAML to JSON\npython3 -c \"import json, yaml, sys; json.dump(yaml.safe_load(sys.stdin), sys.stdout, indent=2)\" \u003C data.yaml\n\n# JSON to CSV\njq -r '.[] | [.id, .name, .email] | @csv' data.json > data.csv\n\n# CSV to JSON\npython3 -c \"\nimport csv, json, sys\nreader = csv.DictReader(open(sys.argv[1]))\nprint(json.dumps(list(reader), indent=2))\n\" data.csv\n\n# JSON to TOML\npython3 -c \"import json, tomli_w, sys; tomli_w.dump(json.load(sys.stdin), sys.stdout.buffer)\" \u003C data.json\n\n# Pretty-print JSON\njq '.' data.json\npython3 -m json.tool data.json\n```\n\n### Binary formats (inspection)\n\n```bash\n# MessagePack → JSON\npython3 -c \"\nimport msgpack, json, sys\ndata = msgpack.unpackb(sys.stdin.buffer.read(), raw=False)\nprint(json.dumps(data, indent=2))\n\" \u003C data.msgpack\n\n# Protobuf (decode without schema — shows field numbers)\nprotoc --decode_raw \u003C data.pb\n\n# CBOR → JSON\npython3 -c \"\nimport cbor2, json, sys\ndata = cbor2.loads(sys.stdin.buffer.read())\nprint(json.dumps(data, indent=2, default=str))\n\" \u003C data.cbor\n```\n\n## Quick Decode Script\n\n```bash\n#!\u002Fbin\u002Fbash\n# decode.sh — Auto-detect and decode common encoded strings\nINPUT=\"${1:-$(cat)}\"\n\n# Try Base64\nB64_DECODED=$(echo \"$INPUT\" | base64 -d 2>\u002Fdev\u002Fnull)\nif [[ $? -eq 0 && -n \"$B64_DECODED\" ]]; then\n    echo \"Base64 → $B64_DECODED\"\nfi\n\n# Try URL encoding\nif echo \"$INPUT\" | grep -q '%[0-9A-Fa-f]\\{2\\}'; then\n    URL_DECODED=$(python3 -c \"from urllib.parse import unquote; print(unquote('$INPUT'))\" 2>\u002Fdev\u002Fnull)\n    echo \"URL   → $URL_DECODED\"\nfi\n\n# Try JWT\nif echo \"$INPUT\" | grep -qP '^eyJ[A-Za-z0-9_-]+\\.eyJ[A-Za-z0-9_-]+\\.'; then\n    echo \"JWT header:\"\n    echo \"$INPUT\" | cut -d. -f1 | tr '-_' '+\u002F' | base64 -d 2>\u002Fdev\u002Fnull | jq\n    echo \"JWT payload:\"\n    echo \"$INPUT\" | cut -d. -f2 | tr '-_' '+\u002F' | base64 -d 2>\u002Fdev\u002Fnull | jq\nfi\n\n# Try hex\nif echo \"$INPUT\" | grep -qP '^[0-9a-fA-F]+$' && [[ $((${#INPUT} % 2)) -eq 0 ]]; then\n    HEX_DECODED=$(echo \"$INPUT\" | xxd -r -p 2>\u002Fdev\u002Fnull)\n    if [[ -n \"$HEX_DECODED\" ]]; then\n        echo \"Hex   → $HEX_DECODED\"\n    fi\nfi\n```\n\n## Tips\n\n- Base64 increases data size by ~33%. Use it for embedding binary data in text formats (JSON, XML, email), not for compression or encryption.\n- Base64url (RFC 4648) uses `-` and `_` instead of `+` and `\u002F`, and omits padding `=`. JWTs and URL parameters use this variant.\n- SHA-256 is the standard for integrity checks. MD5 is fine for dedup and non-security checksums but broken for cryptographic use.\n- JWTs are signed, not encrypted. Anyone can decode the header and payload. Only the signature verifies authenticity. Never put secrets in JWT claims.\n- When files display garbled text (mojibake), the problem is almost always wrong encoding assumption. Check with `file -bi` and re-read with the correct encoding.\n- `xxd -p` (plain hex) and `xxd -r -p` (reverse) are the fastest way to convert between binary and hex on the command line.\n- URL-encode with `encodeURIComponent` (JavaScript) or `urllib.parse.quote` (Python), not by hand. Manual encoding misses edge cases.\n",{"name":5,"metadata":43,"description":7},{"clawdbot":44},{"os":45,"emoji":47,"requires":48},[29,46,36],"darwin","?",{"anyBins":49},[34,20,33,50],"xxd","main","e7e336ac628c3e3bd771aff1aaa469c93778f94a","skills\u002Fgitgoodordietrying\u002Fencoding-formats",0,"https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills","https:\u002F\u002Fgithub.com\u002Fkbarbel640-del","2026-08-23 22:39:37",{"source":8,"skillName":5,"commands":59,"prompt":69,"zipAvailable":70},[60,63,66],{"tool":61,"command":62},"npx","npx skills add kbarbel640-del\u002Fskills --skill encoding-formats",{"tool":64,"command":65},"bunx","bunx skills add kbarbel640-del\u002Fskills --skill encoding-formats",{"tool":67,"command":68},"pnpm","pnpm dlx skills add kbarbel640-del\u002Fskills --skill encoding-formats","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fgitgoodordietrying\u002Fencoding-formats\u002FSKILL.md",true,[72],{"path":73,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":76,"sourceHtmlUrl":77},"_meta.json","REFERENCE","参考文档",298,"https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fgitgoodordietrying\u002Fencoding-formats\u002F_meta.json",[79,108,136,157,163,181],{"skillNo":80,"name":81,"nameZh":6,"description":82,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":90,"publishedAt":38},"01M6G8J55967D6281ACMKAKTEQ","crowd-prompting","A marketplace where AI agents improve prompts, system instructions, tool descriptions, and other text-based content with domain expertise from real-world operations — and earn tokens for valuable contributions.","https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fzzadrian\u002Fcrowd-prompting\u002FSKILL.md",[85],{"categoryNo":86,"categoryName":87,"taxonomyType":88,"taxonomyTypeDesc":89},"01JBSEEDCAT000000000000036","内容创作者","PROFESSION","适用职业",[91,92,93,23,25,94,95,96,97,98,99,100,101,102,103,104,105,106,107],"crowd prompting","prompting","api","domain","world","prompts","operations","instructions","text","tool","system","improve","productivity","marketplace","https","content","ai agents",{"skillNo":109,"name":110,"nameZh":6,"description":111,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":112,"categories":113,"keywords":119,"publishedAt":38},"01M6G8J0PKPAYVET6BWE8W4C26","dilemm-ai","Compete in DilemmAI, the prisoner's dilemma AI arena at dilemm.ai. Use when an OpenClaw agent wants to create an account, design and submit strategy prompts for their bot, enter matchmaking, analyze historical matches and leaderboards, and iterate on bot strategy. Requires headless browser automation (no API available yet).","https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fzyori\u002Fdilemm-ai\u002FSKILL.md",[114],{"categoryNo":115,"categoryName":116,"taxonomyType":117,"taxonomyTypeDesc":118},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[120,93,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,96,28],"dilemm ai","arena","analyze","browser","openclaw agent","create","automation","browser automation","bot","openclaw","design","strategy","headless","compete","submit","matchmaking",{"skillNo":137,"name":138,"nameZh":6,"description":139,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":140,"categories":141,"keywords":145,"publishedAt":38},"01M6G8HWAF3K9SW7NP3Z83CPJ6","arxiv-paper-reviews","Interact with arXiv Crawler API to fetch papers, read reviews, and submit comments. Use when working with arXiv papers, fetching paper lists by date\u002Fcategory\u002Finterest, viewing paper details with comments, or submitting paper reviews via API at http:\u002F\u002F150.158.152.82:8000.","https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fzxrys\u002Fweak-accept\u002FSKILL.md",[142],{"categoryNo":143,"categoryName":144,"taxonomyType":117,"taxonomyTypeDesc":118},"01JBSEEDCAT000000000000021","资料检索与研究",[146,147,148,149,150,151,152,23,153,134,28,154,93,155,156],"arxiv paper reviews","reviews","arxiv","paper","crawler","papers","comments","fetch","read","http","via",{"skillNo":158,"name":138,"nameZh":6,"description":139,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":159,"categories":160,"keywords":162,"publishedAt":38},"01M6G8HQJ7N7WMY6YV7D7SSJ16","https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fzxrys\u002Farxiv-paper-reviews\u002FSKILL.md",[161],{"categoryNo":143,"categoryName":144,"taxonomyType":117,"taxonomyTypeDesc":118},[149,148,147,146,156,151,152,23,150,134,28,153,154,93,155],{"skillNo":164,"name":165,"nameZh":6,"description":166,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":167,"categories":168,"keywords":169,"publishedAt":38},"01M6G8HM95Y64JNVFC2HN4E2BG","git-summary","Get a quick summary of the current Git repository including status, recent commits, branches, and contributors.","https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fzweack\u002Fgit-summary\u002FSKILL.md",[],[170,171,172,173,174,36,175,29,176,177,178,179,180],"git","summary","git summary","commits","repository","status","contributors","true","quick","get","branches",{"skillNo":182,"name":183,"nameZh":6,"description":184,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":185,"categories":186,"keywords":191,"publishedAt":38},"01M6G8HGEGF05CNEJCGVYFSC4R","agent-constitution","Interact with AgentConstitution governance contracts on Base Sepolia. Check compliance, read rules, log actions, query governance state.","https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fztsalexey\u002Fagent-constitution\u002FSKILL.md",[187,188],{"categoryNo":115,"categoryName":116,"taxonomyType":117,"taxonomyTypeDesc":118},{"categoryNo":189,"categoryName":190,"taxonomyType":88,"taxonomyTypeDesc":89},"01JBSEEDCAT000000000000037","律师与法务",[192,193,194,195,23,196,197,51,198,154,199,200,201,202,203,105,204,205,206],"agent constitution","constitution","contracts","tree","query","base","log","check","Rules","projects","compliance","actions","state","governance","github"]