[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6D2727EXMBD1DTZ3ADCT0QT":3,"skill-same:01M6D2727EXMBD1DTZ3ADCT0QT":62},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":34,"linkOnly":35,"markdownContent":36,"markdownBody":37,"frontMatter":38,"refName":39,"commitSha":40,"skillDirectory":41,"forksCount":42,"repositoryHtmlUrl":43,"repositoryHomepageUrl":44,"ownerHtmlUrl":45,"ownerBlogUrl":44,"ownerDescription":46,"updatedAt":47,"install":48,"resources":61},"01M6D2727EXMBD1DTZ3ADCT0QT","systems-formats",null,"Binary formats and serialisation: choosing a format, endianness, alignment and padding, versioning with backward and forward compatibility, parsing untrusted input with bounds and limits, zero-copy parsing, checksums, compression, magic numbers and headers, fuzzing parsers, and writing the spec down. Read before designing, parsing or changing a file format or a wire protocol.","enowdev\u002Fenowxcli","enowdev","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F179420349?v=4","01M6D1MVX2KC0X3Z9NESE7P22K","Enow Dev.","01M6D1MW1FMZGBDKWAJY4GQX81",67,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fenowdev\u002Fenowxcli\u002Fblob\u002Fmain\u002Fcrates\u002Fenowx-core\u002Fskills\u002Fsystems-formats\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33],"systems formats","writing","spec","fuzzing","protocol","file","compression","ZeRO","parsing","format","versioning","parsers","read","alignment","compatibility","2026-11-10 10:12:30",false,"---\nname: systems-formats\ndescription: \"Binary formats and serialisation: choosing a format, endianness, alignment and padding, versioning with backward and forward compatibility, parsing untrusted input with bounds and limits, zero-copy parsing, checksums, compression, magic numbers and headers, fuzzing parsers, and writing the spec down. Read before designing, parsing or changing a file format or a wire protocol.\"\n---\n\n# Binary formats and serialisation\n\nThe default format is a `memcpy` of an in-memory struct: host byte order,\ncompiler padding written to disk, no magic number, no version, and no way\nto add a field next year. The default parser reads a `u32` length, calls\n`Vec::with_capacity(len)` and falls over on the first corrupt or hostile\nfile. This skill: pick an existing format first; when you must design\none, make it explicit, versioned, bounded and written down; parse every\ninput as if an attacker wrote it, because sometimes one did.\n\n## 1. Choose before inventing\n\n| Need | Format | Rust crates |\n|---|---|---|\n| Human-readable interchange | JSON (integers above 2^53 lose precision in JavaScript: send big ids as strings) | `serde_json` |\n| JSON's data model, compact | CBOR (RFC 8949), MessagePack | `ciborium`, `minicbor`, `rmp-serde` |\n| Evolving schemas across languages, RPC | Protocol Buffers | `prost` (with `buf` for lint and breaking checks) |\n| Read in place without parsing | FlatBuffers, Cap'n Proto | `flatbuffers`, `capnp` |\n| Rust to Rust, compact | postcard | `postcard` |\n| Rust to Rust, zero-copy | rkyv | `rkyv` (validated access) |\n| Pipelines with schema evolution | Avro | `apache-avro` |\n| Columnar analytics | Parquet on disk, Arrow in memory and IPC | `parquet`, `arrow` |\n\n- bincode is unmaintained (RUSTSEC-2025-0141): new code uses postcard.\n- Invent a format only when none fits (a log, an index, a container with\n  random access), and then with a written spec (section 11).\n\n## 2. A custom header\n\nEvery file or message starts with a fixed header, all integers\nlittle-endian:\n\n| Offset | Size | Field | Rule |\n|---|---|---|---|\n| 0 | 8 | magic `89 4D 59 46 0D 0A 1A 0A` | non-ASCII first byte, then CR LF, Ctrl-Z, LF: catches text-mode and 7-bit damage (PNG's trick) |\n| 8 | 2 | major version | a reader refuses a major it does not know |\n| 10 | 2 | minor version | adds optional content; older readers skip it |\n| 12 | 4 | header length | lets a newer header grow; readers skip to its end |\n| 16 | 8 | flags | bits 0 to 31 optional, bits 32 to 63 required: an unknown required bit means refuse |\n| 24 | 4 | CRC32C of bytes 0 to 23 | header integrity |\n\nThe body is records of `type u32, length u32, payload, crc32c u32`\n(type-length-value), so a reader skips what it does not understand. A\nbyte for the compression algorithm (section 9) lets it change later.\n\n## 3. Endianness, explicit\n\n- The spec fixes the byte order (little-endian unless a standard says\n  network order) and code converts on every access, even on a match.\n- Rust: `u32::from_le_bytes`, `to_le_bytes`; `zerocopy` byte-order types\n  for struct views. C: assemble from bytes (compilers emit one load):\n\n```c\nstatic inline uint32_t load_le32(const uint8_t *p) {\n    return (uint32_t)p[0] | (uint32_t)p[1] \u003C\u003C 8 | (uint32_t)p[2] \u003C\u003C 16 | (uint32_t)p[3] \u003C\u003C 24;\n}\n```\n\n  C++20 has `std::endian`, C++23 `std::byteswap`; `le32toh` and friends\n  are not portable across Linux, macOS and Windows.\n\n## 4. Alignment and padding stay in memory\n\n- Never `memcpy`, `fwrite(&s, sizeof s, ...)` or `transmute` a struct to\n  or from bytes: padding leaks memory contents, layout changes with the\n  compiler and target, and a `bool` or enum from arbitrary bytes is UB.\n- Encode field by field, or use a view type proven to have no padding and\n  to accept any bytes: `zerocopy` derives or `bytemuck::Pod`.\n\n```rust\nuse zerocopy::byteorder::little_endian::{U16, U32};\nuse zerocopy::{FromBytes, Immutable, IntoBytes, KnownLayout, Unaligned};\n\n#[derive(FromBytes, IntoBytes, KnownLayout, Immutable, Unaligned)]\n#[repr(C)]\nstruct RecordHeader { kind: U16, flags: U16, len: U32 }\n\nlet (header, rest) = RecordHeader::ref_from_prefix(input).map_err(|_| Error::ShortHeader)?;\nlet len = header.len.get() as usize;\n```\n\n- `#[repr(packed)]` and `__attribute__((packed))`: a reference to a\n  misaligned field is undefined; copy the field out or use\n  `ptr::read_unaligned`.\n\n## 5. Versioning and compatibility\n\n- Plan both directions: new readers read old data (backward), old readers\n  read new data and skip what they do not know (forward). Everything that\n  may be added later is tagged (a record type, a field number); unknown\n  fields are preserved on rewrite when other writers may need them.\n- Safe changes: a new optional field with a default, a new skippable\n  record type. Breaking: changing a field's type or meaning, reusing a tag,\n  reordering positional fields, making an optional field required.\n- Protocol Buffers: never reuse a field number (`reserved 4, 7;` and\n  `reserved \"old_name\";` for removed ones); numbers 1 to 15 take one byte,\n  so give them to frequent fields; first enum value `*_UNSPECIFIED = 0`;\n  `optional` where presence matters; `buf breaking --against\n  '.git#branch=main'` in CI.\n- Every version the reader still supports has a sample file in the tests.\n\n## 6. Parsing untrusted input\n\nEverything from outside the process is untrusted, including your own\nformat read back from disk.\n\n- Every length and count is checked against the bytes remaining and a\n  maximum before it is used, and never trusted for allocation: reserve\n  `min(count, remaining \u002F min_item_size)` and grow as items arrive.\n- Limits in one place and in the spec: total size (16 MiB per message,\n  say), element count, nesting depth (64), string length, decompressed\n  size. No recursion without a depth counter; offset and length arithmetic\n  with `checked_add` and `checked_mul`.\n- Strict by default: reject trailing bytes, overlong varints,\n  non-canonical encodings and duplicate keys (two parsers disagreeing on\n  one input is a security bug).\n- Errors carry the offset and the expectation (`truncated record at\n  offset 1024: need 16 bytes, have 5`).\n\n```rust\npub struct Reader\u003C'a> { rest: &'a [u8], offset: usize }\n\nimpl\u003C'a> Reader\u003C'a> {\n    pub fn take(&mut self, n: usize) -> Result\u003C&'a [u8], Error> {\n        let (head, tail) = self.rest.split_at_checked(n).ok_or(Error::Truncated {\n            offset: self.offset, need: n, have: self.rest.len(),\n        })?;\n        self.rest = tail;\n        self.offset += n;\n        Ok(head)\n    }\n\n    pub fn u32_le(&mut self) -> Result\u003Cu32, Error> {\n        let (bytes, tail) = self.rest.split_first_chunk::\u003C4>().ok_or(Error::Truncated {\n            offset: self.offset, need: 4, have: self.rest.len(),\n        })?;\n        self.rest = tail;\n        self.offset += 4;\n        Ok(u32::from_le_bytes(*bytes))\n    }\n\n    \u002F\u002F\u002F A u32 length prefix, checked against `max` before any byte is taken.\n    pub fn bytes_with_len(&mut self, max: usize) -> Result\u003C&'a [u8], Error> {\n        let len = self.u32_le()? as usize;\n        if len > max {\n            return Err(Error::TooLarge { offset: self.offset, len, max });\n        }\n        self.take(len)\n    }\n\n    \u002F\u002F\u002F Strict formats end here: bytes left over are an error.\n    pub fn finish(self) -> Result\u003C(), Error> {\n        if self.rest.is_empty() { Ok(()) } else { Err(Error::Trailing { offset: self.offset }) }\n    }\n}\n```\n\n## 7. Varints and framing\n\n- LEB128 varints (protobuf, WebAssembly): 7 bits per byte, at most 10\n  bytes for a u64 and 5 for a u32; reject longer, and in strict mode\n  reject overlong forms (`80 00` for zero). ZigZag for signed values.\n- Streams need framing: a u32 length prefix checked against a maximum\n  frame before the body is read. `tokio_util::codec::LengthDelimitedCodec::builder()\n  .max_frame_length(16 * 1024 * 1024).new_framed(io)` does it (default\n  8 MiB, a 4-byte big-endian length).\n- Append-only logs: each record `magic, length, crc, payload`, so a reader\n  detects a torn last write and resynchronises by scanning for the magic.\n  Reads are partial: accumulate until the frame is complete.\n\n## 8. Zero-copy and integrity\n\n- Parsed values borrow the input (`&'a [u8]`, `&'a str`) while the owner\n  (a `Vec`, an `Mmap`, `bytes::Bytes`) keeps it alive. Views built from\n  byte-order types are `Unaligned` and accept any offset; an aligned view\n  needs an aligned buffer, and `ref_from_bytes` refuses otherwise.\n- FlatBuffers, Cap'n Proto and rkyv read in place: run their verifier on\n  untrusted bytes (`flatbuffers::root::\u003CT>`, rkyv's validated `access`);\n  the unchecked accessors are for bytes this process wrote.\n- `#[serde(borrow)] name: Cow\u003C'a, str>` borrows when the JSON string has\n  no escapes and allocates when it does.\n\n| Need | Use | Rust |\n|---|---|---|\n| Catch accidental corruption | CRC32C (hardware on x86 SSE4.2 and Arm) | `crc32c` |\n| Interop with zip, gzip, PNG | CRC-32 (IEEE) | `crc32fast` |\n| Fast content hash, no adversary | XXH3 | `xxhash-rust` |\n| Content address, tamper evidence | BLAKE3, SHA-256 | `blake3`, `sha2` |\n| Tamper evidence with a shared key | HMAC-SHA256, or an AEAD | `hmac`, `chacha20poly1305`, `aes-gcm` |\n\nA CRC catches accidents, not attackers: anyone can recompute it. Checksum\neach record or block so damage is local, verify before parsing the\npayload, and compare MACs in constant time (`verify_slice`).\n\n## 9. Compression\n\n- zstd by default (level 3; 19 for archives written once); lz4 where\n  speed matters more than ratio; gzip or deflate only for interop; brotli\n  for web assets.\n- Messages under a few KiB compress poorly alone: train a dictionary\n  (`zstd --train samples\u002F* -o dict`) and version it with the format.\n- Cap the output, whatever the header claims (decompression bombs):\n\n```rust\nuse std::io::Read;\nlet mut out = Vec::new();\nzstd::stream::read::Decoder::new(input)?\n    .take(MAX_DECOMPRESSED as u64 + 1)\n    .read_to_end(&mut out)?;\nif out.len() > MAX_DECOMPRESSED {\n    return Err(Error::DecompressedTooLarge { max: MAX_DECOMPRESSED });\n}\n```\n\n## 10. Fuzz and test the parser\n\nA fuzz target per entry point; the corpus seeded with test vectors and\nreal samples; every crash kept as a regression test.\n\n```rust\n\u002F\u002F fuzz\u002Ffuzz_targets\u002Fdecode.rs (cargo install cargo-fuzz; cargo fuzz init)\n#![no_main]\nuse libfuzzer_sys::fuzz_target;\n\nfuzz_target!(|data: &[u8]| {\n    if let Ok(msg) = myformat::decode(data) {\n        let bytes = myformat::encode(&msg);\n        assert_eq!(myformat::decode(&bytes).ok(), Some(msg), \"round trip changed the value\");\n    }\n});\n```\n\n```sh\ncargo +nightly fuzz run decode -- -max_total_time=600 -max_len=1048576\ncargo +nightly fuzz cmin decode            # shrink the corpus\ncargo +nightly fuzz tmin decode \u003Ccrash>    # shrink a crashing input\n```\n\n- `proptest` for `decode(encode(x)) == x` over generated values; C and C++\n  parsers with libFuzzer or AFL++ (`systems-c-cpp`, section 10). Time the\n  parser on hostile shapes too: a million empty records, maximum depth.\n\n## 11. Write the spec down\n\nA format without a spec is whatever the current code does. Keep\n`docs\u002Fformat.md` (or the schema plus notes) with: purpose and byte order;\nthe header table (offset, size, type, meaning, allowed values); each\nrecord type; checksums and compression; the limits and what a reader does\non each error; what a reader does with an unknown version, flag, record or\nfield; and test vectors (small files with a hex dump and the decoded\nvalue) in `tests\u002Fvectors\u002F`, decoded by a test so other implementations can\ncheck against them. A Kaitai Struct (`.ksy`) description or an ImHex\npattern helps people inspect files by hand.\n\n## Check it\n\n- `xxd file | head` (or `hexdump -C`) of a written file matches the spec\n  table byte for byte.\n- Tests decode every vector and every old-version fixture; each limit is\n  tested at the limit and one past it; every prefix of a valid message\n  (a loop over `0..len`) fails cleanly as truncated, never panics.\n- The fuzz target ran at least 10 minutes on this change without a\n  finding; say so if it could not run here (cargo-fuzz needs nightly).\n\n## Avoid\n\nStructs written with `memcpy` or `transmute`; host byte order on disk; no\nmagic number or version; lengths used before they are checked against\nwhat remains and a maximum; `with_capacity` from an input count;\nrecursion without a depth limit; trailing garbage accepted in a strict\nformat; reused protobuf field numbers; CRCs trusted against tampering;\ndecompression without an output cap; bincode in new code; a format with\nno spec and no test vectors.\n","# Binary formats and serialisation\n\nThe default format is a `memcpy` of an in-memory struct: host byte order,\ncompiler padding written to disk, no magic number, no version, and no way\nto add a field next year. The default parser reads a `u32` length, calls\n`Vec::with_capacity(len)` and falls over on the first corrupt or hostile\nfile. This skill: pick an existing format first; when you must design\none, make it explicit, versioned, bounded and written down; parse every\ninput as if an attacker wrote it, because sometimes one did.\n\n## 1. Choose before inventing\n\n| Need | Format | Rust crates |\n|---|---|---|\n| Human-readable interchange | JSON (integers above 2^53 lose precision in JavaScript: send big ids as strings) | `serde_json` |\n| JSON's data model, compact | CBOR (RFC 8949), MessagePack | `ciborium`, `minicbor`, `rmp-serde` |\n| Evolving schemas across languages, RPC | Protocol Buffers | `prost` (with `buf` for lint and breaking checks) |\n| Read in place without parsing | FlatBuffers, Cap'n Proto | `flatbuffers`, `capnp` |\n| Rust to Rust, compact | postcard | `postcard` |\n| Rust to Rust, zero-copy | rkyv | `rkyv` (validated access) |\n| Pipelines with schema evolution | Avro | `apache-avro` |\n| Columnar analytics | Parquet on disk, Arrow in memory and IPC | `parquet`, `arrow` |\n\n- bincode is unmaintained (RUSTSEC-2025-0141): new code uses postcard.\n- Invent a format only when none fits (a log, an index, a container with\n  random access), and then with a written spec (section 11).\n\n## 2. A custom header\n\nEvery file or message starts with a fixed header, all integers\nlittle-endian:\n\n| Offset | Size | Field | Rule |\n|---|---|---|---|\n| 0 | 8 | magic `89 4D 59 46 0D 0A 1A 0A` | non-ASCII first byte, then CR LF, Ctrl-Z, LF: catches text-mode and 7-bit damage (PNG's trick) |\n| 8 | 2 | major version | a reader refuses a major it does not know |\n| 10 | 2 | minor version | adds optional content; older readers skip it |\n| 12 | 4 | header length | lets a newer header grow; readers skip to its end |\n| 16 | 8 | flags | bits 0 to 31 optional, bits 32 to 63 required: an unknown required bit means refuse |\n| 24 | 4 | CRC32C of bytes 0 to 23 | header integrity |\n\nThe body is records of `type u32, length u32, payload, crc32c u32`\n(type-length-value), so a reader skips what it does not understand. A\nbyte for the compression algorithm (section 9) lets it change later.\n\n## 3. Endianness, explicit\n\n- The spec fixes the byte order (little-endian unless a standard says\n  network order) and code converts on every access, even on a match.\n- Rust: `u32::from_le_bytes`, `to_le_bytes`; `zerocopy` byte-order types\n  for struct views. C: assemble from bytes (compilers emit one load):\n\n```c\nstatic inline uint32_t load_le32(const uint8_t *p) {\n    return (uint32_t)p[0] | (uint32_t)p[1] \u003C\u003C 8 | (uint32_t)p[2] \u003C\u003C 16 | (uint32_t)p[3] \u003C\u003C 24;\n}\n```\n\n  C++20 has `std::endian`, C++23 `std::byteswap`; `le32toh` and friends\n  are not portable across Linux, macOS and Windows.\n\n## 4. Alignment and padding stay in memory\n\n- Never `memcpy`, `fwrite(&s, sizeof s, ...)` or `transmute` a struct to\n  or from bytes: padding leaks memory contents, layout changes with the\n  compiler and target, and a `bool` or enum from arbitrary bytes is UB.\n- Encode field by field, or use a view type proven to have no padding and\n  to accept any bytes: `zerocopy` derives or `bytemuck::Pod`.\n\n```rust\nuse zerocopy::byteorder::little_endian::{U16, U32};\nuse zerocopy::{FromBytes, Immutable, IntoBytes, KnownLayout, Unaligned};\n\n#[derive(FromBytes, IntoBytes, KnownLayout, Immutable, Unaligned)]\n#[repr(C)]\nstruct RecordHeader { kind: U16, flags: U16, len: U32 }\n\nlet (header, rest) = RecordHeader::ref_from_prefix(input).map_err(|_| Error::ShortHeader)?;\nlet len = header.len.get() as usize;\n```\n\n- `#[repr(packed)]` and `__attribute__((packed))`: a reference to a\n  misaligned field is undefined; copy the field out or use\n  `ptr::read_unaligned`.\n\n## 5. Versioning and compatibility\n\n- Plan both directions: new readers read old data (backward), old readers\n  read new data and skip what they do not know (forward). Everything that\n  may be added later is tagged (a record type, a field number); unknown\n  fields are preserved on rewrite when other writers may need them.\n- Safe changes: a new optional field with a default, a new skippable\n  record type. Breaking: changing a field's type or meaning, reusing a tag,\n  reordering positional fields, making an optional field required.\n- Protocol Buffers: never reuse a field number (`reserved 4, 7;` and\n  `reserved \"old_name\";` for removed ones); numbers 1 to 15 take one byte,\n  so give them to frequent fields; first enum value `*_UNSPECIFIED = 0`;\n  `optional` where presence matters; `buf breaking --against\n  '.git#branch=main'` in CI.\n- Every version the reader still supports has a sample file in the tests.\n\n## 6. Parsing untrusted input\n\nEverything from outside the process is untrusted, including your own\nformat read back from disk.\n\n- Every length and count is checked against the bytes remaining and a\n  maximum before it is used, and never trusted for allocation: reserve\n  `min(count, remaining \u002F min_item_size)` and grow as items arrive.\n- Limits in one place and in the spec: total size (16 MiB per message,\n  say), element count, nesting depth (64), string length, decompressed\n  size. No recursion without a depth counter; offset and length arithmetic\n  with `checked_add` and `checked_mul`.\n- Strict by default: reject trailing bytes, overlong varints,\n  non-canonical encodings and duplicate keys (two parsers disagreeing on\n  one input is a security bug).\n- Errors carry the offset and the expectation (`truncated record at\n  offset 1024: need 16 bytes, have 5`).\n\n```rust\npub struct Reader\u003C'a> { rest: &'a [u8], offset: usize }\n\nimpl\u003C'a> Reader\u003C'a> {\n    pub fn take(&mut self, n: usize) -> Result\u003C&'a [u8], Error> {\n        let (head, tail) = self.rest.split_at_checked(n).ok_or(Error::Truncated {\n            offset: self.offset, need: n, have: self.rest.len(),\n        })?;\n        self.rest = tail;\n        self.offset += n;\n        Ok(head)\n    }\n\n    pub fn u32_le(&mut self) -> Result\u003Cu32, Error> {\n        let (bytes, tail) = self.rest.split_first_chunk::\u003C4>().ok_or(Error::Truncated {\n            offset: self.offset, need: 4, have: self.rest.len(),\n        })?;\n        self.rest = tail;\n        self.offset += 4;\n        Ok(u32::from_le_bytes(*bytes))\n    }\n\n    \u002F\u002F\u002F A u32 length prefix, checked against `max` before any byte is taken.\n    pub fn bytes_with_len(&mut self, max: usize) -> Result\u003C&'a [u8], Error> {\n        let len = self.u32_le()? as usize;\n        if len > max {\n            return Err(Error::TooLarge { offset: self.offset, len, max });\n        }\n        self.take(len)\n    }\n\n    \u002F\u002F\u002F Strict formats end here: bytes left over are an error.\n    pub fn finish(self) -> Result\u003C(), Error> {\n        if self.rest.is_empty() { Ok(()) } else { Err(Error::Trailing { offset: self.offset }) }\n    }\n}\n```\n\n## 7. Varints and framing\n\n- LEB128 varints (protobuf, WebAssembly): 7 bits per byte, at most 10\n  bytes for a u64 and 5 for a u32; reject longer, and in strict mode\n  reject overlong forms (`80 00` for zero). ZigZag for signed values.\n- Streams need framing: a u32 length prefix checked against a maximum\n  frame before the body is read. `tokio_util::codec::LengthDelimitedCodec::builder()\n  .max_frame_length(16 * 1024 * 1024).new_framed(io)` does it (default\n  8 MiB, a 4-byte big-endian length).\n- Append-only logs: each record `magic, length, crc, payload`, so a reader\n  detects a torn last write and resynchronises by scanning for the magic.\n  Reads are partial: accumulate until the frame is complete.\n\n## 8. Zero-copy and integrity\n\n- Parsed values borrow the input (`&'a [u8]`, `&'a str`) while the owner\n  (a `Vec`, an `Mmap`, `bytes::Bytes`) keeps it alive. Views built from\n  byte-order types are `Unaligned` and accept any offset; an aligned view\n  needs an aligned buffer, and `ref_from_bytes` refuses otherwise.\n- FlatBuffers, Cap'n Proto and rkyv read in place: run their verifier on\n  untrusted bytes (`flatbuffers::root::\u003CT>`, rkyv's validated `access`);\n  the unchecked accessors are for bytes this process wrote.\n- `#[serde(borrow)] name: Cow\u003C'a, str>` borrows when the JSON string has\n  no escapes and allocates when it does.\n\n| Need | Use | Rust |\n|---|---|---|\n| Catch accidental corruption | CRC32C (hardware on x86 SSE4.2 and Arm) | `crc32c` |\n| Interop with zip, gzip, PNG | CRC-32 (IEEE) | `crc32fast` |\n| Fast content hash, no adversary | XXH3 | `xxhash-rust` |\n| Content address, tamper evidence | BLAKE3, SHA-256 | `blake3`, `sha2` |\n| Tamper evidence with a shared key | HMAC-SHA256, or an AEAD | `hmac`, `chacha20poly1305`, `aes-gcm` |\n\nA CRC catches accidents, not attackers: anyone can recompute it. Checksum\neach record or block so damage is local, verify before parsing the\npayload, and compare MACs in constant time (`verify_slice`).\n\n## 9. Compression\n\n- zstd by default (level 3; 19 for archives written once); lz4 where\n  speed matters more than ratio; gzip or deflate only for interop; brotli\n  for web assets.\n- Messages under a few KiB compress poorly alone: train a dictionary\n  (`zstd --train samples\u002F* -o dict`) and version it with the format.\n- Cap the output, whatever the header claims (decompression bombs):\n\n```rust\nuse std::io::Read;\nlet mut out = Vec::new();\nzstd::stream::read::Decoder::new(input)?\n    .take(MAX_DECOMPRESSED as u64 + 1)\n    .read_to_end(&mut out)?;\nif out.len() > MAX_DECOMPRESSED {\n    return Err(Error::DecompressedTooLarge { max: MAX_DECOMPRESSED });\n}\n```\n\n## 10. Fuzz and test the parser\n\nA fuzz target per entry point; the corpus seeded with test vectors and\nreal samples; every crash kept as a regression test.\n\n```rust\n\u002F\u002F fuzz\u002Ffuzz_targets\u002Fdecode.rs (cargo install cargo-fuzz; cargo fuzz init)\n#![no_main]\nuse libfuzzer_sys::fuzz_target;\n\nfuzz_target!(|data: &[u8]| {\n    if let Ok(msg) = myformat::decode(data) {\n        let bytes = myformat::encode(&msg);\n        assert_eq!(myformat::decode(&bytes).ok(), Some(msg), \"round trip changed the value\");\n    }\n});\n```\n\n```sh\ncargo +nightly fuzz run decode -- -max_total_time=600 -max_len=1048576\ncargo +nightly fuzz cmin decode            # shrink the corpus\ncargo +nightly fuzz tmin decode \u003Ccrash>    # shrink a crashing input\n```\n\n- `proptest` for `decode(encode(x)) == x` over generated values; C and C++\n  parsers with libFuzzer or AFL++ (`systems-c-cpp`, section 10). Time the\n  parser on hostile shapes too: a million empty records, maximum depth.\n\n## 11. Write the spec down\n\nA format without a spec is whatever the current code does. Keep\n`docs\u002Fformat.md` (or the schema plus notes) with: purpose and byte order;\nthe header table (offset, size, type, meaning, allowed values); each\nrecord type; checksums and compression; the limits and what a reader does\non each error; what a reader does with an unknown version, flag, record or\nfield; and test vectors (small files with a hex dump and the decoded\nvalue) in `tests\u002Fvectors\u002F`, decoded by a test so other implementations can\ncheck against them. A Kaitai Struct (`.ksy`) description or an ImHex\npattern helps people inspect files by hand.\n\n## Check it\n\n- `xxd file | head` (or `hexdump -C`) of a written file matches the spec\n  table byte for byte.\n- Tests decode every vector and every old-version fixture; each limit is\n  tested at the limit and one past it; every prefix of a valid message\n  (a loop over `0..len`) fails cleanly as truncated, never panics.\n- The fuzz target ran at least 10 minutes on this change without a\n  finding; say so if it could not run here (cargo-fuzz needs nightly).\n\n## Avoid\n\nStructs written with `memcpy` or `transmute`; host byte order on disk; no\nmagic number or version; lengths used before they are checked against\nwhat remains and a maximum; `with_capacity` from an input count;\nrecursion without a depth limit; trailing garbage accepted in a strict\nformat; reused protobuf field numbers; CRCs trusted against tampering;\ndecompression without an output cap; bincode in new code; a format with\nno spec and no test vectors.\n",{"name":5,"description":7},"main","38293fe1b096e6bf3646c7344c38786ed688f464","crates\u002Fenowx-core\u002Fskills\u002Fsystems-formats",21,"https:\u002F\u002Fgithub.com\u002Fenowdev\u002Fenowxcli","https:\u002F\u002Fenowx.ai","https:\u002F\u002Fgithub.com\u002Fenowdev","Pretending to be a Developer","2026-10-06 02:05:35",{"source":8,"skillName":5,"commands":49,"prompt":59,"zipAvailable":60},[50,53,56],{"tool":51,"command":52},"npx","npx skills add enowdev\u002Fenowxcli --skill systems-formats",{"tool":54,"command":55},"bunx","bunx skills add enowdev\u002Fenowxcli --skill systems-formats",{"tool":57,"command":58},"pnpm","pnpm dlx skills add enowdev\u002Fenowxcli --skill systems-formats","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fenowdev\u002Fenowxcli\u002Fblob\u002Fmain\u002Fcrates\u002Fenowx-core\u002Fskills\u002Fsystems-formats\u002FSKILL.md",true,[],[63,79,103,119,146,168],{"skillNo":64,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":67,"categories":68,"keywords":74,"publishedAt":34},"01M6D2HQYSQD9N7MTNDT4A7CBM","demo-skill","Project-scope demo skill for tests.","https:\u002F\u002Fgithub.com\u002Fenowdev\u002Fenowxcli\u002Fblob\u002Fmain\u002Fworkspace\u002F.agents\u002Fskills\u002Fdemo-skill\u002FSKILL.md",[69],{"categoryNo":70,"categoryName":71,"taxonomyType":72,"taxonomyTypeDesc":73},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[75,76,77,78,31],"demo skill","demo","tests","project",{"skillNo":80,"name":20,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":92,"publishedAt":34},"01M6D2HM0GCPM0TM2HPG9HHR2R","Writing words people read: the specific thing instead of claims, words to drop, sentences, interface copy (buttons, labels, errors, empty states, confirmations), page copy, notifications and emails, alt text and link text, numbers and dates, words that translate well, documentation and voice. Read before writing a page's copy, an interface's text or a document.","https:\u002F\u002Fgithub.com\u002Fenowdev\u002Fenowxcli\u002Fblob\u002Fmain\u002Fcrates\u002Fenowx-core\u002Fskills\u002Fwriting\u002FSKILL.md",[84,87],{"categoryNo":85,"categoryName":86,"taxonomyType":72,"taxonomyTypeDesc":73},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":88,"categoryName":89,"taxonomyType":90,"taxonomyTypeDesc":91},"01JBSEEDCAT000000000000036","内容创作者","PROFESSION","适用职业",[20,93,94,95,96,97,31,98,99,100,101,102],"text","emails","empty","drop","document","page","voice","notifications","documentation","translate",{"skillNo":104,"name":105,"nameZh":6,"description":106,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":107,"categories":108,"keywords":112,"publishedAt":34},"01M6D2HG07YS49NM196Q0R3AZ8","ui","Designing or restyling an interface: direction, layout, type, colour, icons, components, states, responsive behaviour. Read before building a page, a screen or a set of components.","https:\u002F\u002Fgithub.com\u002Fenowdev\u002Fenowxcli\u002Fblob\u002Fmain\u002Fcrates\u002Fenowx-core\u002Fskills\u002Fui\u002FSKILL.md",[109],{"categoryNo":110,"categoryName":111,"taxonomyType":90,"taxonomyTypeDesc":91},"01JBSEEDCAT000000000000032","设计师",[105,113,114,115,116,98,117,31,118],"icons","components","SET","responsive","layout","before building",{"skillNo":120,"name":121,"nameZh":6,"description":122,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":123,"categories":124,"keywords":131,"publishedAt":34},"01M6D2HB59HZHWG1RYZDV4SJ91","ui-themes","Light and dark themes, and colour that holds in both: tokens with a value per theme, and what each kind of content needs to stay visible (icons, SVG drawings and line art, logos, photos, screenshots, charts, code, maps, borders, shadows, focus, form controls), switching without a flash, and checking both. Read before building a second theme or a theme toggle, and before any drawing, logo or chart on a page that has two themes.","https:\u002F\u002Fgithub.com\u002Fenowdev\u002Fenowxcli\u002Fblob\u002Fmain\u002Fcrates\u002Fenowx-core\u002Fskills\u002Fui-themes\u002FSKILL.md",[125,128],{"categoryNo":126,"categoryName":127,"taxonomyType":72,"taxonomyTypeDesc":73},"01JBSEEDCAT000000000000015","视觉设计",{"categoryNo":129,"categoryName":130,"taxonomyType":90,"taxonomyTypeDesc":91},"01JBSEEDCAT000000000000030","开发工程师",[105,132,133,134,135,136,137,138,139,31,118,140,98,141,142,143,113,144,145],"ui themes","switching","photos","tokens","drawing","focus","maps","Art","flash","code","Charts","content","screenshots","logo",{"skillNo":147,"name":148,"nameZh":6,"description":149,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":150,"categories":151,"keywords":155,"publishedAt":34},"01M6D2H5R5KRAE524C8QX0T7Q0","ui-structure","Structure before styling: the blueprint written before any interface code (the screen's job, the regions its kind of page needs, their order and why, the layout skeleton in numbers, the components and their states), the anatomy and order of a homepage, a dashboard, a list, a detail page, settings, a form, sign-in, pricing, docs, an article and a checkout, and how to turn the blueprint into code (tokens, layout primitives, components, sections, page). Read first, before building or rebuilding any page or screen.","https:\u002F\u002Fgithub.com\u002Fenowdev\u002Fenowxcli\u002Fblob\u002Fmain\u002Fcrates\u002Fenowx-core\u002Fskills\u002Fui-structure\u002FSKILL.md",[152,153,154],{"categoryNo":85,"categoryName":86,"taxonomyType":72,"taxonomyTypeDesc":73},{"categoryNo":129,"categoryName":130,"taxonomyType":90,"taxonomyTypeDesc":91},{"categoryNo":88,"categoryName":89,"taxonomyType":90,"taxonomyTypeDesc":91},[156,105,157,114,158,159,141,160,161,162,117,135,163,164,165,31,118,98,166,167],"structure","ui structure","pricing","dashboard","Blueprint","article","anatomy","styling","primitives","list","why","how",{"skillNo":169,"name":170,"nameZh":6,"description":171,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":172,"categories":173,"keywords":181,"publishedAt":34},"01M6D2H0ZZBJG77MBGPZHA6MWD","ui-stack-vue","Building interfaces in Vue 3 and Nuxt: project layout, script setup with typed defineProps, defineEmits and defineModel, attribute fallthrough, slots and scoped slots, template refs, ref, reactive and computed, watch versus watchEffect, composables and VueUse, provide and inject, Pinia, keyed v-for, transitions, Nuxt 4 (app directory, pages, layouts, useFetch and useAsyncData keys, $fetch in handlers, server routes, runtime config, useSeoMeta, NuxtImg, middleware), VeeValidate forms, Reka UI and Nuxt UI, testing, vue-tsc, and the traps. Read when the project uses Vue or Nuxt.","https:\u002F\u002Fgithub.com\u002Fenowdev\u002Fenowxcli\u002Fblob\u002Fmain\u002Fcrates\u002Fenowx-core\u002Fskills\u002Fui-stack-vue\u002FSKILL.md",[174,177,180],{"categoryNo":175,"categoryName":176,"taxonomyType":72,"taxonomyTypeDesc":73},"01JBSEEDCAT000000000000008","测试与质量",{"categoryNo":178,"categoryName":179,"taxonomyType":72,"taxonomyTypeDesc":73},"01JBSEEDCAT000000000000009","前端开发",{"categoryNo":110,"categoryName":111,"taxonomyType":90,"taxonomyTypeDesc":91},[105,182,183,184,185,186,187,188,117,78,189,190,31,191,192,193,194,195,196,197],"vue","stack","ui stack vue","forms","pages","watch","runtime","transitions","fetch","pinia","nuxt","template","server","config","directory","testing"]