[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6ACJMHPEN7E76VPKC3P4X6C":3,"skill-same:01M6ACJMHPEN7E76VPKC3P4X6C":73},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":38,"linkOnly":39,"markdownContent":40,"markdownBody":41,"frontMatter":42,"refName":44,"commitSha":45,"skillDirectory":46,"forksCount":47,"repositoryHtmlUrl":48,"repositoryHomepageUrl":49,"ownerHtmlUrl":50,"ownerBlogUrl":51,"ownerDescription":6,"updatedAt":52,"install":53,"resources":66},"01M6ACJMHPEN7E76VPKC3P4X6C","review-comet-shuffle-pr",null,"Use when reviewing a DataFusion Comet pull request that touches native or JVM columnar shuffle, the shuffle writers and readers, partitioning, the Arrow IPC block format, shuffle compression, or the Celeborn integration. Load alongside review-comet-pr.","apache\u002Fdatafusion-comet","apache","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F47359?v=4","01M6AC83E6K4FDKK7HZ8WNMQKE","The Apache Software Foundation","01M6ACHX42TPNCR8TQJEAR8J5J",1322,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-shuffle-pr\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37],"pr","review","review comet shuffle pr","when","request","load","ipc","number","block","native","jvm","use","format","compression","integration","datafusion","arrow","pull","pull request","2026-11-10 10:12:30",false,"---\nname: review-comet-shuffle-pr\ndescription: Use when reviewing a DataFusion Comet pull request that touches native or JVM columnar shuffle, the shuffle writers and readers, partitioning, the Arrow IPC block format, shuffle compression, or the Celeborn integration. Load alongside review-comet-pr.\nargument-hint: \u003Cpr-number>\n---\n\n\u003C!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n  http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied.  See the License for the\nspecific language governing permissions and limitations\nunder the License.\n-->\n\nShuffle-specific review for Comet PR #$ARGUMENTS.\n\n**REQUIRED BACKGROUND:** Use `review-comet-pr` for PR metadata, existing comments, CI, the review\nbar, and the output format. This skill only covers shuffle.\n\n## Read the Contributor Guide First\n\n| Doc                                                  | What you need from it                                                  |\n| ---------------------------------------------------- | ---------------------------------------------------------------------- |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Fnative_shuffle.md`    | Selection rules, architecture, partitioning, block format, spilling    |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Fjvm_shuffle.md`       | Writer variants, handle selection, the row-based path, spill mechanics |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Fmemory_management.md` | Where shuffle memory comes from, which differs between the two paths   |\n\n**Read both shuffle docs even if the PR only touches one path.** The two implementations share the\nmanager, the dependency, the reader, and the on-disk format, and a change to one side of a shared\npiece is the most common way to break the other.\n\n## 1. Which Implementation\n\n| Implementation                        | Selected when                                                                                             |\n| ------------------------------------- | --------------------------------------------------------------------------------------------------------- |\n| Native, `CometExchange`               | `shuffle.mode` is `native` or `auto`, child is a `CometPlan`, supported partitioning, supported key types |\n| JVM columnar, `CometColumnarExchange` | `shuffle.mode` is `jvm`, or the child is row-based, or a partition key type native shuffle cannot handle  |\n\nComplex types are fully supported as **data** columns in both. The restriction is on **partition\nkeys**, and it is not the same rule for the two partitionings:\n\n- **`RangePartitioning` is primitive-only, unconditionally.**\n  `supportedRangePartitioningDataType` rejects every nested type, because native cannot sort them,\n  and rejects collated strings, because Comet compares raw bytes. That is the whole rule. Scalar\n  float and double are accepted, under `spark.comet.exec.strictFloatingPoint` as well: since\n  [#5981](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fpull\u002F5981) the native range partitioner\n  normalizes its comparison keys and its sampled boundary rows the same way the native sort does.\n  `CometSortOrder` is `Compatible()` for scalar floats regardless of strict mode. It is for floats\n  nested in arrays and structs too, because the native sort normalizes them, unless the key's type\n  can hold a null element or field\n  ([#6476](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F6476),\n  [#6477](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F6477)), which strict mode declines.\n  A nested key is already out as a range key for being nested. `CometNativeShuffleSuite` runs\n  \"range partitioning on floating-point uses native shuffle\" under both settings of the config.\n- **`HashPartitioning` is primitive-only by default only.** With\n  `spark.comet.shuffle.native.partitioning.hash.nested.enabled=true` (default `false`),\n  `supportedHashPartitioningDataType` admits structs and arrays recursively, and maps on Spark\n  4.0+, where Spark's `mapsort` normalization makes physical entry order irrelevant. The recursion\n  still rejects a collated string at any depth, and a map on Spark 3.x. A map key also has to\n  clear the separate expression check on the inserted `mapsort(...)`, which `CometMapSort`\n  (`spark\u002Fsrc\u002Fmain\u002Fspark-4.x\u002F`) supports for scalar map keys only.\n  `CometNativeShuffleSuite` covers both settings of the config.\n\nA native exchange on a struct or array hash key is therefore not automatically a bug, and neither\nis one on a scalar float or double range key in strict mode. Check the config before calling either\none, and do not carry the range-partitioning rule over to hash partitioning.\n\n- [ ] A PR that widens what native shuffle supports updates the fallback conditions in\n      `CometShuffleExchangeExec` **and** both docs' \"When X is Used\" lists\n- [ ] A PR that narrows support does not silently move workloads onto the slower path. The JVM path\n      costs a columnar to row to columnar round trip through `ColumnarToRowExec`.\n- [ ] Fallback decisions stay consistent across a stage. `CometShuffleFallbackStickinessSuite`\n      exists because they did not once.\n\n## 2. Spark Compatibility of Partitioning\n\nPartitioning is where shuffle silently produces wrong answers rather than failing.\n\n- [ ] **Hash partitioning uses Murmur3 with seed 42** and `partition_id = hash % num_partitions`,\n      matching Spark. Any change to the hash, the seed, or the modulo changes which rows land in\n      which partition, which breaks a join between a Comet-shuffled side and a Spark-shuffled side.\n- [ ] **Round robin defaults to a content hash, and is off by default.** Comet's default\n      `RoundRobinStrategy::HashAll` assigns partitions from a Murmur3 hash rather than cycling row\n      by row, because placement has to be reproducible across task retries. Two costs are accepted:\n      low-cardinality data distributes unevenly, and unsorted rows land in different partitions\n      than Spark's `UnsafeRow`-sorted assignment would put them, which is why\n      `spark.comet.shuffle.native.partitioning.roundrobin.enabled` defaults to `false`. Sorted\n      output is identical either way.\n- [ ] **Positional round robin is allowed, but its allowlist is the whole safety argument.**\n      `RoundRobinStrategy::RowGroups` is not a bug by construction. It is a function of row order\n      and never sorts, so it needs a retry to replay the same rows in the same order, which is\n      more than Spark's own round robin needs under its default `sortBeforeRepartition=true`.\n      `CometShuffleExchangeExec.replaysRowsInOrder` is what establishes that, and a PR that widens\n      it needs an argument that each operator it admits replays its rows in order, including\n      that the admitted expressions are deterministic; the RDD-level determinism check is defence\n      in depth and cannot fire under today's allowlist. Also check that placement still counts\n      rows rather than batches, that the start still comes from `positionalStartPartition`, and\n      that the group size is still resolved on the driver rather than from executor state. The reasoning behind all three is in `native_shuffle.md` under\n      \"Round Robin Partitioning\".\n- [ ] **Range partitioning bounds come from the driver.** Spark's `RangePartitioner` samples and\n      computes boundaries, they are serialized into the native plan, and native does a binary\n      search over comparable-row-format keys. A change to the comparison or the row encoding must\n      match Spark's ordering exactly, including nulls and signed zero.\n- [ ] The JVM path uses Spark's own partitioner via `partitioner.getPartition(key)`, so it inherits\n      Spark's semantics for free. A PR that reimplements partitioning on that path is solving a\n      problem that does not exist.\n\n## 3. On-Disk and On-Wire Format\n\nWriter and reader must change together, and they are in different languages.\n\nThe block layout is an 8-byte compressed length header, an 8-byte field count header, then the\ncompressed Arrow IPC stream. It is written by the native `ShuffleBlockWriter` and read by\n`NativeBatchDecoderIterator` calling `Native.decodeShuffleBlock()`.\n\n- [ ] A format change updates the writer, the reader, and the Celeborn reader path\n- [ ] A format change is not silently incompatible with shuffle files written by a previous version\n      in the same cluster during a rolling deployment. If it is, the PR needs to say so.\n- [ ] Compression codec changes apply uniformly to all partitions, and each partition stays\n      independently decompressible so reads can parallelize\n- [ ] The commit path still works. Native records the byte offset where each partition begins plus\n      the total length, `CometNativeShuffleWriter` fetches them with\n      `Native.getShufflePartitionOffsets`, converts them to partition lengths, and commits through\n      Spark's `IndexShuffleBlockResolver.writeMetadataFileAndCommit`. Offsets and lengths are easy\n      to confuse and the failure is a corrupt index file rather than an exception.\n- [ ] Checksums via `CometShuffleChecksumSupport` still cover what Spark expects\n\n## 4. Memory and Spilling\n\nShuffle is the largest memory consumer in most queries, and the two paths draw from different\nbudgets.\n\n**Native shuffle** uses the DataFusion memory pool. Partitions spill when the pool denies an\nallocation, or when buffered bytes reach `spark.comet.shuffle.native.maxBufferBytes`, which\ndefaults to `0`, meaning the fixed limit is disabled and memory pressure is the only trigger.\n\n**The local writer spills every partition into one file per task, not one file per partition.**\n`PartitionedSpill` (`native\u002Fshuffle\u002Fsrc\u002Fwriters\u002Flocal\u002Fspill.rs`) opens a single DataFusion spill\nfile lazily on the first spill and records, per partition, the byte ranges holding that partition's\nblocks in write order. `finish_partition` copies a partition's ranges into the output file in that\norder. `spilling_every_partition_creates_one_file` asserts the file count after spilling 64\npartitions three times. The single-partition writer does not spill, and neither does the RSS\nwriter, which pushes blocks to the remote service instead.\n\n**JVM shuffle** allocates off-heap pages through `CometShuffleMemoryAllocator`, which is an\nordinary Spark `MemoryConsumer` drawing from `spark.memory.offHeap.size`. `CometDiskBlockWriter`\ncoordinates spilling across partition writers, largest first. `TooLargePageException` is the signal\nthat a single record does not fit in a page.\n\n- [ ] A PR that adds buffering on either path says where the reservation is\n- [ ] A change to native spill bookkeeping keeps the recorded ranges and the file in sync.\n      `PartitionedSpill` sets a `failed` flag on a partial write because a mid-write failure makes\n      every later range untrustworthy, and a range that runs past the end of a truncated file must\n      fail the task rather than write a short partition.\n- [ ] A PR that changes spill thresholds has benchmark evidence, because spilling too late is an\n      OOM and spilling too early is a throughput loss\n- [ ] Scratch buffers reused for partition-id computation are correctly reset between batches\n- [ ] Use `review-comet-memory-pr` as well for anything touching reservations or the allocator\n\n## 5. Writer Selection on the JVM Path\n\n`CometShuffleManager.shouldBypassMergeSort()` picks between the two JVM writers. It uses bypass if\nthe partition count is below the threshold **and** partitions times cores is within the max thread\ncount, otherwise sort-based to avoid OOM from too many concurrent writers.\n\n| Handle                              | Writer                                                                        |\n| ----------------------------------- | ----------------------------------------------------------------------------- |\n| `CometBypassMergeSortShuffleHandle` | `CometBypassMergeSortShuffleWriter`, one `CometDiskBlockWriter` per partition |\n| `CometSerializedShuffleHandle`      | `CometUnsafeShuffleWriter`, via `CometShuffleExternalSorter`                  |\n| `CometNativeShuffleHandle`          | `CometNativeShuffleWriter`                                                    |\n\nA change to the selection heuristic changes the memory profile of every job with a large partition\ncount. Ask for the reasoning and the numbers.\n\n## 6. Source Layout\n\nNative shuffle lives in its own crate, `datafusion-comet-shuffle`, under `native\u002Fshuffle\u002F`, with\n`shuffle_writer.rs`, `comet_partitioning.rs`, `ipc.rs`, the `partitioners\u002F` and `writers\u002F` modules,\nand benchmarks in `native\u002Fshuffle\u002Fbenches\u002F`. The JVM side is split between\n`spark\u002Fsrc\u002Fmain\u002Fscala\u002Forg\u002Fapache\u002Fspark\u002Fsql\u002Fcomet\u002Fexecution\u002Fshuffle\u002F`, the Java writers under\n`spark\u002Fsrc\u002Fmain\u002Fjava\u002Forg\u002Fapache\u002Fspark\u002Fsql\u002Fcomet\u002Fexecution\u002Fshuffle\u002F`, and the allocators under\n`spark\u002Fsrc\u002Fmain\u002Fjava\u002Forg\u002Fapache\u002Fspark\u002Fshuffle\u002Fcomet\u002F`.\n\n**Verify the paths cited in the docs still exist.** The native shuffle code has moved between\ncrates before, and the \"Key Classes\" tables in the docs are exactly what goes stale when it does.\n\n## 7. Tests\n\n| Suite                                                               | Covers                                     |\n| ------------------------------------------------------------------- | ------------------------------------------ |\n| `org.apache.comet.exec.CometNativeShuffleSuite`                     | Native shuffle end to end                  |\n| `org.apache.comet.exec.CometColumnarShuffleSuite`                   | JVM columnar shuffle end to end            |\n| `CometShuffle4_0Suite`                                              | Spark 4.x specific behavior                |\n| `CometDiskBlockWriterSuite`                                         | JVM spill and page handling                |\n| `NativeBatchDecoderIteratorLifecycleChecks`, `...ConcurrencyChecks` | Reader lifetime and concurrency            |\n| `CometNativeShuffleInputRDDSuite`                                   | The scheduling-anchor RDD                  |\n| `CometCeleborn*Suite`                                               | The Celeborn path, which is easy to forget |\n| `CometShuffleBenchmark`                                             | Throughput                                 |\n\nAsk specifically:\n\n- [ ] Is the **other** shuffle path tested, if the change touched anything shared?\n- [ ] Is the Celeborn path tested, if the change touched the manager, dependency, writer, or reader?\n- [ ] Are multiple partitions and an actual spill exercised, rather than one small batch?\n- [ ] Are complex types covered as data columns, and is the partition-key restriction tested at its\n      boundary, on **both** settings of\n      `spark.comet.shuffle.native.partitioning.hash.nested.enabled`?\n- [ ] For a partitioning change, is there a test that the same input produces the same partition\n      assignment as Spark, not just that the total row count matches?\n\n## 8. Do the PR Changes Make the Shuffle Docs Stale?\n\nBoth docs are structured as tables of current fact, which is exactly what a refactor invalidates.\nCheck:\n\n- **`native_shuffle.md`**: the \"When Native Shuffle is Used\" conditions, the architecture diagram,\n  the Scala Side and Rust Side \"Key Classes\" tables including their file paths, the write-path and\n  read-path numbered steps, the Partitioning sections and the guarantees they state, the\n  Compression table, the Configuration table **with its defaults**, and the comparison table at the\n  end\n- **`jvm_shuffle.md`**: the \"When JVM Shuffle is Used\" list, the writer architecture diagram, the\n  Shuffle Manager, Handles, Writers and Reader tables, the bypass selection description, the Memory\n  Management section, and the Configuration table\n- A config default changed in `CometConf.scala` but left at its old value in either doc's config\n  table is a specific thing to look for, because those tables are hand-maintained while the\n  user-facing `configs.md` is generated\n- If the PR adds a partitioning scheme, both the native doc's Partitioning section and the\n  comparison table's \"Supported schemes\" row need it\n","\u003C!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n  http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied.  See the License for the\nspecific language governing permissions and limitations\nunder the License.\n-->\n\nShuffle-specific review for Comet PR #$ARGUMENTS.\n\n**REQUIRED BACKGROUND:** Use `review-comet-pr` for PR metadata, existing comments, CI, the review\nbar, and the output format. This skill only covers shuffle.\n\n## Read the Contributor Guide First\n\n| Doc                                                  | What you need from it                                                  |\n| ---------------------------------------------------- | ---------------------------------------------------------------------- |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Fnative_shuffle.md`    | Selection rules, architecture, partitioning, block format, spilling    |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Fjvm_shuffle.md`       | Writer variants, handle selection, the row-based path, spill mechanics |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Fmemory_management.md` | Where shuffle memory comes from, which differs between the two paths   |\n\n**Read both shuffle docs even if the PR only touches one path.** The two implementations share the\nmanager, the dependency, the reader, and the on-disk format, and a change to one side of a shared\npiece is the most common way to break the other.\n\n## 1. Which Implementation\n\n| Implementation                        | Selected when                                                                                             |\n| ------------------------------------- | --------------------------------------------------------------------------------------------------------- |\n| Native, `CometExchange`               | `shuffle.mode` is `native` or `auto`, child is a `CometPlan`, supported partitioning, supported key types |\n| JVM columnar, `CometColumnarExchange` | `shuffle.mode` is `jvm`, or the child is row-based, or a partition key type native shuffle cannot handle  |\n\nComplex types are fully supported as **data** columns in both. The restriction is on **partition\nkeys**, and it is not the same rule for the two partitionings:\n\n- **`RangePartitioning` is primitive-only, unconditionally.**\n  `supportedRangePartitioningDataType` rejects every nested type, because native cannot sort them,\n  and rejects collated strings, because Comet compares raw bytes. That is the whole rule. Scalar\n  float and double are accepted, under `spark.comet.exec.strictFloatingPoint` as well: since\n  [#5981](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fpull\u002F5981) the native range partitioner\n  normalizes its comparison keys and its sampled boundary rows the same way the native sort does.\n  `CometSortOrder` is `Compatible()` for scalar floats regardless of strict mode. It is for floats\n  nested in arrays and structs too, because the native sort normalizes them, unless the key's type\n  can hold a null element or field\n  ([#6476](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F6476),\n  [#6477](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F6477)), which strict mode declines.\n  A nested key is already out as a range key for being nested. `CometNativeShuffleSuite` runs\n  \"range partitioning on floating-point uses native shuffle\" under both settings of the config.\n- **`HashPartitioning` is primitive-only by default only.** With\n  `spark.comet.shuffle.native.partitioning.hash.nested.enabled=true` (default `false`),\n  `supportedHashPartitioningDataType` admits structs and arrays recursively, and maps on Spark\n  4.0+, where Spark's `mapsort` normalization makes physical entry order irrelevant. The recursion\n  still rejects a collated string at any depth, and a map on Spark 3.x. A map key also has to\n  clear the separate expression check on the inserted `mapsort(...)`, which `CometMapSort`\n  (`spark\u002Fsrc\u002Fmain\u002Fspark-4.x\u002F`) supports for scalar map keys only.\n  `CometNativeShuffleSuite` covers both settings of the config.\n\nA native exchange on a struct or array hash key is therefore not automatically a bug, and neither\nis one on a scalar float or double range key in strict mode. Check the config before calling either\none, and do not carry the range-partitioning rule over to hash partitioning.\n\n- [ ] A PR that widens what native shuffle supports updates the fallback conditions in\n      `CometShuffleExchangeExec` **and** both docs' \"When X is Used\" lists\n- [ ] A PR that narrows support does not silently move workloads onto the slower path. The JVM path\n      costs a columnar to row to columnar round trip through `ColumnarToRowExec`.\n- [ ] Fallback decisions stay consistent across a stage. `CometShuffleFallbackStickinessSuite`\n      exists because they did not once.\n\n## 2. Spark Compatibility of Partitioning\n\nPartitioning is where shuffle silently produces wrong answers rather than failing.\n\n- [ ] **Hash partitioning uses Murmur3 with seed 42** and `partition_id = hash % num_partitions`,\n      matching Spark. Any change to the hash, the seed, or the modulo changes which rows land in\n      which partition, which breaks a join between a Comet-shuffled side and a Spark-shuffled side.\n- [ ] **Round robin defaults to a content hash, and is off by default.** Comet's default\n      `RoundRobinStrategy::HashAll` assigns partitions from a Murmur3 hash rather than cycling row\n      by row, because placement has to be reproducible across task retries. Two costs are accepted:\n      low-cardinality data distributes unevenly, and unsorted rows land in different partitions\n      than Spark's `UnsafeRow`-sorted assignment would put them, which is why\n      `spark.comet.shuffle.native.partitioning.roundrobin.enabled` defaults to `false`. Sorted\n      output is identical either way.\n- [ ] **Positional round robin is allowed, but its allowlist is the whole safety argument.**\n      `RoundRobinStrategy::RowGroups` is not a bug by construction. It is a function of row order\n      and never sorts, so it needs a retry to replay the same rows in the same order, which is\n      more than Spark's own round robin needs under its default `sortBeforeRepartition=true`.\n      `CometShuffleExchangeExec.replaysRowsInOrder` is what establishes that, and a PR that widens\n      it needs an argument that each operator it admits replays its rows in order, including\n      that the admitted expressions are deterministic; the RDD-level determinism check is defence\n      in depth and cannot fire under today's allowlist. Also check that placement still counts\n      rows rather than batches, that the start still comes from `positionalStartPartition`, and\n      that the group size is still resolved on the driver rather than from executor state. The reasoning behind all three is in `native_shuffle.md` under\n      \"Round Robin Partitioning\".\n- [ ] **Range partitioning bounds come from the driver.** Spark's `RangePartitioner` samples and\n      computes boundaries, they are serialized into the native plan, and native does a binary\n      search over comparable-row-format keys. A change to the comparison or the row encoding must\n      match Spark's ordering exactly, including nulls and signed zero.\n- [ ] The JVM path uses Spark's own partitioner via `partitioner.getPartition(key)`, so it inherits\n      Spark's semantics for free. A PR that reimplements partitioning on that path is solving a\n      problem that does not exist.\n\n## 3. On-Disk and On-Wire Format\n\nWriter and reader must change together, and they are in different languages.\n\nThe block layout is an 8-byte compressed length header, an 8-byte field count header, then the\ncompressed Arrow IPC stream. It is written by the native `ShuffleBlockWriter` and read by\n`NativeBatchDecoderIterator` calling `Native.decodeShuffleBlock()`.\n\n- [ ] A format change updates the writer, the reader, and the Celeborn reader path\n- [ ] A format change is not silently incompatible with shuffle files written by a previous version\n      in the same cluster during a rolling deployment. If it is, the PR needs to say so.\n- [ ] Compression codec changes apply uniformly to all partitions, and each partition stays\n      independently decompressible so reads can parallelize\n- [ ] The commit path still works. Native records the byte offset where each partition begins plus\n      the total length, `CometNativeShuffleWriter` fetches them with\n      `Native.getShufflePartitionOffsets`, converts them to partition lengths, and commits through\n      Spark's `IndexShuffleBlockResolver.writeMetadataFileAndCommit`. Offsets and lengths are easy\n      to confuse and the failure is a corrupt index file rather than an exception.\n- [ ] Checksums via `CometShuffleChecksumSupport` still cover what Spark expects\n\n## 4. Memory and Spilling\n\nShuffle is the largest memory consumer in most queries, and the two paths draw from different\nbudgets.\n\n**Native shuffle** uses the DataFusion memory pool. Partitions spill when the pool denies an\nallocation, or when buffered bytes reach `spark.comet.shuffle.native.maxBufferBytes`, which\ndefaults to `0`, meaning the fixed limit is disabled and memory pressure is the only trigger.\n\n**The local writer spills every partition into one file per task, not one file per partition.**\n`PartitionedSpill` (`native\u002Fshuffle\u002Fsrc\u002Fwriters\u002Flocal\u002Fspill.rs`) opens a single DataFusion spill\nfile lazily on the first spill and records, per partition, the byte ranges holding that partition's\nblocks in write order. `finish_partition` copies a partition's ranges into the output file in that\norder. `spilling_every_partition_creates_one_file` asserts the file count after spilling 64\npartitions three times. The single-partition writer does not spill, and neither does the RSS\nwriter, which pushes blocks to the remote service instead.\n\n**JVM shuffle** allocates off-heap pages through `CometShuffleMemoryAllocator`, which is an\nordinary Spark `MemoryConsumer` drawing from `spark.memory.offHeap.size`. `CometDiskBlockWriter`\ncoordinates spilling across partition writers, largest first. `TooLargePageException` is the signal\nthat a single record does not fit in a page.\n\n- [ ] A PR that adds buffering on either path says where the reservation is\n- [ ] A change to native spill bookkeeping keeps the recorded ranges and the file in sync.\n      `PartitionedSpill` sets a `failed` flag on a partial write because a mid-write failure makes\n      every later range untrustworthy, and a range that runs past the end of a truncated file must\n      fail the task rather than write a short partition.\n- [ ] A PR that changes spill thresholds has benchmark evidence, because spilling too late is an\n      OOM and spilling too early is a throughput loss\n- [ ] Scratch buffers reused for partition-id computation are correctly reset between batches\n- [ ] Use `review-comet-memory-pr` as well for anything touching reservations or the allocator\n\n## 5. Writer Selection on the JVM Path\n\n`CometShuffleManager.shouldBypassMergeSort()` picks between the two JVM writers. It uses bypass if\nthe partition count is below the threshold **and** partitions times cores is within the max thread\ncount, otherwise sort-based to avoid OOM from too many concurrent writers.\n\n| Handle                              | Writer                                                                        |\n| ----------------------------------- | ----------------------------------------------------------------------------- |\n| `CometBypassMergeSortShuffleHandle` | `CometBypassMergeSortShuffleWriter`, one `CometDiskBlockWriter` per partition |\n| `CometSerializedShuffleHandle`      | `CometUnsafeShuffleWriter`, via `CometShuffleExternalSorter`                  |\n| `CometNativeShuffleHandle`          | `CometNativeShuffleWriter`                                                    |\n\nA change to the selection heuristic changes the memory profile of every job with a large partition\ncount. Ask for the reasoning and the numbers.\n\n## 6. Source Layout\n\nNative shuffle lives in its own crate, `datafusion-comet-shuffle`, under `native\u002Fshuffle\u002F`, with\n`shuffle_writer.rs`, `comet_partitioning.rs`, `ipc.rs`, the `partitioners\u002F` and `writers\u002F` modules,\nand benchmarks in `native\u002Fshuffle\u002Fbenches\u002F`. The JVM side is split between\n`spark\u002Fsrc\u002Fmain\u002Fscala\u002Forg\u002Fapache\u002Fspark\u002Fsql\u002Fcomet\u002Fexecution\u002Fshuffle\u002F`, the Java writers under\n`spark\u002Fsrc\u002Fmain\u002Fjava\u002Forg\u002Fapache\u002Fspark\u002Fsql\u002Fcomet\u002Fexecution\u002Fshuffle\u002F`, and the allocators under\n`spark\u002Fsrc\u002Fmain\u002Fjava\u002Forg\u002Fapache\u002Fspark\u002Fshuffle\u002Fcomet\u002F`.\n\n**Verify the paths cited in the docs still exist.** The native shuffle code has moved between\ncrates before, and the \"Key Classes\" tables in the docs are exactly what goes stale when it does.\n\n## 7. Tests\n\n| Suite                                                               | Covers                                     |\n| ------------------------------------------------------------------- | ------------------------------------------ |\n| `org.apache.comet.exec.CometNativeShuffleSuite`                     | Native shuffle end to end                  |\n| `org.apache.comet.exec.CometColumnarShuffleSuite`                   | JVM columnar shuffle end to end            |\n| `CometShuffle4_0Suite`                                              | Spark 4.x specific behavior                |\n| `CometDiskBlockWriterSuite`                                         | JVM spill and page handling                |\n| `NativeBatchDecoderIteratorLifecycleChecks`, `...ConcurrencyChecks` | Reader lifetime and concurrency            |\n| `CometNativeShuffleInputRDDSuite`                                   | The scheduling-anchor RDD                  |\n| `CometCeleborn*Suite`                                               | The Celeborn path, which is easy to forget |\n| `CometShuffleBenchmark`                                             | Throughput                                 |\n\nAsk specifically:\n\n- [ ] Is the **other** shuffle path tested, if the change touched anything shared?\n- [ ] Is the Celeborn path tested, if the change touched the manager, dependency, writer, or reader?\n- [ ] Are multiple partitions and an actual spill exercised, rather than one small batch?\n- [ ] Are complex types covered as data columns, and is the partition-key restriction tested at its\n      boundary, on **both** settings of\n      `spark.comet.shuffle.native.partitioning.hash.nested.enabled`?\n- [ ] For a partitioning change, is there a test that the same input produces the same partition\n      assignment as Spark, not just that the total row count matches?\n\n## 8. Do the PR Changes Make the Shuffle Docs Stale?\n\nBoth docs are structured as tables of current fact, which is exactly what a refactor invalidates.\nCheck:\n\n- **`native_shuffle.md`**: the \"When Native Shuffle is Used\" conditions, the architecture diagram,\n  the Scala Side and Rust Side \"Key Classes\" tables including their file paths, the write-path and\n  read-path numbered steps, the Partitioning sections and the guarantees they state, the\n  Compression table, the Configuration table **with its defaults**, and the comparison table at the\n  end\n- **`jvm_shuffle.md`**: the \"When JVM Shuffle is Used\" list, the writer architecture diagram, the\n  Shuffle Manager, Handles, Writers and Reader tables, the bypass selection description, the Memory\n  Management section, and the Configuration table\n- A config default changed in `CometConf.scala` but left at its old value in either doc's config\n  table is a specific thing to look for, because those tables are hand-maintained while the\n  user-facing `configs.md` is generated\n- If the PR adds a partitioning scheme, both the native doc's Partitioning section and the\n  comparison table's \"Supported schemes\" row need it\n",{"name":5,"description":7,"argument-hint":43},"\u003Cpr-number>","main","7deeba21a42e4a9cd34f43e7ed6e960e4e851b26",".ai\u002Fskills\u002Freview-comet-shuffle-pr",418,"https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet","https:\u002F\u002Fdatafusion.apache.org\u002Fcomet","https:\u002F\u002Fgithub.com\u002Fapache","https:\u002F\u002Fwww.apache.org\u002F","2026-11-14 06:10:49",{"source":8,"skillName":5,"commands":54,"prompt":64,"zipAvailable":65},[55,58,61],{"tool":56,"command":57},"npx","npx skills add apache\u002Fdatafusion-comet --skill review-comet-shuffle-pr",{"tool":59,"command":60},"bunx","bunx skills add apache\u002Fdatafusion-comet --skill review-comet-shuffle-pr",{"tool":62,"command":63},"pnpm","pnpm dlx skills add apache\u002Fdatafusion-comet --skill review-comet-shuffle-pr","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-shuffle-pr\u002FSKILL.md",true,[67],{"path":68,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":71,"sourceHtmlUrl":72},"OWNERSHIP","OTHER","其他",20,"https:\u002F\u002Fgithub.com\u002FJetBrains\u002Fintellij-community\u002Fblob\u002Fmaster\u002F.agents\u002Fskills\u002Fpseudo-kmp\u002FOWNERSHIP",[74,104,128,145,156,171],{"skillNo":75,"name":76,"nameZh":6,"description":77,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":78,"categories":79,"keywords":85,"publishedAt":38},"01M6ACJR50ZS7F1QX58JEKCGVM","wire-datafusion-function","Use when wiring an existing DataFusion or datafusion-spark function into Comet for a Spark expression. Identifies the right wiring pattern (one-line passthrough, datafusion-spark UDF registration, or custom serde with input massaging \u002F restrictions), applies the Scala serde, registers the UDF in jni_api when needed, and adds SQL file tests. Assumes the function already exists upstream — if not, switch to `implement-comet-expression`.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fwire-datafusion-function\u002FSKILL.md",[80],{"categoryNo":81,"categoryName":82,"taxonomyType":83,"taxonomyTypeDesc":84},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",[34,86,87,88,89,90,91,92,93,94,95,96,97,30,98,22,99,100,101,102,103],"wire datafusion function","sql","scala","implement","spark","registration","switch","for","upstream","custom","expression","tests","Existing","file","api","wiring","with","name",{"skillNo":105,"name":106,"nameZh":6,"description":107,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":108,"categories":109,"keywords":110,"publishedAt":38},"01M6ACJPDDPND6FHE2PC1CJ4D3","suggest-native-expression","Use when picking the next Spark expression to implement natively in Comet, or when asked whether a specific codegen-dispatched expression is worth a native Rust implementation. Scores the candidate on how likely a native path can be 100% Spark-compatible and how much throughput or allocation it would save versus the JVM codegen dispatcher, records the verdict in the per-expression audit log so disqualified candidates are not re-litigated, and files a GitHub issue for the recommendation.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fsuggest-native-expression\u002FSKILL.md",[],[111,96,28,112,113,114,89,90,115,116,22,117,118,29,119,120,121,122,123,124,125,126,127],"suggest native expression","dispatcher","github","rust","re","files","candidate","log","next","how","audit","audit log","allocation","issue","throughput","path","save",{"skillNo":129,"name":130,"nameZh":6,"description":131,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":132,"categories":133,"keywords":134,"publishedAt":38},"01M6ACJHX8BES7ZNNRNVCQ4QWE","review-comet-pr","Use when reviewing a DataFusion Comet pull request. Covers the workflow that applies to every PR and routes to the area-specific review skills for expressions, FFI, memory management, shuffle, Iceberg writes, and documentation. Provides guidance to a human reviewer rather than posting comments.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-pr\u002FSKILL.md",[],[19,20,135,36,136,37,137,34,138,139,22,26,140,141,23,142,93,30,143,144],"review comet pr","iceberg","memory","workflow","documentation","reviewer","human","posting","review skills","comments",{"skillNo":146,"name":147,"nameZh":6,"description":148,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":149,"categories":150,"keywords":151,"publishedAt":38},"01M6ACJFM0QS61Z1RW516HXN2M","review-comet-memory-pr","Use when reviewing a DataFusion Comet pull request that touches memory pools, memory reservations, spilling, the Spark TaskMemoryManager bridge, allocators, or memory-related configuration. Load alongside review-comet-pr.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-memory-pr\u002FSKILL.md",[],[19,137,152,20,36,37,34,90,153,154,30,24,22,26,23,155],"review comet memory pr","bridge","reservations","configuration",{"skillNo":157,"name":158,"nameZh":6,"description":159,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":160,"categories":161,"keywords":162,"publishedAt":38},"01M6ACJCV64V0FZPMVXDE2K3DX","review-comet-iceberg-write-pr","Use when reviewing a DataFusion Comet pull request that touches Iceberg writes, the split-operator write plan (IcebergWriteStrategy, IcebergWriteExec, IcebergCommitExec), the native Iceberg writer (CometIcebergWriteExec, iceberg_write.rs, iceberg_partition_path.rs), its eligibility gate (CometIcebergNativeWrite), or the iceberg-rust pin. Load alongside review-comet-pr.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-iceberg-write-pr\u002FSKILL.md",[],[19,136,163,20,164,36,37,34,114,165,166,167,30,24,168,22,26,169,170,28],"review comet iceberg write pr","write","writer","write plan","partition","pin","split","plan",{"skillNo":172,"name":173,"nameZh":6,"description":174,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":175,"categories":176,"keywords":177,"publishedAt":38},"01M6ACJA72KJRAM6F7WHMX01PN","review-comet-ffi-pr","Use when reviewing a DataFusion Comet pull request that crosses the JVM\u002Fnative boundary, touching Arrow C Data or C Stream interface code, batch export and import, CometExecIterator, ScanExec, NativeUtil, CometVector subclasses, or jni_api. Load alongside review-comet-pr.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-ffi-pr\u002FSKILL.md",[],[19,178,20,179,36,37,34,35,180,181,182,183,30,24,22,26,100,29,184,28,23],"review comet ffi pr","ffi","code","batch","import","interface","boundary"]