[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6ACJCV64V0FZPMVXDE2K3DX":3,"skill-same:01M6ACJCV64V0FZPMVXDE2K3DX":74},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":45,"commitSha":46,"skillDirectory":47,"forksCount":48,"repositoryHtmlUrl":49,"repositoryHomepageUrl":50,"ownerHtmlUrl":51,"ownerBlogUrl":52,"ownerDescription":6,"updatedAt":53,"install":54,"resources":67},"01M6ACJCV64V0FZPMVXDE2K3DX","review-comet-iceberg-write-pr",null,"Use when reviewing a DataFusion Comet pull request that touches Iceberg writes, the split-operator write plan (IcebergWriteStrategy, IcebergWriteExec, IcebergCommitExec), the native Iceberg writer (CometIcebergWriteExec, iceberg_write.rs, iceberg_partition_path.rs), its eligibility gate (CometIcebergNativeWrite), or the iceberg-rust pin. Load alongside review-comet-pr.","apache\u002Fdatafusion-comet","apache","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F47359?v=4","01M6AC83E6K4FDKK7HZ8WNMQKE","The Apache Software Foundation","01M6ACHX42TPNCR8TQJEAR8J5J",1322,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-iceberg-write-pr\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"review","pr","iceberg","review comet iceberg write pr","write","when","load","number","split","native","write plan","writer","use","partition","pin","plan","datafusion","pull","pull request","rust","2026-11-10 10:12:30",false,"---\nname: review-comet-iceberg-write-pr\ndescription: Use when reviewing a DataFusion Comet pull request that touches Iceberg writes, the split-operator write plan (IcebergWriteStrategy, IcebergWriteExec, IcebergCommitExec), the native Iceberg writer (CometIcebergWriteExec, iceberg_write.rs, iceberg_partition_path.rs), its eligibility gate (CometIcebergNativeWrite), or the iceberg-rust pin. Load alongside review-comet-pr.\nargument-hint: \u003Cpr-number>\n---\n\n\u003C!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n  http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied.  See the License for the\nspecific language governing permissions and limitations\nunder the License.\n-->\n\nIceberg-write-specific review for Comet PR #$ARGUMENTS.\n\n**REQUIRED BACKGROUND:** Use `review-comet-pr` for PR metadata, existing comments, CI, the review\nbar, and the output format. This skill only covers Iceberg writes.\n\n## Read the Contributor Guide First\n\n| Doc                                                    | What you need from it                                                                   |\n| ------------------------------------------------------ | --------------------------------------------------------------------------------------- |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Ficeberg-writes.md`      | The two layers, the gate's properties, the wire format both ways, cleanup ownership     |\n| `docs\u002Fsource\u002Fuser-guide\u002Flatest\u002Ficeberg.md` (Writes)    | The eligibility table and the accepted divergences from iceberg-java, as users see them |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Ficeberg-spark-tests.md` | What the Iceberg Spark test diffs enable, and what a green run does and does not prove  |\n\nThe organizing rule is that **the native writer produces what iceberg-java would have produced, or\ndeclines at plan time.** Review every change against it. Iceberg write bugs are quiet: a divergent\ndata file or manifest entry commits successfully, and every later reader of the table, Comet or\nnot, inherits it. There is no query that fails to tell you.\n\n## 1. Which Layer\n\n| Layer                | Code                                                                                                                         |\n| -------------------- | ---------------------------------------------------------------------------------------------------------------------------- |\n| Split-operator plan  | `IcebergWriteStrategy`, `IcebergWriteLogical`, `IcebergWriteExec`, `IcebergCommitExec`, the `spark-*\u002F...\u002Ficeberg\u002F` shims     |\n| Native writer        | `CometIcebergNativeWrite` (gate and serde), `IcebergWriteProtoTranslation`, `CometIcebergWriteExec`, `iceberg_write.rs`      |\n| Shared with the scan | `iceberg_common.rs` (`load_file_io`, `storage_factory_for`, `scheme_of`), `IcebergReflection`, `NativeConfig` S3 translation |\n\n`spark.comet.write.iceberg.enabled` switches both write layers. It is off by default, but once a\nuser turns it on, a change to the split plan affects every append, overwrite and copy-on-write\nIceberg write, including writes that never reach the native writer, and a change to the native\nwriter affects every eligible one. The testing-only\n`spark.comet.write.iceberg.splitOperator.enabled` plans the split operator with the native writer\noff. A change to a shared file affects the native Iceberg scan too.\n\n## 2. Direction of the Gate Change\n\nEstablish first whether the PR widens or narrows what `CometIcebergNativeWrite.getSupportLevel`\nadmits. The two need different evidence.\n\n- **Narrowing** (a new rule, a smaller allow list) costs performance only. Check that the rule\n  reports a specific reason and has a `CometIcebergWriteDetectionSuite` case.\n- **Widening** (lifting a rule, vetting a new property, a new scheme or type) means Comet now claims\n  writes it previously refused. \"The tests pass\" is not evidence here, because the existing tests\n  were written against the narrower gate. Ask for a test that writes the newly admitted\n  configuration through both writers and compares the result, and for the user guide's eligibility\n  table to change in the same PR.\n\nFor every gate change:\n\n- [ ] **It stays an allow list.** New support is added by vetting a key or value, not by widening a\n      prefix or turning a deny list around. Unknown keys in `write.parquet.*`, `parquet.*`, the\n      session Hadoop `parquet.*` keys, and any other namespace that reaches the writer must still\n      decline.\n- [ ] **It reads the effective configuration** (`TriggerContext.properties`, table properties\n      overlaid with `SparkWrite.writeProperties`), not the table properties alone.\n- [ ] **It fails closed.** A reflection lookup that cannot answer returns a reason. Look for\n      `getOrElse(None)`-shaped code, or a `catch` that returns `None`, in a new rule.\n- [ ] **It agrees with the native side.** If the gate and the native code both interpret the same\n      input (a location's scheme, a partition spec, a type, a property value), check that they\n      interpret it identically on the edge cases: case, `scheme:\u002Fpath` versus `scheme:\u002F\u002Fpath`,\n      whitespace, `void` partition fields, missing source columns. A gate more permissive than the\n      native code turns a fallback into a failed query\n      ([#6140](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F6140)).\n- [ ] **Settings the native writer cannot honour decline.** A table, `FileIO` or Hadoop setting\n      that iceberg-java would apply and the native writer ignores produces a successful write that\n      silently differs: credentials, encryption, ACLs, tags, storage class\n      ([#6139](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F6139)). The `gs:\u002F\u002F` path is the\n      model ([#5637](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F5637)).\n- [ ] **New executor-side reflection is probed on the driver.** Any iceberg-java member the task\n      closure or `IcebergReflection`'s commit-message assembly now calls must be added to what\n      `requireExecutorReflectionResolvable` resolves, so a missing member is a plan-time fallback.\n\n## 3. Output Parity with iceberg-java\n\nFor a change to the native writer, the metadata rebuild, or the pin, ask what iceberg-java writes\nfor the same input, and check it at the pinned Iceberg versions (1.5.2, 1.8.1, 1.10.0, 1.11.0), not\nonly the latest.\n\n- [ ] **Logical content and manifest metadata match**: rows, partition values, record counts,\n      metrics, spec id, sort order id. A difference here is a bug or a gate rule, never a new\n      documented divergence.\n- [ ] **Physical differences are justified.** A new file-level difference (encoding, footer\n      metadata, roll points, file names) is added to the user guide's accepted-divergences list with\n      the reason no reader can observe it.\n- [ ] **Metrics stay iceberg-java's.** `rebuildDataFilesWithJavaMetrics` re-derives manifest\n      metrics from each file's footer with `ParquetUtil.footerMetrics` and `MetricsConfig`. A PR\n      that starts trusting iceberg-rust's metrics, or adds a native-tracked value beyond float and\n      double NaN counts and bounds, needs a strong reason.\n- [ ] **Value semantics match, not only formats.** iceberg-rust compares and hashes partition values\n      with its own rules: until apache\u002Ficeberg-rust#3327 its `OrderedFloat` equality treated\n      `-0.0` and `0.0` as equal where Java's `Float.compare` does not\n      ([#6138](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F6138)). Check equality, hashing,\n      ordering and rendering for float, double, timestamp, timestamptz, binary and decimal.\n- [ ] **Timestamp partition values are UTC.** Iceberg's `years`, `months`, `days` and `hours`\n      ignore the session timezone. For date and timestamp sources, `PartitionValueCalculator`\n      (`iceberg_partition_value.rs`) computes them with the same kernels in\n      `iceberg_funcs\u002Ftemporal.rs` that key the sort in front of a clustered write, and those follow\n      iceberg-java's `DateTimeUtil`, including its pre-1970 rounding. A change that moves one of\n      these transforms back to iceberg-rust, or that makes the kernels read the column's timezone\n      label, breaks that agreement. iceberg-rust's `years` and `months` follow the label\n      ([apache\u002Ficeberg-rust#3142](https:\u002F\u002Fgithub.com\u002Fapache\u002Ficeberg-rust\u002Fissues\u002F3142)). Ask for a\n      test in a non-UTC session for a change here. \"Partition values\" in the Writes section of the user-guide `iceberg.md` lists\n      where iceberg-rust differs, and `timezones.md` covers how Comet labels timestamps.\n- [ ] **Partition paths use the Java renderers.** Directory names come from\n      `CometLocationGenerator` \u002F `partition_to_path` in `iceberg_partition_path.rs`, which follow\n      iceberg-java's `partitionToPath`, with `java_float_string` for floats. A PR that calls\n      iceberg-rust's `PartitionKey::to_path` or `format!` on a value reintroduces a fixed bug.\n- [ ] **Nothing nondeterministic reaches the output.** Map iteration order leaks into file names,\n      manifest order and the row order of unordered reads. The fanout arm sorts its `DataFile`s by\n      path for this reason ([#5776](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F5776)).\n- [ ] **Casts stay strict.** `decorate_batch_with_field_ids` casts with `safe: false`, so a type\n      mismatch fails the task. A safe cast would write NULLs.\n\n## 4. Failure Handling and Cleanup\n\nCleanup must never have an ownership gap. During the handoff, native remains armed until the JVM\nhas taken the locations and acknowledged that by polling EOF, so a brief overlap is intentional.\nRead the ownership table in the contributor guide before reviewing any change near `AbortOnDrop`,\n`TrackingLocationGenerator`, `WrittenFileCleanup`, `drainNativePayload` or\n`IcebergCommitExec.collectAndCommit`.\n\n- [ ] A new failure point between writing a file and the JVM taking the locations is covered by the\n      native guard, including the path where the plan is dropped mid-write rather than returning an\n      error.\n- [ ] The locations column is still read, and ownership taken, **before** the manifest is decoded,\n      so a decode failure still cleans up.\n- [ ] Cleanup is best effort and never replaces the original exception (`addSuppressed`, logging).\n- [ ] File names still carry the task attempt id, so a retried or speculative attempt cannot reuse\n      another attempt's names.\n- [ ] A job failure still aborts with the completed tasks' messages and deletes their files, and\n      the commit path still hands Iceberg genuine `SparkWrite$TaskCommit` messages, so\n      `SparkWrite.abort` keeps working.\n- [ ] The user-visible exception type is still what Spark's own write path throws on each Spark\n      version ([#6143](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F6143)).\n- [ ] **Every open file and every held-back row is reserved.** Files must be opened through\n      `MeteredParquetWriterBuilder`, and rows the writer holds outside iceberg-rust (the\n      `PartitionFeed`s, or anything a new feed holds back) must be added to what `run_write_task` reserves. A\n      builder that constructs `ParquetWriterBuilder` directly leaves its files out of the task's\n      memory reservation, so a wide fanout write grows past the pool instead of closing\n      partitions early or failing its task. A fanout file must also report to its partition's\n      `OpenFileMemory`, or the partitions holding the most are not the ones closed.\n      A storage scheme newly supported for writes needs its entry in\n      `StorageWrites::for_location`: a file reports its flushed row groups until its storage has\n      written them out, which a local file does at once and an object store only part by part.\n\n## 5. Plan Shape\n\n- [ ] **AQE off as well as on.** Transition and plan-shape changes must be tested with\n      `spark.sql.adaptive.enabled=false`. Per-stage transition insertion with AQE on has hidden a\n      real bug ([#5689](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F5689)), and Iceberg's own\n      extension tests pick AQE at random per session, so an AQE-dependent bug shows up upstream as\n      intermittent.\n- [ ] **Transitions stay stripped the way they are.** `CometIcebergWriteExec` must not become a\n      `ColumnarToRowTransition`, and the transition beneath it is removed in the separate pass in\n      `EliminateRedundantTransitions`, before the main `transformUp`.\n- [ ] **Exactly one commit.** The committer relies on `V2CommandExec` memoizing `run()` and on\n      `IcebergWriteLogical` anchoring re-plans. A change here needs the AQE re-plan test to still\n      pass.\n- [ ] **Rules that rewrite plans keep the write node.** Anything that restores Spark operators from\n      a Comet node's `originalPlan` must handle the write execs\n      ([#5719](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F5719)).\n- [ ] **The kill switch still works.** Code on the write path respects `spark.comet.enabled`\n      ([#6142](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F6142)).\n- [ ] **Credentials stay out of plan strings.** `CometIcebergWriteExec.stringArgs` must not print\n      the proto; `catalog_properties` carries translated `fs.s3a.*` secrets.\n\n## 6. The iceberg-rust Pin\n\n`native\u002FCargo.toml` pins `iceberg` and `iceberg-storage-opendal` to a git revision.\n\n- [ ] A pin bump is reviewed as a writer change: what changed upstream in the writers, `DataFile`,\n      manifests, partition handling and storage, and does the PR run the write suites and the\n      Iceberg Spark tests?\n- [ ] Code that depends on iceberg-rust behaviour that is not an API, such as\n      `clustered_write_err` matching the clustered writer's message text, still has the test that\n      fails when upstream changes it.\n- [ ] Nothing starts depending on a personal fork. A PR that needs an unmerged iceberg-rust change\n      says so and is not merged until the pin moves to an upstream revision.\n\n## 7. Tests\n\n| Suite                               | Covers                                                                                          |\n| ----------------------------------- | ----------------------------------------------------------------------------------------------- |\n| `CometIcebergWriteActionSuite`      | Both layers end to end, parity with iceberg-java, DML, failure cleanup, AQE re-plans            |\n| `CometIcebergWriteDetectionSuite`   | Most eligibility rules, accepted and declined                                                   |\n| `CometIcebergSystemFunctionSuite`   | Native partition transforms keeping partitioned writes native                                   |\n| `CometIcebergRewriteActionSuite`    | `rewrite_data_files` through the split plan and the native writer                               |\n| `IcebergWriteProtoTranslationSuite` | Property to `IcebergParquetWriteSettings` translation                                           |\n| Rust tests in `iceberg_write.rs`    | Rolling, fanout order, clustered checks, cleanup guard, manifest round trip, memory reservation |\n| `iceberg_partition_path.rs` tests   | Partition path rendering against iceberg-java                                                   |\n| `CometIcebergWriteBenchmark`        | Native versus iceberg-java throughput                                                           |\n| `CometIcebergEndToEndBenchmark`     | Native read, shuffle and write against stock Spark, primitive and nested columns                |\n\nAsk specifically:\n\n- [ ] **Does the test prove the native writer ran?** Every fallback is silent to the query. A test\n      that only compares results passes when both sides used iceberg-java. Look for\n      `assertNativeWriteEngages` or a collected `CometIcebergWriteExec`.\n- [ ] **Is the input native?** `withNativeEnabled` also sets `localTableScan`; a test that sets\n      `spark.comet.write.iceberg.enabled` by hand and inserts `VALUES` usually tests the JVM writer.\n- [ ] **Is the comparison against iceberg-java** (a sibling table written by the JVM writer) rather\n      than hand-written expected values?\n- [ ] **Enough partitions for an ordering bug?** Two partitions pass half the time; use eight or\n      more.\n- [ ] **Does a failure test check storage state**, the files under the data location against those\n      the manifests reference, and not only the table contents?\n- [ ] **Does it run on each Iceberg version the behaviour depends on?** Pull requests run only the\n      default Spark profile. Reflection, metrics conventions and path spelling differ across\n      Iceberg versions, so ask for the `run-iceberg-tests` label or a local run where they matter.\n- [ ] **Is a green Iceberg Spark test run being cited as evidence?** It does not show which writer\n      handled a test. The job summary counts writes per writer without naming tests (see \"Which\n      writer ran each Iceberg write\" in `contributor-guide\u002Ficeberg-spark-tests.md`).\n\n## 8. Do the PR Changes Make the Docs Stale?\n\n- **`contributor-guide\u002Ficeberg-writes.md`**: the component table and its paths, the gate\n  properties, the proto description, the native adaptations list, the payload columns and the JVM\n  steps after them, the cleanup ownership table, the test table, and the Iceberg version per Spark\n  profile\n- **`user-guide\u002Flatest\u002Ficeberg.md`, Writes section**: the eligibility table, the \"falls back\" lists, the\n  failure-handling section, and the accepted-divergences list. A PR that lifts or adds a rule, or\n  changes a divergence, must update these. A divergence the code has since fixed must be removed.\n- **`user-guide\u002Flatest\u002Ficeberg.md`, Native Reader section**: its unsupported-features list mentions writes. The\n  Known Issues section lists the open write issues\n- **`contributor-guide\u002Ficeberg-spark-tests.md`**: the flags the diffs set\n- `CometConf.scala` doc strings for the two flags, when defaults or scope change\n","\u003C!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n  http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied.  See the License for the\nspecific language governing permissions and limitations\nunder the License.\n-->\n\nIceberg-write-specific review for Comet PR #$ARGUMENTS.\n\n**REQUIRED BACKGROUND:** Use `review-comet-pr` for PR metadata, existing comments, CI, the review\nbar, and the output format. This skill only covers Iceberg writes.\n\n## Read the Contributor Guide First\n\n| Doc                                                    | What you need from it                                                                   |\n| ------------------------------------------------------ | --------------------------------------------------------------------------------------- |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Ficeberg-writes.md`      | The two layers, the gate's properties, the wire format both ways, cleanup ownership     |\n| `docs\u002Fsource\u002Fuser-guide\u002Flatest\u002Ficeberg.md` (Writes)    | The eligibility table and the accepted divergences from iceberg-java, as users see them |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Ficeberg-spark-tests.md` | What the Iceberg Spark test diffs enable, and what a green run does and does not prove  |\n\nThe organizing rule is that **the native writer produces what iceberg-java would have produced, or\ndeclines at plan time.** Review every change against it. Iceberg write bugs are quiet: a divergent\ndata file or manifest entry commits successfully, and every later reader of the table, Comet or\nnot, inherits it. There is no query that fails to tell you.\n\n## 1. Which Layer\n\n| Layer                | Code                                                                                                                         |\n| -------------------- | ---------------------------------------------------------------------------------------------------------------------------- |\n| Split-operator plan  | `IcebergWriteStrategy`, `IcebergWriteLogical`, `IcebergWriteExec`, `IcebergCommitExec`, the `spark-*\u002F...\u002Ficeberg\u002F` shims     |\n| Native writer        | `CometIcebergNativeWrite` (gate and serde), `IcebergWriteProtoTranslation`, `CometIcebergWriteExec`, `iceberg_write.rs`      |\n| Shared with the scan | `iceberg_common.rs` (`load_file_io`, `storage_factory_for`, `scheme_of`), `IcebergReflection`, `NativeConfig` S3 translation |\n\n`spark.comet.write.iceberg.enabled` switches both write layers. It is off by default, but once a\nuser turns it on, a change to the split plan affects every append, overwrite and copy-on-write\nIceberg write, including writes that never reach the native writer, and a change to the native\nwriter affects every eligible one. The testing-only\n`spark.comet.write.iceberg.splitOperator.enabled` plans the split operator with the native writer\noff. A change to a shared file affects the native Iceberg scan too.\n\n## 2. Direction of the Gate Change\n\nEstablish first whether the PR widens or narrows what `CometIcebergNativeWrite.getSupportLevel`\nadmits. The two need different evidence.\n\n- **Narrowing** (a new rule, a smaller allow list) costs performance only. Check that the rule\n  reports a specific reason and has a `CometIcebergWriteDetectionSuite` case.\n- **Widening** (lifting a rule, vetting a new property, a new scheme or type) means Comet now claims\n  writes it previously refused. \"The tests pass\" is not evidence here, because the existing tests\n  were written against the narrower gate. Ask for a test that writes the newly admitted\n  configuration through both writers and compares the result, and for the user guide's eligibility\n  table to change in the same PR.\n\nFor every gate change:\n\n- [ ] **It stays an allow list.** New support is added by vetting a key or value, not by widening a\n      prefix or turning a deny list around. Unknown keys in `write.parquet.*`, `parquet.*`, the\n      session Hadoop `parquet.*` keys, and any other namespace that reaches the writer must still\n      decline.\n- [ ] **It reads the effective configuration** (`TriggerContext.properties`, table properties\n      overlaid with `SparkWrite.writeProperties`), not the table properties alone.\n- [ ] **It fails closed.** A reflection lookup that cannot answer returns a reason. Look for\n      `getOrElse(None)`-shaped code, or a `catch` that returns `None`, in a new rule.\n- [ ] **It agrees with the native side.** If the gate and the native code both interpret the same\n      input (a location's scheme, a partition spec, a type, a property value), check that they\n      interpret it identically on the edge cases: case, `scheme:\u002Fpath` versus `scheme:\u002F\u002Fpath`,\n      whitespace, `void` partition fields, missing source columns. A gate more permissive than the\n      native code turns a fallback into a failed query\n      ([#6140](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F6140)).\n- [ ] **Settings the native writer cannot honour decline.** A table, `FileIO` or Hadoop setting\n      that iceberg-java would apply and the native writer ignores produces a successful write that\n      silently differs: credentials, encryption, ACLs, tags, storage class\n      ([#6139](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F6139)). The `gs:\u002F\u002F` path is the\n      model ([#5637](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F5637)).\n- [ ] **New executor-side reflection is probed on the driver.** Any iceberg-java member the task\n      closure or `IcebergReflection`'s commit-message assembly now calls must be added to what\n      `requireExecutorReflectionResolvable` resolves, so a missing member is a plan-time fallback.\n\n## 3. Output Parity with iceberg-java\n\nFor a change to the native writer, the metadata rebuild, or the pin, ask what iceberg-java writes\nfor the same input, and check it at the pinned Iceberg versions (1.5.2, 1.8.1, 1.10.0, 1.11.0), not\nonly the latest.\n\n- [ ] **Logical content and manifest metadata match**: rows, partition values, record counts,\n      metrics, spec id, sort order id. A difference here is a bug or a gate rule, never a new\n      documented divergence.\n- [ ] **Physical differences are justified.** A new file-level difference (encoding, footer\n      metadata, roll points, file names) is added to the user guide's accepted-divergences list with\n      the reason no reader can observe it.\n- [ ] **Metrics stay iceberg-java's.** `rebuildDataFilesWithJavaMetrics` re-derives manifest\n      metrics from each file's footer with `ParquetUtil.footerMetrics` and `MetricsConfig`. A PR\n      that starts trusting iceberg-rust's metrics, or adds a native-tracked value beyond float and\n      double NaN counts and bounds, needs a strong reason.\n- [ ] **Value semantics match, not only formats.** iceberg-rust compares and hashes partition values\n      with its own rules: until apache\u002Ficeberg-rust#3327 its `OrderedFloat` equality treated\n      `-0.0` and `0.0` as equal where Java's `Float.compare` does not\n      ([#6138](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F6138)). Check equality, hashing,\n      ordering and rendering for float, double, timestamp, timestamptz, binary and decimal.\n- [ ] **Timestamp partition values are UTC.** Iceberg's `years`, `months`, `days` and `hours`\n      ignore the session timezone. For date and timestamp sources, `PartitionValueCalculator`\n      (`iceberg_partition_value.rs`) computes them with the same kernels in\n      `iceberg_funcs\u002Ftemporal.rs` that key the sort in front of a clustered write, and those follow\n      iceberg-java's `DateTimeUtil`, including its pre-1970 rounding. A change that moves one of\n      these transforms back to iceberg-rust, or that makes the kernels read the column's timezone\n      label, breaks that agreement. iceberg-rust's `years` and `months` follow the label\n      ([apache\u002Ficeberg-rust#3142](https:\u002F\u002Fgithub.com\u002Fapache\u002Ficeberg-rust\u002Fissues\u002F3142)). Ask for a\n      test in a non-UTC session for a change here. \"Partition values\" in the Writes section of the user-guide `iceberg.md` lists\n      where iceberg-rust differs, and `timezones.md` covers how Comet labels timestamps.\n- [ ] **Partition paths use the Java renderers.** Directory names come from\n      `CometLocationGenerator` \u002F `partition_to_path` in `iceberg_partition_path.rs`, which follow\n      iceberg-java's `partitionToPath`, with `java_float_string` for floats. A PR that calls\n      iceberg-rust's `PartitionKey::to_path` or `format!` on a value reintroduces a fixed bug.\n- [ ] **Nothing nondeterministic reaches the output.** Map iteration order leaks into file names,\n      manifest order and the row order of unordered reads. The fanout arm sorts its `DataFile`s by\n      path for this reason ([#5776](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F5776)).\n- [ ] **Casts stay strict.** `decorate_batch_with_field_ids` casts with `safe: false`, so a type\n      mismatch fails the task. A safe cast would write NULLs.\n\n## 4. Failure Handling and Cleanup\n\nCleanup must never have an ownership gap. During the handoff, native remains armed until the JVM\nhas taken the locations and acknowledged that by polling EOF, so a brief overlap is intentional.\nRead the ownership table in the contributor guide before reviewing any change near `AbortOnDrop`,\n`TrackingLocationGenerator`, `WrittenFileCleanup`, `drainNativePayload` or\n`IcebergCommitExec.collectAndCommit`.\n\n- [ ] A new failure point between writing a file and the JVM taking the locations is covered by the\n      native guard, including the path where the plan is dropped mid-write rather than returning an\n      error.\n- [ ] The locations column is still read, and ownership taken, **before** the manifest is decoded,\n      so a decode failure still cleans up.\n- [ ] Cleanup is best effort and never replaces the original exception (`addSuppressed`, logging).\n- [ ] File names still carry the task attempt id, so a retried or speculative attempt cannot reuse\n      another attempt's names.\n- [ ] A job failure still aborts with the completed tasks' messages and deletes their files, and\n      the commit path still hands Iceberg genuine `SparkWrite$TaskCommit` messages, so\n      `SparkWrite.abort` keeps working.\n- [ ] The user-visible exception type is still what Spark's own write path throws on each Spark\n      version ([#6143](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F6143)).\n- [ ] **Every open file and every held-back row is reserved.** Files must be opened through\n      `MeteredParquetWriterBuilder`, and rows the writer holds outside iceberg-rust (the\n      `PartitionFeed`s, or anything a new feed holds back) must be added to what `run_write_task` reserves. A\n      builder that constructs `ParquetWriterBuilder` directly leaves its files out of the task's\n      memory reservation, so a wide fanout write grows past the pool instead of closing\n      partitions early or failing its task. A fanout file must also report to its partition's\n      `OpenFileMemory`, or the partitions holding the most are not the ones closed.\n      A storage scheme newly supported for writes needs its entry in\n      `StorageWrites::for_location`: a file reports its flushed row groups until its storage has\n      written them out, which a local file does at once and an object store only part by part.\n\n## 5. Plan Shape\n\n- [ ] **AQE off as well as on.** Transition and plan-shape changes must be tested with\n      `spark.sql.adaptive.enabled=false`. Per-stage transition insertion with AQE on has hidden a\n      real bug ([#5689](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F5689)), and Iceberg's own\n      extension tests pick AQE at random per session, so an AQE-dependent bug shows up upstream as\n      intermittent.\n- [ ] **Transitions stay stripped the way they are.** `CometIcebergWriteExec` must not become a\n      `ColumnarToRowTransition`, and the transition beneath it is removed in the separate pass in\n      `EliminateRedundantTransitions`, before the main `transformUp`.\n- [ ] **Exactly one commit.** The committer relies on `V2CommandExec` memoizing `run()` and on\n      `IcebergWriteLogical` anchoring re-plans. A change here needs the AQE re-plan test to still\n      pass.\n- [ ] **Rules that rewrite plans keep the write node.** Anything that restores Spark operators from\n      a Comet node's `originalPlan` must handle the write execs\n      ([#5719](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F5719)).\n- [ ] **The kill switch still works.** Code on the write path respects `spark.comet.enabled`\n      ([#6142](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F6142)).\n- [ ] **Credentials stay out of plan strings.** `CometIcebergWriteExec.stringArgs` must not print\n      the proto; `catalog_properties` carries translated `fs.s3a.*` secrets.\n\n## 6. The iceberg-rust Pin\n\n`native\u002FCargo.toml` pins `iceberg` and `iceberg-storage-opendal` to a git revision.\n\n- [ ] A pin bump is reviewed as a writer change: what changed upstream in the writers, `DataFile`,\n      manifests, partition handling and storage, and does the PR run the write suites and the\n      Iceberg Spark tests?\n- [ ] Code that depends on iceberg-rust behaviour that is not an API, such as\n      `clustered_write_err` matching the clustered writer's message text, still has the test that\n      fails when upstream changes it.\n- [ ] Nothing starts depending on a personal fork. A PR that needs an unmerged iceberg-rust change\n      says so and is not merged until the pin moves to an upstream revision.\n\n## 7. Tests\n\n| Suite                               | Covers                                                                                          |\n| ----------------------------------- | ----------------------------------------------------------------------------------------------- |\n| `CometIcebergWriteActionSuite`      | Both layers end to end, parity with iceberg-java, DML, failure cleanup, AQE re-plans            |\n| `CometIcebergWriteDetectionSuite`   | Most eligibility rules, accepted and declined                                                   |\n| `CometIcebergSystemFunctionSuite`   | Native partition transforms keeping partitioned writes native                                   |\n| `CometIcebergRewriteActionSuite`    | `rewrite_data_files` through the split plan and the native writer                               |\n| `IcebergWriteProtoTranslationSuite` | Property to `IcebergParquetWriteSettings` translation                                           |\n| Rust tests in `iceberg_write.rs`    | Rolling, fanout order, clustered checks, cleanup guard, manifest round trip, memory reservation |\n| `iceberg_partition_path.rs` tests   | Partition path rendering against iceberg-java                                                   |\n| `CometIcebergWriteBenchmark`        | Native versus iceberg-java throughput                                                           |\n| `CometIcebergEndToEndBenchmark`     | Native read, shuffle and write against stock Spark, primitive and nested columns                |\n\nAsk specifically:\n\n- [ ] **Does the test prove the native writer ran?** Every fallback is silent to the query. A test\n      that only compares results passes when both sides used iceberg-java. Look for\n      `assertNativeWriteEngages` or a collected `CometIcebergWriteExec`.\n- [ ] **Is the input native?** `withNativeEnabled` also sets `localTableScan`; a test that sets\n      `spark.comet.write.iceberg.enabled` by hand and inserts `VALUES` usually tests the JVM writer.\n- [ ] **Is the comparison against iceberg-java** (a sibling table written by the JVM writer) rather\n      than hand-written expected values?\n- [ ] **Enough partitions for an ordering bug?** Two partitions pass half the time; use eight or\n      more.\n- [ ] **Does a failure test check storage state**, the files under the data location against those\n      the manifests reference, and not only the table contents?\n- [ ] **Does it run on each Iceberg version the behaviour depends on?** Pull requests run only the\n      default Spark profile. Reflection, metrics conventions and path spelling differ across\n      Iceberg versions, so ask for the `run-iceberg-tests` label or a local run where they matter.\n- [ ] **Is a green Iceberg Spark test run being cited as evidence?** It does not show which writer\n      handled a test. The job summary counts writes per writer without naming tests (see \"Which\n      writer ran each Iceberg write\" in `contributor-guide\u002Ficeberg-spark-tests.md`).\n\n## 8. Do the PR Changes Make the Docs Stale?\n\n- **`contributor-guide\u002Ficeberg-writes.md`**: the component table and its paths, the gate\n  properties, the proto description, the native adaptations list, the payload columns and the JVM\n  steps after them, the cleanup ownership table, the test table, and the Iceberg version per Spark\n  profile\n- **`user-guide\u002Flatest\u002Ficeberg.md`, Writes section**: the eligibility table, the \"falls back\" lists, the\n  failure-handling section, and the accepted-divergences list. A PR that lifts or adds a rule, or\n  changes a divergence, must update these. A divergence the code has since fixed must be removed.\n- **`user-guide\u002Flatest\u002Ficeberg.md`, Native Reader section**: its unsupported-features list mentions writes. The\n  Known Issues section lists the open write issues\n- **`contributor-guide\u002Ficeberg-spark-tests.md`**: the flags the diffs set\n- `CometConf.scala` doc strings for the two flags, when defaults or scope change\n",{"name":5,"description":7,"argument-hint":44},"\u003Cpr-number>","main","7deeba21a42e4a9cd34f43e7ed6e960e4e851b26",".ai\u002Fskills\u002Freview-comet-iceberg-write-pr",418,"https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet","https:\u002F\u002Fdatafusion.apache.org\u002Fcomet","https:\u002F\u002Fgithub.com\u002Fapache","https:\u002F\u002Fwww.apache.org\u002F","2026-11-14 06:10:49",{"source":8,"skillName":5,"commands":55,"prompt":65,"zipAvailable":66},[56,59,62],{"tool":57,"command":58},"npx","npx skills add apache\u002Fdatafusion-comet --skill review-comet-iceberg-write-pr",{"tool":60,"command":61},"bunx","bunx skills add apache\u002Fdatafusion-comet --skill review-comet-iceberg-write-pr",{"tool":63,"command":64},"pnpm","pnpm dlx skills add apache\u002Fdatafusion-comet --skill review-comet-iceberg-write-pr","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-iceberg-write-pr\u002FSKILL.md",true,[68],{"path":69,"resourceType":70,"resourceTypeDesc":71,"sizeBytes":72,"sourceHtmlUrl":73},"README.md","REFERENCE","参考文档",7326,"https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fdevelopment\u002Fagent-md-refactor\u002FREADME.md",[75,105,129,144,160,171],{"skillNo":76,"name":77,"nameZh":6,"description":78,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":79,"categories":80,"keywords":86,"publishedAt":39},"01M6ACJR50ZS7F1QX58JEKCGVM","wire-datafusion-function","Use when wiring an existing DataFusion or datafusion-spark function into Comet for a Spark expression. Identifies the right wiring pattern (one-line passthrough, datafusion-spark UDF registration, or custom serde with input massaging \u002F restrictions), applies the Scala serde, registers the UDF in jni_api when needed, and adds SQL file tests. Assumes the function already exists upstream — if not, switch to `implement-comet-expression`.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fwire-datafusion-function\u002FSKILL.md",[81],{"categoryNo":82,"categoryName":83,"taxonomyType":84,"taxonomyTypeDesc":85},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",[35,87,88,89,90,91,92,93,94,95,96,97,98,31,99,24,100,101,102,103,104],"wire datafusion function","sql","scala","implement","spark","registration","switch","for","upstream","custom","expression","tests","Existing","file","api","wiring","with","name",{"skillNo":106,"name":107,"nameZh":6,"description":108,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":109,"categories":110,"keywords":111,"publishedAt":39},"01M6ACJPDDPND6FHE2PC1CJ4D3","suggest-native-expression","Use when picking the next Spark expression to implement natively in Comet, or when asked whether a specific codegen-dispatched expression is worth a native Rust implementation. Scores the candidate on how likely a native path can be 100% Spark-compatible and how much throughput or allocation it would save versus the JVM codegen dispatcher, records the verdict in the per-expression audit log so disqualified candidates are not re-litigated, and files a GitHub issue for the recommendation.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fsuggest-native-expression\u002FSKILL.md",[],[112,97,28,113,114,38,90,91,115,116,24,117,118,119,120,121,122,123,124,125,126,127,128],"suggest native expression","dispatcher","github","re","files","candidate","log","jvm","next","how","audit","audit log","allocation","issue","throughput","path","save",{"skillNo":130,"name":131,"nameZh":6,"description":132,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":133,"categories":134,"keywords":135,"publishedAt":39},"01M6ACJMHPEN7E76VPKC3P4X6C","review-comet-shuffle-pr","Use when reviewing a DataFusion Comet pull request that touches native or JVM columnar shuffle, the shuffle writers and readers, partitioning, the Arrow IPC block format, shuffle compression, or the Celeborn integration. Load alongside review-comet-pr.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-shuffle-pr\u002FSKILL.md",[],[136,20,19,36,37,35,137,31,25,24,138,26,119,28,139,140,141,142,143],"review comet shuffle pr","arrow","ipc","request","format","integration","block","compression",{"skillNo":145,"name":146,"nameZh":6,"description":147,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":148,"categories":149,"keywords":150,"publishedAt":39},"01M6ACJHX8BES7ZNNRNVCQ4QWE","review-comet-pr","Use when reviewing a DataFusion Comet pull request. Covers the workflow that applies to every PR and routes to the area-specific review skills for expressions, FFI, memory management, shuffle, Iceberg writes, and documentation. Provides guidance to a human reviewer rather than posting comments.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-pr\u002FSKILL.md",[],[20,19,151,36,21,37,152,35,153,154,24,26,155,156,139,157,94,31,158,159],"review comet pr","memory","workflow","documentation","reviewer","human","posting","review skills","comments",{"skillNo":161,"name":162,"nameZh":6,"description":163,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":164,"categories":165,"keywords":166,"publishedAt":39},"01M6ACJFM0QS61Z1RW516HXN2M","review-comet-memory-pr","Use when reviewing a DataFusion Comet pull request that touches memory pools, memory reservations, spilling, the Spark TaskMemoryManager bridge, allocators, or memory-related configuration. Load alongside review-comet-pr.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-memory-pr\u002FSKILL.md",[],[20,152,167,19,36,37,35,91,168,169,31,25,24,26,139,170],"review comet memory pr","bridge","reservations","configuration",{"skillNo":172,"name":173,"nameZh":6,"description":174,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":175,"categories":176,"keywords":177,"publishedAt":39},"01M6ACJA72KJRAM6F7WHMX01PN","review-comet-ffi-pr","Use when reviewing a DataFusion Comet pull request that crosses the JVM\u002Fnative boundary, touching Arrow C Data or C Stream interface code, batch export and import, CometExecIterator, ScanExec, NativeUtil, CometVector subclasses, or jni_api. Load alongside review-comet-pr.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-ffi-pr\u002FSKILL.md",[],[20,178,19,179,36,37,35,137,180,181,182,183,31,25,24,26,101,119,184,28,139],"review comet ffi pr","ffi","code","batch","import","interface","boundary"]