[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6ACJ7TKXE02CWAMZY02XHBY":3,"skill-same:01M6ACJ7TKXE02CWAMZY02XHBY":68},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":45,"commitSha":46,"skillDirectory":47,"forksCount":48,"repositoryHtmlUrl":49,"repositoryHomepageUrl":50,"ownerHtmlUrl":51,"ownerBlogUrl":52,"ownerDescription":6,"updatedAt":53,"install":54,"resources":67},"01M6ACJ7TKXE02CWAMZY02XHBY","review-comet-expression-pr",null,"Use when reviewing a DataFusion Comet pull request that adds or changes a Spark expression, its Scala serde, its protobuf message, or its native Rust implementation. Load alongside review-comet-pr, which covers the parts of the review that apply to every PR.","apache\u002Fdatafusion-comet","apache","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F47359?v=4","01M6AC83E6K4FDKK7HZ8WNMQKE","The Apache Software Foundation","01M6ACHX42TPNCR8TQJEAR8J5J",1322,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-expression-pr\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"pr","review comet expression pr","review","expression","use","when","apply","changes","load","which","number","native","message","datafusion","pull","protobuf","pull request","spark","scala","rust","2026-11-10 10:12:30",false,"---\nname: review-comet-expression-pr\ndescription: Use when reviewing a DataFusion Comet pull request that adds or changes a Spark expression, its Scala serde, its protobuf message, or its native Rust implementation. Load alongside review-comet-pr, which covers the parts of the review that apply to every PR.\nargument-hint: \u003Cpr-number>\n---\n\n\u003C!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n  http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied.  See the License for the\nspecific language governing permissions and limitations\nunder the License.\n-->\n\nExpression-specific review for Comet PR #$ARGUMENTS.\n\n**REQUIRED BACKGROUND:** Use `review-comet-pr` for PR metadata, existing comments, CI, the review\nbar, and the output format. This skill only covers what is specific to expressions.\n\n## Read the Contributor Guide First\n\n| Doc                                                        | What you need from it                                                                                                                                       |\n| ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Fadding_a_new_expression.md` | The serde contract, support levels, when to set the return type explicitly, shimming                                                                        |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Fsql-file-tests.md`          | The test framework expression PRs are expected to use, and every directive it supports                                                                      |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Foptimizing_expressions.md`  | The benchmark workflow and the no-regression rule, for PRs that change an existing kernel                                                                   |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Ftimezones.md`               | The timestamp label rule, where the session timezone comes from, and how to test it, for datetime expressions, casts, and anything that returns a timestamp |\n\nHold the diff against these. If the PR does something a guide says to do differently, either the PR\nis wrong or the guide is out of date, and you need to say which.\n\n## 1. Read the Spark Source\n\n**For any PR that adds or modifies an expression, you must read the Spark source to understand the\ncanonical behavior.** This is the authoritative reference for what Comet must match. Do not review\nan expression PR from the diff alone.\n\nClone Spark into `\u002Ftmp\u002Fspark` if it is not already there, using a shallow clone, then find the\nexpression class under `sql\u002Fcatalyst\u002Fsrc\u002Fmain\u002Fscala` by searching for its `case class` declaration.\n\nRead the Spark implementation carefully. Pay attention to:\n\n- `eval` and `doGenCode` \u002F `nullSafeEval`. These define the exact behavior.\n- `inputTypes` and `dataType`. These define which types Spark accepts and what it returns.\n- Null handling. Does it use `nullable = true`? Does `nullSafeEval` handle nulls implicitly?\n- Special cases, guards, and `require` assertions.\n- ANSI mode branches, usually `SQLConf.get.ansiEnabled` or `failOnError`.\n- Behavior differences across the Spark versions Comet supports. If the expression changed between\n  3.5 and 4.0, the serde needs a shim, not a single implementation.\n\nThen read the Spark tests for the expression, under the test source trees in `sql\u002F`.\n\nCompare Spark's behavior against the PR and identify edge cases Spark tests that the PR does not,\ntypes Spark supports that the PR does not handle, and behavioral differences that should be marked\n`Incompatible`.\n\n## 2. Scala Serde\n\nLocation: `spark\u002Fsrc\u002Fmain\u002Fscala\u002Forg\u002Fapache\u002Fcomet\u002Fserde\u002F`\n\n- [ ] Expression class correctly identified\n- [ ] All child expressions converted via `exprToProtoInternal`\n- [ ] Return type correctly serialized\n- [ ] `getSupportLevel` reflects true compatibility rather than the happy path\n- [ ] Serde lives in the appropriate file (`datetime.scala`, `strings.scala`, `arithmetic.scala`, and so on)\n- [ ] ANSI and `fail_on_error` handling matches the constraints in `adding_a_new_expression.md`\n- [ ] A change that routes a whole class of expressions to the JVM codegen dispatcher lists the new\n      shapes it admits and tests one of each, including decimal results whose scale differs from the\n      declared type and boolean inputs from sliced batches (#6424, #6425)\n- [ ] A timezone-aware expression serializes `CometTimeZone.nativeId(expr.timeZoneId)`, the\n      timezone Spark stamped on it in a form native code can parse, rather than\n      `SQLConf.get.sessionLocalTimeZone` or the JVM default, and returns\n      `CometTimeZone.supportLevel` from `getSupportLevel` when `nativeId` gives `None`\n\n### Registration in `QueryPlanSerde.scala`\n\n- [ ] Added to the correct map (temporal, string, arithmetic, and so on)\n- [ ] No duplicate registrations\n- [ ] Import statement added\n\n### Return type collisions with DataFusion built-ins\n\nIf the PR registers a Spark function whose name is also defined by `datafusion-functions`, such as\n`levenshtein`, `concat`, `coalesce`, `sha2`, or `regexp_replace`, check that the serde sets the\nreturn type explicitly via `scalarFunctionExprToProtoWithReturnType` rather than\n`scalarFunctionExprToProto` or the bare `CometScalarFunction(name)` shortcut.\n\nWithout an explicit return type the native planner consults DataFusion's UDF registry first for\ntype resolution, and any arity or input-type difference between the Spark and DataFusion versions\nfails native execution with `Error from DataFusion: Function 'X' expects N arguments but received\nM`. The Comet UDF is only swapped in after DataFusion's signature validation passes. See \"When to\nset the return type explicitly\" in `adding_a_new_expression.md`.\n\n## 3. Rust Implementation\n\nLocation: `native\u002Fspark-expr\u002Fsrc\u002F`, registered in `comet_scalar_funcs.rs`.\n\n- [ ] Matches DataFusion and Arrow conventions\n- [ ] Null handling is correct, including nulls inside nested types\n- [ ] No panics. Use `Result`.\n- [ ] Batch operations rather than row-by-row where a kernel exists\n- [ ] Invalid UTF-8 going into a native `StringType` goes through `decode_utf8_spark_lossy`\n- [ ] An aggregate merges partial states in Spark's order of operations, not just with Spark's\n      buffer layout. A different floating-point order fails Spark's exact checks, such as `m2 == 0`,\n      when a constant is merged from several partitions (#6423).\n- [ ] Comparisons, grouping and set operations treat `-0.0` and `0.0`, and NaN payloads, as Spark\n      does, including inside arrays and structs. Where a DataFusion kernel treats them differently,\n      the serde falls back (#5507, #5701).\n- [ ] A `TimestampType` result is labelled `\"UTC\"`, in `return_type()` or `data_type()` and in the\n      arrays the kernel builds. Not the session timezone, and not `None`.\n- [ ] Local-time work uses the session timezone from the proto. DataFusion's own datetime functions\n      take the timezone from the input's label, so wiring one in for a timezone-aware Spark\n      expression evaluates it in UTC unless the session timezone is passed explicitly.\n- [ ] A UTC fast path matches a fixed list of UTC aliases, as `is_utc_timezone` in\n      `extract_date_part.rs` does, and sends every other timezone ID down the general path\n\nBefore accepting a hand-written kernel, ask whether the function already exists upstream in\nDataFusion or the `datafusion-spark` crate. Comet prefers wiring an upstream function over carrying\nits own copy.\n\n## 4. Tests\n\n**Expression tests should use Comet SQL Tests (`CometSqlFileTestSuite`) where possible.** The\nframework runs each query through both Spark and Comet and compares results, with no Scala code.\nFall back to Comet Scala Tests in `CometExpressionSuite` only when the SQL framework cannot express\nthe test, for example complex `DataFrame` setup or programmatic data generation.\n\nTest location: `spark\u002Fsrc\u002Ftest\u002Fresources\u002Fsql-tests\u002Fexpressions\u002F\u003Ccategory>\u002F`, with categories\n`aggregate\u002F`, `array\u002F`, `string\u002F`, `math\u002F`, `struct\u002F`, `map\u002F`, `datetime\u002F`, `hash\u002F`, and others.\n\n```sql\nstatement\nCREATE TABLE test_crc32(col string, a int, b float) USING parquet\n\nstatement\nINSERT INTO test_crc32 VALUES ('Spark', 10, 1.5), (NULL, NULL, NULL), ('', 0, 0.0)\n\n-- default mode: verifies native Comet execution and that the result matches Spark\nquery\nSELECT crc32(col) FROM test_crc32\n\n-- compares results without requiring native execution\nquery spark_answer_only\nSELECT crc32(cast(a as string)) FROM test_crc32\n\n-- allows numeric variance for floating-point results\nquery tolerance=0.0001\nSELECT cos(v) FROM test_trig\n\n-- asserts fallback to Spark occurs\nquery expect_fallback(unsupported expression)\nSELECT unsupported_func(v) FROM test_table\n\n-- verifies both engines throw matching exceptions\nquery expect_error(ARITHMETIC_OVERFLOW)\nSELECT 2147483647 + 1\n\n-- skip a query with a known bug, with the issue link\nquery ignore(https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002FNNNN)\nSELECT known_buggy_expr(v) FROM test_table\n```\n\nRun the whole suite with `-Dsuites=\"org.apache.comet.CometSqlFileTestSuite\" -Dtest=none`, or a\nsingle file by appending a substring of its name to the suite argument.\n\n**Verify all test requirements, whichever framework is used:**\n\n- [ ] Basic functionality tested against column data, not only literals\n- [ ] Null handling tested\n- [ ] Edge cases tested: empty input, overflow, boundary values, negative values\n- [ ] Both literal and column arguments tested, in every combination for multi-argument\n      expressions. They take different code paths.\n- [ ] Timestamp and datetime expressions are tested as \"Testing timezone-sensitive code\" in\n      `timezones.md` describes. That means several session timezones through `ConfigMatrix`,\n      including `Etc\u002FUTC` and a zone with DST, both `TIMESTAMP` and `TIMESTAMP_NTZ` inputs, and a\n      result that is compared or fed to another expression rather than only projected.\n- [ ] SQL syntax gated with `MinSparkVersion` when it only parses on newer Spark\n- [ ] For a function from another project, such as an Iceberg transform, the expected values come\n      from that project's Java implementation, which is what Spark runs, not from iceberg-rust\n      (#6426)\n- [ ] A `query ignore(...)` for a result that differs from Spark comes with a fallback for that\n      case, so the divergence doesn't ship natively by default (#5701)\n- [ ] `expect_error` patterns substring-match what both Spark and Comet actually throw\n- [ ] One expression per SQL file\n- [ ] Comet Scala literal tests disable constant folding:\n\n```scala\nwithSQLConf(SQLConf.OPTIMIZER_EXCLUDED_RULES.key ->\n    \"org.apache.spark.sql.catalyst.optimizer.ConstantFolding\") {\n  checkSparkAnswerAndOperator(\"SELECT func(literal)\")\n}\n```\n\n## 5. Performance\n\n**For PRs that add a new native expression, performance is not optional.** The point of a native\nimplementation is to be faster than Spark's codegen. If it is not faster, it may not be worth\ncarrying.\n\n1. Does the PR description contain benchmark numbers comparing Comet against Spark? If not, ask for\n   them.\n2. Does the PR add a benchmark under `spark\u002Fsrc\u002Ftest\u002Fscala\u002Forg\u002Fapache\u002Fspark\u002Fsql\u002Fbenchmark\u002F`?\n3. Are the benchmarks representative, with realistic data sizes and the data shapes that break\n   optimizations, such as nulls, dictionaries, and wide strings?\n4. Does the Rust implementation allocate or copy more than it needs to, process row by row where a\n   kernel exists, redo type conversions, or revalidate UTF-8 repeatedly?\n5. If a benchmark shows Comet slower than Spark, flag it. The PR needs to explain why that is\n   acceptable or include a plan to fix it.\n\nFor PRs optimizing an existing kernel, `optimizing_expressions.md` defines the no-regression rule\nand the requirement to record a performance audit. Hold the PR to both.\n\n## 6. Documentation\n\n### Generated, review the source instead\n\n`GenerateDocs` fills in the expression compatibility tables and the config tables when the docs are\nbuilt, and CI rebuilds and publishes the docs when a merge changes their inputs. The **Generated Content** section of\n`docs\u002Fsource\u002Fcontributor-guide\u002Fstyle_guide.md` says exactly which text is generated. Never ask the\ncontributor to hand-edit that text. Check what feeds it:\n\n- Expression marked `Incompatible(Some(\"...\"))` but `getIncompatibleReasons()` is empty, so the\n  compat page shows it as supported with no caveats.\n- `Unsupported(Some(\"...\"))` for specific types or argument shapes with no\n  `getUnsupportedReasons()` to surface the limitation.\n- Reason strings that drift from the `notes` passed to `Compatible` \u002F `Incompatible` \u002F\n  `Unsupported`. They need not match exactly, but consistency helps.\n- Reason strings too terse to be useful in user-facing docs: a single word, no context, no link to\n  a tracking issue when behavior is known to differ.\n\n### Hand-edited, the PR should update these\n\n- `docs\u002Fsource\u002Fuser-guide\u002Flatest\u002Fexpressions.md`, the supported-expressions list. A new expression\n  belongs here.\n- Other `latest\u002Fcompatibility\u002F` pages such as `floating-point.md`, `regex.md`, `operators.md`.\n\n### Does the PR make the contributor guide stale?\n\nCheck `adding_a_new_expression.md` against the diff. It goes stale when a PR:\n\n- Changes the `CometExpressionSerde` interface, the support-level API, or the\n  `getIncompatibleReasons` \u002F `getUnsupportedReasons` contract\n- Adds or renames an expression map in `QueryPlanSerde.scala`, since the guide names them\n- Changes how the return type is resolved, which would invalidate the \"when to set the return type\n  explicitly\" section\n- Changes the shimming pattern or adds a Spark version to the set the guide lists\n- Changes how scalar functions are registered in `comet_scalar_funcs.rs`\n\nCheck `sql-file-tests.md` when the PR adds, renames, or changes a test directive. The directive\nreference in that doc is the only place they are documented.\n\n## Common Expression Review Findings\n\n1. **Incomplete type support**, Spark supports types the PR does not handle\n2. **Missing edge cases**, null, overflow, empty string, negative values\n3. **Wrong return type**, it must match Spark exactly\n4. **Tests in the wrong framework**, Scala tests where a SQL file test would do\n5. **Missing `getSupportLevel`**, divergences left undeclared rather than marked `Incompatible`\n6. **Version-specific Spark behavior implemented once**, with no shim or version check, or with a\n   protobuf field, native argument, or native function named for the Spark version\n   (`spark_420_plus`) instead of the behavior (`wrap_second_millisecond_overflow`)\n7. **Name collides with a DataFusion built-in** and no explicit return type\n8. **Timestamp result mislabelled**, with the session timezone or no timezone instead of `\"UTC\"`.\n   It only shows once the result is compared or fed to another expression.\n","\u003C!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n  http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied.  See the License for the\nspecific language governing permissions and limitations\nunder the License.\n-->\n\nExpression-specific review for Comet PR #$ARGUMENTS.\n\n**REQUIRED BACKGROUND:** Use `review-comet-pr` for PR metadata, existing comments, CI, the review\nbar, and the output format. This skill only covers what is specific to expressions.\n\n## Read the Contributor Guide First\n\n| Doc                                                        | What you need from it                                                                                                                                       |\n| ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Fadding_a_new_expression.md` | The serde contract, support levels, when to set the return type explicitly, shimming                                                                        |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Fsql-file-tests.md`          | The test framework expression PRs are expected to use, and every directive it supports                                                                      |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Foptimizing_expressions.md`  | The benchmark workflow and the no-regression rule, for PRs that change an existing kernel                                                                   |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Ftimezones.md`               | The timestamp label rule, where the session timezone comes from, and how to test it, for datetime expressions, casts, and anything that returns a timestamp |\n\nHold the diff against these. If the PR does something a guide says to do differently, either the PR\nis wrong or the guide is out of date, and you need to say which.\n\n## 1. Read the Spark Source\n\n**For any PR that adds or modifies an expression, you must read the Spark source to understand the\ncanonical behavior.** This is the authoritative reference for what Comet must match. Do not review\nan expression PR from the diff alone.\n\nClone Spark into `\u002Ftmp\u002Fspark` if it is not already there, using a shallow clone, then find the\nexpression class under `sql\u002Fcatalyst\u002Fsrc\u002Fmain\u002Fscala` by searching for its `case class` declaration.\n\nRead the Spark implementation carefully. Pay attention to:\n\n- `eval` and `doGenCode` \u002F `nullSafeEval`. These define the exact behavior.\n- `inputTypes` and `dataType`. These define which types Spark accepts and what it returns.\n- Null handling. Does it use `nullable = true`? Does `nullSafeEval` handle nulls implicitly?\n- Special cases, guards, and `require` assertions.\n- ANSI mode branches, usually `SQLConf.get.ansiEnabled` or `failOnError`.\n- Behavior differences across the Spark versions Comet supports. If the expression changed between\n  3.5 and 4.0, the serde needs a shim, not a single implementation.\n\nThen read the Spark tests for the expression, under the test source trees in `sql\u002F`.\n\nCompare Spark's behavior against the PR and identify edge cases Spark tests that the PR does not,\ntypes Spark supports that the PR does not handle, and behavioral differences that should be marked\n`Incompatible`.\n\n## 2. Scala Serde\n\nLocation: `spark\u002Fsrc\u002Fmain\u002Fscala\u002Forg\u002Fapache\u002Fcomet\u002Fserde\u002F`\n\n- [ ] Expression class correctly identified\n- [ ] All child expressions converted via `exprToProtoInternal`\n- [ ] Return type correctly serialized\n- [ ] `getSupportLevel` reflects true compatibility rather than the happy path\n- [ ] Serde lives in the appropriate file (`datetime.scala`, `strings.scala`, `arithmetic.scala`, and so on)\n- [ ] ANSI and `fail_on_error` handling matches the constraints in `adding_a_new_expression.md`\n- [ ] A change that routes a whole class of expressions to the JVM codegen dispatcher lists the new\n      shapes it admits and tests one of each, including decimal results whose scale differs from the\n      declared type and boolean inputs from sliced batches (#6424, #6425)\n- [ ] A timezone-aware expression serializes `CometTimeZone.nativeId(expr.timeZoneId)`, the\n      timezone Spark stamped on it in a form native code can parse, rather than\n      `SQLConf.get.sessionLocalTimeZone` or the JVM default, and returns\n      `CometTimeZone.supportLevel` from `getSupportLevel` when `nativeId` gives `None`\n\n### Registration in `QueryPlanSerde.scala`\n\n- [ ] Added to the correct map (temporal, string, arithmetic, and so on)\n- [ ] No duplicate registrations\n- [ ] Import statement added\n\n### Return type collisions with DataFusion built-ins\n\nIf the PR registers a Spark function whose name is also defined by `datafusion-functions`, such as\n`levenshtein`, `concat`, `coalesce`, `sha2`, or `regexp_replace`, check that the serde sets the\nreturn type explicitly via `scalarFunctionExprToProtoWithReturnType` rather than\n`scalarFunctionExprToProto` or the bare `CometScalarFunction(name)` shortcut.\n\nWithout an explicit return type the native planner consults DataFusion's UDF registry first for\ntype resolution, and any arity or input-type difference between the Spark and DataFusion versions\nfails native execution with `Error from DataFusion: Function 'X' expects N arguments but received\nM`. The Comet UDF is only swapped in after DataFusion's signature validation passes. See \"When to\nset the return type explicitly\" in `adding_a_new_expression.md`.\n\n## 3. Rust Implementation\n\nLocation: `native\u002Fspark-expr\u002Fsrc\u002F`, registered in `comet_scalar_funcs.rs`.\n\n- [ ] Matches DataFusion and Arrow conventions\n- [ ] Null handling is correct, including nulls inside nested types\n- [ ] No panics. Use `Result`.\n- [ ] Batch operations rather than row-by-row where a kernel exists\n- [ ] Invalid UTF-8 going into a native `StringType` goes through `decode_utf8_spark_lossy`\n- [ ] An aggregate merges partial states in Spark's order of operations, not just with Spark's\n      buffer layout. A different floating-point order fails Spark's exact checks, such as `m2 == 0`,\n      when a constant is merged from several partitions (#6423).\n- [ ] Comparisons, grouping and set operations treat `-0.0` and `0.0`, and NaN payloads, as Spark\n      does, including inside arrays and structs. Where a DataFusion kernel treats them differently,\n      the serde falls back (#5507, #5701).\n- [ ] A `TimestampType` result is labelled `\"UTC\"`, in `return_type()` or `data_type()` and in the\n      arrays the kernel builds. Not the session timezone, and not `None`.\n- [ ] Local-time work uses the session timezone from the proto. DataFusion's own datetime functions\n      take the timezone from the input's label, so wiring one in for a timezone-aware Spark\n      expression evaluates it in UTC unless the session timezone is passed explicitly.\n- [ ] A UTC fast path matches a fixed list of UTC aliases, as `is_utc_timezone` in\n      `extract_date_part.rs` does, and sends every other timezone ID down the general path\n\nBefore accepting a hand-written kernel, ask whether the function already exists upstream in\nDataFusion or the `datafusion-spark` crate. Comet prefers wiring an upstream function over carrying\nits own copy.\n\n## 4. Tests\n\n**Expression tests should use Comet SQL Tests (`CometSqlFileTestSuite`) where possible.** The\nframework runs each query through both Spark and Comet and compares results, with no Scala code.\nFall back to Comet Scala Tests in `CometExpressionSuite` only when the SQL framework cannot express\nthe test, for example complex `DataFrame` setup or programmatic data generation.\n\nTest location: `spark\u002Fsrc\u002Ftest\u002Fresources\u002Fsql-tests\u002Fexpressions\u002F\u003Ccategory>\u002F`, with categories\n`aggregate\u002F`, `array\u002F`, `string\u002F`, `math\u002F`, `struct\u002F`, `map\u002F`, `datetime\u002F`, `hash\u002F`, and others.\n\n```sql\nstatement\nCREATE TABLE test_crc32(col string, a int, b float) USING parquet\n\nstatement\nINSERT INTO test_crc32 VALUES ('Spark', 10, 1.5), (NULL, NULL, NULL), ('', 0, 0.0)\n\n-- default mode: verifies native Comet execution and that the result matches Spark\nquery\nSELECT crc32(col) FROM test_crc32\n\n-- compares results without requiring native execution\nquery spark_answer_only\nSELECT crc32(cast(a as string)) FROM test_crc32\n\n-- allows numeric variance for floating-point results\nquery tolerance=0.0001\nSELECT cos(v) FROM test_trig\n\n-- asserts fallback to Spark occurs\nquery expect_fallback(unsupported expression)\nSELECT unsupported_func(v) FROM test_table\n\n-- verifies both engines throw matching exceptions\nquery expect_error(ARITHMETIC_OVERFLOW)\nSELECT 2147483647 + 1\n\n-- skip a query with a known bug, with the issue link\nquery ignore(https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002FNNNN)\nSELECT known_buggy_expr(v) FROM test_table\n```\n\nRun the whole suite with `-Dsuites=\"org.apache.comet.CometSqlFileTestSuite\" -Dtest=none`, or a\nsingle file by appending a substring of its name to the suite argument.\n\n**Verify all test requirements, whichever framework is used:**\n\n- [ ] Basic functionality tested against column data, not only literals\n- [ ] Null handling tested\n- [ ] Edge cases tested: empty input, overflow, boundary values, negative values\n- [ ] Both literal and column arguments tested, in every combination for multi-argument\n      expressions. They take different code paths.\n- [ ] Timestamp and datetime expressions are tested as \"Testing timezone-sensitive code\" in\n      `timezones.md` describes. That means several session timezones through `ConfigMatrix`,\n      including `Etc\u002FUTC` and a zone with DST, both `TIMESTAMP` and `TIMESTAMP_NTZ` inputs, and a\n      result that is compared or fed to another expression rather than only projected.\n- [ ] SQL syntax gated with `MinSparkVersion` when it only parses on newer Spark\n- [ ] For a function from another project, such as an Iceberg transform, the expected values come\n      from that project's Java implementation, which is what Spark runs, not from iceberg-rust\n      (#6426)\n- [ ] A `query ignore(...)` for a result that differs from Spark comes with a fallback for that\n      case, so the divergence doesn't ship natively by default (#5701)\n- [ ] `expect_error` patterns substring-match what both Spark and Comet actually throw\n- [ ] One expression per SQL file\n- [ ] Comet Scala literal tests disable constant folding:\n\n```scala\nwithSQLConf(SQLConf.OPTIMIZER_EXCLUDED_RULES.key ->\n    \"org.apache.spark.sql.catalyst.optimizer.ConstantFolding\") {\n  checkSparkAnswerAndOperator(\"SELECT func(literal)\")\n}\n```\n\n## 5. Performance\n\n**For PRs that add a new native expression, performance is not optional.** The point of a native\nimplementation is to be faster than Spark's codegen. If it is not faster, it may not be worth\ncarrying.\n\n1. Does the PR description contain benchmark numbers comparing Comet against Spark? If not, ask for\n   them.\n2. Does the PR add a benchmark under `spark\u002Fsrc\u002Ftest\u002Fscala\u002Forg\u002Fapache\u002Fspark\u002Fsql\u002Fbenchmark\u002F`?\n3. Are the benchmarks representative, with realistic data sizes and the data shapes that break\n   optimizations, such as nulls, dictionaries, and wide strings?\n4. Does the Rust implementation allocate or copy more than it needs to, process row by row where a\n   kernel exists, redo type conversions, or revalidate UTF-8 repeatedly?\n5. If a benchmark shows Comet slower than Spark, flag it. The PR needs to explain why that is\n   acceptable or include a plan to fix it.\n\nFor PRs optimizing an existing kernel, `optimizing_expressions.md` defines the no-regression rule\nand the requirement to record a performance audit. Hold the PR to both.\n\n## 6. Documentation\n\n### Generated, review the source instead\n\n`GenerateDocs` fills in the expression compatibility tables and the config tables when the docs are\nbuilt, and CI rebuilds and publishes the docs when a merge changes their inputs. The **Generated Content** section of\n`docs\u002Fsource\u002Fcontributor-guide\u002Fstyle_guide.md` says exactly which text is generated. Never ask the\ncontributor to hand-edit that text. Check what feeds it:\n\n- Expression marked `Incompatible(Some(\"...\"))` but `getIncompatibleReasons()` is empty, so the\n  compat page shows it as supported with no caveats.\n- `Unsupported(Some(\"...\"))` for specific types or argument shapes with no\n  `getUnsupportedReasons()` to surface the limitation.\n- Reason strings that drift from the `notes` passed to `Compatible` \u002F `Incompatible` \u002F\n  `Unsupported`. They need not match exactly, but consistency helps.\n- Reason strings too terse to be useful in user-facing docs: a single word, no context, no link to\n  a tracking issue when behavior is known to differ.\n\n### Hand-edited, the PR should update these\n\n- `docs\u002Fsource\u002Fuser-guide\u002Flatest\u002Fexpressions.md`, the supported-expressions list. A new expression\n  belongs here.\n- Other `latest\u002Fcompatibility\u002F` pages such as `floating-point.md`, `regex.md`, `operators.md`.\n\n### Does the PR make the contributor guide stale?\n\nCheck `adding_a_new_expression.md` against the diff. It goes stale when a PR:\n\n- Changes the `CometExpressionSerde` interface, the support-level API, or the\n  `getIncompatibleReasons` \u002F `getUnsupportedReasons` contract\n- Adds or renames an expression map in `QueryPlanSerde.scala`, since the guide names them\n- Changes how the return type is resolved, which would invalidate the \"when to set the return type\n  explicitly\" section\n- Changes the shimming pattern or adds a Spark version to the set the guide lists\n- Changes how scalar functions are registered in `comet_scalar_funcs.rs`\n\nCheck `sql-file-tests.md` when the PR adds, renames, or changes a test directive. The directive\nreference in that doc is the only place they are documented.\n\n## Common Expression Review Findings\n\n1. **Incomplete type support**, Spark supports types the PR does not handle\n2. **Missing edge cases**, null, overflow, empty string, negative values\n3. **Wrong return type**, it must match Spark exactly\n4. **Tests in the wrong framework**, Scala tests where a SQL file test would do\n5. **Missing `getSupportLevel`**, divergences left undeclared rather than marked `Incompatible`\n6. **Version-specific Spark behavior implemented once**, with no shim or version check, or with a\n   protobuf field, native argument, or native function named for the Spark version\n   (`spark_420_plus`) instead of the behavior (`wrap_second_millisecond_overflow`)\n7. **Name collides with a DataFusion built-in** and no explicit return type\n8. **Timestamp result mislabelled**, with the session timezone or no timezone instead of `\"UTC\"`.\n   It only shows once the result is compared or fed to another expression.\n",{"name":5,"description":7,"argument-hint":44},"\u003Cpr-number>","main","4de6704391c9a7832a2079fc4e8d80fc09e9b209",".ai\u002Fskills\u002Freview-comet-expression-pr",418,"https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet","https:\u002F\u002Fdatafusion.apache.org\u002Fcomet","https:\u002F\u002Fgithub.com\u002Fapache","https:\u002F\u002Fwww.apache.org\u002F","2026-11-13 03:12:00",{"source":8,"skillName":5,"commands":55,"prompt":65,"zipAvailable":66},[56,59,62],{"tool":57,"command":58},"npx","npx skills add apache\u002Fdatafusion-comet --skill review-comet-expression-pr",{"tool":60,"command":61},"bunx","bunx skills add apache\u002Fdatafusion-comet --skill review-comet-expression-pr",{"tool":63,"command":64},"pnpm","pnpm dlx skills add apache\u002Fdatafusion-comet --skill review-comet-expression-pr","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-expression-pr\u002FSKILL.md",true,[],[69,96,120,135,152,163],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":80,"publishedAt":39},"01M6ACJR50ZS7F1QX58JEKCGVM","wire-datafusion-function","Use when wiring an existing DataFusion or datafusion-spark function into Comet for a Spark expression. Identifies the right wiring pattern (one-line passthrough, datafusion-spark UDF registration, or custom serde with input massaging \u002F restrictions), applies the Scala serde, registers the UDF in jni_api when needed, and adds SQL file tests. Assumes the function already exists upstream — if not, switch to `implement-comet-expression`.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fwire-datafusion-function\u002FSKILL.md",[75],{"categoryNo":76,"categoryName":77,"taxonomyType":78,"taxonomyTypeDesc":79},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",[32,81,82,37,83,36,84,85,86,87,88,22,89,23,90,24,91,92,93,94,95],"wire datafusion function","sql","implement","registration","switch","for","upstream","custom","tests","Existing","file","api","wiring","with","name",{"skillNo":97,"name":98,"nameZh":6,"description":99,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":100,"categories":101,"keywords":102,"publishedAt":39},"01M6ACJPDDPND6FHE2PC1CJ4D3","suggest-native-expression","Use when picking the next Spark expression to implement natively in Comet, or when asked whether a specific codegen-dispatched expression is worth a native Rust implementation. Scores the candidate on how likely a native path can be 100% Spark-compatible and how much throughput or allocation it would save versus the JVM codegen dispatcher, records the verdict in the per-expression audit log so disqualified candidates are not re-litigated, and files a GitHub issue for the recommendation.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fsuggest-native-expression\u002FSKILL.md",[],[103,22,30,104,105,38,83,36,106,107,24,108,109,110,111,112,113,114,115,116,117,118,119],"suggest native expression","dispatcher","github","re","files","candidate","log","jvm","next","how","audit","audit log","allocation","issue","throughput","path","save",{"skillNo":121,"name":122,"nameZh":6,"description":123,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":124,"categories":125,"keywords":126,"publishedAt":39},"01M6ACJMHPEN7E76VPKC3P4X6C","review-comet-shuffle-pr","Use when reviewing a DataFusion Comet pull request that touches native or JVM columnar shuffle, the shuffle writers and readers, partitioning, the Arrow IPC block format, shuffle compression, or the Celeborn integration. Load alongside review-comet-pr.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-shuffle-pr\u002FSKILL.md",[],[127,19,21,33,35,32,128,23,27,24,129,29,110,30,130,131,132,133,134],"review comet shuffle pr","arrow","ipc","request","format","integration","block","compression",{"skillNo":136,"name":137,"nameZh":6,"description":138,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":139,"categories":140,"keywords":141,"publishedAt":39},"01M6ACJHX8BES7ZNNRNVCQ4QWE","review-comet-pr","Use when reviewing a DataFusion Comet pull request. Covers the workflow that applies to every PR and routes to the area-specific review skills for expressions, FFI, memory management, shuffle, Iceberg writes, and documentation. Provides guidance to a human reviewer rather than posting comments.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-pr\u002FSKILL.md",[],[19,21,142,33,143,35,144,32,145,146,24,29,147,148,130,149,86,23,150,151],"review comet pr","iceberg","memory","workflow","documentation","reviewer","human","posting","review skills","comments",{"skillNo":153,"name":154,"nameZh":6,"description":155,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":156,"categories":157,"keywords":158,"publishedAt":39},"01M6ACJFM0QS61Z1RW516HXN2M","review-comet-memory-pr","Use when reviewing a DataFusion Comet pull request that touches memory pools, memory reservations, spilling, the Spark TaskMemoryManager bridge, allocators, or memory-related configuration. Load alongside review-comet-pr.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-memory-pr\u002FSKILL.md",[],[19,144,159,21,33,35,32,36,160,161,23,27,24,29,130,162],"review comet memory pr","bridge","reservations","configuration",{"skillNo":164,"name":165,"nameZh":6,"description":166,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":167,"categories":168,"keywords":169,"publishedAt":39},"01M6ACJCV64V0FZPMVXDE2K3DX","review-comet-iceberg-write-pr","Use when reviewing a DataFusion Comet pull request that touches Iceberg writes, the split-operator write plan (IcebergWriteStrategy, IcebergWriteExec, IcebergCommitExec), the native Iceberg writer (CometIcebergWriteExec, iceberg_write.rs, iceberg_partition_path.rs), its eligibility gate (CometIcebergNativeWrite), or the iceberg-rust pin. Load alongside review-comet-pr.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-iceberg-write-pr\u002FSKILL.md",[],[19,143,170,21,171,33,35,32,38,172,173,174,23,27,175,24,29,176,177,30],"review comet iceberg write pr","write","writer","write plan","partition","pin","split","plan"]