[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6ACDDAAM03F6BHR51296EW6":3,"skill-same:01M6ACDDAAM03F6BHR51296EW6":67},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":38,"linkOnly":39,"markdownContent":40,"markdownBody":41,"frontMatter":42,"refName":44,"commitSha":45,"skillDirectory":46,"forksCount":47,"repositoryHtmlUrl":48,"repositoryHomepageUrl":49,"ownerHtmlUrl":50,"ownerBlogUrl":51,"ownerDescription":6,"updatedAt":52,"install":53,"resources":66},"01M6ACDDAAM03F6BHR51296EW6","audit-datafusion-spark-expression",null,"Audit a datafusion-spark function implementation for correctness against Apache Spark 4.2.0. Studies the Spark source across versions, reviews the Rust implementation and its signature, verifies expected values against a real PySpark, fixes divergences, and captures the rest as issues and disabled tests.","apache\u002Fdatafusion","apache","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F47359?v=4","01M6AC83E6K4FDKK7HZ8WNMQKE","The Apache Software Foundation","01M6ACD7H7PW9G5CPPV291Y97C",9585,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Faudit-datafusion-spark-expression\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,9,35,36,37],"spark","datafusion","audit datafusion spark expression","audit","expression","across","expected","fixes","source","for","correctness","reviews","as","tests","rust","REST","issues","pyspark","apache spark","2026-11-10 10:12:30",false,"---\nname: audit-datafusion-spark-expression\ndescription: Audit a datafusion-spark function implementation for correctness against Apache Spark 4.2.0. Studies the Spark source across versions, reviews the Rust implementation and its signature, verifies expected values against a real PySpark, fixes divergences, and captures the rest as issues and disabled tests.\nargument-hint: \u003Cfunction-name>\n---\n\n\u003C!--\nSPDX-License-Identifier: Apache-2.0\nSPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership\nSPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements\n-->\n\nAudit the `datafusion-spark` implementation of the `$ARGUMENTS` function for\ncorrectness against Apache Spark.\n\n## Baseline Version\n\nThe baseline is **Spark 4.2.0**. Every expected value written into a `.slt`\nfile asserts Spark 4.2.0 behavior.\n\nSpark 3.5.8, 4.0.4, and 4.1.3 are also read, but only to detect that behavior\nchanged between versions. They never set an expectation. There is currently no\nmechanism for version-specific expectations in DataFusion's SLT files, which is\ntracked by https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion\u002Fissues\u002F23887.\n\nOlder versions are read rather than ignored because `datafusion-spark` is meant\nto be reusable by engines that target more than one Spark release. Comet\nsupports Spark 3.4 through 4.0. A function audited against 4.2.0 alone can be\ncorrect there and silently wrong for such a consumer, and nothing in the crate\nrecords which of the two it is. Finding the divergence is useful even though\nthe test suite cannot yet assert it.\n\n## Overview\n\n1. Study the Spark implementation across versions\n2. Harvest Spark's own test coverage\n3. Review the DataFusion implementation\n4. Cross-check Comet and Sail\n5. Review existing SLT coverage\n6. Gap analysis\n7. Establish ground truth with PySpark\n8. Apply findings\n\nAudit one function per invocation.\n\n## Paths in This Skill Are Hints\n\nEvery file path below was correct when this skill was written and several will\nrot. `FunctionRegistry.scala`, the golden-file directories, and\n`datafusion\u002Fcommon\u002Fsrc\u002Fconfig.rs` all move on someone else's schedule.\n\n**A path that does not resolve is a stop condition, not a step to skip.** The\nfailure mode this exists to prevent is a command that quietly matches nothing,\nan audit that reads the silence as \"no ANSI branch\" or \"Spark does not test\nthis\", and a confident report built on an empty grep. A `grep` over a\nnonexistent directory and a `grep` that genuinely found nothing are the same\noutput.\n\nSo when a path in this skill does not exist, or a command returns nothing where\nsomething was expected:\n\n1. Say so out loud in the audit report. Name the path and the step.\n2. Search for where the thing moved to, by content rather than by path. The\n   grep patterns here are more durable than the directories they are pointed at.\n3. If it cannot be found, mark every conclusion that depended on that step as\n   unestablished. Do not let an unread source become an absence of findings.\n\nThe same rule covers a `.slt` file, an implementation file, or a Spark class\nthat is not where this skill says it is.\n\n## Step 1: Study the Spark Implementation\n\nShallow-clone the four version tags. Skip any already present.\n\n```bash\nset -eu\nfor tag in v3.5.8 v4.0.4 v4.1.3 v4.2.0; do\n  dir=\"\u002Ftmp\u002Fspark-${tag}\"\n  [ -d \"$dir\" ] || git clone --depth 1 --branch \"$tag\" \\\n    https:\u002F\u002Fgithub.com\u002Fapache\u002Fspark.git \"$dir\"\ndone\n```\n\nFind the expression class in each version. Spark registers SQL function names\nin `FunctionRegistry.scala`, so start there to map `$ARGUMENTS` to its Scala\nclass name, then locate the class.\n\n```bash\nfor tag in v3.5.8 v4.0.4 v4.1.3 v4.2.0; do\n  dir=\"\u002Ftmp\u002Fspark-${tag}\"\n  echo \"=== $tag ===\"\n  grep -rn \"\\\"$ARGUMENTS\\\"\" \\\n    \"$dir\u002Fsql\u002Fcatalyst\u002Fsrc\u002Fmain\u002Fscala\u002Forg\u002Fapache\u002Fspark\u002Fsql\u002Fcatalyst\u002Fanalysis\u002FFunctionRegistry.scala\"\ndone\n```\n\nWith the class name in hand, read its source in each version:\n\n```bash\nfor tag in v3.5.8 v4.0.4 v4.1.3 v4.2.0; do\n  dir=\"\u002Ftmp\u002Fspark-${tag}\"\n  echo \"=== $tag ===\"\n  find \"$dir\u002Fsql\u002Fcatalyst\u002Fsrc\u002Fmain\u002Fscala\" -name \"*.scala\" \\\n    | xargs grep -lE \"case class \u003CClassName>[ (\\[]\" 2>\u002Fdev\u002Fnull\ndone\n```\n\nA Scala `case class` name is always followed by a space, a `(`, or a `[`, so\nthat bracket expression is the portable way to avoid matching a longer class\nname that starts with the same prefix. Do not reach for `\\b`, which is a GNU\nand ugrep extension and is not available in POSIX or BSD `grep`.\n\nIf it is not in catalyst, widen the search to `$dir\u002Fsql`.\n\nFor each version, record:\n\n- `eval` \u002F `nullSafeEval` \u002F `doGenCode` logic\n- `inputTypes` and `dataType`\n- `nullable`, and how nulls propagate\n- ANSI branches (`ansiEnabled`, `failOnError`)\n- guards, `require` assertions, thrown exceptions\n- any SQLConf the expression reads\n- for string functions in 4.0+, whether it routes through\n  `CollationSupport.X.exec(..., collationId)` or declares\n  `StringTypeWithCollation` inputs\n\nProduce a diff summary across 3.5.8 → 4.0.4 → 4.1.3 → 4.2.0. Note new or\nremoved input types, changed edge-case behavior, new ANSI branches, new\nparameters, and collation changes.\n\nTwo changes affect nearly every function and are easy to miss:\n\n- ANSI mode became the **default** in Spark 4.0. A function whose behavior\n  differs under ANSI has a different default behavior in 3.5 than in 4.x.\n- Collation support landed for many string functions in 4.0. Non-default\n  collations are a divergence axis that does not exist in 3.5.\n\n## Step 2: Harvest Spark's Own Test Coverage\n\nSpark's tests are a ready-made edge-case inventory. Two sources matter.\n\nUnit tests:\n\n```bash\ndir=\u002Ftmp\u002Fspark-v4.2.0\nfind \"$dir\u002Fsql\" -name \"*.scala\" -path \"*\u002Ftest\u002F*\" \\\n  | xargs grep -ln \"$ARGUMENTS\" 2>\u002Fdev\u002Fnull\n```\n\nGolden files, which carry Spark's own verified query results and are the\nhighest-value source:\n\n```bash\nfor tag in v3.5.8 v4.0.4 v4.1.3 v4.2.0; do\n  dir=\"\u002Ftmp\u002Fspark-${tag}\"\n  echo \"=== $tag ===\"\n  grep -rln \"$ARGUMENTS\" \"$dir\u002Fsql\u002Fcore\u002Fsrc\u002Ftest\u002Fresources\u002Fsql-tests\u002Fresults\u002F\" 2>\u002Fdev\u002Fnull\ndone\n```\n\n**The ANSI and non-ANSI golden directories swapped meaning in Spark 4.0.** Read\nthe version banner before trusting a path:\n\n| Version | ANSI-mode results        | Non-ANSI results            |\n| ------- | ------------------------ | --------------------------- |\n| 3.5.8   | `results\u002Fansi\u002FX.sql.out` | `results\u002FX.sql.out`         |\n| 4.x     | `results\u002FX.sql.out`      | `results\u002Fnonansi\u002FX.sql.out` |\n\nANSI became the default in 4.0, so the unqualified path flipped from meaning\nnon-ANSI to meaning ANSI. An auditor who learns the layout on 3.5.8 and then\nreads `results\u002FX.sql.out` on 4.2.0 will record exactly inverted expectations,\nand they will look plausible. Confirm the mode by checking whether the file\ncontains error output or `NULL` for a known-invalid input.\n\nRead every match and extract:\n\n- input types covered\n- edge cases exercised: null, empty, overflow, negative, boundary, special\n  characters\n- ANSI-mode cases\n- error cases and their exact messages\n\nThis list is the reference for the gap matrix in Step 6.\n\n## Step 3: Review the DataFusion Implementation\n\n```bash\ngrep -rln \"$ARGUMENTS\" datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002F --include=\"*.rs\"\n```\n\nThe implementation lives at\n`datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002F\u003Ccategory>\u002F\u003Cfunction>.rs`. **Read every other\nfile that grep returns as well.** In particular\n`datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002F\u003Ccategory>\u002Fmod.rs` carries the `export_functions!`\nregistration and the user-facing doc string, and unimplemented-behavior `TODO`\ncomments are written there too, not only in the function's own file. An audit\nthat reads only `\u003Cfunction>.rs` will miss them and will leave a doc string that\ncontradicts the code after a fix.\n\n**A `.slt` file under `test_files\u002Fspark\u002F` does not imply an implementation in\n`datafusion\u002Fspark`.** `SessionStateBuilderSpark::with_spark_features` layers the\nSpark functions on top of `with_default_features`, so a name the spark crate\ndoes not define resolves to DataFusion's own function of that name. `md5`,\n`coalesce`, and `reverse` are served this way today, and `date_format` resolves\nthrough an alias on core's `to_char`. The query under audit still runs, and the\ncode it runs is not in the directory the grep above searched. If that grep comes\nback empty, widen it before concluding the function is unimplemented:\n\n```bash\ngrep -rln \"\\\"$ARGUMENTS\\\"\" \\\n  datafusion\u002Ffunctions\u002Fsrc datafusion\u002Ffunctions-nested\u002Fsrc \\\n  datafusion\u002Ffunctions-aggregate\u002Fsrc datafusion\u002Ffunctions-window\u002Fsrc \\\n  datafusion\u002Ffunctions-table\u002Fsrc --include=\"*.rs\"\n```\n\nA hit may be in a file named after something else, because core registers\naliases. `date_format` lives in `to_char.rs`. Read the `aliases` vector, not\njust the file name.\n\nIf that grep is also empty, the function is genuinely unimplemented and the\n`.slt` file is a porting stub with every query commented out. Check with\n`grep -cE '^(query|statement)' \u003Cfile>.slt`, which returns zero for a stub.\nThere is no implementation to audit in that case. Say so and stop, rather than\nreporting a clean audit of nothing.\n\nOtherwise audit what the grep found against Spark with the rest of this step's\nchecklist. The findings land differently. A core function has non-Spark users,\nso Spark's semantics cannot be imposed on it. Where the two agree, record the\ncoverage and leave the core function alone. Where they diverge, the fix is a\nSpark-specific implementation under `datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002F\u003Ccategory>\u002F`\nthat shadows the core one, never an edit to the core function.\n`datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002Fmath\u002Fabs.rs` shadowing\n`datafusion\u002Ffunctions\u002Fsrc\u002Fmath\u002Fabs.rs` is the in-repo precedent. Say in the\naudit report which crate the audited code came from, because a reader who\nassumes `datafusion\u002Fspark` will look for a file that does not exist.\n\nCheck each of the following. The first two have no Comet analog and are the\nmost common source of silent divergence.\n\n**Signature and coercion.** Compare `Signature` (and `coerce_types`, if\nimplemented) against Spark's `inputTypes`. Two failure directions, both real:\n\n- DataFusion rejects a type Spark accepts. The user gets a coercion error\n  where Spark returns a value.\n- DataFusion accepts a type Spark rejects. The user gets a value where Spark\n  raises an analysis error.\n\nNote that `Signature::exact` is narrow. A Spark function accepting any string\ntype needs all of `Utf8`, `LargeUtf8`, and `Utf8View` handled, and a function\naccepting any integral type needs the full width range.\n\n**Return nullability.** Compare `return_field_from_args` (or `return_type`)\nagainst Spark's `nullable`. A function that reports non-nullable while Spark\ncan return NULL produces wrong plans, not merely wrong values: downstream\noptimizer rules will fold away null checks that were load-bearing.\n\n**Null propagation.** Does every input-null combination return what Spark\nreturns? Check the order in which nullness and validity are tested. Spark\nexpressions that are null intolerant (a `NullIntolerant` mixin in 3.5, a\n`nullIntolerant` override in 4.x) short circuit to NULL *before* validating the\nother arguments, so a NULL in one argument suppresses an error the other\nargument would otherwise raise, even under ANSI mode. An implementation that\nvalidates first and checks nullness second raises where Spark returns NULL.\n\n**Arrow kernel semantics are not Java semantics.** A `datafusion-spark`\nfunction is usually a thin arrangement of Arrow kernels, and the kernels do not\nshare Java's rules. Check each one the implementation leans on:\n\n- **Comparison kernels order floats by total order**, so `-0.0` compares as\n  *less than* `0.0` rather than equal to it. Any `eq`\u002F`lt` used to detect a\n  special value, such as a zero divisor, silently misses `-0.0`. Java compares\n  them as equal. This is easy to miss because `-0.0` almost never appears in\n  hand-written test values, so write it in deliberately.\n- **Arithmetic kernels are checked, not wrapping.** Java wraps silently on\n  overflow, so a checked Arrow kernel raises where Spark returns a wrapped\n  value. Where Spark's own expression is written in `Int` arithmetic, a\n  wrapping kernel is the faithful choice, not the reckless one.\n- **Java promotes `byte` and `short` operands to `int`.** A Spark expression\n  whose Scala source reads `(r + n)` on `Byte` operands is evaluated at `Int`\n  width and cannot overflow, while the same source on `Int` operands can.\n  Reproducing it at `Int8` width wraps where Spark does not.\n- **Kernel coverage is type dependent.** Arrow's `rem` raises on a zero divisor\n  for integers and decimals but returns `NaN` for floats, so an ANSI check that\n  relies on the kernel to raise silently does nothing on the float path.\n\nRead the Scala source as arithmetic to be reproduced, not as pseudocode to be\nparaphrased. Then test each of these against a live Spark rather than reasoning\nabout them.\n\n**`ColumnarValue` shape dispatch.** A two-argument function has four shape\ncombinations, and each needs its own branch:\n`(Scalar, Scalar)`, `(Array, Scalar)`, `(Scalar, Array)`, `(Array, Array)`.\nA missing arm falls through to a catch-all `exec_err!` and turns a working\nSpark query into a hard error. `(Scalar, Array)` is the one most often\nforgotten, because the natural test query puts the column first. Write a query\nfor every combination and run it. Do not infer coverage from reading the\n`match`.\n\n**How to run an ad-hoc query.** The only in-repo way to execute a\n`datafusion-spark` function is to add the query to\n`datafusion\u002Fsqllogictest\u002Ftest_files\u002Fspark\u002F\u003Ccategory>\u002F$ARGUMENTS.slt` and run\n\n```bash\ncargo test --test sqllogictests -- spark\u002F\u003Ccategory>\u002F$ARGUMENTS\n```\n\n`datafusion-cli` does **not** register the `datafusion-spark` function library,\nso reaching for it fails in a way that looks like the function does not exist:\n\n```\n$ .\u002Ftarget\u002Fdebug\u002Fdatafusion-cli -c \"SELECT next_day(arrow_cast(95026236,'Date32'),'Mon');\"\nError during planning: Invalid function 'next_day'. Did you mean 'today'?\n```\n\nGiving the CLI an opt-in for the Spark library is tracked by\nhttps:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion\u002Fissues\u002F24146. Until that lands, treat the\nerror above as a registration gap and not as evidence about the function.\n\nFor a throwaway probe that should not end up in the committed file, add a\ntemporary `.slt` alongside it, run it by name, and delete it afterwards. To\nexercise a raw storage value that has no literal syntax, such as a `Date32`\npast the last date DataFusion can render, wrap it in `arrow_cast`.\n\n**Cross the shapes with the NULL cases, and with ANSI mode.** These are not\nthree independent checklists to tick off separately, they are one product to\nenumerate. The bug this instruction exists to catch is a branch that gets NULL\nhandling right in the row-wise path and wrong in the vectorized one, so the two\nshapes disagree with each other while each looks correct in isolation. For\nevery shape, ask what happens when the *other* argument is NULL, and when it is\nNULL in only some rows. A shape that takes an array and a scalar has to make\nthe scalar's validation depend on the array's null mask, because a per-row null\nintolerant function skips validation on the rows it short circuits. All-NULL,\nsome-NULL, and no-NULL versions of the same array are three different tests.\n\nNote that a `(Array, Scalar)` branch returning `ColumnarValue::Scalar` is *not*\nby itself a bug. In DataFusion a returned `Scalar` means \"this value for every\nrow\" and the caller broadcasts it to the batch length. Before reporting one,\nrun an N-row query and count the rows that come back. Report it only if the\nrow count is actually wrong, or if the scalar's value depends on per-row input.\nOtherwise it is at most a readability cleanup, and reporting it as a\ncorrectness bug costs the audit credibility.\n\n**Overflow and underflow.** Does it return `Err`, wrap, or panic? Spark's\nanswer depends on ANSI mode. A panic is always a bug.\n\n**Type dispatch.** Every type the signature admits must have a branch. An\n`exec_err!` fallthrough for a type the signature accepts is a bug.\n\n**ANSI handling.** Compare against `datafusion.execution.enable_ansi_mode`\n(defined in `datafusion\u002Fcommon\u002Fsrc\u002Fconfig.rs`). The crate has scattered `TODO`\ncomments about unimplemented ANSI paths. Treat every one found in this\nfunction as a finding, not as a known-acceptable state. Grep for them across\nevery file Step 3's grep returned, including `mod.rs`:\n\n```bash\ngrep -rn \"TODO\" datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002F\u003Ccategory>\u002F$ARGUMENTS.rs \\\n                datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002F\u003Ccategory>\u002Fmod.rs\n```\n\nWiring ANSI up is usually mechanical rather than a semantics decision. The flag\nis already threaded through `ScalarFunctionArgs`, so the implementation reads\nit directly:\n\n```rust\nlet ScalarFunctionArgs { args, config_options, .. } = args;\nlet ansi_mode = config_options.execution.enable_ansi_mode;\n```\n\n`datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002Fmath\u002Fabs.rs` and\n`datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002Fmath\u002Fmodulus.rs` are the in-repo precedents for\nboth the plumbing and the test layout. DataFusion does not model Spark's error\nclasses or SQLSTATE values, so say that in the audit either way.\n\nWhether to match Spark's message text depends on who constructs the error:\n\n- **The function constructs it** (an `exec_err!` you write). Match Spark's text\n  exactly, including trailing punctuation, and assert it in the `.slt`.\n- **An Arrow kernel constructs it** (`rem`, `add`, a cast). You do not control\n  the wording. Assert only that an error is raised, note the divergence in a\n  comment, and file an issue for the message text rather than contorting the\n  implementation to restate an Arrow error in Spark's words.\n\nDo not read the `abs.rs` precedent as licence to invent DataFusion-flavoured\ntext for an error you *do* construct. `abs` reports `Int32 overflow on\nabs(-2147483648)` where Spark 4.2.0 reports `[ARITHMETIC_OVERFLOW] overflow.`,\nwhich is a known divergence tracked by\nhttps:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion\u002Fissues\u002F24145. Follow its ANSI plumbing and\ntest layout, not its wording.\n\n`abs` also shows why the \"who constructs it\" question is worth asking rather\nthan assuming. Its scalar path builds the message in the spark crate, but its\narray path borrows `make_try_abs_function!` from\n`datafusion\u002Ffunctions\u002Fsrc\u002Fmath\u002Fabs.rs`, so that half of the text is owned by\ncore DataFusion and shared with core's own `abs`. Wherever a spark function\nreuses a core macro or kernel, Spark-specific error text is not available\nwithout un-sharing the code first. Check where the message is actually built\nbefore promising to match Spark's.\n\n**Rust unit tests.** Read the `#[test]` blocks in the same file. Per the crate\nREADME, direct invocation through `test_scalar_function!()` bypasses input\ncoercion, so passing unit tests never establish end-to-end correctness. Note\nwhich cases are covered by unit tests only.\n\n## Step 4: Cross-Check Comet and Sail\n\nThe `datafusion-spark` README recommends checking both projects for existing\nimplementations. Diff against them to surface edge cases they handle that this\ncrate misses.\n\nNeither project needs to be checked out. Both are searched over HTTP, by\ncontent rather than by path, because both have moved their directory layouts\nmore than once:\n\n```bash\nfor repo in apache\u002Fdatafusion-comet lakehq\u002Fsail; do\n  echo \"=== $repo ===\"\n  gh api -X GET search\u002Fcode -f q=\"$ARGUMENTS repo:$repo\" --jq '.items[].path'\ndone\n\n# Then read a specific hit\ngh api \"repos\u002F\u003Crepo>\u002Fcontents\u002F\u003Cpath from above>\" --jq '.content' | base64 -d\n```\n\nIf a clone happens to be on disk, grep it instead. It is faster and it is not\nrate limited. Do not silence the failure with `2>\u002Fdev\u002Fnull`, though: a grep over\na directory that is not there prints an error, and swallowing it turns \"Comet\nnot checked out\" into \"Comet has no implementation\".\n\n```bash\n[ -d ..\u002Fdatafusion-comet ] \\\n  && grep -rln \"$ARGUMENTS\" ..\u002Fdatafusion-comet\u002Fnative\u002Fspark-expr\u002Fsrc\u002F --include=\"*.rs\" \\\n  || echo \"no local Comet clone, use the API above\"\n```\n\n`gh api search\u002Fcode` requires authentication and is rate limited to a handful of\nrequests per minute. If it fails, say so in the audit report and mark Step 4 as\nnot performed rather than reporting no differences found.\n\nAs of this writing the implementations live under\n`crates\u002Fsail-function\u002Fsrc\u002Fscalar\u002F\u003Ccategory>\u002Fspark_\u003Cfunction>.rs`, and Sail also\nkeeps behavior fixtures at\n`python\u002Fpysail\u002Ftests\u002Fspark\u002Ffunction\u002Ffeatures\u002F\u003Cfunction>.feature`. Treat both\npaths as hints, not as addresses.\n\nSail matters more than a generic peer here: many `datafusion-spark` functions\nwere ported from Sail, so Sail's current code is often the same code with\nsubsequent fixes applied. A difference between the two is a strong lead.\n\nTreat both as peer implementations, not as authority. Each has its own audit\nbacklog, and a shared assumption between them is not evidence of correctness.\nSpark's source and a running Spark are the only authorities.\n\n## Step 5: Review Existing SLT Coverage\n\n```bash\nls datafusion\u002Fsqllogictest\u002Ftest_files\u002Fspark\u002F*\u002F$ARGUMENTS.slt\n```\n\nRead the file and list the queries, the types exercised, which cases are\ncommented out, and whether the file has an audit header comment from a\nprevious audit.\n\nAlso read `datafusion\u002Fsqllogictest\u002Ftest_files\u002Fspark\u002FREADME.md`. It is the\nbinding local convention for this tree and it prescribes things this skill does\nnot repeat: every function must be tested on both scalar and array inputs,\nevery literal needs an explicit cast because DataFusion and Spark infer types\ndifferently, test cases contain `SELECT` statements only, and a function whose\nbehavior differs under ANSI is wrapped in\n\n```sql\nstatement ok\nset datafusion.execution.enable_ansi_mode = true;\n\n# queries\n\nstatement ok\nset datafusion.execution.enable_ansi_mode = false;\n```\n\nReset the flag to `false` at the end of the block. The setting leaks to\nwhatever follows it in the file otherwise.\n\nMany files in this tree were machine-generated from Sail's gold data. Their\nexpected values were not necessarily verified against Spark, so an existing\npassing expectation is not evidence of correctness. Re-verify in Step 7.\n\n## Step 6: Gap Analysis\n\nBuild this matrix, filling each cell with yes, no, or n\u002Fa:\n\n| Dimension                                      | Spark tests it | DataFusion SLT | Rust unit test | Gap? |\n| ---------------------------------------------- | -------------- | -------------- | -------------- | ---- |\n| All-scalar arguments                           |                |                |                |      |\n| All-array arguments                            |                |                |                |      |\n| Array first argument, scalar second            |                |                |                |      |\n| Scalar first argument, array second            |                |                |                |      |\n| NULL input, each argument independently        |                |                |                |      |\n| Empty string \u002F array \u002F map                     |                |                |                |      |\n| Zero, negative zero, negatives                 |                |                |                |      |\n| Overflow, underflow                            |                |                |                |      |\n| Boundary values (INT_MIN, INT_MAX, i64 bounds) |                |                |                |      |\n| NaN, Infinity, -Infinity, subnormal            |                |                |                |      |\n| Multibyte \u002F special UTF-8                      |                |                |                |      |\n| ANSI mode enabled                              |                |                |                |      |\n| ANSI mode disabled                             |                |                |                |      |\n| Every type in Spark's `inputTypes`             |                |                |                |      |\n| Types Spark rejects                            |                |                |                |      |\n| Return nullability                             |                |                |                |      |\n| Cross-version divergence                       |                |                |                |      |\n\nRows marked n\u002Fa need a one-line justification, for example \"no float inputs\"\nagainst the NaN row.\n\nBoth ANSI rows matter independently. DataFusion defaults `enable_ansi_mode` to\n`false` while Spark 4.x defaults ANSI to `true`, so the mode users actually hit\non Spark 4 is the one most likely to be untested here.\n\n## Step 7: Establish Ground Truth with PySpark\n\nNo expected value goes into a `.slt` file without being observed from a real\nSpark. Reading `nullSafeEval` and predicting the output is exactly how wrong\ngolden values get written in the first place.\n\n### Set up the environment\n\nRequires a JDK 17 or later on `PATH`. Use `uv` to create the venv rather than\n`python3 -m venv`: `pyspark==4.2.0` requires Python 3.10 or later, and the\nsystem `python3` on many machines is older. `uv` is already used by this\nrepository and will fetch a suitable interpreter if none is installed.\n\nUnset `SPARK_HOME` and `PYSPARK_PYTHON` before doing anything else. If either\nis set, a local Spark install on `SPARK_HOME` silently overrides the\npip-installed `pyspark` JARs, and `PYSPARK_PYTHON` can point PySpark's workers\nat a different Python than the venv you just built. The `SPARK_HOME` case is\nnot hypothetical: it reproduces reliably on a machine with any other Spark\ninstall on `PATH`, and it fails late and unhelpfully, inside `getOrCreate()`,\nas `TypeError: 'JavaPackage' object is not callable`. If you hit that exact\nerror, this is the cause: check `echo $SPARK_HOME` and unset it.\n\n**`unset` does not persist between tool calls.** Each shell invocation starts\nfrom the user's profile, which is where `SPARK_HOME` was exported in the first\nplace. Unsetting it once in the setup block and then running the harness in a\nlater call reintroduces the failure. Put `unset SPARK_HOME PYSPARK_PYTHON` in\nthe same command as every JVM-starting invocation, not in a preceding one.\n\n```bash\nunset SPARK_HOME PYSPARK_PYTHON\nuv venv --python 3.12 \u002Ftmp\u002Fpyspark-4.2.0-venv\nuv pip install --python \u002Ftmp\u002Fpyspark-4.2.0-venv --quiet 'pyspark==4.2.0'\n\u002Ftmp\u002Fpyspark-4.2.0-venv\u002Fbin\u002Fpython -c \"import pyspark; print(pyspark.__version__)\"\n```\n\nThe last command must print `4.2.0`. If it prints anything else, stop and fix\nthe environment: an audit against the wrong Spark version is worse than no\naudit, because it writes confident wrong expectations.\n\nThat check is necessary but not sufficient. It only reads the installed\npackage's metadata: it never starts a JVM, so it passes cleanly even on a\nmachine primed to hit the `SPARK_HOME` shadowing failure above. The check\nthat actually proves which Spark you are talking to is the\n`# Spark 4.2.0 tz=UTC` banner the harness prints from a live `SparkSession` in\nthe next section. Do not trust the environment until you have seen that\nbanner.\n\n### The harness\n\nWrite this to the scratchpad as `spark_ground_truth.py`. It is deliberately not\ncommitted to the repo: `pyspark` is too heavy a dependency to add to the `uv`\nworkspace in `dev\u002F`, and `datafusion\u002Fspark\u002Fscripts\u002F` is claimed by other\nin-flight work.\n\nWrite it unconditionally, overwriting whatever is already there. A previous\naudit may have left an older copy in the same scratchpad, and silently reusing\nit means auditing with a harness whose fixes you do not have.\n\n```python\n#!\u002Fusr\u002Fbin\u002Fenv python3\n\"\"\"Run SQL queries against a local PySpark and print SLT-ready results.\n\nUsage: spark_ground_truth.py QUERIES.sql\n\nInput is a file of SQL statements, each terminated by a semicolon at the end\nof a line. Each query is run under both spark.sql.ansi.enabled=true and\n=false, and the observed result is printed in a form that can be pasted into\na .slt file.\n\nThe session time zone is pinned to UTC so that results do not depend on the\nauditing machine's local zone. DataFusion's sqllogictest runner leaves\nexecution.time_zone unset, which is effectively UTC, so anything time zone\nsensitive would otherwise produce golden values that are wrong everywhere\nexcept on the machine that generated them.\n\nStatement splitting is naive: it splits on a semicolon followed by a newline.\nThat keeps a semicolon inside a string literal intact, as in\nSELECT ascii(';'), as long as the literal is not the last thing on the line.\nA query whose string literal ends a line with a semicolon still splits\nwrongly. Put such a query on a single line with trailing text after the\nliteral, or run it on its own.\n\"\"\"\n\nimport sys\n\nfrom pyspark.sql import SparkSession\n\n\ndef format_value(value):\n    \"\"\"Render one cell the way sqllogictest expects it.\"\"\"\n    if value is None:\n        return \"NULL\"\n    if isinstance(value, bool):\n        return \"true\" if value else \"false\"\n    return str(value)\n\n\ndef run_query(spark, sql):\n    try:\n        rows = spark.sql(sql).collect()\n    except Exception as exc:  # noqa: BLE001 - any failure is a result here\n        first_line = str(exc).strip().split(\"\\n\")[0]\n        return f\"ERROR: {type(exc).__name__}: {first_line}\"\n    if not rows:\n        return \"(no rows)\"\n    return \"\\n\".join(\"\\t\".join(format_value(v) for v in row) for row in rows)\n\n\ndef main(path):\n    spark = (\n        SparkSession.builder.appName(\"slt-ground-truth\")\n        .master(\"local[1]\")\n        .getOrCreate()\n    )\n    spark.conf.set(\"spark.sql.session.timeZone\", \"UTC\")\n    spark.sparkContext.setLogLevel(\"ERROR\")\n    print(f\"# Spark {spark.version} tz={spark.conf.get('spark.sql.session.timeZone')}\")\n\n    raw = open(path).read()\n    if not raw.endswith(\"\\n\"):\n        raw += \"\\n\"\n    queries = [q.strip() for q in raw.split(\";\\n\") if q.strip()]\n\n    for sql in queries:\n        for ansi in (True, False):\n            spark.conf.set(\"spark.sql.ansi.enabled\", str(ansi).lower())\n            print(f\"### ansi={str(ansi).lower()}\")\n            print(f\"# {sql}\")\n            print(run_query(spark, sql))\n            print()\n\n    spark.stop()\n\n\nif __name__ == \"__main__\":\n    if len(sys.argv) != 2:\n        sys.exit(\"usage: spark_ground_truth.py QUERIES.sql\")\n    main(sys.argv[1])\n```\n\n### Using it\n\nWrite every candidate query, including the ones already in the `.slt` file, to\na scratch `.sql` file and run:\n\n```bash\ncd \u003Cscratchpad>\nunset SPARK_HOME PYSPARK_PYTHON\n\u002Ftmp\u002Fpyspark-4.2.0-venv\u002Fbin\u002Fpython spark_ground_truth.py queries.sql \\\n  > ground_truth.txt 2> spark_logs.txt\n```\n\nKeep the streams separate. Spark writes log4j output, including structured JSON\nerror records, on a stream that will interleave with the results if you merge\nthem with `2>&1`. The merged form is what an agent reaches for by reflex and it\nmakes the output unreadable exactly on the error cases that matter most.\n\nVerify the banner line reads `# Spark 4.2.0 tz=UTC` before trusting any\noutput. Both halves matter. The version half proves you are not talking to a\nSpark install that shadowed the pip-installed one, and the time zone half\nproves the session is not silently inheriting the auditing machine's local\nzone. DataFusion's sqllogictest runner leaves `execution.time_zone` unset,\nwhich behaves as UTC, so a session on any other zone generates golden values\nthat only reproduce on the machine that generated them. This is invisible for\nfunctions that take a `DATE` and never touch a time zone, and it is a silent\ncorruption for `to_utc_timestamp`, `date_format`, `from_unixtime`, and every\nother timestamp function. If the banner shows anything but `tz=UTC`, stop and\nfix the harness before recording a single expected value.\n\n**Wrap date, timestamp, and interval results in `CAST(... AS STRING)`.** The\nharness calls `collect()`, which converts every cell to a Python object, and\nPython's `datetime.date` cannot represent everything Spark's `DateType` can. A\nvalue outside Python's range surfaces as a line like\n\n```\nERROR: ValueError: year 10000 is out of range\n```\n\nwhich reads exactly like a Spark error and is not one. Spark returned a value\nfine. Any `ValueError` or `OverflowError` in the output is a harness artifact,\nnever a Spark result. Re-run the query wrapped in a cast to see what Spark\nactually produced:\n\n```sql\nSELECT CAST(next_day(DATE '9999-12-31', 'Mon') AS STRING);\n-- +10000-01-03\n```\n\nRe-verify expectations that already exist in the `.slt` file. Many files in\nthis tree were machine-generated and their values were never checked against\nSpark. An existing expectation is a claim, not evidence.\n\nWrite the queries in Spark SQL. The `.slt` file needs DataFusion syntax, which\ndiffers: DataFusion uses `'2015-01-14'::DATE` where Spark SQL uses\n`DATE '2015-01-14'`. Translate when transcribing, and keep the semantics\nidentical. If a query cannot be expressed in both dialects, note it as a\nfinding rather than silently testing something different.\n\n### Never use `--complete`\n\nThe sqllogictest runner supports `--complete`, which fills in expected results\nwith whatever DataFusion currently returns.\n\n**Never run it on files under `test_files\u002Fspark\u002F`.** On an audit it would\nsilently cement the bug being audited as the golden answer, which inverts the\npurpose of the whole exercise. Expected values come from PySpark and are\nwritten by hand.\n\n### If PySpark cannot be installed\n\nContinue the audit using values derived from the Spark source and the golden\nfiles from Step 2, and mark every such expectation as unverified in both the\naudit report and a comment in the `.slt` file:\n\n```sql\n# UNVERIFIED: derived from Spark source, not confirmed against a running Spark\n```\n\nNever present a derived value as a verified one. An audit that overstates its\nconfidence is worse than one that admits a gap.\n\n## Step 8: Apply Findings\n\nFindings do not survive as prose. A recommendation in a PR description is\nburied the moment the PR merges. Every finding becomes code, a test, or a\nfiled issue.\n\n### Fix what can be fixed\n\nDivergences from Spark 4.2.0 are fixed in the audit PR. Attempt the fix\nregardless of size. Fall back to filing an issue only when the fix needs a\nsemantics decision a human should make, for example choosing which of two\ndefensible behaviors DataFusion should adopt when Spark's own behavior is\ninconsistent across versions.\n\nEvery fix is locked in by an SLT test that fails before the fix and passes\nafter. A test that passes before the fix was testing something else.\n\nVerify both directions before committing. Stash only the implementation\nchange, so the new `.slt` query runs against the unfixed code:\n\n```bash\n# Stash the implementation only, leaving the new .slt query in place\ngit stash push -- datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002F\u003Ccategory>\u002F\u003Cfunction>.rs\ncargo test --test sqllogictests -- spark\u002F\u003Ccategory>\u002F$ARGUMENTS   # expect FAIL\ngit stash pop\ncargo test --test sqllogictests -- spark\u002F\u003Ccategory>\u002F$ARGUMENTS   # expect PASS\n```\n\nThis verifies SLT tests only. Rust unit tests live inside the implementation\nfile, so stashing that file stashes the tests with it and the run proves\nnothing about them. For a unit test, get the fail-before evidence by reverting\nthe implementation body by hand, or by writing the test first and watching it\nfail before the fix exists.\n\nRead the failure list from the first run and confirm it names the queries you\nadded. Do not judge the run by an exit code you captured through a pipe:\n`cargo test ... | head` reports the exit status of `head`, not of `cargo`, so a\npiped command that \"succeeded\" tells you nothing. Either run it unpiped or\ninspect the `External error: N errors in file ...` block directly.\n\nIf the first run passes, the test is not exercising the fix. Either the\nquery was stashed along with the implementation, or the assertion does not\nactually cover the divergent case. Investigate before continuing: a test\nthat passes against the unfixed code locks in nothing.\n\nIf the fix also changed a `mod.rs` doc string or a helper in another file, that\nfile is not covered by the stash above. Add it to the `git stash push` argument\nlist, or the \"expect FAIL\" run may fail for the wrong reason.\n\nRun this before committing the fix. If `git stash push` reports\n`No local changes to save`, the change is already committed and the check is\ninvalid, because the \"expect FAIL\" run just tested the fixed code. Redo it\nagainst the pre-fix commit by checking that one file out of history and back,\nwhich touches only the working tree and never moves `HEAD` or a branch:\n\n```bash\n# \u003Cpre-fix-sha> is the commit before the one that introduced the fix\ngit checkout \u003Cpre-fix-sha> -- datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002F\u003Ccategory>\u002F\u003Cfunction>.rs\ncargo test --test sqllogictests -- spark\u002F\u003Ccategory>\u002F$ARGUMENTS   # expect FAIL\ngit checkout HEAD -- datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002F\u003Ccategory>\u002F\u003Cfunction>.rs\ncargo test --test sqllogictests -- spark\u002F\u003Ccategory>\u002F$ARGUMENTS   # expect PASS\n```\n\nFind `\u003Cpre-fix-sha>` with `git log --oneline -- \u003Cthe implementation file>` and\ntake the commit before the fix. If the fix also touched other files, list every\none of them in both `git checkout` commands.\n\nDo not reach for `git reset`, `git revert`, or `git rebase` here. None of them\nare needed and all of them can leave the branch in a state the audit then has\nto repair.\n\n### Capture what cannot be fixed\n\nA deferred divergence becomes a commented-out SLT query carrying the correct\nSpark 4.2.0 result and the issue link:\n\n```sql\n# Spark 4.2.0 raises ILLEGAL_DAY_OF_WEEK under ANSI mode; DataFusion returns NULL\n# https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion\u002Fissues\u002F23889   \u003C- the real number, not a placeholder\n# query error\n# SELECT next_day('2015-01-14'::DATE, 'NOT_A_DAY'::string);\n```\n\nUncommenting the block is the contract for verifying the eventual fix. Note\nthat DataFusion `.slt` files use `#` for comments, not `--`.\n\n**File the issue first, then write the block.** The block cannot be written\nuntil its issue number exists, so do the filing described below before typing\nthe comment. Never commit a placeholder in place of the number. A string like\n`\u003CISSUE-URL-PENDING>`, `NNNNN`, or `TBD` inside a URL ships a dead link, and\nbecause the surrounding block is commented out no test ever catches it.\n\nIf the issue cannot be filed at all, for example because `gh` is not\nauthenticated or the user asked that nothing be filed yet, omit the link line\nentirely. A block with no link is honest and self-contained. Then say so\nexplicitly in the audit report: name the divergence, state that no issue was\nfiled and why, and ask the user to file it. Do not leave the reader to\ndiscover the gap from a broken URL.\n\nWhen you do assert an error, remember that the runner normalizes whitespace in\nthe `query error` regex, so literal leading, trailing, and repeated spaces in\nthe expected message cannot be written out. An input like `' MO '` produces\n`Illegal input for day of week:  MO .` with two spaces, and a regex containing\nthose two spaces is collapsed to one before matching. Write `\\s+` instead,\nwhich carries no literal whitespace and survives normalization:\n\n```sql\nquery error Illegal input for day of week:\\s+MO \\.\nSELECT next_day('2015-01-14'::DATE, ' MO '::string);\n```\n\nThat asserts the padding is present without pinning its exact width. Anchor\nthe right-hand end of the match on a non-whitespace character from Spark's\nmessage, such as the trailing period in Spark 4.x error templates, so the\nassertion still fails if the padding runs into the rest of the message. If\neven that cannot be expressed, fall back to the stable prefix and say in a\ncomment why the assertion stops where it does.\n\nMatch Spark's message text exactly, including trailing punctuation. Spark 4.x\nerror templates in `error-conditions.json` generally end in a period, and the\npre-4.0 exception strings they replaced generally did not. Copying the older\nstring leaves the implementation asserting a version the audit does not claim,\nand a `query error` prefix match will not catch it.\n\nBefore filing, search for an existing issue:\n\n```bash\ngh issue list --repo apache\u002Fdatafusion --search \"$ARGUMENTS \u003Csymptom> in:title,body\" --state all --limit 5\n```\n\nIf a candidate comes back, open it with `gh issue view \u003CN> --repo apache\u002Fdatafusion`\nand confirm it actually describes this divergence and is still open. A closed\nissue cited as a known divergence is worse than no citation, because the reader\nfollows the link and finds a fix that already shipped.\n\nIf no issue exists, file one with the `bug` label plus `spark`. Title format:\n`[Bug] \u003Cfunction> \u003Cone-line symptom>`. The body includes the Spark version\nrange affected, a minimal repro in both Spark SQL and DataFusion SQL, the\ndivergent result, and the relevant file and line.\n\n### Capture cross-version divergences\n\nA behavior difference between Spark versions cannot be tested today. Capture it\nin two places.\n\nA comment at the query site naming the versions:\n\n```sql\n# Spark 3.5.8 returns NULL here; Spark 4.0.4 and later throw under ANSI mode.\n# Expectation below asserts 4.2.0. Version-specific expectations are tracked by\n# https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion\u002Fissues\u002F23887\n```\n\nAnd its own filed issue, so the divergence is scheduled once the version\nmechanism lands.\n\n### Record the audit\n\nAdd a header comment block to the `.slt` file, directly below the license\nheader and the existing provenance comment:\n\n```sql\n# Audited against Spark 4.2.0 on YYYY-MM-DD.\n# Expected values verified with pyspark==4.2.0.\n# Known divergences are captured as commented-out queries below.\n```\n\nUse today's real date. If PySpark was unavailable, say so on the second line\ninstead of claiming verification.\n\n### Attribution\n\nEvery issue and PR the skill files ends with:\n\n```\nSurfaced by the audit-datafusion-spark-expression skill.\n```\n\nDo not mention Claude, AI, or automated generation anywhere in an issue or PR.\n\n### Before committing\n\nPer the repository `CLAUDE.md`:\n\n```bash\ncargo fmt --all\ncargo clippy --all-targets --all-features -- -D warnings\ncargo test -p datafusion-spark --lib $ARGUMENTS\ncargo test --test sqllogictests -- spark\u002F\u003Ccategory>\u002F$ARGUMENTS\n```\n\nThe `--lib` run is not optional. `cargo test --test sqllogictests` does not\ncompile or run the `#[test]` blocks inside `datafusion\u002Fspark\u002Fsrc\u002F`, so a\nrefactor that renames or changes the signature of a helper those tests call\npasses the SLT gate and fails CI.\n\nDo not use `cargo build -p datafusion-spark` as a quick compile check. The\ncrate does not currently build in isolation and the resulting error is\nunrelated to your change. Use `cargo test -p datafusion-spark --lib` or the\nsqllogictest target instead.\n\nIf the audit touched documentation:\n\n```bash\n.\u002Fci\u002Fscripts\u002Fdoc_prettier_check.sh --write --allow-dirty\n```\n\nFollow `.github\u002Fpull_request_template.md` for the PR.\n\n## Output Format\n\nPresent the audit as:\n\n1. **Function summary.** What `$ARGUMENTS` does, its input and output types,\n   and its null behavior.\n2. **Spark version differences.** Across 3.5.8, 4.0.4, 4.1.3, and 4.2.0. Say\n   explicitly if there are none.\n3. **DataFusion implementation notes.** Including signature, coercion, and\n   nullability findings.\n4. **Gap matrix.** From Step 6, plus implementation gaps.\n5. **Findings.** Split into fixed in this PR, deferred with a filed issue, and\n   low-risk coverage gaps.\n6. **Offer to add the low-risk coverage tests.**\n\nLow-risk coverage gaps are the only category that pauses for user input.\nCorrectness findings are always either fixed or filed.\n\nThe line between the two is the code you touched. Coverage for a path this PR\nchanged goes into the `.slt` file without asking: it is part of the fix. The\noffer is for cases on paths the PR did not touch and that already look correct,\nwhere adding tests is optional scope the user may not want. Phrase the offer\nas:\n\n> Spark exercises the following cases that have no DataFusion test, on code\n> paths that appear correct. Would you like me to add them?\n>\n> - [list each case]\n\n## Tone and Style\n\n- Write in clear, concise prose.\n- Use backticks around function names, file paths, type names, and config keys.\n- Acknowledge what is already well covered before raising gaps.\n- Avoid em dashes and semicolons. Use separate sentences.\n- Do not overstate confidence. An unverified expectation is labeled as one.\n","\u003C!--\nSPDX-License-Identifier: Apache-2.0\nSPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership\nSPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements\n-->\n\nAudit the `datafusion-spark` implementation of the `$ARGUMENTS` function for\ncorrectness against Apache Spark.\n\n## Baseline Version\n\nThe baseline is **Spark 4.2.0**. Every expected value written into a `.slt`\nfile asserts Spark 4.2.0 behavior.\n\nSpark 3.5.8, 4.0.4, and 4.1.3 are also read, but only to detect that behavior\nchanged between versions. They never set an expectation. There is currently no\nmechanism for version-specific expectations in DataFusion's SLT files, which is\ntracked by https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion\u002Fissues\u002F23887.\n\nOlder versions are read rather than ignored because `datafusion-spark` is meant\nto be reusable by engines that target more than one Spark release. Comet\nsupports Spark 3.4 through 4.0. A function audited against 4.2.0 alone can be\ncorrect there and silently wrong for such a consumer, and nothing in the crate\nrecords which of the two it is. Finding the divergence is useful even though\nthe test suite cannot yet assert it.\n\n## Overview\n\n1. Study the Spark implementation across versions\n2. Harvest Spark's own test coverage\n3. Review the DataFusion implementation\n4. Cross-check Comet and Sail\n5. Review existing SLT coverage\n6. Gap analysis\n7. Establish ground truth with PySpark\n8. Apply findings\n\nAudit one function per invocation.\n\n## Paths in This Skill Are Hints\n\nEvery file path below was correct when this skill was written and several will\nrot. `FunctionRegistry.scala`, the golden-file directories, and\n`datafusion\u002Fcommon\u002Fsrc\u002Fconfig.rs` all move on someone else's schedule.\n\n**A path that does not resolve is a stop condition, not a step to skip.** The\nfailure mode this exists to prevent is a command that quietly matches nothing,\nan audit that reads the silence as \"no ANSI branch\" or \"Spark does not test\nthis\", and a confident report built on an empty grep. A `grep` over a\nnonexistent directory and a `grep` that genuinely found nothing are the same\noutput.\n\nSo when a path in this skill does not exist, or a command returns nothing where\nsomething was expected:\n\n1. Say so out loud in the audit report. Name the path and the step.\n2. Search for where the thing moved to, by content rather than by path. The\n   grep patterns here are more durable than the directories they are pointed at.\n3. If it cannot be found, mark every conclusion that depended on that step as\n   unestablished. Do not let an unread source become an absence of findings.\n\nThe same rule covers a `.slt` file, an implementation file, or a Spark class\nthat is not where this skill says it is.\n\n## Step 1: Study the Spark Implementation\n\nShallow-clone the four version tags. Skip any already present.\n\n```bash\nset -eu\nfor tag in v3.5.8 v4.0.4 v4.1.3 v4.2.0; do\n  dir=\"\u002Ftmp\u002Fspark-${tag}\"\n  [ -d \"$dir\" ] || git clone --depth 1 --branch \"$tag\" \\\n    https:\u002F\u002Fgithub.com\u002Fapache\u002Fspark.git \"$dir\"\ndone\n```\n\nFind the expression class in each version. Spark registers SQL function names\nin `FunctionRegistry.scala`, so start there to map `$ARGUMENTS` to its Scala\nclass name, then locate the class.\n\n```bash\nfor tag in v3.5.8 v4.0.4 v4.1.3 v4.2.0; do\n  dir=\"\u002Ftmp\u002Fspark-${tag}\"\n  echo \"=== $tag ===\"\n  grep -rn \"\\\"$ARGUMENTS\\\"\" \\\n    \"$dir\u002Fsql\u002Fcatalyst\u002Fsrc\u002Fmain\u002Fscala\u002Forg\u002Fapache\u002Fspark\u002Fsql\u002Fcatalyst\u002Fanalysis\u002FFunctionRegistry.scala\"\ndone\n```\n\nWith the class name in hand, read its source in each version:\n\n```bash\nfor tag in v3.5.8 v4.0.4 v4.1.3 v4.2.0; do\n  dir=\"\u002Ftmp\u002Fspark-${tag}\"\n  echo \"=== $tag ===\"\n  find \"$dir\u002Fsql\u002Fcatalyst\u002Fsrc\u002Fmain\u002Fscala\" -name \"*.scala\" \\\n    | xargs grep -lE \"case class \u003CClassName>[ (\\[]\" 2>\u002Fdev\u002Fnull\ndone\n```\n\nA Scala `case class` name is always followed by a space, a `(`, or a `[`, so\nthat bracket expression is the portable way to avoid matching a longer class\nname that starts with the same prefix. Do not reach for `\\b`, which is a GNU\nand ugrep extension and is not available in POSIX or BSD `grep`.\n\nIf it is not in catalyst, widen the search to `$dir\u002Fsql`.\n\nFor each version, record:\n\n- `eval` \u002F `nullSafeEval` \u002F `doGenCode` logic\n- `inputTypes` and `dataType`\n- `nullable`, and how nulls propagate\n- ANSI branches (`ansiEnabled`, `failOnError`)\n- guards, `require` assertions, thrown exceptions\n- any SQLConf the expression reads\n- for string functions in 4.0+, whether it routes through\n  `CollationSupport.X.exec(..., collationId)` or declares\n  `StringTypeWithCollation` inputs\n\nProduce a diff summary across 3.5.8 → 4.0.4 → 4.1.3 → 4.2.0. Note new or\nremoved input types, changed edge-case behavior, new ANSI branches, new\nparameters, and collation changes.\n\nTwo changes affect nearly every function and are easy to miss:\n\n- ANSI mode became the **default** in Spark 4.0. A function whose behavior\n  differs under ANSI has a different default behavior in 3.5 than in 4.x.\n- Collation support landed for many string functions in 4.0. Non-default\n  collations are a divergence axis that does not exist in 3.5.\n\n## Step 2: Harvest Spark's Own Test Coverage\n\nSpark's tests are a ready-made edge-case inventory. Two sources matter.\n\nUnit tests:\n\n```bash\ndir=\u002Ftmp\u002Fspark-v4.2.0\nfind \"$dir\u002Fsql\" -name \"*.scala\" -path \"*\u002Ftest\u002F*\" \\\n  | xargs grep -ln \"$ARGUMENTS\" 2>\u002Fdev\u002Fnull\n```\n\nGolden files, which carry Spark's own verified query results and are the\nhighest-value source:\n\n```bash\nfor tag in v3.5.8 v4.0.4 v4.1.3 v4.2.0; do\n  dir=\"\u002Ftmp\u002Fspark-${tag}\"\n  echo \"=== $tag ===\"\n  grep -rln \"$ARGUMENTS\" \"$dir\u002Fsql\u002Fcore\u002Fsrc\u002Ftest\u002Fresources\u002Fsql-tests\u002Fresults\u002F\" 2>\u002Fdev\u002Fnull\ndone\n```\n\n**The ANSI and non-ANSI golden directories swapped meaning in Spark 4.0.** Read\nthe version banner before trusting a path:\n\n| Version | ANSI-mode results        | Non-ANSI results            |\n| ------- | ------------------------ | --------------------------- |\n| 3.5.8   | `results\u002Fansi\u002FX.sql.out` | `results\u002FX.sql.out`         |\n| 4.x     | `results\u002FX.sql.out`      | `results\u002Fnonansi\u002FX.sql.out` |\n\nANSI became the default in 4.0, so the unqualified path flipped from meaning\nnon-ANSI to meaning ANSI. An auditor who learns the layout on 3.5.8 and then\nreads `results\u002FX.sql.out` on 4.2.0 will record exactly inverted expectations,\nand they will look plausible. Confirm the mode by checking whether the file\ncontains error output or `NULL` for a known-invalid input.\n\nRead every match and extract:\n\n- input types covered\n- edge cases exercised: null, empty, overflow, negative, boundary, special\n  characters\n- ANSI-mode cases\n- error cases and their exact messages\n\nThis list is the reference for the gap matrix in Step 6.\n\n## Step 3: Review the DataFusion Implementation\n\n```bash\ngrep -rln \"$ARGUMENTS\" datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002F --include=\"*.rs\"\n```\n\nThe implementation lives at\n`datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002F\u003Ccategory>\u002F\u003Cfunction>.rs`. **Read every other\nfile that grep returns as well.** In particular\n`datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002F\u003Ccategory>\u002Fmod.rs` carries the `export_functions!`\nregistration and the user-facing doc string, and unimplemented-behavior `TODO`\ncomments are written there too, not only in the function's own file. An audit\nthat reads only `\u003Cfunction>.rs` will miss them and will leave a doc string that\ncontradicts the code after a fix.\n\n**A `.slt` file under `test_files\u002Fspark\u002F` does not imply an implementation in\n`datafusion\u002Fspark`.** `SessionStateBuilderSpark::with_spark_features` layers the\nSpark functions on top of `with_default_features`, so a name the spark crate\ndoes not define resolves to DataFusion's own function of that name. `md5`,\n`coalesce`, and `reverse` are served this way today, and `date_format` resolves\nthrough an alias on core's `to_char`. The query under audit still runs, and the\ncode it runs is not in the directory the grep above searched. If that grep comes\nback empty, widen it before concluding the function is unimplemented:\n\n```bash\ngrep -rln \"\\\"$ARGUMENTS\\\"\" \\\n  datafusion\u002Ffunctions\u002Fsrc datafusion\u002Ffunctions-nested\u002Fsrc \\\n  datafusion\u002Ffunctions-aggregate\u002Fsrc datafusion\u002Ffunctions-window\u002Fsrc \\\n  datafusion\u002Ffunctions-table\u002Fsrc --include=\"*.rs\"\n```\n\nA hit may be in a file named after something else, because core registers\naliases. `date_format` lives in `to_char.rs`. Read the `aliases` vector, not\njust the file name.\n\nIf that grep is also empty, the function is genuinely unimplemented and the\n`.slt` file is a porting stub with every query commented out. Check with\n`grep -cE '^(query|statement)' \u003Cfile>.slt`, which returns zero for a stub.\nThere is no implementation to audit in that case. Say so and stop, rather than\nreporting a clean audit of nothing.\n\nOtherwise audit what the grep found against Spark with the rest of this step's\nchecklist. The findings land differently. A core function has non-Spark users,\nso Spark's semantics cannot be imposed on it. Where the two agree, record the\ncoverage and leave the core function alone. Where they diverge, the fix is a\nSpark-specific implementation under `datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002F\u003Ccategory>\u002F`\nthat shadows the core one, never an edit to the core function.\n`datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002Fmath\u002Fabs.rs` shadowing\n`datafusion\u002Ffunctions\u002Fsrc\u002Fmath\u002Fabs.rs` is the in-repo precedent. Say in the\naudit report which crate the audited code came from, because a reader who\nassumes `datafusion\u002Fspark` will look for a file that does not exist.\n\nCheck each of the following. The first two have no Comet analog and are the\nmost common source of silent divergence.\n\n**Signature and coercion.** Compare `Signature` (and `coerce_types`, if\nimplemented) against Spark's `inputTypes`. Two failure directions, both real:\n\n- DataFusion rejects a type Spark accepts. The user gets a coercion error\n  where Spark returns a value.\n- DataFusion accepts a type Spark rejects. The user gets a value where Spark\n  raises an analysis error.\n\nNote that `Signature::exact` is narrow. A Spark function accepting any string\ntype needs all of `Utf8`, `LargeUtf8`, and `Utf8View` handled, and a function\naccepting any integral type needs the full width range.\n\n**Return nullability.** Compare `return_field_from_args` (or `return_type`)\nagainst Spark's `nullable`. A function that reports non-nullable while Spark\ncan return NULL produces wrong plans, not merely wrong values: downstream\noptimizer rules will fold away null checks that were load-bearing.\n\n**Null propagation.** Does every input-null combination return what Spark\nreturns? Check the order in which nullness and validity are tested. Spark\nexpressions that are null intolerant (a `NullIntolerant` mixin in 3.5, a\n`nullIntolerant` override in 4.x) short circuit to NULL *before* validating the\nother arguments, so a NULL in one argument suppresses an error the other\nargument would otherwise raise, even under ANSI mode. An implementation that\nvalidates first and checks nullness second raises where Spark returns NULL.\n\n**Arrow kernel semantics are not Java semantics.** A `datafusion-spark`\nfunction is usually a thin arrangement of Arrow kernels, and the kernels do not\nshare Java's rules. Check each one the implementation leans on:\n\n- **Comparison kernels order floats by total order**, so `-0.0` compares as\n  *less than* `0.0` rather than equal to it. Any `eq`\u002F`lt` used to detect a\n  special value, such as a zero divisor, silently misses `-0.0`. Java compares\n  them as equal. This is easy to miss because `-0.0` almost never appears in\n  hand-written test values, so write it in deliberately.\n- **Arithmetic kernels are checked, not wrapping.** Java wraps silently on\n  overflow, so a checked Arrow kernel raises where Spark returns a wrapped\n  value. Where Spark's own expression is written in `Int` arithmetic, a\n  wrapping kernel is the faithful choice, not the reckless one.\n- **Java promotes `byte` and `short` operands to `int`.** A Spark expression\n  whose Scala source reads `(r + n)` on `Byte` operands is evaluated at `Int`\n  width and cannot overflow, while the same source on `Int` operands can.\n  Reproducing it at `Int8` width wraps where Spark does not.\n- **Kernel coverage is type dependent.** Arrow's `rem` raises on a zero divisor\n  for integers and decimals but returns `NaN` for floats, so an ANSI check that\n  relies on the kernel to raise silently does nothing on the float path.\n\nRead the Scala source as arithmetic to be reproduced, not as pseudocode to be\nparaphrased. Then test each of these against a live Spark rather than reasoning\nabout them.\n\n**`ColumnarValue` shape dispatch.** A two-argument function has four shape\ncombinations, and each needs its own branch:\n`(Scalar, Scalar)`, `(Array, Scalar)`, `(Scalar, Array)`, `(Array, Array)`.\nA missing arm falls through to a catch-all `exec_err!` and turns a working\nSpark query into a hard error. `(Scalar, Array)` is the one most often\nforgotten, because the natural test query puts the column first. Write a query\nfor every combination and run it. Do not infer coverage from reading the\n`match`.\n\n**How to run an ad-hoc query.** The only in-repo way to execute a\n`datafusion-spark` function is to add the query to\n`datafusion\u002Fsqllogictest\u002Ftest_files\u002Fspark\u002F\u003Ccategory>\u002F$ARGUMENTS.slt` and run\n\n```bash\ncargo test --test sqllogictests -- spark\u002F\u003Ccategory>\u002F$ARGUMENTS\n```\n\n`datafusion-cli` does **not** register the `datafusion-spark` function library,\nso reaching for it fails in a way that looks like the function does not exist:\n\n```\n$ .\u002Ftarget\u002Fdebug\u002Fdatafusion-cli -c \"SELECT next_day(arrow_cast(95026236,'Date32'),'Mon');\"\nError during planning: Invalid function 'next_day'. Did you mean 'today'?\n```\n\nGiving the CLI an opt-in for the Spark library is tracked by\nhttps:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion\u002Fissues\u002F24146. Until that lands, treat the\nerror above as a registration gap and not as evidence about the function.\n\nFor a throwaway probe that should not end up in the committed file, add a\ntemporary `.slt` alongside it, run it by name, and delete it afterwards. To\nexercise a raw storage value that has no literal syntax, such as a `Date32`\npast the last date DataFusion can render, wrap it in `arrow_cast`.\n\n**Cross the shapes with the NULL cases, and with ANSI mode.** These are not\nthree independent checklists to tick off separately, they are one product to\nenumerate. The bug this instruction exists to catch is a branch that gets NULL\nhandling right in the row-wise path and wrong in the vectorized one, so the two\nshapes disagree with each other while each looks correct in isolation. For\nevery shape, ask what happens when the *other* argument is NULL, and when it is\nNULL in only some rows. A shape that takes an array and a scalar has to make\nthe scalar's validation depend on the array's null mask, because a per-row null\nintolerant function skips validation on the rows it short circuits. All-NULL,\nsome-NULL, and no-NULL versions of the same array are three different tests.\n\nNote that a `(Array, Scalar)` branch returning `ColumnarValue::Scalar` is *not*\nby itself a bug. In DataFusion a returned `Scalar` means \"this value for every\nrow\" and the caller broadcasts it to the batch length. Before reporting one,\nrun an N-row query and count the rows that come back. Report it only if the\nrow count is actually wrong, or if the scalar's value depends on per-row input.\nOtherwise it is at most a readability cleanup, and reporting it as a\ncorrectness bug costs the audit credibility.\n\n**Overflow and underflow.** Does it return `Err`, wrap, or panic? Spark's\nanswer depends on ANSI mode. A panic is always a bug.\n\n**Type dispatch.** Every type the signature admits must have a branch. An\n`exec_err!` fallthrough for a type the signature accepts is a bug.\n\n**ANSI handling.** Compare against `datafusion.execution.enable_ansi_mode`\n(defined in `datafusion\u002Fcommon\u002Fsrc\u002Fconfig.rs`). The crate has scattered `TODO`\ncomments about unimplemented ANSI paths. Treat every one found in this\nfunction as a finding, not as a known-acceptable state. Grep for them across\nevery file Step 3's grep returned, including `mod.rs`:\n\n```bash\ngrep -rn \"TODO\" datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002F\u003Ccategory>\u002F$ARGUMENTS.rs \\\n                datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002F\u003Ccategory>\u002Fmod.rs\n```\n\nWiring ANSI up is usually mechanical rather than a semantics decision. The flag\nis already threaded through `ScalarFunctionArgs`, so the implementation reads\nit directly:\n\n```rust\nlet ScalarFunctionArgs { args, config_options, .. } = args;\nlet ansi_mode = config_options.execution.enable_ansi_mode;\n```\n\n`datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002Fmath\u002Fabs.rs` and\n`datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002Fmath\u002Fmodulus.rs` are the in-repo precedents for\nboth the plumbing and the test layout. DataFusion does not model Spark's error\nclasses or SQLSTATE values, so say that in the audit either way.\n\nWhether to match Spark's message text depends on who constructs the error:\n\n- **The function constructs it** (an `exec_err!` you write). Match Spark's text\n  exactly, including trailing punctuation, and assert it in the `.slt`.\n- **An Arrow kernel constructs it** (`rem`, `add`, a cast). You do not control\n  the wording. Assert only that an error is raised, note the divergence in a\n  comment, and file an issue for the message text rather than contorting the\n  implementation to restate an Arrow error in Spark's words.\n\nDo not read the `abs.rs` precedent as licence to invent DataFusion-flavoured\ntext for an error you *do* construct. `abs` reports `Int32 overflow on\nabs(-2147483648)` where Spark 4.2.0 reports `[ARITHMETIC_OVERFLOW] overflow.`,\nwhich is a known divergence tracked by\nhttps:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion\u002Fissues\u002F24145. Follow its ANSI plumbing and\ntest layout, not its wording.\n\n`abs` also shows why the \"who constructs it\" question is worth asking rather\nthan assuming. Its scalar path builds the message in the spark crate, but its\narray path borrows `make_try_abs_function!` from\n`datafusion\u002Ffunctions\u002Fsrc\u002Fmath\u002Fabs.rs`, so that half of the text is owned by\ncore DataFusion and shared with core's own `abs`. Wherever a spark function\nreuses a core macro or kernel, Spark-specific error text is not available\nwithout un-sharing the code first. Check where the message is actually built\nbefore promising to match Spark's.\n\n**Rust unit tests.** Read the `#[test]` blocks in the same file. Per the crate\nREADME, direct invocation through `test_scalar_function!()` bypasses input\ncoercion, so passing unit tests never establish end-to-end correctness. Note\nwhich cases are covered by unit tests only.\n\n## Step 4: Cross-Check Comet and Sail\n\nThe `datafusion-spark` README recommends checking both projects for existing\nimplementations. Diff against them to surface edge cases they handle that this\ncrate misses.\n\nNeither project needs to be checked out. Both are searched over HTTP, by\ncontent rather than by path, because both have moved their directory layouts\nmore than once:\n\n```bash\nfor repo in apache\u002Fdatafusion-comet lakehq\u002Fsail; do\n  echo \"=== $repo ===\"\n  gh api -X GET search\u002Fcode -f q=\"$ARGUMENTS repo:$repo\" --jq '.items[].path'\ndone\n\n# Then read a specific hit\ngh api \"repos\u002F\u003Crepo>\u002Fcontents\u002F\u003Cpath from above>\" --jq '.content' | base64 -d\n```\n\nIf a clone happens to be on disk, grep it instead. It is faster and it is not\nrate limited. Do not silence the failure with `2>\u002Fdev\u002Fnull`, though: a grep over\na directory that is not there prints an error, and swallowing it turns \"Comet\nnot checked out\" into \"Comet has no implementation\".\n\n```bash\n[ -d ..\u002Fdatafusion-comet ] \\\n  && grep -rln \"$ARGUMENTS\" ..\u002Fdatafusion-comet\u002Fnative\u002Fspark-expr\u002Fsrc\u002F --include=\"*.rs\" \\\n  || echo \"no local Comet clone, use the API above\"\n```\n\n`gh api search\u002Fcode` requires authentication and is rate limited to a handful of\nrequests per minute. If it fails, say so in the audit report and mark Step 4 as\nnot performed rather than reporting no differences found.\n\nAs of this writing the implementations live under\n`crates\u002Fsail-function\u002Fsrc\u002Fscalar\u002F\u003Ccategory>\u002Fspark_\u003Cfunction>.rs`, and Sail also\nkeeps behavior fixtures at\n`python\u002Fpysail\u002Ftests\u002Fspark\u002Ffunction\u002Ffeatures\u002F\u003Cfunction>.feature`. Treat both\npaths as hints, not as addresses.\n\nSail matters more than a generic peer here: many `datafusion-spark` functions\nwere ported from Sail, so Sail's current code is often the same code with\nsubsequent fixes applied. A difference between the two is a strong lead.\n\nTreat both as peer implementations, not as authority. Each has its own audit\nbacklog, and a shared assumption between them is not evidence of correctness.\nSpark's source and a running Spark are the only authorities.\n\n## Step 5: Review Existing SLT Coverage\n\n```bash\nls datafusion\u002Fsqllogictest\u002Ftest_files\u002Fspark\u002F*\u002F$ARGUMENTS.slt\n```\n\nRead the file and list the queries, the types exercised, which cases are\ncommented out, and whether the file has an audit header comment from a\nprevious audit.\n\nAlso read `datafusion\u002Fsqllogictest\u002Ftest_files\u002Fspark\u002FREADME.md`. It is the\nbinding local convention for this tree and it prescribes things this skill does\nnot repeat: every function must be tested on both scalar and array inputs,\nevery literal needs an explicit cast because DataFusion and Spark infer types\ndifferently, test cases contain `SELECT` statements only, and a function whose\nbehavior differs under ANSI is wrapped in\n\n```sql\nstatement ok\nset datafusion.execution.enable_ansi_mode = true;\n\n# queries\n\nstatement ok\nset datafusion.execution.enable_ansi_mode = false;\n```\n\nReset the flag to `false` at the end of the block. The setting leaks to\nwhatever follows it in the file otherwise.\n\nMany files in this tree were machine-generated from Sail's gold data. Their\nexpected values were not necessarily verified against Spark, so an existing\npassing expectation is not evidence of correctness. Re-verify in Step 7.\n\n## Step 6: Gap Analysis\n\nBuild this matrix, filling each cell with yes, no, or n\u002Fa:\n\n| Dimension                                      | Spark tests it | DataFusion SLT | Rust unit test | Gap? |\n| ---------------------------------------------- | -------------- | -------------- | -------------- | ---- |\n| All-scalar arguments                           |                |                |                |      |\n| All-array arguments                            |                |                |                |      |\n| Array first argument, scalar second            |                |                |                |      |\n| Scalar first argument, array second            |                |                |                |      |\n| NULL input, each argument independently        |                |                |                |      |\n| Empty string \u002F array \u002F map                     |                |                |                |      |\n| Zero, negative zero, negatives                 |                |                |                |      |\n| Overflow, underflow                            |                |                |                |      |\n| Boundary values (INT_MIN, INT_MAX, i64 bounds) |                |                |                |      |\n| NaN, Infinity, -Infinity, subnormal            |                |                |                |      |\n| Multibyte \u002F special UTF-8                      |                |                |                |      |\n| ANSI mode enabled                              |                |                |                |      |\n| ANSI mode disabled                             |                |                |                |      |\n| Every type in Spark's `inputTypes`             |                |                |                |      |\n| Types Spark rejects                            |                |                |                |      |\n| Return nullability                             |                |                |                |      |\n| Cross-version divergence                       |                |                |                |      |\n\nRows marked n\u002Fa need a one-line justification, for example \"no float inputs\"\nagainst the NaN row.\n\nBoth ANSI rows matter independently. DataFusion defaults `enable_ansi_mode` to\n`false` while Spark 4.x defaults ANSI to `true`, so the mode users actually hit\non Spark 4 is the one most likely to be untested here.\n\n## Step 7: Establish Ground Truth with PySpark\n\nNo expected value goes into a `.slt` file without being observed from a real\nSpark. Reading `nullSafeEval` and predicting the output is exactly how wrong\ngolden values get written in the first place.\n\n### Set up the environment\n\nRequires a JDK 17 or later on `PATH`. Use `uv` to create the venv rather than\n`python3 -m venv`: `pyspark==4.2.0` requires Python 3.10 or later, and the\nsystem `python3` on many machines is older. `uv` is already used by this\nrepository and will fetch a suitable interpreter if none is installed.\n\nUnset `SPARK_HOME` and `PYSPARK_PYTHON` before doing anything else. If either\nis set, a local Spark install on `SPARK_HOME` silently overrides the\npip-installed `pyspark` JARs, and `PYSPARK_PYTHON` can point PySpark's workers\nat a different Python than the venv you just built. The `SPARK_HOME` case is\nnot hypothetical: it reproduces reliably on a machine with any other Spark\ninstall on `PATH`, and it fails late and unhelpfully, inside `getOrCreate()`,\nas `TypeError: 'JavaPackage' object is not callable`. If you hit that exact\nerror, this is the cause: check `echo $SPARK_HOME` and unset it.\n\n**`unset` does not persist between tool calls.** Each shell invocation starts\nfrom the user's profile, which is where `SPARK_HOME` was exported in the first\nplace. Unsetting it once in the setup block and then running the harness in a\nlater call reintroduces the failure. Put `unset SPARK_HOME PYSPARK_PYTHON` in\nthe same command as every JVM-starting invocation, not in a preceding one.\n\n```bash\nunset SPARK_HOME PYSPARK_PYTHON\nuv venv --python 3.12 \u002Ftmp\u002Fpyspark-4.2.0-venv\nuv pip install --python \u002Ftmp\u002Fpyspark-4.2.0-venv --quiet 'pyspark==4.2.0'\n\u002Ftmp\u002Fpyspark-4.2.0-venv\u002Fbin\u002Fpython -c \"import pyspark; print(pyspark.__version__)\"\n```\n\nThe last command must print `4.2.0`. If it prints anything else, stop and fix\nthe environment: an audit against the wrong Spark version is worse than no\naudit, because it writes confident wrong expectations.\n\nThat check is necessary but not sufficient. It only reads the installed\npackage's metadata: it never starts a JVM, so it passes cleanly even on a\nmachine primed to hit the `SPARK_HOME` shadowing failure above. The check\nthat actually proves which Spark you are talking to is the\n`# Spark 4.2.0 tz=UTC` banner the harness prints from a live `SparkSession` in\nthe next section. Do not trust the environment until you have seen that\nbanner.\n\n### The harness\n\nWrite this to the scratchpad as `spark_ground_truth.py`. It is deliberately not\ncommitted to the repo: `pyspark` is too heavy a dependency to add to the `uv`\nworkspace in `dev\u002F`, and `datafusion\u002Fspark\u002Fscripts\u002F` is claimed by other\nin-flight work.\n\nWrite it unconditionally, overwriting whatever is already there. A previous\naudit may have left an older copy in the same scratchpad, and silently reusing\nit means auditing with a harness whose fixes you do not have.\n\n```python\n#!\u002Fusr\u002Fbin\u002Fenv python3\n\"\"\"Run SQL queries against a local PySpark and print SLT-ready results.\n\nUsage: spark_ground_truth.py QUERIES.sql\n\nInput is a file of SQL statements, each terminated by a semicolon at the end\nof a line. Each query is run under both spark.sql.ansi.enabled=true and\n=false, and the observed result is printed in a form that can be pasted into\na .slt file.\n\nThe session time zone is pinned to UTC so that results do not depend on the\nauditing machine's local zone. DataFusion's sqllogictest runner leaves\nexecution.time_zone unset, which is effectively UTC, so anything time zone\nsensitive would otherwise produce golden values that are wrong everywhere\nexcept on the machine that generated them.\n\nStatement splitting is naive: it splits on a semicolon followed by a newline.\nThat keeps a semicolon inside a string literal intact, as in\nSELECT ascii(';'), as long as the literal is not the last thing on the line.\nA query whose string literal ends a line with a semicolon still splits\nwrongly. Put such a query on a single line with trailing text after the\nliteral, or run it on its own.\n\"\"\"\n\nimport sys\n\nfrom pyspark.sql import SparkSession\n\n\ndef format_value(value):\n    \"\"\"Render one cell the way sqllogictest expects it.\"\"\"\n    if value is None:\n        return \"NULL\"\n    if isinstance(value, bool):\n        return \"true\" if value else \"false\"\n    return str(value)\n\n\ndef run_query(spark, sql):\n    try:\n        rows = spark.sql(sql).collect()\n    except Exception as exc:  # noqa: BLE001 - any failure is a result here\n        first_line = str(exc).strip().split(\"\\n\")[0]\n        return f\"ERROR: {type(exc).__name__}: {first_line}\"\n    if not rows:\n        return \"(no rows)\"\n    return \"\\n\".join(\"\\t\".join(format_value(v) for v in row) for row in rows)\n\n\ndef main(path):\n    spark = (\n        SparkSession.builder.appName(\"slt-ground-truth\")\n        .master(\"local[1]\")\n        .getOrCreate()\n    )\n    spark.conf.set(\"spark.sql.session.timeZone\", \"UTC\")\n    spark.sparkContext.setLogLevel(\"ERROR\")\n    print(f\"# Spark {spark.version} tz={spark.conf.get('spark.sql.session.timeZone')}\")\n\n    raw = open(path).read()\n    if not raw.endswith(\"\\n\"):\n        raw += \"\\n\"\n    queries = [q.strip() for q in raw.split(\";\\n\") if q.strip()]\n\n    for sql in queries:\n        for ansi in (True, False):\n            spark.conf.set(\"spark.sql.ansi.enabled\", str(ansi).lower())\n            print(f\"### ansi={str(ansi).lower()}\")\n            print(f\"# {sql}\")\n            print(run_query(spark, sql))\n            print()\n\n    spark.stop()\n\n\nif __name__ == \"__main__\":\n    if len(sys.argv) != 2:\n        sys.exit(\"usage: spark_ground_truth.py QUERIES.sql\")\n    main(sys.argv[1])\n```\n\n### Using it\n\nWrite every candidate query, including the ones already in the `.slt` file, to\na scratch `.sql` file and run:\n\n```bash\ncd \u003Cscratchpad>\nunset SPARK_HOME PYSPARK_PYTHON\n\u002Ftmp\u002Fpyspark-4.2.0-venv\u002Fbin\u002Fpython spark_ground_truth.py queries.sql \\\n  > ground_truth.txt 2> spark_logs.txt\n```\n\nKeep the streams separate. Spark writes log4j output, including structured JSON\nerror records, on a stream that will interleave with the results if you merge\nthem with `2>&1`. The merged form is what an agent reaches for by reflex and it\nmakes the output unreadable exactly on the error cases that matter most.\n\nVerify the banner line reads `# Spark 4.2.0 tz=UTC` before trusting any\noutput. Both halves matter. The version half proves you are not talking to a\nSpark install that shadowed the pip-installed one, and the time zone half\nproves the session is not silently inheriting the auditing machine's local\nzone. DataFusion's sqllogictest runner leaves `execution.time_zone` unset,\nwhich behaves as UTC, so a session on any other zone generates golden values\nthat only reproduce on the machine that generated them. This is invisible for\nfunctions that take a `DATE` and never touch a time zone, and it is a silent\ncorruption for `to_utc_timestamp`, `date_format`, `from_unixtime`, and every\nother timestamp function. If the banner shows anything but `tz=UTC`, stop and\nfix the harness before recording a single expected value.\n\n**Wrap date, timestamp, and interval results in `CAST(... AS STRING)`.** The\nharness calls `collect()`, which converts every cell to a Python object, and\nPython's `datetime.date` cannot represent everything Spark's `DateType` can. A\nvalue outside Python's range surfaces as a line like\n\n```\nERROR: ValueError: year 10000 is out of range\n```\n\nwhich reads exactly like a Spark error and is not one. Spark returned a value\nfine. Any `ValueError` or `OverflowError` in the output is a harness artifact,\nnever a Spark result. Re-run the query wrapped in a cast to see what Spark\nactually produced:\n\n```sql\nSELECT CAST(next_day(DATE '9999-12-31', 'Mon') AS STRING);\n-- +10000-01-03\n```\n\nRe-verify expectations that already exist in the `.slt` file. Many files in\nthis tree were machine-generated and their values were never checked against\nSpark. An existing expectation is a claim, not evidence.\n\nWrite the queries in Spark SQL. The `.slt` file needs DataFusion syntax, which\ndiffers: DataFusion uses `'2015-01-14'::DATE` where Spark SQL uses\n`DATE '2015-01-14'`. Translate when transcribing, and keep the semantics\nidentical. If a query cannot be expressed in both dialects, note it as a\nfinding rather than silently testing something different.\n\n### Never use `--complete`\n\nThe sqllogictest runner supports `--complete`, which fills in expected results\nwith whatever DataFusion currently returns.\n\n**Never run it on files under `test_files\u002Fspark\u002F`.** On an audit it would\nsilently cement the bug being audited as the golden answer, which inverts the\npurpose of the whole exercise. Expected values come from PySpark and are\nwritten by hand.\n\n### If PySpark cannot be installed\n\nContinue the audit using values derived from the Spark source and the golden\nfiles from Step 2, and mark every such expectation as unverified in both the\naudit report and a comment in the `.slt` file:\n\n```sql\n# UNVERIFIED: derived from Spark source, not confirmed against a running Spark\n```\n\nNever present a derived value as a verified one. An audit that overstates its\nconfidence is worse than one that admits a gap.\n\n## Step 8: Apply Findings\n\nFindings do not survive as prose. A recommendation in a PR description is\nburied the moment the PR merges. Every finding becomes code, a test, or a\nfiled issue.\n\n### Fix what can be fixed\n\nDivergences from Spark 4.2.0 are fixed in the audit PR. Attempt the fix\nregardless of size. Fall back to filing an issue only when the fix needs a\nsemantics decision a human should make, for example choosing which of two\ndefensible behaviors DataFusion should adopt when Spark's own behavior is\ninconsistent across versions.\n\nEvery fix is locked in by an SLT test that fails before the fix and passes\nafter. A test that passes before the fix was testing something else.\n\nVerify both directions before committing. Stash only the implementation\nchange, so the new `.slt` query runs against the unfixed code:\n\n```bash\n# Stash the implementation only, leaving the new .slt query in place\ngit stash push -- datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002F\u003Ccategory>\u002F\u003Cfunction>.rs\ncargo test --test sqllogictests -- spark\u002F\u003Ccategory>\u002F$ARGUMENTS   # expect FAIL\ngit stash pop\ncargo test --test sqllogictests -- spark\u002F\u003Ccategory>\u002F$ARGUMENTS   # expect PASS\n```\n\nThis verifies SLT tests only. Rust unit tests live inside the implementation\nfile, so stashing that file stashes the tests with it and the run proves\nnothing about them. For a unit test, get the fail-before evidence by reverting\nthe implementation body by hand, or by writing the test first and watching it\nfail before the fix exists.\n\nRead the failure list from the first run and confirm it names the queries you\nadded. Do not judge the run by an exit code you captured through a pipe:\n`cargo test ... | head` reports the exit status of `head`, not of `cargo`, so a\npiped command that \"succeeded\" tells you nothing. Either run it unpiped or\ninspect the `External error: N errors in file ...` block directly.\n\nIf the first run passes, the test is not exercising the fix. Either the\nquery was stashed along with the implementation, or the assertion does not\nactually cover the divergent case. Investigate before continuing: a test\nthat passes against the unfixed code locks in nothing.\n\nIf the fix also changed a `mod.rs` doc string or a helper in another file, that\nfile is not covered by the stash above. Add it to the `git stash push` argument\nlist, or the \"expect FAIL\" run may fail for the wrong reason.\n\nRun this before committing the fix. If `git stash push` reports\n`No local changes to save`, the change is already committed and the check is\ninvalid, because the \"expect FAIL\" run just tested the fixed code. Redo it\nagainst the pre-fix commit by checking that one file out of history and back,\nwhich touches only the working tree and never moves `HEAD` or a branch:\n\n```bash\n# \u003Cpre-fix-sha> is the commit before the one that introduced the fix\ngit checkout \u003Cpre-fix-sha> -- datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002F\u003Ccategory>\u002F\u003Cfunction>.rs\ncargo test --test sqllogictests -- spark\u002F\u003Ccategory>\u002F$ARGUMENTS   # expect FAIL\ngit checkout HEAD -- datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002F\u003Ccategory>\u002F\u003Cfunction>.rs\ncargo test --test sqllogictests -- spark\u002F\u003Ccategory>\u002F$ARGUMENTS   # expect PASS\n```\n\nFind `\u003Cpre-fix-sha>` with `git log --oneline -- \u003Cthe implementation file>` and\ntake the commit before the fix. If the fix also touched other files, list every\none of them in both `git checkout` commands.\n\nDo not reach for `git reset`, `git revert`, or `git rebase` here. None of them\nare needed and all of them can leave the branch in a state the audit then has\nto repair.\n\n### Capture what cannot be fixed\n\nA deferred divergence becomes a commented-out SLT query carrying the correct\nSpark 4.2.0 result and the issue link:\n\n```sql\n# Spark 4.2.0 raises ILLEGAL_DAY_OF_WEEK under ANSI mode; DataFusion returns NULL\n# https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion\u002Fissues\u002F23889   \u003C- the real number, not a placeholder\n# query error\n# SELECT next_day('2015-01-14'::DATE, 'NOT_A_DAY'::string);\n```\n\nUncommenting the block is the contract for verifying the eventual fix. Note\nthat DataFusion `.slt` files use `#` for comments, not `--`.\n\n**File the issue first, then write the block.** The block cannot be written\nuntil its issue number exists, so do the filing described below before typing\nthe comment. Never commit a placeholder in place of the number. A string like\n`\u003CISSUE-URL-PENDING>`, `NNNNN`, or `TBD` inside a URL ships a dead link, and\nbecause the surrounding block is commented out no test ever catches it.\n\nIf the issue cannot be filed at all, for example because `gh` is not\nauthenticated or the user asked that nothing be filed yet, omit the link line\nentirely. A block with no link is honest and self-contained. Then say so\nexplicitly in the audit report: name the divergence, state that no issue was\nfiled and why, and ask the user to file it. Do not leave the reader to\ndiscover the gap from a broken URL.\n\nWhen you do assert an error, remember that the runner normalizes whitespace in\nthe `query error` regex, so literal leading, trailing, and repeated spaces in\nthe expected message cannot be written out. An input like `' MO '` produces\n`Illegal input for day of week:  MO .` with two spaces, and a regex containing\nthose two spaces is collapsed to one before matching. Write `\\s+` instead,\nwhich carries no literal whitespace and survives normalization:\n\n```sql\nquery error Illegal input for day of week:\\s+MO \\.\nSELECT next_day('2015-01-14'::DATE, ' MO '::string);\n```\n\nThat asserts the padding is present without pinning its exact width. Anchor\nthe right-hand end of the match on a non-whitespace character from Spark's\nmessage, such as the trailing period in Spark 4.x error templates, so the\nassertion still fails if the padding runs into the rest of the message. If\neven that cannot be expressed, fall back to the stable prefix and say in a\ncomment why the assertion stops where it does.\n\nMatch Spark's message text exactly, including trailing punctuation. Spark 4.x\nerror templates in `error-conditions.json` generally end in a period, and the\npre-4.0 exception strings they replaced generally did not. Copying the older\nstring leaves the implementation asserting a version the audit does not claim,\nand a `query error` prefix match will not catch it.\n\nBefore filing, search for an existing issue:\n\n```bash\ngh issue list --repo apache\u002Fdatafusion --search \"$ARGUMENTS \u003Csymptom> in:title,body\" --state all --limit 5\n```\n\nIf a candidate comes back, open it with `gh issue view \u003CN> --repo apache\u002Fdatafusion`\nand confirm it actually describes this divergence and is still open. A closed\nissue cited as a known divergence is worse than no citation, because the reader\nfollows the link and finds a fix that already shipped.\n\nIf no issue exists, file one with the `bug` label plus `spark`. Title format:\n`[Bug] \u003Cfunction> \u003Cone-line symptom>`. The body includes the Spark version\nrange affected, a minimal repro in both Spark SQL and DataFusion SQL, the\ndivergent result, and the relevant file and line.\n\n### Capture cross-version divergences\n\nA behavior difference between Spark versions cannot be tested today. Capture it\nin two places.\n\nA comment at the query site naming the versions:\n\n```sql\n# Spark 3.5.8 returns NULL here; Spark 4.0.4 and later throw under ANSI mode.\n# Expectation below asserts 4.2.0. Version-specific expectations are tracked by\n# https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion\u002Fissues\u002F23887\n```\n\nAnd its own filed issue, so the divergence is scheduled once the version\nmechanism lands.\n\n### Record the audit\n\nAdd a header comment block to the `.slt` file, directly below the license\nheader and the existing provenance comment:\n\n```sql\n# Audited against Spark 4.2.0 on YYYY-MM-DD.\n# Expected values verified with pyspark==4.2.0.\n# Known divergences are captured as commented-out queries below.\n```\n\nUse today's real date. If PySpark was unavailable, say so on the second line\ninstead of claiming verification.\n\n### Attribution\n\nEvery issue and PR the skill files ends with:\n\n```\nSurfaced by the audit-datafusion-spark-expression skill.\n```\n\nDo not mention Claude, AI, or automated generation anywhere in an issue or PR.\n\n### Before committing\n\nPer the repository `CLAUDE.md`:\n\n```bash\ncargo fmt --all\ncargo clippy --all-targets --all-features -- -D warnings\ncargo test -p datafusion-spark --lib $ARGUMENTS\ncargo test --test sqllogictests -- spark\u002F\u003Ccategory>\u002F$ARGUMENTS\n```\n\nThe `--lib` run is not optional. `cargo test --test sqllogictests` does not\ncompile or run the `#[test]` blocks inside `datafusion\u002Fspark\u002Fsrc\u002F`, so a\nrefactor that renames or changes the signature of a helper those tests call\npasses the SLT gate and fails CI.\n\nDo not use `cargo build -p datafusion-spark` as a quick compile check. The\ncrate does not currently build in isolation and the resulting error is\nunrelated to your change. Use `cargo test -p datafusion-spark --lib` or the\nsqllogictest target instead.\n\nIf the audit touched documentation:\n\n```bash\n.\u002Fci\u002Fscripts\u002Fdoc_prettier_check.sh --write --allow-dirty\n```\n\nFollow `.github\u002Fpull_request_template.md` for the PR.\n\n## Output Format\n\nPresent the audit as:\n\n1. **Function summary.** What `$ARGUMENTS` does, its input and output types,\n   and its null behavior.\n2. **Spark version differences.** Across 3.5.8, 4.0.4, 4.1.3, and 4.2.0. Say\n   explicitly if there are none.\n3. **DataFusion implementation notes.** Including signature, coercion, and\n   nullability findings.\n4. **Gap matrix.** From Step 6, plus implementation gaps.\n5. **Findings.** Split into fixed in this PR, deferred with a filed issue, and\n   low-risk coverage gaps.\n6. **Offer to add the low-risk coverage tests.**\n\nLow-risk coverage gaps are the only category that pauses for user input.\nCorrectness findings are always either fixed or filed.\n\nThe line between the two is the code you touched. Coverage for a path this PR\nchanged goes into the `.slt` file without asking: it is part of the fix. The\noffer is for cases on paths the PR did not touch and that already look correct,\nwhere adding tests is optional scope the user may not want. Phrase the offer\nas:\n\n> Spark exercises the following cases that have no DataFusion test, on code\n> paths that appear correct. Would you like me to add them?\n>\n> - [list each case]\n\n## Tone and Style\n\n- Write in clear, concise prose.\n- Use backticks around function names, file paths, type names, and config keys.\n- Acknowledge what is already well covered before raising gaps.\n- Avoid em dashes and semicolons. Use separate sentences.\n- Do not overstate confidence. An unverified expectation is labeled as one.\n",{"name":5,"description":7,"argument-hint":43},"\u003Cfunction-name>","main","ee55effcccfe9d5634c6df3038e4ad78d4af0b8e",".ai\u002Fskills\u002Faudit-datafusion-spark-expression",2644,"https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion","https:\u002F\u002Fdatafusion.apache.org\u002F","https:\u002F\u002Fgithub.com\u002Fapache","https:\u002F\u002Fwww.apache.org\u002F","2026-11-11 10:40:31",{"source":8,"skillName":5,"commands":54,"prompt":64,"zipAvailable":65},[55,58,61],{"tool":56,"command":57},"npx","npx skills add apache\u002Fdatafusion --skill audit-datafusion-spark-expression",{"tool":59,"command":60},"bunx","bunx skills add apache\u002Fdatafusion --skill audit-datafusion-spark-expression",{"tool":62,"command":63},"pnpm","pnpm dlx skills add apache\u002Fdatafusion --skill audit-datafusion-spark-expression","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Faudit-datafusion-spark-expression\u002FSKILL.md",true,[],[68,88,109],{"skillNo":69,"name":70,"nameZh":6,"description":71,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":72,"categories":73,"keywords":74,"publishedAt":38},"01M6ACDJXN4NB50439K7E7KKMP","pr_review","Review Apache DataFusion pull requests following the project's PR review guide. Use whenever asked to review a DataFusion PR or PR URL, and whenever creating a PR, to check the changes against the same criteria before submitting.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fpr_review\u002FSKILL.md",[],[75,76,77,78,79,9,20,80,81,82,83,84,85,86,87],"review","pr review","pr","pull requests","pull","check","guide","use","review guide","project","URL","requests","changes",{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":38},"01M6ACDG4GVV9PQ2ZACWXJNR2B","datafusion-ffi","Patterns and review checklist for the `datafusion-ffi` crate. Use whenever the user adds, edits, or reviews code under `datafusion\u002Fffi\u002F` — new `FFI_X` wrappers, `ForeignX` implementations, codec changes, or expanding an existing wrapper to cover more of a trait's surface. Also use when reviewing PRs that touch this crate.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fdatafusion-ffi\u002FSKILL.md",[],[20,95,96,97,28,98,99,75,100,101,82,102,103,104,105,30,106,107,87,108],"datafusion ffi","ffi","when","new","checklist","code","review checklist","Wrapper","prs","patterns","touch","more","cover","Existing",{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":38},"01M6ACDA8M3Z9W6BXCXVHZSB4M","add-benchmarks","Guidelines for designing or extending DataFusion benchmarks.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fadd-benchmarks\u002FSKILL.md",[],[116,117,118,20,28,119,120],"add benchmarks","benchmarks","add","extending","guidelines"]