[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M699RG78PC9EE6KBWWG6Z5BB":3,"skill-same:01M699RG78PC9EE6KBWWG6Z5BB":94},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":20,"linkOnly":21,"markdownContent":22,"markdownBody":23,"frontMatter":24,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":28,"repositoryHtmlUrl":29,"repositoryHomepageUrl":6,"ownerHtmlUrl":30,"ownerBlogUrl":31,"ownerDescription":32,"updatedAt":33,"install":34,"resources":47},"01M699RG78PC9EE6KBWWG6Z5BB","hdinsight-migration",null,"Ports HDInsight Spark and Hive workloads to Fabric, converting HiveContext and SparkContext to SparkSession, WASB and ABFS paths to OneLake shortcuts, Hive DDL to Delta tables in a Lakehouse, and Oozie coordinators and actions to pipeline activities and triggers. Use when the source workload is HDInsight.","microsoft\u002Fskills-for-fabric","microsoft","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F6154722?v=4","01M698VYDXZ6JW5WH3CRV650R8","Microsoft","01M699NAPJKS2WBHHDN86Q5MM6",1339,"MIT","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fskills\u002Fhdinsight-migration\u002FSKILL.md",[],[19],"hdinsight migration","2026-11-10 10:12:30",false,"---\nname: hdinsight-migration\ndescription: \"Ports HDInsight Spark and Hive workloads to Fabric, converting HiveContext and SparkContext to SparkSession, WASB and ABFS paths to OneLake shortcuts, Hive DDL to Delta tables in a Lakehouse, and Oozie coordinators and actions to pipeline activities and triggers. Use when the source workload is HDInsight.\"\n---\n\n> **Telemetry — MANDATORY.** Every `api.fabric.microsoft.com` call must carry\n> `x-ms-fabric-skill: hdinsight-migration` (`az rest`: `--headers \"x-ms-fabric-skill=hdinsight-migration\"`),\n> including every LRO poll, `fabric_lro` and retry. Snippets omit it — add it anyway.\n\n> **CRITICAL NOTES**\n> 1. To find workspace details (including its ID) from a workspace name: list all workspaces, then use JMESPath filtering\n> 2. To find item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace, then use JMESPath filtering\n> 3. HDInsight has no `mssparkutils` or `dbutils` equivalent — `notebookutils` is net-new capability being introduced\n> 4. `HiveContext` and `SQLContext` are legacy Spark 1.x\u002F2.x APIs — Fabric uses Spark 3.x `SparkSession` exclusively\n> 5. `wasb:\u002F\u002F` paths are deprecated and require a Storage Account key or SAS — replace with OneLake shortcuts\n> 6. Fabric cannot read or shortcut `hdfs:\u002F\u002F` directly. Its Pipeline HDFS connector supports Anonymous authentication only; export or bridge Kerberos sources to supported storage first.\n\n# HDInsight → Microsoft Fabric Migration\n\n## Prerequisite Knowledge\n\nRead these companion documents before executing migration tasks:\n\n- [COMMON-CORE.md](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md) — Fabric REST API patterns, authentication, token audiences, item discovery\n- [COMMON-CLI.md](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md) — `az rest`, `az login`, token acquisition, Fabric REST via CLI\n- [SPARK-AUTHORING-CORE.md](..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md) — Notebook deployment, lakehouse creation, Spark job execution\n\nFor notebook and Lakehouse creation, see [spark-cli](..\u002Fspark-cli\u002FSKILL.md).\nFor Fabric Warehouse DDL\u002FDML authoring, see [sqldw-cli](..\u002Fsqldw-cli\u002FSKILL.md).\n\n---\n\n## Table of Contents\n\n| Topic | Reference |\n|---|---|\n| Migration Workload Map | [§ Migration Workload Map](#migration-workload-map) |\n| SparkSession & Context API Changes | [§ SparkSession API Changes](#sparksession--context-api-changes) |\n| WASB \u002F ABFS → OneLake Path Migration | [path-migration.md](resources\u002Fpath-migration.md) |\n| Hive DDL → Delta Lake \u002F Lakehouse Schemas | [hive-to-delta.md](resources\u002Fhive-to-delta.md) |\n| Oozie → Fabric Pipelines | [§ Oozie → Fabric Pipelines](#oozie--fabric-pipelines) |\n| Introducing `notebookutils` | [§ Introducing notebookutils](#introducing-notebookutils) |\n| Before\u002FAfter Code Patterns | [code-patterns.md](resources\u002Fcode-patterns.md) |\n| Spark Configuration Differences | [§ Spark Configuration Differences](#spark-configuration-differences) |\n| Must \u002F Prefer \u002F Avoid | [§ Must \u002F Prefer \u002F Avoid](#must--prefer--avoid) |\n| Authentication & Token Acquisition | [COMMON-CORE.md § Authentication](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#authentication--token-acquisition) |\n| Lakehouse Management | [SPARK-AUTHORING-CORE.md § Lakehouse Management](..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#lakehouse-management) |\n\n---\n\n## Migration Workload Map\n\n| HDInsight Component | Fabric Target | Notes |\n|---|---|---|\n| **Spark cluster** (notebooks, scripts) | Fabric Spark (Lakehouse \u002F Notebooks \u002F SJD) | No persistent cluster — Starter Pool or Custom Pool provides on-demand Spark |\n| **Hive \u002F HiveServer2** | **Lakehouse SQL Endpoint** + Lakehouse schemas | Delta Lake replaces Hive metastore; schemas provide namespace equivalent |\n| **HBase** | **Fabric Warehouse** or **Azure Cosmos DB** (separate from Fabric) | HBase has no direct Fabric equivalent — assess workload access patterns |\n| **Oozie workflows** | **Fabric Data Pipelines** | Map Oozie actions to Fabric activities; see [§ Oozie → Fabric Pipelines](#oozie--fabric-pipelines) |\n| **YARN Resource Manager** | **Fabric Spark monitoring** (Spark UI, Monitoring Hub) | No YARN — Fabric manages compute automatically |\n| **Ambari** | **Fabric Monitoring Hub** + **Admin Portal** | Cluster health, capacity, and job monitoring |\n| **WASB \u002F ABFS storage** | **OneLake Shortcuts** → `abfss:\u002F\u002Fworkspace@onelake.dfs.fabric.microsoft.com\u002F` | See [path-migration.md](resources\u002Fpath-migration.md) |\n| **Ranger policies** | **Fabric workspace roles** + **OneLake data access roles** | Map Ranger row\u002Fcolumn filters to Lakehouse row-level security |\n| **Livy REST server** | **Fabric Livy API** | Compatible endpoint — see SPARK-AUTHORING-CORE.md |\n\n---\n\n## SparkSession & Context API Changes\n\nHDInsight Spark clusters often use legacy Spark 1.x \u002F 2.x API styles. Replace all of these with the unified `SparkSession`:\n\n| Legacy HDInsight Pattern | Fabric Spark 3.x Replacement |\n|---|---|\n| `from pyspark import SparkContext; sc = SparkContext()` | Not needed — `sc = spark.sparkContext` (pre-instantiated) |\n| `from pyspark.sql import HiveContext; hc = HiveContext(sc)` | Not needed — `spark` session has Hive-compatible SQL support via Delta schemas |\n| `from pyspark.sql import SQLContext; sqlc = SQLContext(sc)` | Not needed — use `spark.sql(...)` directly |\n| `SparkSession.builder.enableHiveSupport().getOrCreate()` | Not needed in Fabric — `spark` is pre-built and available |\n| `sc.textFile(\"wasb:\u002F\u002Fcontainer@account.blob.core.windows.net\u002Fpath\")` | `spark.read.text(\"abfss:\u002F\u002Fworkspace@onelake.dfs.fabric.microsoft.com\u002Flh.Lakehouse\u002FFiles\u002Fpath\")` |\n| `sqlContext.sql(\"CREATE TABLE ... STORED AS ORC\")` | See [hive-to-delta.md](resources\u002Fhive-to-delta.md) for Delta DDL equivalent |\n\n> In Fabric notebooks, `spark` (SparkSession) and `sc` (SparkContext) are **pre-instantiated** — do not call `SparkContext()` or `SparkSession.builder...getOrCreate()` at the top of migrated notebooks.\n\n---\n\n## Oozie → Fabric Pipelines\n\nMap Oozie workflow actions to Fabric Data Pipeline activities:\n\n| Oozie Action Type | Fabric Pipeline Activity | Notes |\n|---|---|---|\n| `\u003Cspark>` action | **Notebook activity** or **Spark Job Definition activity** | Pass parameters via notebook cell parameters or SJD arguments |\n| `\u003Chive>` action | **Script activity** (SQL) against Lakehouse SQL Endpoint | Convert HiveQL to Spark SQL or Delta SQL |\n| `\u003Cshell>` action | **Azure Function activity** or **Web activity** | Shell scripts must be refactored; no direct shell execution in Fabric Pipelines |\n| `\u003Cjava>` action | **Azure Batch activity** (external) or refactor to PySpark | Java MapReduce jobs must be rewritten |\n| `\u003Csqoop>` action | **Copy Data activity** (Fabric Data Factory connector) | Sqoop import\u002Fexport maps to Fabric Copy Data with JDBC source\u002Fsink |\n| `\u003Ccoordinator>` (time-based schedule) | **Pipeline schedule trigger** | Set recurrence in pipeline trigger; supports cron-like expressions |\n| `\u003Ccoordinator>` (data-triggered) | **Storage Event trigger** | Trigger on OneLake file arrival |\n\n> **Delegate to `spark-cli`** for notebook and SJD creation after mapping pipeline activities.\n\n---\n\n## Introducing `notebookutils`\n\nHDInsight Spark had no built-in utility framework equivalent to `mssparkutils` or `dbutils`. When migrating to Fabric, introduce `notebookutils` for common operations:\n\n| Operation | Old HDInsight Approach | `notebookutils` Equivalent |\n|---|---|---|\n| List files | `dbutils` (N\u002FA) \u002F HDFS CLI | `notebookutils.fs.ls(\"abfss:\u002F\u002F...\")` |\n| Copy file | HDFS API \u002F `shutil` | `notebookutils.fs.cp(src, dest)` |\n| Read secret | Azure Key Vault REST call | `notebookutils.credentials.getSecret(keyVaultUrl, secretName)` |\n| Get notebook context | Not available | `notebookutils.runtime.context` — returns workspace ID, notebook ID, etc. |\n| Run child notebook | Not available | `notebookutils.notebook.run(\"notebook_name\", timeout, {\"param\": \"value\"})` |\n| Exit notebook with value | `sys.exit()` | `notebookutils.notebook.exit(\"value\")` |\n| Mount storage | WASB config in `spark-defaults.conf` | OneLake Shortcut (no runtime mount needed) |\n\n---\n\n## Spark Configuration Differences\n\n| HDInsight Concept | Fabric Spark Equivalent | Migration Action |\n|---|---|---|\n| `spark-defaults.conf` (cluster-wide) | Fabric **Spark Workspace Settings** + **Environment** item | Move config properties to Environment or use `%%configure` in notebooks |\n| `%%configure` magic | `%%configure` magic — **identical** | No change needed |\n| YARN queue \u002F resource allocation | **Fabric Spark pool** node size and autoscale settings | Map queue SLAs to Custom Pool configuration |\n| Ambari service configs (HDFS, YARN tuning) | Not applicable — Fabric manages infrastructure | Remove; focus on application-level Spark configs |\n| HDI Spark version (e.g., Spark 2.4) | Fabric Runtime 1.3 = Spark 3.5 (latest) | Test for deprecated API removals (e.g., `HiveContext`, RDD-style ML) |\n| Conda environment \u002F `bootstrap.sh` | **Fabric Environment** item with custom libraries | Recreate conda\u002Fpip dependencies in a Fabric Environment |\n| `hive-site.xml` (metastore connection) | Not needed — Delta Lake IS the metastore in Fabric | Remove metastore config; use Lakehouse schemas for namespace organization |\n\n---\n\n## Must \u002F Prefer \u002F Avoid\n\n### MUST DO\n- **Replace all `wasb:\u002F\u002F` \u002F `wasbs:\u002F\u002F` paths** with OneLake `abfss:\u002F\u002F` paths or OneLake Shortcuts — `wasb:\u002F\u002F` requires storage account keys which are not the Fabric-preferred auth model\n- **Replace `HiveContext`, `SQLContext`, and standalone `SparkContext()`** — use the pre-instantiated `spark` session in Fabric notebooks\n- **Migrate Hive DDL** (`STORED AS ORC`, `LOCATION`, `TBLPROPERTIES`) to Delta Lake DDL — see [hive-to-delta.md](resources\u002Fhive-to-delta.md)\n- **Introduce `notebookutils`** for file system operations, secret retrieval, and child notebook orchestration where HDInsight used custom scripts or direct API calls\n- **Replace Oozie XML workflows** with Fabric Data Pipelines — see [§ Oozie → Fabric Pipelines](#oozie--fabric-pipelines)\n- **Align library management** to Fabric Environments — remove `bootstrap.sh`, conda envs, and runtime `%pip install` patterns for production workloads\n- **Treat `hdfs:\u002F\u002F` as source-cluster-only** — export or ingest into ADLS Gen2 or OneLake; do not claim that a gateway adds Kerberos support to Fabric's Anonymous-only HDFS connector\n\n### PREFER\n- **OneLake Shortcuts** over copying data — mount existing ADLS Gen2 containers as shortcuts to avoid re-ingestion during migration\n- **Delta Lake** for all tables migrated from Hive ORC\u002FParquet — ACID guarantees, time travel, and schema enforcement improve data quality\n- **Fabric Starter Pool** for initial migration validation — no pool configuration overhead, fast session startup\n- **Lakehouse schemas** (database namespaces) for organizing migrated Hive databases — one schema per Hive database within a single Lakehouse\n- **Medallion architecture** for restructuring migrated data layers during migration — align Bronze\u002FSilver\u002FGold with raw Hive → validated Delta → serving Gold patterns\n\n### AVOID\n- **Do not use `SparkContext()` or `HiveContext()` constructors** in Fabric notebooks — they conflict with the pre-instantiated `spark` session and will raise errors\n- **Do not use `hive-site.xml` or external Hive metastore configuration** — Fabric's Delta Lake-backed Lakehouse IS the metastore\n- **Do not assume YARN queue mappings translate to Fabric pools** — re-design resource allocation based on Fabric Spark pool SLAs\n- **Do not attempt to run Oozie shell actions or Java MapReduce jobs** directly in Fabric — these must be refactored (see [§ Oozie → Fabric Pipelines](#oozie--fabric-pipelines))\n- **Do not use `%sh` magic for file system operations** in production notebooks — use `notebookutils.fs.*` for portability and OneLake token-based auth\n\n---\n\n## Examples\n\nSee [code-patterns.md](resources\u002Fcode-patterns.md) for full before\u002Fafter examples. Key quick references:\n\n**Legacy context → Fabric pre-instantiated session**\n\n```python\n# HDInsight (remove entirely)\nfrom pyspark.sql import HiveContext\nhc = HiveContext(sc)\n\n# Fabric — use pre-instantiated spark directly\ndf = spark.sql(\"SELECT * FROM sales.fact_orders\")\n```\n\n**WASB path → OneLake path (after shortcut creation)**\n\n```python\n# HDInsight\ndf = spark.read.parquet(\"wasb:\u002F\u002Fraw@myaccount.blob.core.windows.net\u002Forders\u002F\")\n\n# Fabric\ndf = spark.read.parquet(\"Files\u002Fraw\u002Forders\u002F\")\n```\n\n**Hive DDL → Delta DDL**\n\n```sql\n-- HDInsight\nCREATE TABLE sales_db.fact_orders (...) STORED AS ORC LOCATION 'wasb:\u002F\u002F...';\n\n-- Fabric\nCREATE SCHEMA IF NOT EXISTS sales_db;\nCREATE TABLE sales_db.fact_orders (...) USING DELTA;\n```\n","> **Telemetry — MANDATORY.** Every `api.fabric.microsoft.com` call must carry\n> `x-ms-fabric-skill: hdinsight-migration` (`az rest`: `--headers \"x-ms-fabric-skill=hdinsight-migration\"`),\n> including every LRO poll, `fabric_lro` and retry. Snippets omit it — add it anyway.\n\n> **CRITICAL NOTES**\n> 1. To find workspace details (including its ID) from a workspace name: list all workspaces, then use JMESPath filtering\n> 2. To find item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace, then use JMESPath filtering\n> 3. HDInsight has no `mssparkutils` or `dbutils` equivalent — `notebookutils` is net-new capability being introduced\n> 4. `HiveContext` and `SQLContext` are legacy Spark 1.x\u002F2.x APIs — Fabric uses Spark 3.x `SparkSession` exclusively\n> 5. `wasb:\u002F\u002F` paths are deprecated and require a Storage Account key or SAS — replace with OneLake shortcuts\n> 6. Fabric cannot read or shortcut `hdfs:\u002F\u002F` directly. Its Pipeline HDFS connector supports Anonymous authentication only; export or bridge Kerberos sources to supported storage first.\n\n# HDInsight → Microsoft Fabric Migration\n\n## Prerequisite Knowledge\n\nRead these companion documents before executing migration tasks:\n\n- [COMMON-CORE.md](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md) — Fabric REST API patterns, authentication, token audiences, item discovery\n- [COMMON-CLI.md](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md) — `az rest`, `az login`, token acquisition, Fabric REST via CLI\n- [SPARK-AUTHORING-CORE.md](..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md) — Notebook deployment, lakehouse creation, Spark job execution\n\nFor notebook and Lakehouse creation, see [spark-cli](..\u002Fspark-cli\u002FSKILL.md).\nFor Fabric Warehouse DDL\u002FDML authoring, see [sqldw-cli](..\u002Fsqldw-cli\u002FSKILL.md).\n\n---\n\n## Table of Contents\n\n| Topic | Reference |\n|---|---|\n| Migration Workload Map | [§ Migration Workload Map](#migration-workload-map) |\n| SparkSession & Context API Changes | [§ SparkSession API Changes](#sparksession--context-api-changes) |\n| WASB \u002F ABFS → OneLake Path Migration | [path-migration.md](resources\u002Fpath-migration.md) |\n| Hive DDL → Delta Lake \u002F Lakehouse Schemas | [hive-to-delta.md](resources\u002Fhive-to-delta.md) |\n| Oozie → Fabric Pipelines | [§ Oozie → Fabric Pipelines](#oozie--fabric-pipelines) |\n| Introducing `notebookutils` | [§ Introducing notebookutils](#introducing-notebookutils) |\n| Before\u002FAfter Code Patterns | [code-patterns.md](resources\u002Fcode-patterns.md) |\n| Spark Configuration Differences | [§ Spark Configuration Differences](#spark-configuration-differences) |\n| Must \u002F Prefer \u002F Avoid | [§ Must \u002F Prefer \u002F Avoid](#must--prefer--avoid) |\n| Authentication & Token Acquisition | [COMMON-CORE.md § Authentication](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#authentication--token-acquisition) |\n| Lakehouse Management | [SPARK-AUTHORING-CORE.md § Lakehouse Management](..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#lakehouse-management) |\n\n---\n\n## Migration Workload Map\n\n| HDInsight Component | Fabric Target | Notes |\n|---|---|---|\n| **Spark cluster** (notebooks, scripts) | Fabric Spark (Lakehouse \u002F Notebooks \u002F SJD) | No persistent cluster — Starter Pool or Custom Pool provides on-demand Spark |\n| **Hive \u002F HiveServer2** | **Lakehouse SQL Endpoint** + Lakehouse schemas | Delta Lake replaces Hive metastore; schemas provide namespace equivalent |\n| **HBase** | **Fabric Warehouse** or **Azure Cosmos DB** (separate from Fabric) | HBase has no direct Fabric equivalent — assess workload access patterns |\n| **Oozie workflows** | **Fabric Data Pipelines** | Map Oozie actions to Fabric activities; see [§ Oozie → Fabric Pipelines](#oozie--fabric-pipelines) |\n| **YARN Resource Manager** | **Fabric Spark monitoring** (Spark UI, Monitoring Hub) | No YARN — Fabric manages compute automatically |\n| **Ambari** | **Fabric Monitoring Hub** + **Admin Portal** | Cluster health, capacity, and job monitoring |\n| **WASB \u002F ABFS storage** | **OneLake Shortcuts** → `abfss:\u002F\u002Fworkspace@onelake.dfs.fabric.microsoft.com\u002F` | See [path-migration.md](resources\u002Fpath-migration.md) |\n| **Ranger policies** | **Fabric workspace roles** + **OneLake data access roles** | Map Ranger row\u002Fcolumn filters to Lakehouse row-level security |\n| **Livy REST server** | **Fabric Livy API** | Compatible endpoint — see SPARK-AUTHORING-CORE.md |\n\n---\n\n## SparkSession & Context API Changes\n\nHDInsight Spark clusters often use legacy Spark 1.x \u002F 2.x API styles. Replace all of these with the unified `SparkSession`:\n\n| Legacy HDInsight Pattern | Fabric Spark 3.x Replacement |\n|---|---|\n| `from pyspark import SparkContext; sc = SparkContext()` | Not needed — `sc = spark.sparkContext` (pre-instantiated) |\n| `from pyspark.sql import HiveContext; hc = HiveContext(sc)` | Not needed — `spark` session has Hive-compatible SQL support via Delta schemas |\n| `from pyspark.sql import SQLContext; sqlc = SQLContext(sc)` | Not needed — use `spark.sql(...)` directly |\n| `SparkSession.builder.enableHiveSupport().getOrCreate()` | Not needed in Fabric — `spark` is pre-built and available |\n| `sc.textFile(\"wasb:\u002F\u002Fcontainer@account.blob.core.windows.net\u002Fpath\")` | `spark.read.text(\"abfss:\u002F\u002Fworkspace@onelake.dfs.fabric.microsoft.com\u002Flh.Lakehouse\u002FFiles\u002Fpath\")` |\n| `sqlContext.sql(\"CREATE TABLE ... STORED AS ORC\")` | See [hive-to-delta.md](resources\u002Fhive-to-delta.md) for Delta DDL equivalent |\n\n> In Fabric notebooks, `spark` (SparkSession) and `sc` (SparkContext) are **pre-instantiated** — do not call `SparkContext()` or `SparkSession.builder...getOrCreate()` at the top of migrated notebooks.\n\n---\n\n## Oozie → Fabric Pipelines\n\nMap Oozie workflow actions to Fabric Data Pipeline activities:\n\n| Oozie Action Type | Fabric Pipeline Activity | Notes |\n|---|---|---|\n| `\u003Cspark>` action | **Notebook activity** or **Spark Job Definition activity** | Pass parameters via notebook cell parameters or SJD arguments |\n| `\u003Chive>` action | **Script activity** (SQL) against Lakehouse SQL Endpoint | Convert HiveQL to Spark SQL or Delta SQL |\n| `\u003Cshell>` action | **Azure Function activity** or **Web activity** | Shell scripts must be refactored; no direct shell execution in Fabric Pipelines |\n| `\u003Cjava>` action | **Azure Batch activity** (external) or refactor to PySpark | Java MapReduce jobs must be rewritten |\n| `\u003Csqoop>` action | **Copy Data activity** (Fabric Data Factory connector) | Sqoop import\u002Fexport maps to Fabric Copy Data with JDBC source\u002Fsink |\n| `\u003Ccoordinator>` (time-based schedule) | **Pipeline schedule trigger** | Set recurrence in pipeline trigger; supports cron-like expressions |\n| `\u003Ccoordinator>` (data-triggered) | **Storage Event trigger** | Trigger on OneLake file arrival |\n\n> **Delegate to `spark-cli`** for notebook and SJD creation after mapping pipeline activities.\n\n---\n\n## Introducing `notebookutils`\n\nHDInsight Spark had no built-in utility framework equivalent to `mssparkutils` or `dbutils`. When migrating to Fabric, introduce `notebookutils` for common operations:\n\n| Operation | Old HDInsight Approach | `notebookutils` Equivalent |\n|---|---|---|\n| List files | `dbutils` (N\u002FA) \u002F HDFS CLI | `notebookutils.fs.ls(\"abfss:\u002F\u002F...\")` |\n| Copy file | HDFS API \u002F `shutil` | `notebookutils.fs.cp(src, dest)` |\n| Read secret | Azure Key Vault REST call | `notebookutils.credentials.getSecret(keyVaultUrl, secretName)` |\n| Get notebook context | Not available | `notebookutils.runtime.context` — returns workspace ID, notebook ID, etc. |\n| Run child notebook | Not available | `notebookutils.notebook.run(\"notebook_name\", timeout, {\"param\": \"value\"})` |\n| Exit notebook with value | `sys.exit()` | `notebookutils.notebook.exit(\"value\")` |\n| Mount storage | WASB config in `spark-defaults.conf` | OneLake Shortcut (no runtime mount needed) |\n\n---\n\n## Spark Configuration Differences\n\n| HDInsight Concept | Fabric Spark Equivalent | Migration Action |\n|---|---|---|\n| `spark-defaults.conf` (cluster-wide) | Fabric **Spark Workspace Settings** + **Environment** item | Move config properties to Environment or use `%%configure` in notebooks |\n| `%%configure` magic | `%%configure` magic — **identical** | No change needed |\n| YARN queue \u002F resource allocation | **Fabric Spark pool** node size and autoscale settings | Map queue SLAs to Custom Pool configuration |\n| Ambari service configs (HDFS, YARN tuning) | Not applicable — Fabric manages infrastructure | Remove; focus on application-level Spark configs |\n| HDI Spark version (e.g., Spark 2.4) | Fabric Runtime 1.3 = Spark 3.5 (latest) | Test for deprecated API removals (e.g., `HiveContext`, RDD-style ML) |\n| Conda environment \u002F `bootstrap.sh` | **Fabric Environment** item with custom libraries | Recreate conda\u002Fpip dependencies in a Fabric Environment |\n| `hive-site.xml` (metastore connection) | Not needed — Delta Lake IS the metastore in Fabric | Remove metastore config; use Lakehouse schemas for namespace organization |\n\n---\n\n## Must \u002F Prefer \u002F Avoid\n\n### MUST DO\n- **Replace all `wasb:\u002F\u002F` \u002F `wasbs:\u002F\u002F` paths** with OneLake `abfss:\u002F\u002F` paths or OneLake Shortcuts — `wasb:\u002F\u002F` requires storage account keys which are not the Fabric-preferred auth model\n- **Replace `HiveContext`, `SQLContext`, and standalone `SparkContext()`** — use the pre-instantiated `spark` session in Fabric notebooks\n- **Migrate Hive DDL** (`STORED AS ORC`, `LOCATION`, `TBLPROPERTIES`) to Delta Lake DDL — see [hive-to-delta.md](resources\u002Fhive-to-delta.md)\n- **Introduce `notebookutils`** for file system operations, secret retrieval, and child notebook orchestration where HDInsight used custom scripts or direct API calls\n- **Replace Oozie XML workflows** with Fabric Data Pipelines — see [§ Oozie → Fabric Pipelines](#oozie--fabric-pipelines)\n- **Align library management** to Fabric Environments — remove `bootstrap.sh`, conda envs, and runtime `%pip install` patterns for production workloads\n- **Treat `hdfs:\u002F\u002F` as source-cluster-only** — export or ingest into ADLS Gen2 or OneLake; do not claim that a gateway adds Kerberos support to Fabric's Anonymous-only HDFS connector\n\n### PREFER\n- **OneLake Shortcuts** over copying data — mount existing ADLS Gen2 containers as shortcuts to avoid re-ingestion during migration\n- **Delta Lake** for all tables migrated from Hive ORC\u002FParquet — ACID guarantees, time travel, and schema enforcement improve data quality\n- **Fabric Starter Pool** for initial migration validation — no pool configuration overhead, fast session startup\n- **Lakehouse schemas** (database namespaces) for organizing migrated Hive databases — one schema per Hive database within a single Lakehouse\n- **Medallion architecture** for restructuring migrated data layers during migration — align Bronze\u002FSilver\u002FGold with raw Hive → validated Delta → serving Gold patterns\n\n### AVOID\n- **Do not use `SparkContext()` or `HiveContext()` constructors** in Fabric notebooks — they conflict with the pre-instantiated `spark` session and will raise errors\n- **Do not use `hive-site.xml` or external Hive metastore configuration** — Fabric's Delta Lake-backed Lakehouse IS the metastore\n- **Do not assume YARN queue mappings translate to Fabric pools** — re-design resource allocation based on Fabric Spark pool SLAs\n- **Do not attempt to run Oozie shell actions or Java MapReduce jobs** directly in Fabric — these must be refactored (see [§ Oozie → Fabric Pipelines](#oozie--fabric-pipelines))\n- **Do not use `%sh` magic for file system operations** in production notebooks — use `notebookutils.fs.*` for portability and OneLake token-based auth\n\n---\n\n## Examples\n\nSee [code-patterns.md](resources\u002Fcode-patterns.md) for full before\u002Fafter examples. Key quick references:\n\n**Legacy context → Fabric pre-instantiated session**\n\n```python\n# HDInsight (remove entirely)\nfrom pyspark.sql import HiveContext\nhc = HiveContext(sc)\n\n# Fabric — use pre-instantiated spark directly\ndf = spark.sql(\"SELECT * FROM sales.fact_orders\")\n```\n\n**WASB path → OneLake path (after shortcut creation)**\n\n```python\n# HDInsight\ndf = spark.read.parquet(\"wasb:\u002F\u002Fraw@myaccount.blob.core.windows.net\u002Forders\u002F\")\n\n# Fabric\ndf = spark.read.parquet(\"Files\u002Fraw\u002Forders\u002F\")\n```\n\n**Hive DDL → Delta DDL**\n\n```sql\n-- HDInsight\nCREATE TABLE sales_db.fact_orders (...) STORED AS ORC LOCATION 'wasb:\u002F\u002F...';\n\n-- Fabric\nCREATE SCHEMA IF NOT EXISTS sales_db;\nCREATE TABLE sales_db.fact_orders (...) USING DELTA;\n```\n",{"name":5,"description":7},"main","4c3986d2e5b498ce2d6e6025533b94b5e5bb2ccd","skills\u002Fhdinsight-migration",393,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric","https:\u002F\u002Fgithub.com\u002Fmicrosoft","https:\u002F\u002Fopensource.microsoft.com","Open source projects and samples from Microsoft","2026-11-06 03:33:28",{"source":8,"skillName":5,"commands":35,"prompt":45,"zipAvailable":46},[36,39,42],{"tool":37,"command":38},"npx","npx skills add microsoft\u002Fskills-for-fabric --skill hdinsight-migration",{"tool":40,"command":41},"bunx","bunx skills add microsoft\u002Fskills-for-fabric --skill hdinsight-migration",{"tool":43,"command":44},"pnpm","pnpm dlx skills add microsoft\u002Fskills-for-fabric --skill hdinsight-migration","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fskills\u002Fhdinsight-migration\u002FSKILL.md",true,[48,54,58,62,68,74,78,82,86,90],{"path":49,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":52,"sourceHtmlUrl":53},"anthropic-best-practices.md","REFERENCE","参考文档",45825,"https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002F_collections\u002Fdot_config\u002Fopencode\u002Fsuperpowers\u002Fskills\u002Fwriting-skills\u002Fanthropic-best-practices.md",{"path":55,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":56,"sourceHtmlUrl":57},"apm.yml",459,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fskills\u002Fhdinsight-migration\u002Fapm.yml",{"path":59,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":60,"sourceHtmlUrl":61},"examples\u002FCLAUDE_MD_TESTING.md",5423,"https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002F_collections\u002Fdot_config\u002Fopencode\u002Fsuperpowers\u002Fskills\u002Fwriting-skills\u002Fexamples\u002FCLAUDE_MD_TESTING.md",{"path":63,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":66,"sourceHtmlUrl":67},"executable_render-graphs.js","SCRIPT","脚本",4857,"https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002F_collections\u002Fdot_config\u002Fopencode\u002Fsuperpowers\u002Fskills\u002Fwriting-skills\u002Fexecutable_render-graphs.js",{"path":69,"resourceType":70,"resourceTypeDesc":71,"sizeBytes":72,"sourceHtmlUrl":73},"graphviz-conventions.dot","OTHER","其他",5970,"https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002F_collections\u002Fdot_config\u002Fopencode\u002Fsuperpowers\u002Fskills\u002Fwriting-skills\u002Fgraphviz-conventions.dot",{"path":75,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":76,"sourceHtmlUrl":77},"persuasion-principles.md",5908,"https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002F_collections\u002Fdot_config\u002Fopencode\u002Fsuperpowers\u002Fskills\u002Fwriting-skills\u002Fpersuasion-principles.md",{"path":79,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":80,"sourceHtmlUrl":81},"resources\u002Fcode-patterns.md",5557,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fskills\u002Fhdinsight-migration\u002Fresources\u002Fcode-patterns.md",{"path":83,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":84,"sourceHtmlUrl":85},"resources\u002Fhive-to-delta.md",6680,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fskills\u002Fhdinsight-migration\u002Fresources\u002Fhive-to-delta.md",{"path":87,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":88,"sourceHtmlUrl":89},"resources\u002Fpath-migration.md",6881,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fskills\u002Fhdinsight-migration\u002Fresources\u002Fpath-migration.md",{"path":91,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":92,"sourceHtmlUrl":93},"testing-skills-with-subagents.md",12557,"https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002F_collections\u002Fdot_config\u002Fopencode\u002Fsuperpowers\u002Fskills\u002Fwriting-skills\u002Ftesting-skills-with-subagents.md",[95,121,150,155,161,169],{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":20},"01M6PHZNVB6K7DCCJH4YZHKC4F","ontology-cli","Manages Microsoft Fabric Ontology items: authors legacy V1 entity\u002Frelationship definitions and answers questions for new-experience (V2) Ontologies through ask_ontology. Use for Ontology item operations or grounded questions; V2 authoring\u002Fupgrades and V1 queries are unsupported. Power BI data questions use fabriciq, report authoring powerbi-report-cli, and model\u002FDAX changes semantic-model-authoring.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fskills\u002Fontology-cli\u002FSKILL.md",[],[102,103,104,105,106,107,108,109,110,111,112,113,9,114,115,116,117,118,119,120],"ontology cli","ontology","cli","use","changes","for","v1","bi","relationship","queries","Legacy","power","operations","upgrades","ask","model","power bi","microsoft fabric","fabriciq",{"skillNo":122,"name":123,"nameZh":6,"description":124,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":125,"categories":126,"keywords":132,"publishedAt":20},"01M6PHZ91CSTP67DM6JN8TYEFV","databasehub-cli","Analyzes Database Hub inventory, health, and security posture through Entra-authenticated Fabric APIs. Use for estate counts; SQL\u002FPostgreSQL CPU, storage, and memory; Cosmos DB availability\u002FRU; authentication\u002Fauditing\u002FCMK warnings; and Fabric SQL inventory. Fabric SQL tuning routes to sqldb-cli; Warehouse\u002FLakehouse\u002FMirrored Database SQL to sqldw-cli. Triggers: data estate performance, database estate health, database security posture.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fskills\u002Fdatabasehub-cli\u002FSKILL.md",[127],{"categoryNo":128,"categoryName":129,"taxonomyType":130,"taxonomyTypeDesc":131},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",[133,104,134,135,107,136,137,138,139,140,141,142,143,144,105,145,146,147,148,149],"databasehub cli","triggers","authentication","warehouse","database security","availability","CPU","cosmos","database sql","postgresql","db","security posture","health","database","inventory","performance","memory",{"skillNo":151,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":152,"categories":153,"keywords":154,"publishedAt":20},"01M6PHYVDVBD0VQN4FFAMN428Z","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fplugins\u002Ffabric-skills\u002Fskills\u002Fontology-cli\u002FSKILL.md",[],[102,103,104,119,120,9,118,106,107,108,109,110,111,112,113,105,114,115,116,117],{"skillNo":156,"name":123,"nameZh":6,"description":124,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":157,"categories":158,"keywords":160,"publishedAt":20},"01M6PHYDZ17VY69E3GS71KFW09","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fplugins\u002Ffabric-skills\u002Fskills\u002Fdatabasehub-cli\u002FSKILL.md",[159],{"categoryNo":128,"categoryName":129,"taxonomyType":130,"taxonomyTypeDesc":131},[104,133,149,139,146,107,136,137,138,140,141,134,143,144,105,145,147,135,148,142],{"skillNo":162,"name":163,"nameZh":6,"description":164,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":165,"categories":166,"keywords":167,"publishedAt":20},"01M699SMYSV15Z37ZCCX9EMP85","variable-library-cli","Manages Fabric Variable Library items, including definitions, libraryVariables and valueSets overrides. Covers which consumers can reference a variable and with what syntax -- pipelines, notebooks via notebookutils variableLibrary, Dataflow Gen2, Copy Jobs, shortcuts, UDFs and Plan items -- the active value set per workspace or deployment stage, and Git serialization for CI\u002FCD.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fskills\u002Fvariable-library-cli\u002FSKILL.md",[],[104,168],"variable library cli",{"skillNo":170,"name":171,"nameZh":6,"description":172,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":173,"categories":174,"keywords":180,"publishedAt":20},"01M699SGQ4VSWBR73HQKX1YH3Y","synapse-migration","Converts Synapse workspaces to Fabric: Spark items, Lake Databases, Linked Services, and Dedicated SQL Pool schema\u002Fprocedures for Lakehouse or Warehouse. During Pool-to-Lakehouse migration only, can migrate approved dependent procedure callers after notebook readiness. Excludes standalone pipelines and Warehouse administration. Triggers: Synapse workspace to Fabric, Dedicated SQL Pool to Lakehouse, Dedicated SQL Pool to Fabric Warehouse.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fskills\u002Fsynapse-migration\u002FSKILL.md",[175],{"categoryNo":176,"categoryName":177,"taxonomyType":178,"taxonomyTypeDesc":179},"01JBSEEDCAT000000000000033","数据分析师","PROFESSION","适用职业",[181],"synapse migration"]