[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M73G97WNX56VE7YS2FFHX4AV":3,"skill-same:01M73G97WNX56VE7YS2FFHX4AV":48},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":29,"ownerHtmlUrl":30,"ownerBlogUrl":31,"ownerDescription":32,"updatedAt":33,"install":34,"resources":47},"01M73G97WNX56VE7YS2FFHX4AV","dask-parallel-computing",null,"Parallel\u002Fdistributed computing for larger-than-RAM data. Components: DataFrames (parallel pandas), Arrays (parallel NumPy), Bags, Futures, Schedulers. Scales laptop to HPC cluster. For single-machine speed use polars; for out-of-core without cluster use vaex.","bg-szy\u002FTOP-SKILLS","bg-szy","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F106576190?v=4","01M73A4E8KPV67SFXRY8XZGNBX","ShuZhiyong","01M73A4E96MQ294MHM74TZBVQD",6,"BSD-3-Clause","https:\u002F\u002Fgithub.com\u002Fbg-szy\u002FTOP-SKILLS\u002Fblob\u002Fmaster\u002Fskills\u002Fawesome-skills\u002Fdask-parallel-computing\u002FSKILL.md",[],[],"2026-11-11 15:11:14",false,"---\nname: dask-parallel-computing\ndescription: \"Parallel\u002Fdistributed computing for larger-than-RAM data. Components: DataFrames (parallel pandas), Arrays (parallel NumPy), Bags, Futures, Schedulers. Scales laptop to HPC cluster. For single-machine speed use polars; for out-of-core without cluster use vaex.\"\nlicense: BSD-3-Clause\n---\n\n# Dask — Parallel & Distributed Computing\n\n## Overview\n\nDask is a Python library for parallel and distributed computing that scales familiar pandas\u002FNumPy APIs to larger-than-memory datasets. It provides five main components (DataFrames, Arrays, Bags, Futures, Schedulers) and scales from single-machine multi-core to multi-node HPC clusters.\n\n## When to Use\n\n- Processing datasets that exceed available RAM (10 GB–100 TB)\n- Parallelizing pandas or NumPy operations across multiple cores\n- Processing multiple files efficiently (CSV, Parquet, JSON, HDF5, Zarr)\n- Building custom parallel workflows with task dependencies\n- Distributing workloads across HPC clusters (SLURM, Kubernetes)\n- Streaming\u002FETL pipelines for unstructured data (logs, JSON records)\n- For **in-memory single-machine speed**: use polars instead\n- For **out-of-core single-machine analytics**: use vaex instead\n\n## Prerequisites\n\n```bash\npip install dask[complete]        # All components\npip install dask[dataframe]       # DataFrames only\npip install dask[distributed]     # Distributed scheduler + dashboard\npip install dask-jobqueue          # HPC cluster integration (SLURM, PBS)\n```\n\n## Core API\n\n### 1. DataFrames — Parallel Pandas\n\n```python\nimport dask.dataframe as dd\n\n# Read multiple files as a single DataFrame\nddf = dd.read_csv('data\u002F2024-*.csv')\nddf = dd.read_parquet('data\u002F', columns=['id', 'value', 'category'])\n\n# Operations are lazy until .compute()\nfiltered = ddf[ddf['value'] > 100]\nresult = filtered.groupby('category').agg({'value': ['mean', 'sum']}).compute()\nprint(result.shape)  # (n_categories, 2)\n\n# Custom operations via map_partitions (preferred over apply)\ndef normalize_partition(df):\n    df['norm_value'] = (df['value'] - df['value'].mean()) \u002F df['value'].std()\n    return df\n\nddf = ddf.map_partitions(normalize_partition)\n\n# Joins\nddf_merged = ddf.merge(lookup_ddf, on='category', how='left')\n\n# Write results\nddf.to_parquet('output\u002F', engine='pyarrow')\n```\n\n```python\n# Repartitioning for optimal chunk sizes\nddf = ddf.repartition(npartitions=20)         # By count\nddf = ddf.repartition(partition_size='100MB')  # By size\n\n# Index management for sorted operations\nddf = ddf.set_index('timestamp', sorted=True)\n\n# Debugging\nprint(f\"Partitions: {ddf.npartitions}\")\nprint(f\"Dtypes: {ddf.dtypes}\")\nsample = ddf.get_partition(0).compute()  # Inspect first partition\n```\n\n### 2. Arrays — Parallel NumPy\n\n```python\nimport dask.array as da\nimport numpy as np\n\n# Create from various sources\nx = da.random.random((100000, 1000), chunks=(10000, 1000))\nx = da.from_array(np_array, chunks=(10000, 1000))\nx = da.from_zarr('large_dataset.zarr')\n\n# Standard operations (lazy)\ny = (x - x.mean(axis=0)) \u002F x.std(axis=0)  # Normalize\nz = da.dot(x.T, x)                          # Matrix multiply\nu, s, v = da.linalg.svd(x)                  # SVD\n\n# Compute and persist\nresult = y.mean(axis=0).compute()\nprint(result.shape)  # (1000,)\n```\n\n```python\n# Custom operations with map_blocks\ndef custom_filter(block):\n    from scipy.ndimage import gaussian_filter\n    return gaussian_filter(block, sigma=2)\n\nfiltered = da.map_blocks(custom_filter, x, dtype=x.dtype)\n\n# Rechunking for different access patterns\nx_rechunked = x.rechunk({0: 5000, 1: 500})\n\n# Save to disk\nda.to_zarr(y, 'normalized.zarr')\n```\n\n### 3. Bags — Unstructured Data Processing\n\n```python\nimport dask.bag as db\nimport json\n\n# Read unstructured data\nbag = db.read_text('logs\u002F*.json').map(json.loads)\n\n# Functional operations\nvalid = bag.filter(lambda x: x['status'] == 'success')\nids = valid.pluck('user_id')\nflat = bag.map(lambda x: x['tags']).flatten()\n\n# Aggregation — use foldby instead of groupby (much faster)\ncounts = bag.foldby(\n    key='category',\n    binop=lambda total, x: total + x['amount'],\n    initial=0,\n    combine=lambda a, b: a + b,\n    combine_initial=0\n).compute()\n\n# Convert to DataFrame for structured analysis\nddf = valid.to_dataframe(meta={'user_id': 'str', 'amount': 'float64', 'category': 'str'})\n```\n\n### 4. Futures — Task-Based Parallelism\n\n```python\nfrom dask.distributed import Client\n\nclient = Client()  # Local cluster with all cores\nprint(client.dashboard_link)  # http:\u002F\u002Flocalhost:8787\n\n# Submit individual tasks (executes immediately, not lazy)\ndef process(x, param):\n    return x ** param\n\nfuture = client.submit(process, 42, param=2)\nprint(future.result())  # 1764\n\n# Map over many inputs\nfutures = client.map(process, range(100), param=2)\nresults = client.gather(futures)\nprint(len(results))  # 100\n```\n\n```python\n# Scatter large data to workers (avoids repeated transfers)\nimport numpy as np\nbig_data = np.random.random((10000, 1000))\ndata_future = client.scatter(big_data, broadcast=True)\n\n# Submit tasks using scattered data\nfutures = [client.submit(process_chunk, data_future, i) for i in range(10)]\nresults = client.gather(futures)\n\n# Progressive result processing\nfrom dask.distributed import as_completed\nfor future in as_completed(futures):\n    result = future.result()\n    print(f\"Completed: {result}\")\n\n# Coordination primitives\nfrom dask.distributed import Lock, Queue, Event\nlock = Lock('resource-lock')\nwith lock:\n    # Thread-safe operation across workers\n    pass\n\nclient.close()\n```\n\n### 5. Schedulers & Configuration\n\n```python\nimport dask\n\n# Global scheduler setting\ndask.config.set(scheduler='threads')       # Default: GIL-releasing numeric work\ndask.config.set(scheduler='processes')     # Pure Python, GIL-bound work\ndask.config.set(scheduler='synchronous')   # Debugging with pdb\n\n# Context manager for temporary change\nwith dask.config.set(scheduler='synchronous'):\n    result = computation.compute()  # Can use pdb here\n\n# Per-compute override\nresult = ddf.mean().compute(scheduler='processes')\n\n# Distributed scheduler with resource control\nfrom dask.distributed import Client\nclient = Client(n_workers=4, threads_per_worker=2, memory_limit='4GB')\nprint(client.dashboard_link)\n```\n\n```python\n# HPC cluster integration\nfrom dask_jobqueue import SLURMCluster\nfrom dask.distributed import Client\n\ncluster = SLURMCluster(\n    cores=24, memory='100GB',\n    walltime='02:00:00', queue='regular'\n)\ncluster.scale(jobs=10)  # Request 10 SLURM jobs\nclient = Client(cluster)\n\n# Adaptive scaling\ncluster.adapt(minimum=2, maximum=20)\n\nresult = computation.compute()\nclient.close()\n```\n\n## Key Concepts\n\n### Component Selection Guide\n\n| Data Type | Component | When to Use |\n|-----------|-----------|-------------|\n| Tabular (CSV, Parquet) | **DataFrames** | Standard pandas-like operations at scale |\n| Numeric arrays (HDF5, Zarr) | **Arrays** | NumPy operations, linear algebra, image processing |\n| Text, JSON, logs | **Bags** | ETL\u002Fcleaning → convert to DataFrame for analysis |\n| Custom parallel tasks | **Futures** | Dynamic workflows, parameter sweeps, task dependencies |\n| Any of above | **Schedulers** | Control execution backend (threads\u002Fprocesses\u002Fdistributed) |\n\n**Control level**: DataFrames\u002FArrays\u002FBags = high-level lazy API. Futures = low-level immediate execution.\n\n### Lazy Evaluation Model\n\nAll DataFrames, Arrays, and Bags build a **task graph** — nothing executes until `.compute()` or `.persist()`.\n\n- `.compute()` — execute and return result to local memory\n- `.persist()` — execute and keep result on workers (for reuse across multiple computations)\n- `dask.compute(a, b, c)` — compute multiple results in a single pass (shares intermediates)\n\n### Chunk Size Strategy\n\nTarget: **~100 MB per chunk** (or 10 chunks per core in worker memory).\n\n| Chunk Size | Effect |\n|-----------|--------|\n| Too large (>1 GB) | Memory overflow, poor parallelization |\n| Optimal (~100 MB) | Good parallelism, manageable memory |\n| Too small (\u003C1 MB) | Excessive scheduling overhead |\n\n**Example**: 8 cores, 32 GB RAM → target ~400 MB per chunk (32 GB \u002F 8 cores \u002F 10).\n\n### Scheduler Selection Guide\n\n| Scheduler | Overhead | Best For | GIL |\n|-----------|----------|----------|-----|\n| `threads` (default) | ~10 µs\u002Ftask | NumPy, pandas, scikit-learn | Affected |\n| `processes` | ~10 ms\u002Ftask | Pure Python, text processing | Not affected |\n| `synchronous` | ~1 µs\u002Ftask | Debugging with pdb | N\u002FA |\n| `distributed` | ~1 ms\u002Ftask | Dashboard, clusters, advanced features | Configurable |\n\n## Common Workflows\n\n### Workflow 1: Multi-File ETL Pipeline\n\n```python\nimport dask.dataframe as dd\nimport dask\n\n# Extract: Read all CSV files\nddf = dd.read_csv('raw_data\u002F*.csv', dtype={'amount': 'float64'})\n\n# Transform: Clean and process\nddf = ddf[ddf['status'] == 'valid']\nddf['amount'] = ddf['amount'].fillna(0)\nddf = ddf.dropna(subset=['category'])\n\n# Aggregate\nsummary = ddf.groupby('category').agg({'amount': ['sum', 'mean', 'count']})\n\n# Load: Save as Parquet (columnar, compressed)\nsummary.to_parquet('output\u002Fsummary.parquet')\nprint(f\"Processed {len(ddf)} rows across {ddf.npartitions} partitions\")\n```\n\n### Workflow 2: Large-Scale Array Processing\n\n```python\nimport dask.array as da\n\n# Load large scientific dataset\nx = da.from_zarr('experiment_data.zarr')  # e.g., (50000, 50000) float64\nprint(f\"Shape: {x.shape}, Chunks: {x.chunks}\")\n\n# Normalize per-column\nx_norm = (x - x.mean(axis=0)) \u002F x.std(axis=0)\n\n# Compute covariance matrix\ncov = da.dot(x_norm.T, x_norm) \u002F (x_norm.shape[0] - 1)\n\n# SVD for dimensionality reduction (top-k)\nu, s, v = da.linalg.svd_compressed(x_norm, k=50)\n\n# Save results\nda.to_zarr(u, 'pca_components.zarr')\nprint(f\"Explained variance (top 5): {(s[:5]**2 \u002F (s**2).sum()).compute()}\")\n```\n\n### Workflow 3: Unstructured Data to Analysis\n\nThis workflow is a simple combination of Bags (Section 3) → DataFrames (Section 1): read JSON logs with Bags, filter\u002Ftransform, convert to DataFrame for groupby analysis. Each step maps directly to Core API examples above.\n\n## Key Parameters\n\n| Parameter | Module | Default | Description |\n|-----------|--------|---------|-------------|\n| `npartitions` | DataFrame | auto | Number of partitions (controls parallelism) |\n| `partition_size` | DataFrame | — | Target size per partition (e.g., '100MB') |\n| `chunks` | Array | required | Chunk dimensions (e.g., `(10000, 1000)`) |\n| `blocksize` | Bag | '128 MiB' | File read block size |\n| `scheduler` | All | 'threads' | Execution backend ('threads', 'processes', 'synchronous') |\n| `n_workers` | Distributed | auto | Number of worker processes |\n| `threads_per_worker` | Distributed | auto | Threads per worker |\n| `memory_limit` | Distributed | auto | Per-worker memory limit (e.g., '4GB') |\n| `sorted` | `set_index` | False | Whether data is pre-sorted (enables optimizations) |\n| `meta` | `map_partitions` | — | Output DataFrame\u002FSeries structure template |\n\n## Best Practices\n\n1. **Let Dask handle data loading** — Never load data into pandas\u002Fnumpy first then convert. Use `dd.read_csv()` \u002F `da.from_zarr()` directly.\n\n2. **Batch compute calls** — Use `dask.compute(a, b, c)` instead of calling `.compute()` in loops. Allows sharing intermediates.\n\n3. **Use `map_partitions` over `apply`** — `ddf.apply(func, axis=1)` creates one task per row. `ddf.map_partitions(func)` creates one task per partition.\n\n4. **Persist reused intermediates** — Call `.persist()` on data accessed multiple times, then `del` when done.\n\n5. **Use the dashboard** — `client.dashboard_link` shows task progress, memory usage, worker states. Essential for diagnosing performance issues.\n\n6. **Anti-pattern — Excessively large task graphs**: If `len(ddf.__dask_graph__())` returns millions, increase chunk sizes or use `map_partitions`\u002F`map_blocks` to fuse operations.\n\n7. **Anti-pattern — Wrong scheduler for workload**: Using threads for pure Python text processing (GIL-bound) or processes for NumPy operations (unnecessary serialization overhead).\n\n## Common Recipes\n\n### Recipe: Dask-ML Integration\n\n```python\nfrom dask_ml.preprocessing import StandardScaler\nfrom dask_ml.model_selection import train_test_split\nimport dask.array as da\n\nX = da.random.random((100000, 50), chunks=(10000, 50))\ny = da.random.randint(0, 2, size=100000, chunks=10000)\n\n# Preprocessing\nscaler = StandardScaler()\nX_scaled = scaler.fit_transform(X)\n\n# Train\u002Ftest split\nX_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2)\nprint(f\"Train: {X_train.shape}, Test: {X_test.shape}\")\n```\n\n### Recipe: Actors for Stateful Computation\n\n```python\nfrom dask.distributed import Client\n\nclient = Client()\n\nclass RunningStats:\n    def __init__(self):\n        self.count = 0\n        self.total = 0.0\n\n    def add(self, value):\n        self.count += 1\n        self.total += value\n        return self.total \u002F self.count\n\n# Create actor on a worker\nstats = client.submit(RunningStats, actor=True).result()\n\n# Call methods (~1ms roundtrip)\nfor v in [10, 20, 30]:\n    mean = stats.add(v).result()\n    print(f\"Running mean: {mean}\")\n\nclient.close()\n```\n\n### Recipe: Kubernetes Cluster\n\n```python\nfrom dask_kubernetes import KubeCluster\nfrom dask.distributed import Client\n\ncluster = KubeCluster()\ncluster.adapt(minimum=2, maximum=50)\nclient = Client(cluster)\n\n# Run computation on auto-scaling cluster\nresult = large_computation.compute()\nclient.close()\n```\n\n## Troubleshooting\n\n| Problem | Cause | Solution |\n|---------|-------|----------|\n| `MemoryError` during `.compute()` | Result too large for local memory | Use `.to_parquet()` or `.persist()` instead of `.compute()` |\n| Slow computation start | Task graph has millions of tasks | Increase chunk sizes; use `map_partitions`\u002F`map_blocks` |\n| Poor parallelization | GIL contention with threads scheduler | Switch to `scheduler='processes'` for Python-heavy code |\n| `TypeError` in `map_partitions` | Missing or wrong `meta` parameter | Provide `meta=pd.DataFrame({'col': pd.Series(dtype='float64')})` |\n| Workers killed (OOM) | Chunks exceed worker memory | Decrease chunk size; increase `memory_limit` |\n| `KilledWorker` exception | Worker process crashed | Check worker logs; reduce memory per task; increase `memory_limit` |\n| Slow joins\u002Fmerges | Data not pre-sorted on join key | Call `ddf.set_index('key', sorted=True)` before join |\n| `NotImplementedError` | Operation not supported by Dask | Use `map_partitions` with pandas equivalent |\n| Dashboard not accessible | Distributed client not started | Use `client = Client()` to enable distributed scheduler |\n| Data type mismatch across partitions | Inconsistent CSV files | Specify `dtype` explicitly in `dd.read_csv()` |\n\n## Bundled Resources\n\n- **`references\u002Fcollections_guide.md`** — Detailed DataFrames, Arrays, and Bags guide with comprehensive code examples for reading, transforming, aggregating, and writing data. Covers `map_partitions` patterns, meta parameter, chunking strategies, `map_blocks`, `foldby`, and collection conversion. Consolidated from original `dataframes.md`, `arrays.md`, and `bags.md`. Original `best-practices.md` content relocated to Best Practices section and Key Concepts inline.\n- **`references\u002Fdistributed_computing.md`** — Futures API, distributed coordination primitives (Locks, Queues, Events, Variables), Actors, scheduler configuration, HPC cluster setup (SLURM, Kubernetes), adaptive scaling, dashboard monitoring, and performance profiling. Consolidated from original `futures.md` and `schedulers.md`.\n\nNot migrated: Original had 6 reference files. `best-practices.md` content consolidated into Best Practices section and Key Concepts (chunk strategy, scheduler selection). Remaining content organized into 2 reference files covering the 5 main components.\n\n## Related Skills\n\n- **polars-dataframes** — In-memory single-machine DataFrame library; faster than Dask for data that fits in RAM\n- **zarr-python** — Chunked array storage format; primary Dask Array persistence backend\n- **scikit-learn-machine-learning** — ML library; Dask-ML provides distributed wrappers\n\n## References\n\n- Official Documentation: https:\u002F\u002Fdocs.dask.org\u002F\n- Dask Best Practices: https:\u002F\u002Fdocs.dask.org\u002Fen\u002Fstable\u002Fbest-practices.html\n- Dask-ML: https:\u002F\u002Fml.dask.org\u002F\n- Dask-Jobqueue (HPC): https:\u002F\u002Fjobqueue.dask.org\u002F\n","# Dask — Parallel & Distributed Computing\n\n## Overview\n\nDask is a Python library for parallel and distributed computing that scales familiar pandas\u002FNumPy APIs to larger-than-memory datasets. It provides five main components (DataFrames, Arrays, Bags, Futures, Schedulers) and scales from single-machine multi-core to multi-node HPC clusters.\n\n## When to Use\n\n- Processing datasets that exceed available RAM (10 GB–100 TB)\n- Parallelizing pandas or NumPy operations across multiple cores\n- Processing multiple files efficiently (CSV, Parquet, JSON, HDF5, Zarr)\n- Building custom parallel workflows with task dependencies\n- Distributing workloads across HPC clusters (SLURM, Kubernetes)\n- Streaming\u002FETL pipelines for unstructured data (logs, JSON records)\n- For **in-memory single-machine speed**: use polars instead\n- For **out-of-core single-machine analytics**: use vaex instead\n\n## Prerequisites\n\n```bash\npip install dask[complete]        # All components\npip install dask[dataframe]       # DataFrames only\npip install dask[distributed]     # Distributed scheduler + dashboard\npip install dask-jobqueue          # HPC cluster integration (SLURM, PBS)\n```\n\n## Core API\n\n### 1. DataFrames — Parallel Pandas\n\n```python\nimport dask.dataframe as dd\n\n# Read multiple files as a single DataFrame\nddf = dd.read_csv('data\u002F2024-*.csv')\nddf = dd.read_parquet('data\u002F', columns=['id', 'value', 'category'])\n\n# Operations are lazy until .compute()\nfiltered = ddf[ddf['value'] > 100]\nresult = filtered.groupby('category').agg({'value': ['mean', 'sum']}).compute()\nprint(result.shape)  # (n_categories, 2)\n\n# Custom operations via map_partitions (preferred over apply)\ndef normalize_partition(df):\n    df['norm_value'] = (df['value'] - df['value'].mean()) \u002F df['value'].std()\n    return df\n\nddf = ddf.map_partitions(normalize_partition)\n\n# Joins\nddf_merged = ddf.merge(lookup_ddf, on='category', how='left')\n\n# Write results\nddf.to_parquet('output\u002F', engine='pyarrow')\n```\n\n```python\n# Repartitioning for optimal chunk sizes\nddf = ddf.repartition(npartitions=20)         # By count\nddf = ddf.repartition(partition_size='100MB')  # By size\n\n# Index management for sorted operations\nddf = ddf.set_index('timestamp', sorted=True)\n\n# Debugging\nprint(f\"Partitions: {ddf.npartitions}\")\nprint(f\"Dtypes: {ddf.dtypes}\")\nsample = ddf.get_partition(0).compute()  # Inspect first partition\n```\n\n### 2. Arrays — Parallel NumPy\n\n```python\nimport dask.array as da\nimport numpy as np\n\n# Create from various sources\nx = da.random.random((100000, 1000), chunks=(10000, 1000))\nx = da.from_array(np_array, chunks=(10000, 1000))\nx = da.from_zarr('large_dataset.zarr')\n\n# Standard operations (lazy)\ny = (x - x.mean(axis=0)) \u002F x.std(axis=0)  # Normalize\nz = da.dot(x.T, x)                          # Matrix multiply\nu, s, v = da.linalg.svd(x)                  # SVD\n\n# Compute and persist\nresult = y.mean(axis=0).compute()\nprint(result.shape)  # (1000,)\n```\n\n```python\n# Custom operations with map_blocks\ndef custom_filter(block):\n    from scipy.ndimage import gaussian_filter\n    return gaussian_filter(block, sigma=2)\n\nfiltered = da.map_blocks(custom_filter, x, dtype=x.dtype)\n\n# Rechunking for different access patterns\nx_rechunked = x.rechunk({0: 5000, 1: 500})\n\n# Save to disk\nda.to_zarr(y, 'normalized.zarr')\n```\n\n### 3. Bags — Unstructured Data Processing\n\n```python\nimport dask.bag as db\nimport json\n\n# Read unstructured data\nbag = db.read_text('logs\u002F*.json').map(json.loads)\n\n# Functional operations\nvalid = bag.filter(lambda x: x['status'] == 'success')\nids = valid.pluck('user_id')\nflat = bag.map(lambda x: x['tags']).flatten()\n\n# Aggregation — use foldby instead of groupby (much faster)\ncounts = bag.foldby(\n    key='category',\n    binop=lambda total, x: total + x['amount'],\n    initial=0,\n    combine=lambda a, b: a + b,\n    combine_initial=0\n).compute()\n\n# Convert to DataFrame for structured analysis\nddf = valid.to_dataframe(meta={'user_id': 'str', 'amount': 'float64', 'category': 'str'})\n```\n\n### 4. Futures — Task-Based Parallelism\n\n```python\nfrom dask.distributed import Client\n\nclient = Client()  # Local cluster with all cores\nprint(client.dashboard_link)  # http:\u002F\u002Flocalhost:8787\n\n# Submit individual tasks (executes immediately, not lazy)\ndef process(x, param):\n    return x ** param\n\nfuture = client.submit(process, 42, param=2)\nprint(future.result())  # 1764\n\n# Map over many inputs\nfutures = client.map(process, range(100), param=2)\nresults = client.gather(futures)\nprint(len(results))  # 100\n```\n\n```python\n# Scatter large data to workers (avoids repeated transfers)\nimport numpy as np\nbig_data = np.random.random((10000, 1000))\ndata_future = client.scatter(big_data, broadcast=True)\n\n# Submit tasks using scattered data\nfutures = [client.submit(process_chunk, data_future, i) for i in range(10)]\nresults = client.gather(futures)\n\n# Progressive result processing\nfrom dask.distributed import as_completed\nfor future in as_completed(futures):\n    result = future.result()\n    print(f\"Completed: {result}\")\n\n# Coordination primitives\nfrom dask.distributed import Lock, Queue, Event\nlock = Lock('resource-lock')\nwith lock:\n    # Thread-safe operation across workers\n    pass\n\nclient.close()\n```\n\n### 5. Schedulers & Configuration\n\n```python\nimport dask\n\n# Global scheduler setting\ndask.config.set(scheduler='threads')       # Default: GIL-releasing numeric work\ndask.config.set(scheduler='processes')     # Pure Python, GIL-bound work\ndask.config.set(scheduler='synchronous')   # Debugging with pdb\n\n# Context manager for temporary change\nwith dask.config.set(scheduler='synchronous'):\n    result = computation.compute()  # Can use pdb here\n\n# Per-compute override\nresult = ddf.mean().compute(scheduler='processes')\n\n# Distributed scheduler with resource control\nfrom dask.distributed import Client\nclient = Client(n_workers=4, threads_per_worker=2, memory_limit='4GB')\nprint(client.dashboard_link)\n```\n\n```python\n# HPC cluster integration\nfrom dask_jobqueue import SLURMCluster\nfrom dask.distributed import Client\n\ncluster = SLURMCluster(\n    cores=24, memory='100GB',\n    walltime='02:00:00', queue='regular'\n)\ncluster.scale(jobs=10)  # Request 10 SLURM jobs\nclient = Client(cluster)\n\n# Adaptive scaling\ncluster.adapt(minimum=2, maximum=20)\n\nresult = computation.compute()\nclient.close()\n```\n\n## Key Concepts\n\n### Component Selection Guide\n\n| Data Type | Component | When to Use |\n|-----------|-----------|-------------|\n| Tabular (CSV, Parquet) | **DataFrames** | Standard pandas-like operations at scale |\n| Numeric arrays (HDF5, Zarr) | **Arrays** | NumPy operations, linear algebra, image processing |\n| Text, JSON, logs | **Bags** | ETL\u002Fcleaning → convert to DataFrame for analysis |\n| Custom parallel tasks | **Futures** | Dynamic workflows, parameter sweeps, task dependencies |\n| Any of above | **Schedulers** | Control execution backend (threads\u002Fprocesses\u002Fdistributed) |\n\n**Control level**: DataFrames\u002FArrays\u002FBags = high-level lazy API. Futures = low-level immediate execution.\n\n### Lazy Evaluation Model\n\nAll DataFrames, Arrays, and Bags build a **task graph** — nothing executes until `.compute()` or `.persist()`.\n\n- `.compute()` — execute and return result to local memory\n- `.persist()` — execute and keep result on workers (for reuse across multiple computations)\n- `dask.compute(a, b, c)` — compute multiple results in a single pass (shares intermediates)\n\n### Chunk Size Strategy\n\nTarget: **~100 MB per chunk** (or 10 chunks per core in worker memory).\n\n| Chunk Size | Effect |\n|-----------|--------|\n| Too large (>1 GB) | Memory overflow, poor parallelization |\n| Optimal (~100 MB) | Good parallelism, manageable memory |\n| Too small (\u003C1 MB) | Excessive scheduling overhead |\n\n**Example**: 8 cores, 32 GB RAM → target ~400 MB per chunk (32 GB \u002F 8 cores \u002F 10).\n\n### Scheduler Selection Guide\n\n| Scheduler | Overhead | Best For | GIL |\n|-----------|----------|----------|-----|\n| `threads` (default) | ~10 µs\u002Ftask | NumPy, pandas, scikit-learn | Affected |\n| `processes` | ~10 ms\u002Ftask | Pure Python, text processing | Not affected |\n| `synchronous` | ~1 µs\u002Ftask | Debugging with pdb | N\u002FA |\n| `distributed` | ~1 ms\u002Ftask | Dashboard, clusters, advanced features | Configurable |\n\n## Common Workflows\n\n### Workflow 1: Multi-File ETL Pipeline\n\n```python\nimport dask.dataframe as dd\nimport dask\n\n# Extract: Read all CSV files\nddf = dd.read_csv('raw_data\u002F*.csv', dtype={'amount': 'float64'})\n\n# Transform: Clean and process\nddf = ddf[ddf['status'] == 'valid']\nddf['amount'] = ddf['amount'].fillna(0)\nddf = ddf.dropna(subset=['category'])\n\n# Aggregate\nsummary = ddf.groupby('category').agg({'amount': ['sum', 'mean', 'count']})\n\n# Load: Save as Parquet (columnar, compressed)\nsummary.to_parquet('output\u002Fsummary.parquet')\nprint(f\"Processed {len(ddf)} rows across {ddf.npartitions} partitions\")\n```\n\n### Workflow 2: Large-Scale Array Processing\n\n```python\nimport dask.array as da\n\n# Load large scientific dataset\nx = da.from_zarr('experiment_data.zarr')  # e.g., (50000, 50000) float64\nprint(f\"Shape: {x.shape}, Chunks: {x.chunks}\")\n\n# Normalize per-column\nx_norm = (x - x.mean(axis=0)) \u002F x.std(axis=0)\n\n# Compute covariance matrix\ncov = da.dot(x_norm.T, x_norm) \u002F (x_norm.shape[0] - 1)\n\n# SVD for dimensionality reduction (top-k)\nu, s, v = da.linalg.svd_compressed(x_norm, k=50)\n\n# Save results\nda.to_zarr(u, 'pca_components.zarr')\nprint(f\"Explained variance (top 5): {(s[:5]**2 \u002F (s**2).sum()).compute()}\")\n```\n\n### Workflow 3: Unstructured Data to Analysis\n\nThis workflow is a simple combination of Bags (Section 3) → DataFrames (Section 1): read JSON logs with Bags, filter\u002Ftransform, convert to DataFrame for groupby analysis. Each step maps directly to Core API examples above.\n\n## Key Parameters\n\n| Parameter | Module | Default | Description |\n|-----------|--------|---------|-------------|\n| `npartitions` | DataFrame | auto | Number of partitions (controls parallelism) |\n| `partition_size` | DataFrame | — | Target size per partition (e.g., '100MB') |\n| `chunks` | Array | required | Chunk dimensions (e.g., `(10000, 1000)`) |\n| `blocksize` | Bag | '128 MiB' | File read block size |\n| `scheduler` | All | 'threads' | Execution backend ('threads', 'processes', 'synchronous') |\n| `n_workers` | Distributed | auto | Number of worker processes |\n| `threads_per_worker` | Distributed | auto | Threads per worker |\n| `memory_limit` | Distributed | auto | Per-worker memory limit (e.g., '4GB') |\n| `sorted` | `set_index` | False | Whether data is pre-sorted (enables optimizations) |\n| `meta` | `map_partitions` | — | Output DataFrame\u002FSeries structure template |\n\n## Best Practices\n\n1. **Let Dask handle data loading** — Never load data into pandas\u002Fnumpy first then convert. Use `dd.read_csv()` \u002F `da.from_zarr()` directly.\n\n2. **Batch compute calls** — Use `dask.compute(a, b, c)` instead of calling `.compute()` in loops. Allows sharing intermediates.\n\n3. **Use `map_partitions` over `apply`** — `ddf.apply(func, axis=1)` creates one task per row. `ddf.map_partitions(func)` creates one task per partition.\n\n4. **Persist reused intermediates** — Call `.persist()` on data accessed multiple times, then `del` when done.\n\n5. **Use the dashboard** — `client.dashboard_link` shows task progress, memory usage, worker states. Essential for diagnosing performance issues.\n\n6. **Anti-pattern — Excessively large task graphs**: If `len(ddf.__dask_graph__())` returns millions, increase chunk sizes or use `map_partitions`\u002F`map_blocks` to fuse operations.\n\n7. **Anti-pattern — Wrong scheduler for workload**: Using threads for pure Python text processing (GIL-bound) or processes for NumPy operations (unnecessary serialization overhead).\n\n## Common Recipes\n\n### Recipe: Dask-ML Integration\n\n```python\nfrom dask_ml.preprocessing import StandardScaler\nfrom dask_ml.model_selection import train_test_split\nimport dask.array as da\n\nX = da.random.random((100000, 50), chunks=(10000, 50))\ny = da.random.randint(0, 2, size=100000, chunks=10000)\n\n# Preprocessing\nscaler = StandardScaler()\nX_scaled = scaler.fit_transform(X)\n\n# Train\u002Ftest split\nX_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2)\nprint(f\"Train: {X_train.shape}, Test: {X_test.shape}\")\n```\n\n### Recipe: Actors for Stateful Computation\n\n```python\nfrom dask.distributed import Client\n\nclient = Client()\n\nclass RunningStats:\n    def __init__(self):\n        self.count = 0\n        self.total = 0.0\n\n    def add(self, value):\n        self.count += 1\n        self.total += value\n        return self.total \u002F self.count\n\n# Create actor on a worker\nstats = client.submit(RunningStats, actor=True).result()\n\n# Call methods (~1ms roundtrip)\nfor v in [10, 20, 30]:\n    mean = stats.add(v).result()\n    print(f\"Running mean: {mean}\")\n\nclient.close()\n```\n\n### Recipe: Kubernetes Cluster\n\n```python\nfrom dask_kubernetes import KubeCluster\nfrom dask.distributed import Client\n\ncluster = KubeCluster()\ncluster.adapt(minimum=2, maximum=50)\nclient = Client(cluster)\n\n# Run computation on auto-scaling cluster\nresult = large_computation.compute()\nclient.close()\n```\n\n## Troubleshooting\n\n| Problem | Cause | Solution |\n|---------|-------|----------|\n| `MemoryError` during `.compute()` | Result too large for local memory | Use `.to_parquet()` or `.persist()` instead of `.compute()` |\n| Slow computation start | Task graph has millions of tasks | Increase chunk sizes; use `map_partitions`\u002F`map_blocks` |\n| Poor parallelization | GIL contention with threads scheduler | Switch to `scheduler='processes'` for Python-heavy code |\n| `TypeError` in `map_partitions` | Missing or wrong `meta` parameter | Provide `meta=pd.DataFrame({'col': pd.Series(dtype='float64')})` |\n| Workers killed (OOM) | Chunks exceed worker memory | Decrease chunk size; increase `memory_limit` |\n| `KilledWorker` exception | Worker process crashed | Check worker logs; reduce memory per task; increase `memory_limit` |\n| Slow joins\u002Fmerges | Data not pre-sorted on join key | Call `ddf.set_index('key', sorted=True)` before join |\n| `NotImplementedError` | Operation not supported by Dask | Use `map_partitions` with pandas equivalent |\n| Dashboard not accessible | Distributed client not started | Use `client = Client()` to enable distributed scheduler |\n| Data type mismatch across partitions | Inconsistent CSV files | Specify `dtype` explicitly in `dd.read_csv()` |\n\n## Bundled Resources\n\n- **`references\u002Fcollections_guide.md`** — Detailed DataFrames, Arrays, and Bags guide with comprehensive code examples for reading, transforming, aggregating, and writing data. Covers `map_partitions` patterns, meta parameter, chunking strategies, `map_blocks`, `foldby`, and collection conversion. Consolidated from original `dataframes.md`, `arrays.md`, and `bags.md`. Original `best-practices.md` content relocated to Best Practices section and Key Concepts inline.\n- **`references\u002Fdistributed_computing.md`** — Futures API, distributed coordination primitives (Locks, Queues, Events, Variables), Actors, scheduler configuration, HPC cluster setup (SLURM, Kubernetes), adaptive scaling, dashboard monitoring, and performance profiling. Consolidated from original `futures.md` and `schedulers.md`.\n\nNot migrated: Original had 6 reference files. `best-practices.md` content consolidated into Best Practices section and Key Concepts (chunk strategy, scheduler selection). Remaining content organized into 2 reference files covering the 5 main components.\n\n## Related Skills\n\n- **polars-dataframes** — In-memory single-machine DataFrame library; faster than Dask for data that fits in RAM\n- **zarr-python** — Chunked array storage format; primary Dask Array persistence backend\n- **scikit-learn-machine-learning** — ML library; Dask-ML provides distributed wrappers\n\n## References\n\n- Official Documentation: https:\u002F\u002Fdocs.dask.org\u002F\n- Dask Best Practices: https:\u002F\u002Fdocs.dask.org\u002Fen\u002Fstable\u002Fbest-practices.html\n- Dask-ML: https:\u002F\u002Fml.dask.org\u002F\n- Dask-Jobqueue (HPC): https:\u002F\u002Fjobqueue.dask.org\u002F\n",{"name":5,"license":15,"description":7},"master","63f17866001c5ca854bf2fa0b678ce277133de5b","skills\u002Fawesome-skills\u002Fdask-parallel-computing",0,"https:\u002F\u002Fgithub.com\u002Fbg-szy\u002FTOP-SKILLS","https:\u002F\u002Fbg-szy.github.io\u002FTOP-SKILLS\u002F","https:\u002F\u002Fgithub.com\u002Fbg-szy","https:\u002F\u002Fwww.ecust.edu.cn\u002F","a student of East China University of Science and Technology","2026-11-15 03:38:02",{"source":8,"skillName":5,"commands":35,"prompt":45,"zipAvailable":46},[36,39,42],{"tool":37,"command":38},"npx","npx skills add bg-szy\u002FTOP-SKILLS --skill dask-parallel-computing",{"tool":40,"command":41},"bunx","bunx skills add bg-szy\u002FTOP-SKILLS --skill dask-parallel-computing",{"tool":43,"command":44},"pnpm","pnpm dlx skills add bg-szy\u002FTOP-SKILLS --skill dask-parallel-computing","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fbg-szy\u002FTOP-SKILLS\u002Fblob\u002Fmaster\u002Fskills\u002Fawesome-skills\u002Fdask-parallel-computing\u002FSKILL.md",true,[],[49,58,65,72,79,86],{"skillNo":50,"name":51,"nameZh":6,"description":52,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":53,"licenseFileUrl":6,"sourceHtmlUrl":54,"categories":55,"keywords":56,"publishedAt":57},"01M7DXW9N826PKXK174PVW3038","serpmaven","Scrape Google search results (SERP) to JSON with SERP Maven (serpmaven) - a real Chrome browser on a residential proxy with sticky sessions. Run it as a CLI, a Python library, a local HTTP search API (`serve`), an MCP server or a Docker image; set the proxy login, workers, pages and cost tracking. Use when the user wants Google results, rank tracking, a self-hosted SERP API, or live Google search for an AI agent.","MIT","https:\u002F\u002Fgithub.com\u002Fbg-szy\u002FTOP-SKILLS\u002Fblob\u002Fmaster\u002Fskills\u002Fmarketplace\u002Fserpmaven\u002FSKILL.md",[],[],"2026-11-15 14:19:17",{"skillNo":59,"name":60,"nameZh":6,"description":61,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":53,"licenseFileUrl":6,"sourceHtmlUrl":62,"categories":63,"keywords":64,"publishedAt":57},"01M7DXVHSTWDV65475QEQ915M7","revnu-landing-page-audit","Find out why a landing page isn't turning visitors into signups, demos or sales. Audits one page against what the ad, post or search that sends people there promised, then checks the first screen, proof, call to action, friction and mobile. Returns at most 7 prioritized fixes, plus what is fine and what could not be checked; rewrites specific lines and fixes the page in the repo when asked. Never promises a conversion lift. Works without a Revnu account. Use when the user says or asks: \"audit my landing page\", \"why isn't this page converting\", \"review my homepage\", \"my ads get clicks but no signups\", \"people bounce from the pricing page\", \"CRO review\", \"improve conversion rate\", \"is my hero section clear\", \"roast my landing page\", \"rewrite my headline\", \"what's wrong with this page\", \"landing page feedback\", \"make the signup page convert better\". Not for SEO or AI-search audits, page speed engineering on its own, or building a new page from scratch.","https:\u002F\u002Fgithub.com\u002Fbg-szy\u002FTOP-SKILLS\u002Fblob\u002Fmaster\u002Fskills\u002Fmarketplace\u002Frevnu-landing-page-audit\u002FSKILL.md",[],[],{"skillNo":66,"name":67,"nameZh":6,"description":68,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":53,"licenseFileUrl":6,"sourceHtmlUrl":69,"categories":70,"keywords":71,"publishedAt":57},"01M7DXV8X2D7B3VTD4HHYYHTGR","revnu-growth-context","Write down once who buys, what they're stuck on, what it costs, what proof you have and what you've already tried, so every growth task starts from facts instead of a questionnaire. Creates or updates .agents\u002Frevnu-growth.md from the repo, the live site and the user, and imports an existing product-marketing.md. Use when the user says \"set up our marketing context\", \"who is our ideal customer\", \"write our ICP\", \"describe the product for marketing\", \"positioning doc\", \"what's our brand voice\", \"remember this about our customers\", \"log what we tried\", \"what have we learned from marketing\", \"we changed pricing, update the context\", \"update our growth notes\", \"summarize the business for the agent\", \"import my product-marketing.md\", \"start a growth doc\", or before a first marketing, SEO, outreach or launch task when no context file exists. Works without a Revnu account.","https:\u002F\u002Fgithub.com\u002Fbg-szy\u002FTOP-SKILLS\u002Fblob\u002Fmaster\u002Fskills\u002Fmarketplace\u002Frevnu-growth-context\u002FSKILL.md",[],[],{"skillNo":73,"name":74,"nameZh":6,"description":75,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":53,"licenseFileUrl":6,"sourceHtmlUrl":76,"categories":77,"keywords":78,"publishedAt":57},"01M7DXV0DGJ1G0A60PAHC6J3C5","revnu-get-listed","Get onto the third-party pages AI answers and search cite when buyers ask for the best tool in your category: \"best X\" roundups, directories, review sites, GitHub awesome lists, community threads and comparison pages. Finds which ones are cited, ranks them by fit and effort, and drafts each submission, PR or pitch by the target's own rules. Use when the user says \"get us listed\", \"which directories should we submit to\", \"how do we get into best-of lists\", \"why does ChatGPT recommend competitors and not us\", \"get mentioned in AI answers\", \"submit to awesome lists\", \"get on G2\", \"where are our competitors listed\", \"pitch listicle authors\", \"find backlink opportunities\", \"launch directories\", \"get into roundups\", \"Product Hunt alternatives\", \"where should we be listed\", \"get cited by Perplexity\". Never submits, pays or posts without a yes. Works without a Revnu account.","https:\u002F\u002Fgithub.com\u002Fbg-szy\u002FTOP-SKILLS\u002Fblob\u002Fmaster\u002Fskills\u002Fmarketplace\u002Frevnu-get-listed\u002FSKILL.md",[],[],{"skillNo":80,"name":81,"nameZh":6,"description":82,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":53,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":57},"01M7DXTSPMGT9D6DSS0HP9BSTA","revnu-ai-readiness-audit","Audit whether your website gives ChatGPT, Perplexity, Claude, Gemini and Google's AI Overviews something worth quoting, and fix the pages that don't. Use when the user says or asks \"is my site ready for AI search\", \"audit my site for AI answers\", \"AEO audit\", \"GEO audit\", \"why doesn't ChatGPT quote our pricing\", \"make our pages citable by AI\", \"do I need llms.txt\", \"does schema help with ChatGPT\", \"check our structured data\", \"is our content visible without JavaScript\", \"AI SEO audit\", \"optimize our homepage for AI answers\", \"LLM SEO check\", \"why does AI get our product wrong\", \"fix our site for AI Overviews\". Checks fetchability and indexability, answers in the raw HTML, entity facts, schema (including JS-injected markup), headings, dates and llms.txt, then gives at most 10 prioritised fixes and edits the repo when asked. Works without a Revnu account.","https:\u002F\u002Fgithub.com\u002Fbg-szy\u002FTOP-SKILLS\u002Fblob\u002Fmaster\u002Fskills\u002Fmarketplace\u002Frevnu-ai-readiness-audit\u002FSKILL.md",[],[],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":53,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":57},"01M7DXSZHCVTRS620GC3S97JZV","proxy-troubleshooting","Diagnose residential proxy failures, with NodeMaven's gateway as the reference - 407 with correct credentials, 406, 410, 500, a connection that hangs, ERR_TUNNEL_CONNECTION_FAILED in a browser, Playwright or Puppeteer ignoring proxy credentials, a sticky session that changes IP, geo-targeting that is silently ignored, passwords with special characters, and unexpected traffic from headless Chrome. Use when a proxy connection fails, returns the wrong location, or costs more traffic than expected.","https:\u002F\u002Fgithub.com\u002Fbg-szy\u002FTOP-SKILLS\u002Fblob\u002Fmaster\u002Fskills\u002Fmarketplace\u002Fproxy-troubleshooting\u002FSKILL.md",[],[]]