[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6EC47ESPJ6Q30QPBTWSFS3Q":3,"skill-same:01M6EC47ESPJ6Q30QPBTWSFS3Q":50},{"skillNo":4,"name":5,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":22,"publishedAt":23,"linkOnly":24,"markdownContent":25,"markdownBody":25,"frontMatter":26,"refName":27,"commitSha":28,"skillDirectory":29,"forksCount":30,"repositoryHtmlUrl":31,"repositoryHomepageUrl":32,"ownerHtmlUrl":33,"ownerBlogUrl":6,"ownerDescription":34,"updatedAt":35,"install":36,"resources":49},"01M6EC47ESPJ6Q30QPBTWSFS3Q","dask",null,"ComeOnOliver\u002Fskillshub","ComeOnOliver","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F43215937?v=4","01M6E5JNFECJAAVFE0W7SMWWD2","Haoran Shu","01M6E5JNNDQS7AJZDYFQ8TPZCK",66,"MIT","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002FK-Dense-AI\u002Fclaude-scientific-skills\u002Fdask\u002FSKILL.md",[17],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[5],"2026-11-10 10:12:30",false,"# Dask\n\n## Overview\n\nDask is a Python library for parallel and distributed computing that enables three critical capabilities:\n- **Larger-than-memory execution** on single machines for data exceeding available RAM\n- **Parallel processing** for improved computational speed across multiple cores\n- **Distributed computation** supporting terabyte-scale datasets across multiple machines\n\nDask scales from laptops (processing ~100 GiB) to clusters (processing ~100 TiB) while maintaining familiar Python APIs.\n\n## When to Use This Skill\n\nThis skill should be used when:\n- Process datasets that exceed available RAM\n- Scale pandas or NumPy operations to larger datasets\n- Parallelize computations for performance improvements\n- Process multiple files efficiently (CSVs, Parquet, JSON, text logs)\n- Build custom parallel workflows with task dependencies\n- Distribute workloads across multiple cores or machines\n\n## Core Capabilities\n\nDask provides five main components, each suited to different use cases:\n\n### 1. DataFrames - Parallel Pandas Operations\n\n**Purpose**: Scale pandas operations to larger datasets through parallel processing.\n\n**When to Use**:\n- Tabular data exceeds available RAM\n- Need to process multiple CSV\u002FParquet files together\n- Pandas operations are slow and need parallelization\n- Scaling from pandas prototype to production\n\n**Reference Documentation**: For comprehensive guidance on Dask DataFrames, refer to `references\u002Fdataframes.md` which includes:\n- Reading data (single files, multiple files, glob patterns)\n- Common operations (filtering, groupby, joins, aggregations)\n- Custom operations with `map_partitions`\n- Performance optimization tips\n- Common patterns (ETL, time series, multi-file processing)\n\n**Quick Example**:\n```python\nimport dask.dataframe as dd\n\n# Read multiple files as single DataFrame\nddf = dd.read_csv('data\u002F2024-*.csv')\n\n# Operations are lazy until compute()\nfiltered = ddf[ddf['value'] > 100]\nresult = filtered.groupby('category').mean().compute()\n```\n\n**Key Points**:\n- Operations are lazy (build task graph) until `.compute()` called\n- Use `map_partitions` for efficient custom operations\n- Convert to DataFrame early when working with structured data from other sources\n\n### 2. Arrays - Parallel NumPy Operations\n\n**Purpose**: Extend NumPy capabilities to datasets larger than memory using blocked algorithms.\n\n**When to Use**:\n- Arrays exceed available RAM\n- NumPy operations need parallelization\n- Working with scientific datasets (HDF5, Zarr, NetCDF)\n- Need parallel linear algebra or array operations\n\n**Reference Documentation**: For comprehensive guidance on Dask Arrays, refer to `references\u002Farrays.md` which includes:\n- Creating arrays (from NumPy, random, from disk)\n- Chunking strategies and optimization\n- Common operations (arithmetic, reductions, linear algebra)\n- Custom operations with `map_blocks`\n- Integration with HDF5, Zarr, and XArray\n\n**Quick Example**:\n```python\nimport dask.array as da\n\n# Create large array with chunks\nx = da.random.random((100000, 100000), chunks=(10000, 10000))\n\n# Operations are lazy\ny = x + 100\nz = y.mean(axis=0)\n\n# Compute result\nresult = z.compute()\n```\n\n**Key Points**:\n- Chunk size is critical (aim for ~100 MB per chunk)\n- Operations work on chunks in parallel\n- Rechunk data when needed for efficient operations\n- Use `map_blocks` for operations not available in Dask\n\n### 3. Bags - Parallel Processing of Unstructured Data\n\n**Purpose**: Process unstructured or semi-structured data (text, JSON, logs) with functional operations.\n\n**When to Use**:\n- Processing text files, logs, or JSON records\n- Data cleaning and ETL before structured analysis\n- Working with Python objects that don't fit array\u002Fdataframe formats\n- Need memory-efficient streaming processing\n\n**Reference Documentation**: For comprehensive guidance on Dask Bags, refer to `references\u002Fbags.md` which includes:\n- Reading text and JSON files\n- Functional operations (map, filter, fold, groupby)\n- Converting to DataFrames\n- Common patterns (log analysis, JSON processing, text processing)\n- Performance considerations\n\n**Quick Example**:\n```python\nimport dask.bag as db\nimport json\n\n# Read and parse JSON files\nbag = db.read_text('logs\u002F*.json').map(json.loads)\n\n# Filter and transform\nvalid = bag.filter(lambda x: x['status'] == 'valid')\nprocessed = valid.map(lambda x: {'id': x['id'], 'value': x['value']})\n\n# Convert to DataFrame for analysis\nddf = processed.to_dataframe()\n```\n\n**Key Points**:\n- Use for initial data cleaning, then convert to DataFrame\u002FArray\n- Use `foldby` instead of `groupby` for better performance\n- Operations are streaming and memory-efficient\n- Convert to structured formats (DataFrame) for complex operations\n\n### 4. Futures - Task-Based Parallelization\n\n**Purpose**: Build custom parallel workflows with fine-grained control over task execution and dependencies.\n\n**When to Use**:\n- Building dynamic, evolving workflows\n- Need immediate task execution (not lazy)\n- Computations depend on runtime conditions\n- Implementing custom parallel algorithms\n- Need stateful computations\n\n**Reference Documentation**: For comprehensive guidance on Dask Futures, refer to `references\u002Ffutures.md` which includes:\n- Setting up distributed client\n- Submitting tasks and working with futures\n- Task dependencies and data movement\n- Advanced coordination (queues, locks, events, actors)\n- Common patterns (parameter sweeps, dynamic tasks, iterative algorithms)\n\n**Quick Example**:\n```python\nfrom dask.distributed import Client\n\nclient = Client()  # Create local cluster\n\n# Submit tasks (executes immediately)\ndef process(x):\n    return x ** 2\n\nfutures = client.map(process, range(100))\n\n# Gather results\nresults = client.gather(futures)\n\nclient.close()\n```\n\n**Key Points**:\n- Requires distributed client (even for single machine)\n- Tasks execute immediately when submitted\n- Pre-scatter large data to avoid repeated transfers\n- ~1ms overhead per task (not suitable for millions of tiny tasks)\n- Use actors for stateful workflows\n\n### 5. Schedulers - Execution Backends\n\n**Purpose**: Control how and where Dask tasks execute (threads, processes, distributed).\n\n**When to Choose Scheduler**:\n- **Threads** (default): NumPy\u002FPandas operations, GIL-releasing libraries, shared memory benefit\n- **Processes**: Pure Python code, text processing, GIL-bound operations\n- **Synchronous**: Debugging with pdb, profiling, understanding errors\n- **Distributed**: Need dashboard, multi-machine clusters, advanced features\n\n**Reference Documentation**: For comprehensive guidance on Dask Schedulers, refer to `references\u002Fschedulers.md` which includes:\n- Detailed scheduler descriptions and characteristics\n- Configuration methods (global, context manager, per-compute)\n- Performance considerations and overhead\n- Common patterns and troubleshooting\n- Thread configuration for optimal performance\n\n**Quick Example**:\n```python\nimport dask\nimport dask.dataframe as dd\n\n# Use threads for DataFrame (default, good for numeric)\nddf = dd.read_csv('data.csv')\nresult1 = ddf.mean().compute()  # Uses threads\n\n# Use processes for Python-heavy work\nimport dask.bag as db\nbag = db.read_text('logs\u002F*.txt')\nresult2 = bag.map(python_function).compute(scheduler='processes')\n\n# Use synchronous for debugging\ndask.config.set(scheduler='synchronous')\nresult3 = problematic_computation.compute()  # Can use pdb\n\n# Use distributed for monitoring and scaling\nfrom dask.distributed import Client\nclient = Client()\nresult4 = computation.compute()  # Uses distributed with dashboard\n```\n\n**Key Points**:\n- Threads: Lowest overhead (~10 µs\u002Ftask), best for numeric work\n- Processes: Avoids GIL (~10 ms\u002Ftask), best for Python work\n- Distributed: Monitoring dashboard (~1 ms\u002Ftask), scales to clusters\n- Can switch schedulers per computation or globally\n\n## Best Practices\n\nFor comprehensive performance optimization guidance, memory management strategies, and common pitfalls to avoid, refer to `references\u002Fbest-practices.md`. Key principles include:\n\n### Start with Simpler Solutions\nBefore using Dask, explore:\n- Better algorithms\n- Efficient file formats (Parquet instead of CSV)\n- Compiled code (Numba, Cython)\n- Data sampling\n\n### Critical Performance Rules\n\n**1. Don't Load Data Locally Then Hand to Dask**\n```python\n# Wrong: Loads all data in memory first\nimport pandas as pd\ndf = pd.read_csv('large.csv')\nddf = dd.from_pandas(df, npartitions=10)\n\n# Correct: Let Dask handle loading\nimport dask.dataframe as dd\nddf = dd.read_csv('large.csv')\n```\n\n**2. Avoid Repeated compute() Calls**\n```python\n# Wrong: Each compute is separate\nfor item in items:\n    result = dask_computation(item).compute()\n\n# Correct: Single compute for all\ncomputations = [dask_computation(item) for item in items]\nresults = dask.compute(*computations)\n```\n\n**3. Don't Build Excessively Large Task Graphs**\n- Increase chunk sizes if millions of tasks\n- Use `map_partitions`\u002F`map_blocks` to fuse operations\n- Check task graph size: `len(ddf.__dask_graph__())`\n\n**4. Choose Appropriate Chunk Sizes**\n- Target: ~100 MB per chunk (or 10 chunks per core in worker memory)\n- Too large: Memory overflow\n- Too small: Scheduling overhead\n\n**5. Use the Dashboard**\n```python\nfrom dask.distributed import Client\nclient = Client()\nprint(client.dashboard_link)  # Monitor performance, identify bottlenecks\n```\n\n## Common Workflow Patterns\n\n### ETL Pipeline\n```python\nimport dask.dataframe as dd\n\n# Extract: Read data\nddf = dd.read_csv('raw_data\u002F*.csv')\n\n# Transform: Clean and process\nddf = ddf[ddf['status'] == 'valid']\nddf['amount'] = ddf['amount'].astype('float64')\nddf = ddf.dropna(subset=['important_col'])\n\n# Load: Aggregate and save\nsummary = ddf.groupby('category').agg({'amount': ['sum', 'mean']})\nsummary.to_parquet('output\u002Fsummary.parquet')\n```\n\n### Unstructured to Structured Pipeline\n```python\nimport dask.bag as db\nimport json\n\n# Start with Bag for unstructured data\nbag = db.read_text('logs\u002F*.json').map(json.loads)\nbag = bag.filter(lambda x: x['status'] == 'valid')\n\n# Convert to DataFrame for structured analysis\nddf = bag.to_dataframe()\nresult = ddf.groupby('category').mean().compute()\n```\n\n### Large-Scale Array Computation\n```python\nimport dask.array as da\n\n# Load or create large array\nx = da.from_zarr('large_dataset.zarr')\n\n# Process in chunks\nnormalized = (x - x.mean()) \u002F x.std()\n\n# Save result\nda.to_zarr(normalized, 'normalized.zarr')\n```\n\n### Custom Parallel Workflow\n```python\nfrom dask.distributed import Client\n\nclient = Client()\n\n# Scatter large dataset once\ndata = client.scatter(large_dataset)\n\n# Process in parallel with dependencies\nfutures = []\nfor param in parameters:\n    future = client.submit(process, data, param)\n    futures.append(future)\n\n# Gather results\nresults = client.gather(futures)\n```\n\n## Selecting the Right Component\n\nUse this decision guide to choose the appropriate Dask component:\n\n**Data Type**:\n- Tabular data → **DataFrames**\n- Numeric arrays → **Arrays**\n- Text\u002FJSON\u002Flogs → **Bags** (then convert to DataFrame)\n- Custom Python objects → **Bags** or **Futures**\n\n**Operation Type**:\n- Standard pandas operations → **DataFrames**\n- Standard NumPy operations → **Arrays**\n- Custom parallel tasks → **Futures**\n- Text processing\u002FETL → **Bags**\n\n**Control Level**:\n- High-level, automatic → **DataFrames\u002FArrays**\n- Low-level, manual → **Futures**\n\n**Workflow Type**:\n- Static computation graph → **DataFrames\u002FArrays\u002FBags**\n- Dynamic, evolving → **Futures**\n\n## Integration Considerations\n\n### File Formats\n- **Efficient**: Parquet, HDF5, Zarr (columnar, compressed, parallel-friendly)\n- **Compatible but slower**: CSV (use for initial ingestion only)\n- **For Arrays**: HDF5, Zarr, NetCDF\n\n### Conversion Between Collections\n```python\n# Bag → DataFrame\nddf = bag.to_dataframe()\n\n# DataFrame → Array (for numeric data)\narr = ddf.to_dask_array(lengths=True)\n\n# Array → DataFrame\nddf = dd.from_dask_array(arr, columns=['col1', 'col2'])\n```\n\n### With Other Libraries\n- **XArray**: Wraps Dask arrays with labeled dimensions (geospatial, imaging)\n- **Dask-ML**: Machine learning with scikit-learn compatible APIs\n- **Distributed**: Advanced cluster management and monitoring\n\n## Debugging and Development\n\n### Iterative Development Workflow\n\n1. **Test on small data with synchronous scheduler**:\n```python\ndask.config.set(scheduler='synchronous')\nresult = computation.compute()  # Can use pdb, easy debugging\n```\n\n2. **Validate with threads on sample**:\n```python\nsample = ddf.head(1000)  # Small sample\n# Test logic, then scale to full dataset\n```\n\n3. **Scale with distributed for monitoring**:\n```python\nfrom dask.distributed import Client\nclient = Client()\nprint(client.dashboard_link)  # Monitor performance\nresult = computation.compute()\n```\n\n### Common Issues\n\n**Memory Errors**:\n- Decrease chunk sizes\n- Use `persist()` strategically and delete when done\n- Check for memory leaks in custom functions\n\n**Slow Start**:\n- Task graph too large (increase chunk sizes)\n- Use `map_partitions` or `map_blocks` to reduce tasks\n\n**Poor Parallelization**:\n- Chunks too large (increase number of partitions)\n- Using threads with Python code (switch to processes)\n- Data dependencies preventing parallelism\n\n## Reference Files\n\nAll reference documentation files can be read as needed for detailed information:\n\n- `references\u002Fdataframes.md` - Complete Dask DataFrame guide\n- `references\u002Farrays.md` - Complete Dask Array guide\n- `references\u002Fbags.md` - Complete Dask Bag guide\n- `references\u002Ffutures.md` - Complete Dask Futures and distributed computing guide\n- `references\u002Fschedulers.md` - Complete scheduler selection and configuration guide\n- `references\u002Fbest-practices.md` - Comprehensive performance optimization and troubleshooting\n\nLoad these files when users need detailed information about specific Dask components, operations, or patterns beyond the quick guidance provided here.\n",{},"main","def8531e65114c0fca8fb8551c1871ee0eed705c","skills\u002FK-Dense-AI\u002Fclaude-scientific-skills\u002Fdask",24,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub","https:\u002F\u002Fskillshub.wtf","https:\u002F\u002Fgithub.com\u002FComeOnOliver","AI & Blockchain Engineer\r\n","2026-06-24 22:07:19",{"source":7,"skillName":5,"commands":37,"prompt":47,"zipAvailable":48},[38,41,44],{"tool":39,"command":40},"npx","npx skills add ComeOnOliver\u002Fskillshub --skill dask",{"tool":42,"command":43},"bunx","bunx skills add ComeOnOliver\u002Fskillshub --skill dask",{"tool":45,"command":46},"pnpm","pnpm dlx skills add ComeOnOliver\u002Fskillshub --skill dask","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002FK-Dense-AI\u002Fclaude-scientific-skills\u002Fdask\u002FSKILL.md",true,[],[51,61,71,80,86,93],{"skillNo":52,"name":53,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":54,"categories":55,"keywords":57,"publishedAt":23},"01M6FKJN13GMG4KY57J3QD8GY5","model-hierarchy-skill","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzscole\u002Fmodel-hierarchy-skill\u002Fmodel-hierarchy-skill\u002FSKILL.md",[56],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},[58,59,60],"model","model hierarchy","model hierarchy skill",{"skillNo":62,"name":63,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":64,"categories":65,"keywords":67,"publishedAt":23},"01M6FKJJJ6GCAMR4PPPRD12M5X","zoom-mcp","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-mcp\u002FSKILL.md",[66],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},[68,69,70],"MCP","zoom","zoom mcp",{"skillNo":72,"name":73,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":23},"01M6FKJE3KYT3KPEZMEEHYKA6K","zoom-apps-sdk","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-apps-sdk\u002FSKILL.md",[],[77,78,79,69],"sdk","zoom apps sdk","apps",{"skillNo":81,"name":82,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":23},"01M6FKJ9EDH9Q5VWDF8HAJRSP0","windows","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwindows\u002FSKILL.md",[],[82],{"skillNo":87,"name":88,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":92,"publishedAt":23},"01M6FKJ666BEEFC3AE9GJ72Q93","whiteboard","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwhiteboard\u002FSKILL.md",[91],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},[88],{"skillNo":94,"name":95,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":23},"01M6FKJ21WFCDGTJQ2Q91T9DVW","websockets","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwebsockets\u002FSKILL.md",[],[95]]