[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7BFZBVWN57VPKF062X0KFEV":3,"skill-same:01M7BFZBVWN57VPKF062X0KFEV":78},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":47,"commitSha":48,"skillDirectory":49,"forksCount":50,"repositoryHtmlUrl":51,"repositoryHomepageUrl":6,"ownerHtmlUrl":52,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":53,"install":54,"resources":67},"01M7BFZBVWN57VPKF062X0KFEV","nemo-curator",null,"Curate LLM training data: dedupe, filter, PII redaction.","LIJALEN23\u002FBeryl","LIJALEN23","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F141412346?v=4","01M7AQK5G18E0M3NKP1S35F4KE","oneplusnine","01M7AQK5GNQE3MMPFJ9TCCJQCN",1,"MIT","https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Foptional-skills\u002Fmlops\u002Fnemo-curator\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: nemo-curator\ndescription: \"Curate LLM training data: dedupe, filter, PII redaction.\"\nversion: 1.0.1\nauthor: Orchestra Research\nlicense: MIT\ndependencies: [nemo-curator, cudf, dask, rapids]\nplatforms: [linux, macos]\nmetadata:\n  hermes:\n    tags: [Data Processing, NeMo Curator, Data Curation, GPU Acceleration, Deduplication, Quality Filtering, NVIDIA, RAPIDS, PII Redaction, Multimodal, LLM Training Data]\n\n---\n\n# NeMo Curator - GPU-Accelerated Data Curation\n\nNVIDIA's toolkit for preparing high-quality training data for LLMs.\n\n## When to use NeMo Curator\n\n**Use NeMo Curator when:**\n- Preparing LLM training data from web scrapes (Common Crawl)\n- Need fast deduplication (16× faster than CPU)\n- Curating multi-modal datasets (text, images, video, audio)\n- Filtering low-quality or toxic content\n- Scaling data processing across GPU cluster\n\n**Performance**:\n- **16× faster** fuzzy deduplication (8TB RedPajama v2)\n- **40% lower TCO** vs CPU alternatives\n- **Near-linear scaling** across GPU nodes\n\n**Use alternatives instead**:\n- **datatrove**: CPU-based, open-source data processing\n- **dolma**: Allen AI's data toolkit\n- **Ray Data**: General ML data processing (no curation focus)\n\n## Quick start\n\n### Installation\n\n```bash\n# NeMo Curator 1.x installs with uv. Extras use hyphens (PyPI-normalized):\n#   text-cuda12 \u002F text-cpu (and image\u002Fvideo\u002Faudio\u002Fmath variants), or `all`.\n\n# Text curation (CUDA 12)\nuv pip install \"nemo-curator[text-cuda12]\"\n\n# All modalities\nuv pip install \"nemo-curator[all]\"\n\n# CPU-only text (slower)\nuv pip install \"nemo-curator[text-cpu]\"\n```\n\n### Basic text curation pipeline\n\n> **Major version rewrite (1.x):** NeMo Curator was rewritten around a **Ray-based\n> pipeline\u002Fstage architecture**. The old `DocumentDataset` + `nemo_curator.modules.*` \u002F\n> `ScoreFilter` \u002F `Modify` call-the-object-on-a-dataset API from 0.x is gone. In 1.x you\n> compose `ProcessingStage`s into a `Pipeline` and run it with an executor. The exact\n> stage\u002Fimport surface differs per modality — treat the examples in this skill below as\n> **conceptual** (0.x-style) and follow the current\n> [quickstart](https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FCurator\u002Fblob\u002Fmain\u002Ftutorials\u002Fquickstart.py)\n> and [text guide](https:\u002F\u002Fdocs.nvidia.com\u002Fnemo\u002Fcurator\u002Flatest\u002Fget-started\u002Ftext) for the\n> exact 1.x APIs rather than copying imports verbatim.\n\nShape of a 1.x pipeline (from the upstream quickstart):\n\n```python\nfrom nemo_curator.pipeline import Pipeline\nfrom nemo_curator.stages.base import ProcessingStage\nfrom nemo_curator.stages.resources import Resources\nfrom nemo_curator.backends.xenna import XennaExecutor\nfrom nemo_curator.core.client import RayClient\n\n# 1. Define\u002Fcompose stages (load -> filter -> dedupe -> classify -> write).\n#    Each stage declares its own Resources (CPU cores, GPU memory, replicas).\npipeline = Pipeline(name=\"curation\", stages=[...])\n\n# 2. Run it with an executor (Ray-backed).\nclient = RayClient()\nclient.start()\npipeline.run(XennaExecutor())\nclient.stop()\n```\n\nThe 0.x-style snippets in the sections that follow illustrate the *concepts* (quality\nfiltering, exact\u002Ffuzzy\u002Fsemantic dedup, PII redaction, classifier filtering). For runnable\n1.x code, map each concept onto the corresponding stage from the modality guide.\n\n## Data curation pipeline\n\n### Stage 1: Quality filtering\n\n```python\nfrom nemo_curator.filters import (\n    WordCountFilter,\n    RepeatedLinesFilter,\n    UrlRatioFilter,\n    NonAlphaNumericFilter\n)\n\n# Apply 30+ heuristic filters\nfrom nemo_curator import ScoreFilter\n\n# Word count filter\ndataset = dataset.filter(WordCountFilter(min_words=50, max_words=100000))\n\n# Remove repetitive content\ndataset = dataset.filter(RepeatedLinesFilter(max_repeated_line_fraction=0.3))\n\n# URL ratio filter\ndataset = dataset.filter(UrlRatioFilter(max_url_ratio=0.2))\n```\n\n### Stage 2: Deduplication\n\n**Exact deduplication**:\n```python\nfrom nemo_curator.modules import ExactDuplicates\n\n# Remove exact duplicates\ndeduped = ExactDuplicates(id_field=\"id\", text_field=\"text\")(dataset)\n```\n\n**Fuzzy deduplication** (16× faster on GPU):\n```python\nfrom nemo_curator.modules import FuzzyDuplicates\n\n# MinHash + LSH deduplication\nfuzzy_dedup = FuzzyDuplicates(\n    id_field=\"id\",\n    text_field=\"text\",\n    num_hashes=260,      # MinHash parameters\n    num_buckets=20,\n    hash_method=\"md5\"\n)\n\ndeduped = fuzzy_dedup(dataset)\n```\n\n**Semantic deduplication**:\n```python\nfrom nemo_curator.modules import SemanticDuplicates\n\n# Embedding-based deduplication\nsemantic_dedup = SemanticDuplicates(\n    id_field=\"id\",\n    text_field=\"text\",\n    embedding_model=\"sentence-transformers\u002Fall-MiniLM-L6-v2\",\n    threshold=0.8  # Cosine similarity threshold\n)\n\ndeduped = semantic_dedup(dataset)\n```\n\n### Stage 3: PII redaction\n\n```python\nfrom nemo_curator.modules import Modify\nfrom nemo_curator.modifiers import PIIRedactor\n\n# Redact personally identifiable information\npii_redactor = PIIRedactor(\n    supported_entities=[\"EMAIL_ADDRESS\", \"PHONE_NUMBER\", \"PERSON\", \"LOCATION\"],\n    anonymize_action=\"replace\"  # or \"redact\"\n)\n\nredacted = Modify(pii_redactor)(dataset)\n```\n\n### Stage 4: Classifier filtering\n\n```python\nfrom nemo_curator.classifiers import QualityClassifier\n\n# Quality classification\nquality_clf = QualityClassifier(\n    model_path=\"nvidia\u002Fquality-classifier-deberta\",\n    batch_size=256,\n    device=\"cuda\"\n)\n\n# Filter low-quality documents\nhigh_quality = dataset.filter(lambda doc: quality_clf(doc[\"text\"]) > 0.5)\n```\n\n## GPU acceleration\n\n### GPU vs CPU performance\n\n| Operation | CPU (16 cores) | GPU (A100) | Speedup |\n|-----------|----------------|------------|---------|\n| Fuzzy dedup (8TB) | 120 hours | 7.5 hours | 16× |\n| Exact dedup (1TB) | 8 hours | 0.5 hours | 16× |\n| Quality filtering | 2 hours | 0.2 hours | 10× |\n\n### Multi-GPU scaling\n\n```python\nfrom nemo_curator import get_client\nimport dask_cuda\n\n# Initialize GPU cluster\nclient = get_client(cluster_type=\"gpu\", n_workers=8)\n\n# Process with 8 GPUs\ndeduped = FuzzyDuplicates(...)(dataset)\n```\n\n## Multi-modal curation\n\n### Image curation\n\n```python\nfrom nemo_curator.image import (\n    AestheticFilter,\n    NSFWFilter,\n    CLIPEmbedder\n)\n\n# Aesthetic scoring\naesthetic_filter = AestheticFilter(threshold=5.0)\nfiltered_images = aesthetic_filter(image_dataset)\n\n# NSFW detection\nnsfw_filter = NSFWFilter(threshold=0.9)\nsafe_images = nsfw_filter(filtered_images)\n\n# Generate CLIP embeddings\nclip_embedder = CLIPEmbedder(model=\"openai\u002Fclip-vit-base-patch32\")\nimage_embeddings = clip_embedder(safe_images)\n```\n\n### Video curation\n\n```python\nfrom nemo_curator.video import (\n    SceneDetector,\n    ClipExtractor,\n    InternVideo2Embedder\n)\n\n# Detect scenes\nscene_detector = SceneDetector(threshold=27.0)\nscenes = scene_detector(video_dataset)\n\n# Extract clips\nclip_extractor = ClipExtractor(min_duration=2.0, max_duration=10.0)\nclips = clip_extractor(scenes)\n\n# Generate embeddings\nvideo_embedder = InternVideo2Embedder()\nvideo_embeddings = video_embedder(clips)\n```\n\n### Audio curation\n\n```python\nfrom nemo_curator.audio import (\n    ASRInference,\n    WERFilter,\n    DurationFilter\n)\n\n# ASR transcription\nasr = ASRInference(model=\"nvidia\u002Fstt_en_fastconformer_hybrid_large_pc\")\ntranscribed = asr(audio_dataset)\n\n# Filter by WER (word error rate)\nwer_filter = WERFilter(max_wer=0.3)\nhigh_quality_audio = wer_filter(transcribed)\n\n# Duration filtering\nduration_filter = DurationFilter(min_duration=1.0, max_duration=30.0)\nfiltered_audio = duration_filter(high_quality_audio)\n```\n\n## Common patterns\n\n### Web scrape curation (Common Crawl)\n\n```python\nfrom nemo_curator import ScoreFilter, Modify\nfrom nemo_curator.filters import *\nfrom nemo_curator.modules import *\nfrom nemo_curator.datasets import DocumentDataset\n\n# Load Common Crawl data\ndataset = DocumentDataset.read_parquet(\"common_crawl\u002F*.parquet\")\n\n# Pipeline\npipeline = [\n    # 1. Quality filtering\n    WordCountFilter(min_words=100, max_words=50000),\n    RepeatedLinesFilter(max_repeated_line_fraction=0.2),\n    SymbolToWordRatioFilter(max_symbol_to_word_ratio=0.3),\n    UrlRatioFilter(max_url_ratio=0.3),\n\n    # 2. Language filtering\n    LanguageIdentificationFilter(target_languages=[\"en\"]),\n\n    # 3. Deduplication\n    ExactDuplicates(id_field=\"id\", text_field=\"text\"),\n    FuzzyDuplicates(id_field=\"id\", text_field=\"text\", num_hashes=260),\n\n    # 4. PII redaction\n    PIIRedactor(),\n\n    # 5. NSFW filtering\n    NSFWClassifier(threshold=0.8)\n]\n\n# Execute\nfor stage in pipeline:\n    dataset = stage(dataset)\n\n# Save\ndataset.to_parquet(\"curated_common_crawl\u002F\")\n```\n\n### Distributed processing\n\n```python\nfrom nemo_curator import get_client\nfrom dask_cuda import LocalCUDACluster\n\n# Multi-GPU cluster\ncluster = LocalCUDACluster(n_workers=8)\nclient = get_client(cluster=cluster)\n\n# Process large dataset\ndataset = DocumentDataset.read_parquet(\"s3:\u002F\u002Flarge_dataset\u002F*.parquet\")\ndeduped = FuzzyDuplicates(...)(dataset)\n\n# Cleanup\nclient.close()\ncluster.close()\n```\n\n## Performance benchmarks\n\n### Fuzzy deduplication (8TB RedPajama v2)\n\n- **CPU (256 cores)**: 120 hours\n- **GPU (8× A100)**: 7.5 hours\n- **Speedup**: 16×\n\n### Exact deduplication (1TB)\n\n- **CPU (64 cores)**: 8 hours\n- **GPU (4× A100)**: 0.5 hours\n- **Speedup**: 16×\n\n### Quality filtering (100GB)\n\n- **CPU (32 cores)**: 2 hours\n- **GPU (2× A100)**: 0.2 hours\n- **Speedup**: 10×\n\n## Cost comparison\n\n**CPU-based curation** (AWS c5.18xlarge × 10):\n- Cost: $3.60\u002Fhour × 10 = $36\u002Fhour\n- Time for 8TB: 120 hours\n- **Total**: $4,320\n\n**GPU-based curation** (AWS p4d.24xlarge × 2):\n- Cost: $32.77\u002Fhour × 2 = $65.54\u002Fhour\n- Time for 8TB: 7.5 hours\n- **Total**: $491.55\n\n**Savings**: 89% reduction ($3,828 saved)\n\n## Supported data formats\n\n- **Input**: Parquet, JSONL, CSV\n- **Output**: Parquet (recommended), JSONL\n- **WebDataset**: TAR archives for multi-modal\n\n## Use cases\n\n**Production deployments**:\n- NVIDIA used NeMo Curator to prepare Nemotron-4 training data\n- Open-source datasets curated: RedPajama v2, The Pile\n\n## References\n\n- **[Filtering Guide](references\u002Ffiltering.md)** - 30+ quality filters, heuristics\n- **[Deduplication Guide](references\u002Fdeduplication.md)** - Exact, fuzzy, semantic methods\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FCurator\n- **Docs**: https:\u002F\u002Fdocs.nvidia.com\u002Fnemo\u002Fcurator\u002Flatest\u002F\n- **Version**: 1.2.0 (1.x is a Ray-based pipeline rewrite — see the quickstart before copying 0.x snippets)\n- **License**: Apache 2.0\n\n\n\n","# NeMo Curator - GPU-Accelerated Data Curation\n\nNVIDIA's toolkit for preparing high-quality training data for LLMs.\n\n## When to use NeMo Curator\n\n**Use NeMo Curator when:**\n- Preparing LLM training data from web scrapes (Common Crawl)\n- Need fast deduplication (16× faster than CPU)\n- Curating multi-modal datasets (text, images, video, audio)\n- Filtering low-quality or toxic content\n- Scaling data processing across GPU cluster\n\n**Performance**:\n- **16× faster** fuzzy deduplication (8TB RedPajama v2)\n- **40% lower TCO** vs CPU alternatives\n- **Near-linear scaling** across GPU nodes\n\n**Use alternatives instead**:\n- **datatrove**: CPU-based, open-source data processing\n- **dolma**: Allen AI's data toolkit\n- **Ray Data**: General ML data processing (no curation focus)\n\n## Quick start\n\n### Installation\n\n```bash\n# NeMo Curator 1.x installs with uv. Extras use hyphens (PyPI-normalized):\n#   text-cuda12 \u002F text-cpu (and image\u002Fvideo\u002Faudio\u002Fmath variants), or `all`.\n\n# Text curation (CUDA 12)\nuv pip install \"nemo-curator[text-cuda12]\"\n\n# All modalities\nuv pip install \"nemo-curator[all]\"\n\n# CPU-only text (slower)\nuv pip install \"nemo-curator[text-cpu]\"\n```\n\n### Basic text curation pipeline\n\n> **Major version rewrite (1.x):** NeMo Curator was rewritten around a **Ray-based\n> pipeline\u002Fstage architecture**. The old `DocumentDataset` + `nemo_curator.modules.*` \u002F\n> `ScoreFilter` \u002F `Modify` call-the-object-on-a-dataset API from 0.x is gone. In 1.x you\n> compose `ProcessingStage`s into a `Pipeline` and run it with an executor. The exact\n> stage\u002Fimport surface differs per modality — treat the examples in this skill below as\n> **conceptual** (0.x-style) and follow the current\n> [quickstart](https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FCurator\u002Fblob\u002Fmain\u002Ftutorials\u002Fquickstart.py)\n> and [text guide](https:\u002F\u002Fdocs.nvidia.com\u002Fnemo\u002Fcurator\u002Flatest\u002Fget-started\u002Ftext) for the\n> exact 1.x APIs rather than copying imports verbatim.\n\nShape of a 1.x pipeline (from the upstream quickstart):\n\n```python\nfrom nemo_curator.pipeline import Pipeline\nfrom nemo_curator.stages.base import ProcessingStage\nfrom nemo_curator.stages.resources import Resources\nfrom nemo_curator.backends.xenna import XennaExecutor\nfrom nemo_curator.core.client import RayClient\n\n# 1. Define\u002Fcompose stages (load -> filter -> dedupe -> classify -> write).\n#    Each stage declares its own Resources (CPU cores, GPU memory, replicas).\npipeline = Pipeline(name=\"curation\", stages=[...])\n\n# 2. Run it with an executor (Ray-backed).\nclient = RayClient()\nclient.start()\npipeline.run(XennaExecutor())\nclient.stop()\n```\n\nThe 0.x-style snippets in the sections that follow illustrate the *concepts* (quality\nfiltering, exact\u002Ffuzzy\u002Fsemantic dedup, PII redaction, classifier filtering). For runnable\n1.x code, map each concept onto the corresponding stage from the modality guide.\n\n## Data curation pipeline\n\n### Stage 1: Quality filtering\n\n```python\nfrom nemo_curator.filters import (\n    WordCountFilter,\n    RepeatedLinesFilter,\n    UrlRatioFilter,\n    NonAlphaNumericFilter\n)\n\n# Apply 30+ heuristic filters\nfrom nemo_curator import ScoreFilter\n\n# Word count filter\ndataset = dataset.filter(WordCountFilter(min_words=50, max_words=100000))\n\n# Remove repetitive content\ndataset = dataset.filter(RepeatedLinesFilter(max_repeated_line_fraction=0.3))\n\n# URL ratio filter\ndataset = dataset.filter(UrlRatioFilter(max_url_ratio=0.2))\n```\n\n### Stage 2: Deduplication\n\n**Exact deduplication**:\n```python\nfrom nemo_curator.modules import ExactDuplicates\n\n# Remove exact duplicates\ndeduped = ExactDuplicates(id_field=\"id\", text_field=\"text\")(dataset)\n```\n\n**Fuzzy deduplication** (16× faster on GPU):\n```python\nfrom nemo_curator.modules import FuzzyDuplicates\n\n# MinHash + LSH deduplication\nfuzzy_dedup = FuzzyDuplicates(\n    id_field=\"id\",\n    text_field=\"text\",\n    num_hashes=260,      # MinHash parameters\n    num_buckets=20,\n    hash_method=\"md5\"\n)\n\ndeduped = fuzzy_dedup(dataset)\n```\n\n**Semantic deduplication**:\n```python\nfrom nemo_curator.modules import SemanticDuplicates\n\n# Embedding-based deduplication\nsemantic_dedup = SemanticDuplicates(\n    id_field=\"id\",\n    text_field=\"text\",\n    embedding_model=\"sentence-transformers\u002Fall-MiniLM-L6-v2\",\n    threshold=0.8  # Cosine similarity threshold\n)\n\ndeduped = semantic_dedup(dataset)\n```\n\n### Stage 3: PII redaction\n\n```python\nfrom nemo_curator.modules import Modify\nfrom nemo_curator.modifiers import PIIRedactor\n\n# Redact personally identifiable information\npii_redactor = PIIRedactor(\n    supported_entities=[\"EMAIL_ADDRESS\", \"PHONE_NUMBER\", \"PERSON\", \"LOCATION\"],\n    anonymize_action=\"replace\"  # or \"redact\"\n)\n\nredacted = Modify(pii_redactor)(dataset)\n```\n\n### Stage 4: Classifier filtering\n\n```python\nfrom nemo_curator.classifiers import QualityClassifier\n\n# Quality classification\nquality_clf = QualityClassifier(\n    model_path=\"nvidia\u002Fquality-classifier-deberta\",\n    batch_size=256,\n    device=\"cuda\"\n)\n\n# Filter low-quality documents\nhigh_quality = dataset.filter(lambda doc: quality_clf(doc[\"text\"]) > 0.5)\n```\n\n## GPU acceleration\n\n### GPU vs CPU performance\n\n| Operation | CPU (16 cores) | GPU (A100) | Speedup |\n|-----------|----------------|------------|---------|\n| Fuzzy dedup (8TB) | 120 hours | 7.5 hours | 16× |\n| Exact dedup (1TB) | 8 hours | 0.5 hours | 16× |\n| Quality filtering | 2 hours | 0.2 hours | 10× |\n\n### Multi-GPU scaling\n\n```python\nfrom nemo_curator import get_client\nimport dask_cuda\n\n# Initialize GPU cluster\nclient = get_client(cluster_type=\"gpu\", n_workers=8)\n\n# Process with 8 GPUs\ndeduped = FuzzyDuplicates(...)(dataset)\n```\n\n## Multi-modal curation\n\n### Image curation\n\n```python\nfrom nemo_curator.image import (\n    AestheticFilter,\n    NSFWFilter,\n    CLIPEmbedder\n)\n\n# Aesthetic scoring\naesthetic_filter = AestheticFilter(threshold=5.0)\nfiltered_images = aesthetic_filter(image_dataset)\n\n# NSFW detection\nnsfw_filter = NSFWFilter(threshold=0.9)\nsafe_images = nsfw_filter(filtered_images)\n\n# Generate CLIP embeddings\nclip_embedder = CLIPEmbedder(model=\"openai\u002Fclip-vit-base-patch32\")\nimage_embeddings = clip_embedder(safe_images)\n```\n\n### Video curation\n\n```python\nfrom nemo_curator.video import (\n    SceneDetector,\n    ClipExtractor,\n    InternVideo2Embedder\n)\n\n# Detect scenes\nscene_detector = SceneDetector(threshold=27.0)\nscenes = scene_detector(video_dataset)\n\n# Extract clips\nclip_extractor = ClipExtractor(min_duration=2.0, max_duration=10.0)\nclips = clip_extractor(scenes)\n\n# Generate embeddings\nvideo_embedder = InternVideo2Embedder()\nvideo_embeddings = video_embedder(clips)\n```\n\n### Audio curation\n\n```python\nfrom nemo_curator.audio import (\n    ASRInference,\n    WERFilter,\n    DurationFilter\n)\n\n# ASR transcription\nasr = ASRInference(model=\"nvidia\u002Fstt_en_fastconformer_hybrid_large_pc\")\ntranscribed = asr(audio_dataset)\n\n# Filter by WER (word error rate)\nwer_filter = WERFilter(max_wer=0.3)\nhigh_quality_audio = wer_filter(transcribed)\n\n# Duration filtering\nduration_filter = DurationFilter(min_duration=1.0, max_duration=30.0)\nfiltered_audio = duration_filter(high_quality_audio)\n```\n\n## Common patterns\n\n### Web scrape curation (Common Crawl)\n\n```python\nfrom nemo_curator import ScoreFilter, Modify\nfrom nemo_curator.filters import *\nfrom nemo_curator.modules import *\nfrom nemo_curator.datasets import DocumentDataset\n\n# Load Common Crawl data\ndataset = DocumentDataset.read_parquet(\"common_crawl\u002F*.parquet\")\n\n# Pipeline\npipeline = [\n    # 1. Quality filtering\n    WordCountFilter(min_words=100, max_words=50000),\n    RepeatedLinesFilter(max_repeated_line_fraction=0.2),\n    SymbolToWordRatioFilter(max_symbol_to_word_ratio=0.3),\n    UrlRatioFilter(max_url_ratio=0.3),\n\n    # 2. Language filtering\n    LanguageIdentificationFilter(target_languages=[\"en\"]),\n\n    # 3. Deduplication\n    ExactDuplicates(id_field=\"id\", text_field=\"text\"),\n    FuzzyDuplicates(id_field=\"id\", text_field=\"text\", num_hashes=260),\n\n    # 4. PII redaction\n    PIIRedactor(),\n\n    # 5. NSFW filtering\n    NSFWClassifier(threshold=0.8)\n]\n\n# Execute\nfor stage in pipeline:\n    dataset = stage(dataset)\n\n# Save\ndataset.to_parquet(\"curated_common_crawl\u002F\")\n```\n\n### Distributed processing\n\n```python\nfrom nemo_curator import get_client\nfrom dask_cuda import LocalCUDACluster\n\n# Multi-GPU cluster\ncluster = LocalCUDACluster(n_workers=8)\nclient = get_client(cluster=cluster)\n\n# Process large dataset\ndataset = DocumentDataset.read_parquet(\"s3:\u002F\u002Flarge_dataset\u002F*.parquet\")\ndeduped = FuzzyDuplicates(...)(dataset)\n\n# Cleanup\nclient.close()\ncluster.close()\n```\n\n## Performance benchmarks\n\n### Fuzzy deduplication (8TB RedPajama v2)\n\n- **CPU (256 cores)**: 120 hours\n- **GPU (8× A100)**: 7.5 hours\n- **Speedup**: 16×\n\n### Exact deduplication (1TB)\n\n- **CPU (64 cores)**: 8 hours\n- **GPU (4× A100)**: 0.5 hours\n- **Speedup**: 16×\n\n### Quality filtering (100GB)\n\n- **CPU (32 cores)**: 2 hours\n- **GPU (2× A100)**: 0.2 hours\n- **Speedup**: 10×\n\n## Cost comparison\n\n**CPU-based curation** (AWS c5.18xlarge × 10):\n- Cost: $3.60\u002Fhour × 10 = $36\u002Fhour\n- Time for 8TB: 120 hours\n- **Total**: $4,320\n\n**GPU-based curation** (AWS p4d.24xlarge × 2):\n- Cost: $32.77\u002Fhour × 2 = $65.54\u002Fhour\n- Time for 8TB: 7.5 hours\n- **Total**: $491.55\n\n**Savings**: 89% reduction ($3,828 saved)\n\n## Supported data formats\n\n- **Input**: Parquet, JSONL, CSV\n- **Output**: Parquet (recommended), JSONL\n- **WebDataset**: TAR archives for multi-modal\n\n## Use cases\n\n**Production deployments**:\n- NVIDIA used NeMo Curator to prepare Nemotron-4 training data\n- Open-source datasets curated: RedPajama v2, The Pile\n\n## References\n\n- **[Filtering Guide](references\u002Ffiltering.md)** - 30+ quality filters, heuristics\n- **[Deduplication Guide](references\u002Fdeduplication.md)** - Exact, fuzzy, semantic methods\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FCurator\n- **Docs**: https:\u002F\u002Fdocs.nvidia.com\u002Fnemo\u002Fcurator\u002Flatest\u002F\n- **Version**: 1.2.0 (1.x is a Ray-based pipeline rewrite — see the quickstart before copying 0.x snippets)\n- **License**: Apache 2.0\n\n\n\n",{"name":5,"author":24,"license":15,"version":25,"metadata":26,"platforms":40,"description":7,"dependencies":43},"Orchestra Research","1.0.1",{"hermes":27},{"tags":28},[29,30,31,32,33,34,35,36,37,38,39],"Data Processing","NeMo Curator","Data Curation","GPU Acceleration","Deduplication","Quality Filtering","NVIDIA","RAPIDS","PII Redaction","Multimodal","LLM Training Data",[41,42],"linux","macos",[5,44,45,46],"cudf","dask","rapids","master","4a5caebee4b5f9068d3a68c9a7ecd8df12bd4962","local-agent\u002Foptional-skills\u002Fmlops\u002Fnemo-curator",0,"https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl","https:\u002F\u002Fgithub.com\u002FLIJALEN23","2026-09-01 14:02:08",{"source":8,"skillName":5,"commands":55,"prompt":65,"zipAvailable":66},[56,59,62],{"tool":57,"command":58},"npx","npx skills add LIJALEN23\u002FBeryl --skill nemo-curator",{"tool":60,"command":61},"bunx","bunx skills add LIJALEN23\u002FBeryl --skill nemo-curator",{"tool":63,"command":64},"pnpm","pnpm dlx skills add LIJALEN23\u002FBeryl --skill nemo-curator","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Foptional-skills\u002Fmlops\u002Fnemo-curator\u002FSKILL.md",true,[68,74],{"path":69,"resourceType":70,"resourceTypeDesc":71,"sizeBytes":72,"sourceHtmlUrl":73},"references\u002Fdeduplication.md","REFERENCE","参考文档",2142,"https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Foptional-skills\u002Fmlops\u002Fnemo-curator\u002Freferences\u002Fdeduplication.md",{"path":75,"resourceType":70,"resourceTypeDesc":71,"sizeBytes":76,"sourceHtmlUrl":77},"references\u002Ffiltering.md",2351,"https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Foptional-skills\u002Fmlops\u002Fnemo-curator\u002Freferences\u002Ffiltering.md",[79,86,93,100,107,114],{"skillNo":80,"name":81,"nameZh":6,"description":82,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":19},"01M7BGCRH58RARQTF3R8CZ84DB","test-driven-development","TDD: enforce RED-GREEN-REFACTOR, tests before code.","https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Fskills\u002Fsoftware-development\u002Ftest-driven-development\u002FSKILL.md",[],[],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":19},"01M7BGCK0GGNV3K6NWMM0MDN3W","systematic-debugging","4-phase root cause debugging: understand bugs before fixing.","https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Fskills\u002Fsoftware-development\u002Fsystematic-debugging\u002FSKILL.md",[],[],{"skillNo":94,"name":95,"nameZh":6,"description":96,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":97,"categories":98,"keywords":99,"publishedAt":19},"01M7BGCF0TAZPKSM4WMYA27XGD","spike","Throwaway experiments to validate an idea before build.","https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Fskills\u002Fsoftware-development\u002Fspike\u002FSKILL.md",[],[],{"skillNo":101,"name":102,"nameZh":6,"description":103,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":19},"01M7BGCBH01YYYBQWR7ZFRMX27","simplify-code","Parallel 4-agent cleanup of recent code changes.","https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Fskills\u002Fsoftware-development\u002Fsimplify-code\u002FSKILL.md",[],[],{"skillNo":108,"name":109,"nameZh":6,"description":110,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":111,"categories":112,"keywords":113,"publishedAt":19},"01M7BGC8MGHS5SRJ3NZYYNAKJY","requesting-code-review","Pre-commit review: security scan, quality gates, auto-fix.","https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Fskills\u002Fsoftware-development\u002Frequesting-code-review\u002FSKILL.md",[],[],{"skillNo":115,"name":116,"nameZh":6,"description":117,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":118,"categories":119,"keywords":120,"publishedAt":19},"01M7BGC4HB5ZJP5N19MZTZXKZ1","python-debugpy","Debug Python: pdb REPL + debugpy remote (DAP).","https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Fskills\u002Fsoftware-development\u002Fpython-debugpy\u002FSKILL.md",[],[]]