[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M74EV74FNFGZN9EF5DA4R16F":3,"skill-same:01M74EV74FNFGZN9EF5DA4R16F":79},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":47,"commitSha":48,"skillDirectory":49,"forksCount":14,"repositoryHtmlUrl":50,"repositoryHomepageUrl":6,"ownerHtmlUrl":51,"ownerBlogUrl":52,"ownerDescription":53,"updatedAt":54,"install":55,"resources":68},"01M74EV74FNFGZN9EF5DA4R16F","nemo-curator",null,"GPU-accelerated data curation for LLM training. Supports text\u002Fimage\u002Fvideo\u002Faudio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.","aniruddhaadak80\u002Fherclew","aniruddhaadak80","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F127435065?v=4","01M74EJ1XW3EF2EYN26PFN1TG8","ANIRUDDHA ADAK","01M74EJ1YFT2GM6APEHR1CTTVF",0,"MIT","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fcore-agent\u002Foptional-skills\u002Fmlops\u002Fnemo-curator\u002FSKILL.md",[],[],"2026-11-11 15:11:14",false,"---\nname: nemo-curator\ndescription: GPU-accelerated data curation for LLM training. Supports text\u002Fimage\u002Fvideo\u002Faudio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ndependencies: [nemo-curator, cudf, dask, rapids]\nplatforms: [linux, macos]\nmetadata:\n  hermes:\n    tags: [Data Processing, NeMo Curator, Data Curation, GPU Acceleration, Deduplication, Quality Filtering, NVIDIA, RAPIDS, PII Redaction, Multimodal, LLM Training Data]\n\n---\n\n# NeMo Curator - GPU-Accelerated Data Curation\n\nNVIDIA's toolkit for preparing high-quality training data for LLMs.\n\n## When to use NeMo Curator\n\n**Use NeMo Curator when:**\n- Preparing LLM training data from web scrapes (Common Crawl)\n- Need fast deduplication (16× faster than CPU)\n- Curating multi-modal datasets (text, images, video, audio)\n- Filtering low-quality or toxic content\n- Scaling data processing across GPU cluster\n\n**Performance**:\n- **16× faster** fuzzy deduplication (8TB RedPajama v2)\n- **40% lower TCO** vs CPU alternatives\n- **Near-linear scaling** across GPU nodes\n\n**Use alternatives instead**:\n- **datatrove**: CPU-based, open-source data processing\n- **dolma**: Allen AI's data toolkit\n- **Ray Data**: General ML data processing (no curation focus)\n\n## Quick start\n\n### Installation\n\n```bash\n# Text curation (CUDA 12)\nuv pip install \"nemo-curator[text_cuda12]\"\n\n# All modalities\nuv pip install \"nemo-curator[all_cuda12]\"\n\n# CPU-only (slower)\nuv pip install \"nemo-curator[cpu]\"\n```\n\n### Basic text curation pipeline\n\n```python\nfrom nemo_curator import ScoreFilter, Modify\nfrom nemo_curator.datasets import DocumentDataset\nimport pandas as pd\n\n# Load data\ndf = pd.DataFrame({\"text\": [\"Good document\", \"Bad doc\", \"Excellent text\"]})\ndataset = DocumentDataset(df)\n\n# Quality filtering\ndef quality_score(doc):\n    return len(doc[\"text\"].split()) > 5  # Filter short docs\n\nfiltered = ScoreFilter(quality_score)(dataset)\n\n# Deduplication\nfrom nemo_curator.modules import ExactDuplicates\ndeduped = ExactDuplicates()(filtered)\n\n# Save\ndeduped.to_parquet(\"curated_data\u002F\")\n```\n\n## Data curation pipeline\n\n### Stage 1: Quality filtering\n\n```python\nfrom nemo_curator.filters import (\n    WordCountFilter,\n    RepeatedLinesFilter,\n    UrlRatioFilter,\n    NonAlphaNumericFilter\n)\n\n# Apply 30+ heuristic filters\nfrom nemo_curator import ScoreFilter\n\n# Word count filter\ndataset = dataset.filter(WordCountFilter(min_words=50, max_words=100000))\n\n# Remove repetitive content\ndataset = dataset.filter(RepeatedLinesFilter(max_repeated_line_fraction=0.3))\n\n# URL ratio filter\ndataset = dataset.filter(UrlRatioFilter(max_url_ratio=0.2))\n```\n\n### Stage 2: Deduplication\n\n**Exact deduplication**:\n```python\nfrom nemo_curator.modules import ExactDuplicates\n\n# Remove exact duplicates\ndeduped = ExactDuplicates(id_field=\"id\", text_field=\"text\")(dataset)\n```\n\n**Fuzzy deduplication** (16× faster on GPU):\n```python\nfrom nemo_curator.modules import FuzzyDuplicates\n\n# MinHash + LSH deduplication\nfuzzy_dedup = FuzzyDuplicates(\n    id_field=\"id\",\n    text_field=\"text\",\n    num_hashes=260,      # MinHash parameters\n    num_buckets=20,\n    hash_method=\"md5\"\n)\n\ndeduped = fuzzy_dedup(dataset)\n```\n\n**Semantic deduplication**:\n```python\nfrom nemo_curator.modules import SemanticDuplicates\n\n# Embedding-based deduplication\nsemantic_dedup = SemanticDuplicates(\n    id_field=\"id\",\n    text_field=\"text\",\n    embedding_model=\"sentence-transformers\u002Fall-MiniLM-L6-v2\",\n    threshold=0.8  # Cosine similarity threshold\n)\n\ndeduped = semantic_dedup(dataset)\n```\n\n### Stage 3: PII redaction\n\n```python\nfrom nemo_curator.modules import Modify\nfrom nemo_curator.modifiers import PIIRedactor\n\n# Redact personally identifiable information\npii_redactor = PIIRedactor(\n    supported_entities=[\"EMAIL_ADDRESS\", \"PHONE_NUMBER\", \"PERSON\", \"LOCATION\"],\n    anonymize_action=\"replace\"  # or \"redact\"\n)\n\nredacted = Modify(pii_redactor)(dataset)\n```\n\n### Stage 4: Classifier filtering\n\n```python\nfrom nemo_curator.classifiers import QualityClassifier\n\n# Quality classification\nquality_clf = QualityClassifier(\n    model_path=\"nvidia\u002Fquality-classifier-deberta\",\n    batch_size=256,\n    device=\"cuda\"\n)\n\n# Filter low-quality documents\nhigh_quality = dataset.filter(lambda doc: quality_clf(doc[\"text\"]) > 0.5)\n```\n\n## GPU acceleration\n\n### GPU vs CPU performance\n\n| Operation | CPU (16 cores) | GPU (A100) | Speedup |\n|-----------|----------------|------------|---------|\n| Fuzzy dedup (8TB) | 120 hours | 7.5 hours | 16× |\n| Exact dedup (1TB) | 8 hours | 0.5 hours | 16× |\n| Quality filtering | 2 hours | 0.2 hours | 10× |\n\n### Multi-GPU scaling\n\n```python\nfrom nemo_curator import get_client\nimport dask_cuda\n\n# Initialize GPU cluster\nclient = get_client(cluster_type=\"gpu\", n_workers=8)\n\n# Process with 8 GPUs\ndeduped = FuzzyDuplicates(...)(dataset)\n```\n\n## Multi-modal curation\n\n### Image curation\n\n```python\nfrom nemo_curator.image import (\n    AestheticFilter,\n    NSFWFilter,\n    CLIPEmbedder\n)\n\n# Aesthetic scoring\naesthetic_filter = AestheticFilter(threshold=5.0)\nfiltered_images = aesthetic_filter(image_dataset)\n\n# NSFW detection\nnsfw_filter = NSFWFilter(threshold=0.9)\nsafe_images = nsfw_filter(filtered_images)\n\n# Generate CLIP embeddings\nclip_embedder = CLIPEmbedder(model=\"openai\u002Fclip-vit-base-patch32\")\nimage_embeddings = clip_embedder(safe_images)\n```\n\n### Video curation\n\n```python\nfrom nemo_curator.video import (\n    SceneDetector,\n    ClipExtractor,\n    InternVideo2Embedder\n)\n\n# Detect scenes\nscene_detector = SceneDetector(threshold=27.0)\nscenes = scene_detector(video_dataset)\n\n# Extract clips\nclip_extractor = ClipExtractor(min_duration=2.0, max_duration=10.0)\nclips = clip_extractor(scenes)\n\n# Generate embeddings\nvideo_embedder = InternVideo2Embedder()\nvideo_embeddings = video_embedder(clips)\n```\n\n### Audio curation\n\n```python\nfrom nemo_curator.audio import (\n    ASRInference,\n    WERFilter,\n    DurationFilter\n)\n\n# ASR transcription\nasr = ASRInference(model=\"nvidia\u002Fstt_en_fastconformer_hybrid_large_pc\")\ntranscribed = asr(audio_dataset)\n\n# Filter by WER (word error rate)\nwer_filter = WERFilter(max_wer=0.3)\nhigh_quality_audio = wer_filter(transcribed)\n\n# Duration filtering\nduration_filter = DurationFilter(min_duration=1.0, max_duration=30.0)\nfiltered_audio = duration_filter(high_quality_audio)\n```\n\n## Common patterns\n\n### Web scrape curation (Common Crawl)\n\n```python\nfrom nemo_curator import ScoreFilter, Modify\nfrom nemo_curator.filters import *\nfrom nemo_curator.modules import *\nfrom nemo_curator.datasets import DocumentDataset\n\n# Load Common Crawl data\ndataset = DocumentDataset.read_parquet(\"common_crawl\u002F*.parquet\")\n\n# Pipeline\npipeline = [\n    # 1. Quality filtering\n    WordCountFilter(min_words=100, max_words=50000),\n    RepeatedLinesFilter(max_repeated_line_fraction=0.2),\n    SymbolToWordRatioFilter(max_symbol_to_word_ratio=0.3),\n    UrlRatioFilter(max_url_ratio=0.3),\n\n    # 2. Language filtering\n    LanguageIdentificationFilter(target_languages=[\"en\"]),\n\n    # 3. Deduplication\n    ExactDuplicates(id_field=\"id\", text_field=\"text\"),\n    FuzzyDuplicates(id_field=\"id\", text_field=\"text\", num_hashes=260),\n\n    # 4. PII redaction\n    PIIRedactor(),\n\n    # 5. NSFW filtering\n    NSFWClassifier(threshold=0.8)\n]\n\n# Execute\nfor stage in pipeline:\n    dataset = stage(dataset)\n\n# Save\ndataset.to_parquet(\"curated_common_crawl\u002F\")\n```\n\n### Distributed processing\n\n```python\nfrom nemo_curator import get_client\nfrom dask_cuda import LocalCUDACluster\n\n# Multi-GPU cluster\ncluster = LocalCUDACluster(n_workers=8)\nclient = get_client(cluster=cluster)\n\n# Process large dataset\ndataset = DocumentDataset.read_parquet(\"s3:\u002F\u002Flarge_dataset\u002F*.parquet\")\ndeduped = FuzzyDuplicates(...)(dataset)\n\n# Cleanup\nclient.close()\ncluster.close()\n```\n\n## Performance benchmarks\n\n### Fuzzy deduplication (8TB RedPajama v2)\n\n- **CPU (256 cores)**: 120 hours\n- **GPU (8× A100)**: 7.5 hours\n- **Speedup**: 16×\n\n### Exact deduplication (1TB)\n\n- **CPU (64 cores)**: 8 hours\n- **GPU (4× A100)**: 0.5 hours\n- **Speedup**: 16×\n\n### Quality filtering (100GB)\n\n- **CPU (32 cores)**: 2 hours\n- **GPU (2× A100)**: 0.2 hours\n- **Speedup**: 10×\n\n## Cost comparison\n\n**CPU-based curation** (AWS c5.18xlarge × 10):\n- Cost: $3.60\u002Fhour × 10 = $36\u002Fhour\n- Time for 8TB: 120 hours\n- **Total**: $4,320\n\n**GPU-based curation** (AWS p4d.24xlarge × 2):\n- Cost: $32.77\u002Fhour × 2 = $65.54\u002Fhour\n- Time for 8TB: 7.5 hours\n- **Total**: $491.55\n\n**Savings**: 89% reduction ($3,828 saved)\n\n## Supported data formats\n\n- **Input**: Parquet, JSONL, CSV\n- **Output**: Parquet (recommended), JSONL\n- **WebDataset**: TAR archives for multi-modal\n\n## Use cases\n\n**Production deployments**:\n- NVIDIA used NeMo Curator to prepare Nemotron-4 training data\n- Open-source datasets curated: RedPajama v2, The Pile\n\n## References\n\n- **[Filtering Guide](references\u002Ffiltering.md)** - 30+ quality filters, heuristics\n- **[Deduplication Guide](references\u002Fdeduplication.md)** - Exact, fuzzy, semantic methods\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNeMo-Curator ⭐ 500+\n- **Docs**: https:\u002F\u002Fdocs.nvidia.com\u002Fnemo-framework\u002Fuser-guide\u002Flatest\u002Fdatacuration\u002F\n- **Version**: 0.4.0+\n- **License**: Apache 2.0\n\n\n\n","# NeMo Curator - GPU-Accelerated Data Curation\n\nNVIDIA's toolkit for preparing high-quality training data for LLMs.\n\n## When to use NeMo Curator\n\n**Use NeMo Curator when:**\n- Preparing LLM training data from web scrapes (Common Crawl)\n- Need fast deduplication (16× faster than CPU)\n- Curating multi-modal datasets (text, images, video, audio)\n- Filtering low-quality or toxic content\n- Scaling data processing across GPU cluster\n\n**Performance**:\n- **16× faster** fuzzy deduplication (8TB RedPajama v2)\n- **40% lower TCO** vs CPU alternatives\n- **Near-linear scaling** across GPU nodes\n\n**Use alternatives instead**:\n- **datatrove**: CPU-based, open-source data processing\n- **dolma**: Allen AI's data toolkit\n- **Ray Data**: General ML data processing (no curation focus)\n\n## Quick start\n\n### Installation\n\n```bash\n# Text curation (CUDA 12)\nuv pip install \"nemo-curator[text_cuda12]\"\n\n# All modalities\nuv pip install \"nemo-curator[all_cuda12]\"\n\n# CPU-only (slower)\nuv pip install \"nemo-curator[cpu]\"\n```\n\n### Basic text curation pipeline\n\n```python\nfrom nemo_curator import ScoreFilter, Modify\nfrom nemo_curator.datasets import DocumentDataset\nimport pandas as pd\n\n# Load data\ndf = pd.DataFrame({\"text\": [\"Good document\", \"Bad doc\", \"Excellent text\"]})\ndataset = DocumentDataset(df)\n\n# Quality filtering\ndef quality_score(doc):\n    return len(doc[\"text\"].split()) > 5  # Filter short docs\n\nfiltered = ScoreFilter(quality_score)(dataset)\n\n# Deduplication\nfrom nemo_curator.modules import ExactDuplicates\ndeduped = ExactDuplicates()(filtered)\n\n# Save\ndeduped.to_parquet(\"curated_data\u002F\")\n```\n\n## Data curation pipeline\n\n### Stage 1: Quality filtering\n\n```python\nfrom nemo_curator.filters import (\n    WordCountFilter,\n    RepeatedLinesFilter,\n    UrlRatioFilter,\n    NonAlphaNumericFilter\n)\n\n# Apply 30+ heuristic filters\nfrom nemo_curator import ScoreFilter\n\n# Word count filter\ndataset = dataset.filter(WordCountFilter(min_words=50, max_words=100000))\n\n# Remove repetitive content\ndataset = dataset.filter(RepeatedLinesFilter(max_repeated_line_fraction=0.3))\n\n# URL ratio filter\ndataset = dataset.filter(UrlRatioFilter(max_url_ratio=0.2))\n```\n\n### Stage 2: Deduplication\n\n**Exact deduplication**:\n```python\nfrom nemo_curator.modules import ExactDuplicates\n\n# Remove exact duplicates\ndeduped = ExactDuplicates(id_field=\"id\", text_field=\"text\")(dataset)\n```\n\n**Fuzzy deduplication** (16× faster on GPU):\n```python\nfrom nemo_curator.modules import FuzzyDuplicates\n\n# MinHash + LSH deduplication\nfuzzy_dedup = FuzzyDuplicates(\n    id_field=\"id\",\n    text_field=\"text\",\n    num_hashes=260,      # MinHash parameters\n    num_buckets=20,\n    hash_method=\"md5\"\n)\n\ndeduped = fuzzy_dedup(dataset)\n```\n\n**Semantic deduplication**:\n```python\nfrom nemo_curator.modules import SemanticDuplicates\n\n# Embedding-based deduplication\nsemantic_dedup = SemanticDuplicates(\n    id_field=\"id\",\n    text_field=\"text\",\n    embedding_model=\"sentence-transformers\u002Fall-MiniLM-L6-v2\",\n    threshold=0.8  # Cosine similarity threshold\n)\n\ndeduped = semantic_dedup(dataset)\n```\n\n### Stage 3: PII redaction\n\n```python\nfrom nemo_curator.modules import Modify\nfrom nemo_curator.modifiers import PIIRedactor\n\n# Redact personally identifiable information\npii_redactor = PIIRedactor(\n    supported_entities=[\"EMAIL_ADDRESS\", \"PHONE_NUMBER\", \"PERSON\", \"LOCATION\"],\n    anonymize_action=\"replace\"  # or \"redact\"\n)\n\nredacted = Modify(pii_redactor)(dataset)\n```\n\n### Stage 4: Classifier filtering\n\n```python\nfrom nemo_curator.classifiers import QualityClassifier\n\n# Quality classification\nquality_clf = QualityClassifier(\n    model_path=\"nvidia\u002Fquality-classifier-deberta\",\n    batch_size=256,\n    device=\"cuda\"\n)\n\n# Filter low-quality documents\nhigh_quality = dataset.filter(lambda doc: quality_clf(doc[\"text\"]) > 0.5)\n```\n\n## GPU acceleration\n\n### GPU vs CPU performance\n\n| Operation | CPU (16 cores) | GPU (A100) | Speedup |\n|-----------|----------------|------------|---------|\n| Fuzzy dedup (8TB) | 120 hours | 7.5 hours | 16× |\n| Exact dedup (1TB) | 8 hours | 0.5 hours | 16× |\n| Quality filtering | 2 hours | 0.2 hours | 10× |\n\n### Multi-GPU scaling\n\n```python\nfrom nemo_curator import get_client\nimport dask_cuda\n\n# Initialize GPU cluster\nclient = get_client(cluster_type=\"gpu\", n_workers=8)\n\n# Process with 8 GPUs\ndeduped = FuzzyDuplicates(...)(dataset)\n```\n\n## Multi-modal curation\n\n### Image curation\n\n```python\nfrom nemo_curator.image import (\n    AestheticFilter,\n    NSFWFilter,\n    CLIPEmbedder\n)\n\n# Aesthetic scoring\naesthetic_filter = AestheticFilter(threshold=5.0)\nfiltered_images = aesthetic_filter(image_dataset)\n\n# NSFW detection\nnsfw_filter = NSFWFilter(threshold=0.9)\nsafe_images = nsfw_filter(filtered_images)\n\n# Generate CLIP embeddings\nclip_embedder = CLIPEmbedder(model=\"openai\u002Fclip-vit-base-patch32\")\nimage_embeddings = clip_embedder(safe_images)\n```\n\n### Video curation\n\n```python\nfrom nemo_curator.video import (\n    SceneDetector,\n    ClipExtractor,\n    InternVideo2Embedder\n)\n\n# Detect scenes\nscene_detector = SceneDetector(threshold=27.0)\nscenes = scene_detector(video_dataset)\n\n# Extract clips\nclip_extractor = ClipExtractor(min_duration=2.0, max_duration=10.0)\nclips = clip_extractor(scenes)\n\n# Generate embeddings\nvideo_embedder = InternVideo2Embedder()\nvideo_embeddings = video_embedder(clips)\n```\n\n### Audio curation\n\n```python\nfrom nemo_curator.audio import (\n    ASRInference,\n    WERFilter,\n    DurationFilter\n)\n\n# ASR transcription\nasr = ASRInference(model=\"nvidia\u002Fstt_en_fastconformer_hybrid_large_pc\")\ntranscribed = asr(audio_dataset)\n\n# Filter by WER (word error rate)\nwer_filter = WERFilter(max_wer=0.3)\nhigh_quality_audio = wer_filter(transcribed)\n\n# Duration filtering\nduration_filter = DurationFilter(min_duration=1.0, max_duration=30.0)\nfiltered_audio = duration_filter(high_quality_audio)\n```\n\n## Common patterns\n\n### Web scrape curation (Common Crawl)\n\n```python\nfrom nemo_curator import ScoreFilter, Modify\nfrom nemo_curator.filters import *\nfrom nemo_curator.modules import *\nfrom nemo_curator.datasets import DocumentDataset\n\n# Load Common Crawl data\ndataset = DocumentDataset.read_parquet(\"common_crawl\u002F*.parquet\")\n\n# Pipeline\npipeline = [\n    # 1. Quality filtering\n    WordCountFilter(min_words=100, max_words=50000),\n    RepeatedLinesFilter(max_repeated_line_fraction=0.2),\n    SymbolToWordRatioFilter(max_symbol_to_word_ratio=0.3),\n    UrlRatioFilter(max_url_ratio=0.3),\n\n    # 2. Language filtering\n    LanguageIdentificationFilter(target_languages=[\"en\"]),\n\n    # 3. Deduplication\n    ExactDuplicates(id_field=\"id\", text_field=\"text\"),\n    FuzzyDuplicates(id_field=\"id\", text_field=\"text\", num_hashes=260),\n\n    # 4. PII redaction\n    PIIRedactor(),\n\n    # 5. NSFW filtering\n    NSFWClassifier(threshold=0.8)\n]\n\n# Execute\nfor stage in pipeline:\n    dataset = stage(dataset)\n\n# Save\ndataset.to_parquet(\"curated_common_crawl\u002F\")\n```\n\n### Distributed processing\n\n```python\nfrom nemo_curator import get_client\nfrom dask_cuda import LocalCUDACluster\n\n# Multi-GPU cluster\ncluster = LocalCUDACluster(n_workers=8)\nclient = get_client(cluster=cluster)\n\n# Process large dataset\ndataset = DocumentDataset.read_parquet(\"s3:\u002F\u002Flarge_dataset\u002F*.parquet\")\ndeduped = FuzzyDuplicates(...)(dataset)\n\n# Cleanup\nclient.close()\ncluster.close()\n```\n\n## Performance benchmarks\n\n### Fuzzy deduplication (8TB RedPajama v2)\n\n- **CPU (256 cores)**: 120 hours\n- **GPU (8× A100)**: 7.5 hours\n- **Speedup**: 16×\n\n### Exact deduplication (1TB)\n\n- **CPU (64 cores)**: 8 hours\n- **GPU (4× A100)**: 0.5 hours\n- **Speedup**: 16×\n\n### Quality filtering (100GB)\n\n- **CPU (32 cores)**: 2 hours\n- **GPU (2× A100)**: 0.2 hours\n- **Speedup**: 10×\n\n## Cost comparison\n\n**CPU-based curation** (AWS c5.18xlarge × 10):\n- Cost: $3.60\u002Fhour × 10 = $36\u002Fhour\n- Time for 8TB: 120 hours\n- **Total**: $4,320\n\n**GPU-based curation** (AWS p4d.24xlarge × 2):\n- Cost: $32.77\u002Fhour × 2 = $65.54\u002Fhour\n- Time for 8TB: 7.5 hours\n- **Total**: $491.55\n\n**Savings**: 89% reduction ($3,828 saved)\n\n## Supported data formats\n\n- **Input**: Parquet, JSONL, CSV\n- **Output**: Parquet (recommended), JSONL\n- **WebDataset**: TAR archives for multi-modal\n\n## Use cases\n\n**Production deployments**:\n- NVIDIA used NeMo Curator to prepare Nemotron-4 training data\n- Open-source datasets curated: RedPajama v2, The Pile\n\n## References\n\n- **[Filtering Guide](references\u002Ffiltering.md)** - 30+ quality filters, heuristics\n- **[Deduplication Guide](references\u002Fdeduplication.md)** - Exact, fuzzy, semantic methods\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNeMo-Curator ⭐ 500+\n- **Docs**: https:\u002F\u002Fdocs.nvidia.com\u002Fnemo-framework\u002Fuser-guide\u002Flatest\u002Fdatacuration\u002F\n- **Version**: 0.4.0+\n- **License**: Apache 2.0\n\n\n\n",{"name":5,"author":24,"license":15,"version":25,"metadata":26,"platforms":40,"description":7,"dependencies":43},"Orchestra Research","1.0.0",{"hermes":27},{"tags":28},[29,30,31,32,33,34,35,36,37,38,39],"Data Processing","NeMo Curator","Data Curation","GPU Acceleration","Deduplication","Quality Filtering","NVIDIA","RAPIDS","PII Redaction","Multimodal","LLM Training Data",[41,42],"linux","macos",[5,44,45,46],"cudf","dask","rapids","main","ff3476b880e3f7eddb4ffe2b62141766402e972b","core-agent\u002Foptional-skills\u002Fmlops\u002Fnemo-curator","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80","https:\u002F\u002Faniruddha-adak.vercel.app","I build agentic harnesses and production-grade agent skills. \r\n\r\nContributor :) @genspark-ai @CopilotKit @openclaw @NousResearch  @synthetic-sciences @PriorLabs","2026-11-14 20:45:37",{"source":8,"skillName":5,"commands":56,"prompt":66,"zipAvailable":67},[57,60,63],{"tool":58,"command":59},"npx","npx skills add aniruddhaadak80\u002Fherclew --skill nemo-curator",{"tool":61,"command":62},"bunx","bunx skills add aniruddhaadak80\u002Fherclew --skill nemo-curator",{"tool":64,"command":65},"pnpm","pnpm dlx skills add aniruddhaadak80\u002Fherclew --skill nemo-curator","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fcore-agent\u002Foptional-skills\u002Fmlops\u002Fnemo-curator\u002FSKILL.md",true,[69,75],{"path":70,"resourceType":71,"resourceTypeDesc":72,"sizeBytes":73,"sourceHtmlUrl":74},"references\u002Fdeduplication.md","REFERENCE","参考文档",2142,"https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fcore-agent\u002Foptional-skills\u002Fmlops\u002Fnemo-curator\u002Freferences\u002Fdeduplication.md",{"path":76,"resourceType":71,"resourceTypeDesc":72,"sizeBytes":77,"sourceHtmlUrl":78},"references\u002Ffiltering.md",2351,"https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fcore-agent\u002Foptional-skills\u002Fmlops\u002Fnemo-curator\u002Freferences\u002Ffiltering.md",[80,89,97,104,111,118],{"skillNo":81,"name":82,"nameZh":6,"description":83,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":84,"licenseFileUrl":85,"sourceHtmlUrl":86,"categories":87,"keywords":88,"publishedAt":19},"01M74FJW4M7QZ5KZHYH1W3SYET","skill-creator","Create, edit, audit, tidy, validate, or restructure AgentSkills and SKILL.md files.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fgateway-node\u002Fskills\u002Fsoftware-development\u002Fskill-creator\u002Flicense.txt","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fgateway-node\u002Fskills\u002Fsoftware-development\u002Fskill-creator\u002FSKILL.md",[],[],{"skillNo":90,"name":91,"nameZh":6,"description":92,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":93,"sourceHtmlUrl":94,"categories":95,"keywords":96,"publishedAt":19},"01M74FDY7EK673NW905BRFR3ZD","prose","OpenProse VM skill pack. Activate on any `prose` command, .prose files, or OpenProse mentions; orchestrates multi-agent workflows.","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fgateway-node\u002Fextensions\u002Fopen-prose\u002Fskills\u002Fprose\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fgateway-node\u002Fextensions\u002Fopen-prose\u002Fskills\u002Fprose\u002FSKILL.md",[],[],{"skillNo":98,"name":99,"nameZh":6,"description":100,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":101,"categories":102,"keywords":103,"publishedAt":19},"01M74FAMG647DMCPE12FZE3PKF","writing-plans","Write implementation plans: bite-sized tasks, paths, code.","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fcore-agent\u002Fskills\u002Fsoftware-development\u002Fwriting-plans\u002FSKILL.md",[],[],{"skillNo":105,"name":106,"nameZh":6,"description":107,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":108,"categories":109,"keywords":110,"publishedAt":19},"01M74FAHVK9XY77HSP589QYQYB","test-driven-development","TDD: enforce RED-GREEN-REFACTOR, tests before code.","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fcore-agent\u002Fskills\u002Fsoftware-development\u002Ftest-driven-development\u002FSKILL.md",[],[],{"skillNo":112,"name":113,"nameZh":6,"description":114,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":115,"categories":116,"keywords":117,"publishedAt":19},"01M74FAECYXZ2DQXQSQ44FHZJZ","systematic-debugging","4-phase root cause debugging: understand bugs before fixing.","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fcore-agent\u002Fskills\u002Fsoftware-development\u002Fsystematic-debugging\u002FSKILL.md",[],[],{"skillNo":119,"name":120,"nameZh":6,"description":121,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":122,"categories":123,"keywords":124,"publishedAt":19},"01M74FABCBB1Z2Y5F67699RPX6","subagent-driven-development","Execute plans via delegate_task subagents (2-stage review).","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fcore-agent\u002Fskills\u002Fsoftware-development\u002Fsubagent-driven-development\u002FSKILL.md",[],[]]