[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M71PHHM7HVQDFPRQZWJ1BMAK":3,"skill-same:01M71PHHM7HVQDFPRQZWJ1BMAK":100},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":27,"publishedAt":48,"linkOnly":49,"markdownContent":50,"markdownBody":51,"frontMatter":52,"refName":70,"commitSha":71,"skillDirectory":72,"forksCount":13,"repositoryHtmlUrl":73,"repositoryHomepageUrl":6,"ownerHtmlUrl":74,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":75,"install":76,"resources":89},"01M71PHHM7HVQDFPRQZWJ1BMAK","nemo-curator",null,"GPU-accelerated data curation for LLM training. Supports text\u002Fimage\u002Fvideo\u002Faudio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.","gjbvsjjvgm-sketch\u002FWORM_V2_SOVEREIGN","gjbvsjjvgm-sketch","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F278511911?v=4","01M71F3JXKZQ70H2KCHHTXKZDA","01M71PENEVFN19DBZFNHJPCHQ7",0,"MIT","https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002FWORM_V2_SOVEREIGN\u002Fblob\u002Fmain\u002Fcore\u002Fhermes-agent\u002Foptional-skills\u002Fmlops\u002Fnemo-curator\u002FSKILL.md",[17,22],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",{"categoryNo":23,"categoryName":24,"taxonomyType":25,"taxonomyTypeDesc":26},"01JBSEEDCAT000000000000033","数据分析师","PROFESSION","适用职业",[28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47],"curator","NeMo","nemo curator","NVIDIA","dask","curation","data","multimodal","features","text","across","image","deduplication","GPU Acceleration","training","heuristics","pii","Audio","processing","Video","2026-11-10 13:02:02",false,"---\nname: nemo-curator\ndescription: GPU-accelerated data curation for LLM training. Supports text\u002Fimage\u002Fvideo\u002Faudio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ndependencies: [nemo-curator, cudf, dask, rapids]\nmetadata:\n  hermes:\n    tags: [Data Processing, NeMo Curator, Data Curation, GPU Acceleration, Deduplication, Quality Filtering, NVIDIA, RAPIDS, PII Redaction, Multimodal, LLM Training Data]\n\n---\n\n# NeMo Curator - GPU-Accelerated Data Curation\n\nNVIDIA's toolkit for preparing high-quality training data for LLMs.\n\n## When to use NeMo Curator\n\n**Use NeMo Curator when:**\n- Preparing LLM training data from web scrapes (Common Crawl)\n- Need fast deduplication (16× faster than CPU)\n- Curating multi-modal datasets (text, images, video, audio)\n- Filtering low-quality or toxic content\n- Scaling data processing across GPU cluster\n\n**Performance**:\n- **16× faster** fuzzy deduplication (8TB RedPajama v2)\n- **40% lower TCO** vs CPU alternatives\n- **Near-linear scaling** across GPU nodes\n\n**Use alternatives instead**:\n- **datatrove**: CPU-based, open-source data processing\n- **dolma**: Allen AI's data toolkit\n- **Ray Data**: General ML data processing (no curation focus)\n\n## Quick start\n\n### Installation\n\n```bash\n# Text curation (CUDA 12)\nuv pip install \"nemo-curator[text_cuda12]\"\n\n# All modalities\nuv pip install \"nemo-curator[all_cuda12]\"\n\n# CPU-only (slower)\nuv pip install \"nemo-curator[cpu]\"\n```\n\n### Basic text curation pipeline\n\n```python\nfrom nemo_curator import ScoreFilter, Modify\nfrom nemo_curator.datasets import DocumentDataset\nimport pandas as pd\n\n# Load data\ndf = pd.DataFrame({\"text\": [\"Good document\", \"Bad doc\", \"Excellent text\"]})\ndataset = DocumentDataset(df)\n\n# Quality filtering\ndef quality_score(doc):\n    return len(doc[\"text\"].split()) > 5  # Filter short docs\n\nfiltered = ScoreFilter(quality_score)(dataset)\n\n# Deduplication\nfrom nemo_curator.modules import ExactDuplicates\ndeduped = ExactDuplicates()(filtered)\n\n# Save\ndeduped.to_parquet(\"curated_data\u002F\")\n```\n\n## Data curation pipeline\n\n### Stage 1: Quality filtering\n\n```python\nfrom nemo_curator.filters import (\n    WordCountFilter,\n    RepeatedLinesFilter,\n    UrlRatioFilter,\n    NonAlphaNumericFilter\n)\n\n# Apply 30+ heuristic filters\nfrom nemo_curator import ScoreFilter\n\n# Word count filter\ndataset = dataset.filter(WordCountFilter(min_words=50, max_words=100000))\n\n# Remove repetitive content\ndataset = dataset.filter(RepeatedLinesFilter(max_repeated_line_fraction=0.3))\n\n# URL ratio filter\ndataset = dataset.filter(UrlRatioFilter(max_url_ratio=0.2))\n```\n\n### Stage 2: Deduplication\n\n**Exact deduplication**:\n```python\nfrom nemo_curator.modules import ExactDuplicates\n\n# Remove exact duplicates\ndeduped = ExactDuplicates(id_field=\"id\", text_field=\"text\")(dataset)\n```\n\n**Fuzzy deduplication** (16× faster on GPU):\n```python\nfrom nemo_curator.modules import FuzzyDuplicates\n\n# MinHash + LSH deduplication\nfuzzy_dedup = FuzzyDuplicates(\n    id_field=\"id\",\n    text_field=\"text\",\n    num_hashes=260,      # MinHash parameters\n    num_buckets=20,\n    hash_method=\"md5\"\n)\n\ndeduped = fuzzy_dedup(dataset)\n```\n\n**Semantic deduplication**:\n```python\nfrom nemo_curator.modules import SemanticDuplicates\n\n# Embedding-based deduplication\nsemantic_dedup = SemanticDuplicates(\n    id_field=\"id\",\n    text_field=\"text\",\n    embedding_model=\"sentence-transformers\u002Fall-MiniLM-L6-v2\",\n    threshold=0.8  # Cosine similarity threshold\n)\n\ndeduped = semantic_dedup(dataset)\n```\n\n### Stage 3: PII redaction\n\n```python\nfrom nemo_curator.modules import Modify\nfrom nemo_curator.modifiers import PIIRedactor\n\n# Redact personally identifiable information\npii_redactor = PIIRedactor(\n    supported_entities=[\"EMAIL_ADDRESS\", \"PHONE_NUMBER\", \"PERSON\", \"LOCATION\"],\n    anonymize_action=\"replace\"  # or \"redact\"\n)\n\nredacted = Modify(pii_redactor)(dataset)\n```\n\n### Stage 4: Classifier filtering\n\n```python\nfrom nemo_curator.classifiers import QualityClassifier\n\n# Quality classification\nquality_clf = QualityClassifier(\n    model_path=\"nvidia\u002Fquality-classifier-deberta\",\n    batch_size=256,\n    device=\"cuda\"\n)\n\n# Filter low-quality documents\nhigh_quality = dataset.filter(lambda doc: quality_clf(doc[\"text\"]) > 0.5)\n```\n\n## GPU acceleration\n\n### GPU vs CPU performance\n\n| Operation | CPU (16 cores) | GPU (A100) | Speedup |\n|-----------|----------------|------------|---------|\n| Fuzzy dedup (8TB) | 120 hours | 7.5 hours | 16× |\n| Exact dedup (1TB) | 8 hours | 0.5 hours | 16× |\n| Quality filtering | 2 hours | 0.2 hours | 10× |\n\n### Multi-GPU scaling\n\n```python\nfrom nemo_curator import get_client\nimport dask_cuda\n\n# Initialize GPU cluster\nclient = get_client(cluster_type=\"gpu\", n_workers=8)\n\n# Process with 8 GPUs\ndeduped = FuzzyDuplicates(...)(dataset)\n```\n\n## Multi-modal curation\n\n### Image curation\n\n```python\nfrom nemo_curator.image import (\n    AestheticFilter,\n    NSFWFilter,\n    CLIPEmbedder\n)\n\n# Aesthetic scoring\naesthetic_filter = AestheticFilter(threshold=5.0)\nfiltered_images = aesthetic_filter(image_dataset)\n\n# NSFW detection\nnsfw_filter = NSFWFilter(threshold=0.9)\nsafe_images = nsfw_filter(filtered_images)\n\n# Generate CLIP embeddings\nclip_embedder = CLIPEmbedder(model=\"openai\u002Fclip-vit-base-patch32\")\nimage_embeddings = clip_embedder(safe_images)\n```\n\n### Video curation\n\n```python\nfrom nemo_curator.video import (\n    SceneDetector,\n    ClipExtractor,\n    InternVideo2Embedder\n)\n\n# Detect scenes\nscene_detector = SceneDetector(threshold=27.0)\nscenes = scene_detector(video_dataset)\n\n# Extract clips\nclip_extractor = ClipExtractor(min_duration=2.0, max_duration=10.0)\nclips = clip_extractor(scenes)\n\n# Generate embeddings\nvideo_embedder = InternVideo2Embedder()\nvideo_embeddings = video_embedder(clips)\n```\n\n### Audio curation\n\n```python\nfrom nemo_curator.audio import (\n    ASRInference,\n    WERFilter,\n    DurationFilter\n)\n\n# ASR transcription\nasr = ASRInference(model=\"nvidia\u002Fstt_en_fastconformer_hybrid_large_pc\")\ntranscribed = asr(audio_dataset)\n\n# Filter by WER (word error rate)\nwer_filter = WERFilter(max_wer=0.3)\nhigh_quality_audio = wer_filter(transcribed)\n\n# Duration filtering\nduration_filter = DurationFilter(min_duration=1.0, max_duration=30.0)\nfiltered_audio = duration_filter(high_quality_audio)\n```\n\n## Common patterns\n\n### Web scrape curation (Common Crawl)\n\n```python\nfrom nemo_curator import ScoreFilter, Modify\nfrom nemo_curator.filters import *\nfrom nemo_curator.modules import *\nfrom nemo_curator.datasets import DocumentDataset\n\n# Load Common Crawl data\ndataset = DocumentDataset.read_parquet(\"common_crawl\u002F*.parquet\")\n\n# Pipeline\npipeline = [\n    # 1. Quality filtering\n    WordCountFilter(min_words=100, max_words=50000),\n    RepeatedLinesFilter(max_repeated_line_fraction=0.2),\n    SymbolToWordRatioFilter(max_symbol_to_word_ratio=0.3),\n    UrlRatioFilter(max_url_ratio=0.3),\n\n    # 2. Language filtering\n    LanguageIdentificationFilter(target_languages=[\"en\"]),\n\n    # 3. Deduplication\n    ExactDuplicates(id_field=\"id\", text_field=\"text\"),\n    FuzzyDuplicates(id_field=\"id\", text_field=\"text\", num_hashes=260),\n\n    # 4. PII redaction\n    PIIRedactor(),\n\n    # 5. NSFW filtering\n    NSFWClassifier(threshold=0.8)\n]\n\n# Execute\nfor stage in pipeline:\n    dataset = stage(dataset)\n\n# Save\ndataset.to_parquet(\"curated_common_crawl\u002F\")\n```\n\n### Distributed processing\n\n```python\nfrom nemo_curator import get_client\nfrom dask_cuda import LocalCUDACluster\n\n# Multi-GPU cluster\ncluster = LocalCUDACluster(n_workers=8)\nclient = get_client(cluster=cluster)\n\n# Process large dataset\ndataset = DocumentDataset.read_parquet(\"s3:\u002F\u002Flarge_dataset\u002F*.parquet\")\ndeduped = FuzzyDuplicates(...)(dataset)\n\n# Cleanup\nclient.close()\ncluster.close()\n```\n\n## Performance benchmarks\n\n### Fuzzy deduplication (8TB RedPajama v2)\n\n- **CPU (256 cores)**: 120 hours\n- **GPU (8× A100)**: 7.5 hours\n- **Speedup**: 16×\n\n### Exact deduplication (1TB)\n\n- **CPU (64 cores)**: 8 hours\n- **GPU (4× A100)**: 0.5 hours\n- **Speedup**: 16×\n\n### Quality filtering (100GB)\n\n- **CPU (32 cores)**: 2 hours\n- **GPU (2× A100)**: 0.2 hours\n- **Speedup**: 10×\n\n## Cost comparison\n\n**CPU-based curation** (AWS c5.18xlarge × 10):\n- Cost: $3.60\u002Fhour × 10 = $36\u002Fhour\n- Time for 8TB: 120 hours\n- **Total**: $4,320\n\n**GPU-based curation** (AWS p4d.24xlarge × 2):\n- Cost: $32.77\u002Fhour × 2 = $65.54\u002Fhour\n- Time for 8TB: 7.5 hours\n- **Total**: $491.55\n\n**Savings**: 89% reduction ($3,828 saved)\n\n## Supported data formats\n\n- **Input**: Parquet, JSONL, CSV\n- **Output**: Parquet (recommended), JSONL\n- **WebDataset**: TAR archives for multi-modal\n\n## Use cases\n\n**Production deployments**:\n- NVIDIA used NeMo Curator to prepare Nemotron-4 training data\n- Open-source datasets curated: RedPajama v2, The Pile\n\n## References\n\n- **[Filtering Guide](references\u002Ffiltering.md)** - 30+ quality filters, heuristics\n- **[Deduplication Guide](references\u002Fdeduplication.md)** - Exact, fuzzy, semantic methods\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNeMo-Curator ⭐ 500+\n- **Docs**: https:\u002F\u002Fdocs.nvidia.com\u002Fnemo-framework\u002Fuser-guide\u002Flatest\u002Fdatacuration\u002F\n- **Version**: 0.4.0+\n- **License**: Apache 2.0\n\n\n\n","# NeMo Curator - GPU-Accelerated Data Curation\n\nNVIDIA's toolkit for preparing high-quality training data for LLMs.\n\n## When to use NeMo Curator\n\n**Use NeMo Curator when:**\n- Preparing LLM training data from web scrapes (Common Crawl)\n- Need fast deduplication (16× faster than CPU)\n- Curating multi-modal datasets (text, images, video, audio)\n- Filtering low-quality or toxic content\n- Scaling data processing across GPU cluster\n\n**Performance**:\n- **16× faster** fuzzy deduplication (8TB RedPajama v2)\n- **40% lower TCO** vs CPU alternatives\n- **Near-linear scaling** across GPU nodes\n\n**Use alternatives instead**:\n- **datatrove**: CPU-based, open-source data processing\n- **dolma**: Allen AI's data toolkit\n- **Ray Data**: General ML data processing (no curation focus)\n\n## Quick start\n\n### Installation\n\n```bash\n# Text curation (CUDA 12)\nuv pip install \"nemo-curator[text_cuda12]\"\n\n# All modalities\nuv pip install \"nemo-curator[all_cuda12]\"\n\n# CPU-only (slower)\nuv pip install \"nemo-curator[cpu]\"\n```\n\n### Basic text curation pipeline\n\n```python\nfrom nemo_curator import ScoreFilter, Modify\nfrom nemo_curator.datasets import DocumentDataset\nimport pandas as pd\n\n# Load data\ndf = pd.DataFrame({\"text\": [\"Good document\", \"Bad doc\", \"Excellent text\"]})\ndataset = DocumentDataset(df)\n\n# Quality filtering\ndef quality_score(doc):\n    return len(doc[\"text\"].split()) > 5  # Filter short docs\n\nfiltered = ScoreFilter(quality_score)(dataset)\n\n# Deduplication\nfrom nemo_curator.modules import ExactDuplicates\ndeduped = ExactDuplicates()(filtered)\n\n# Save\ndeduped.to_parquet(\"curated_data\u002F\")\n```\n\n## Data curation pipeline\n\n### Stage 1: Quality filtering\n\n```python\nfrom nemo_curator.filters import (\n    WordCountFilter,\n    RepeatedLinesFilter,\n    UrlRatioFilter,\n    NonAlphaNumericFilter\n)\n\n# Apply 30+ heuristic filters\nfrom nemo_curator import ScoreFilter\n\n# Word count filter\ndataset = dataset.filter(WordCountFilter(min_words=50, max_words=100000))\n\n# Remove repetitive content\ndataset = dataset.filter(RepeatedLinesFilter(max_repeated_line_fraction=0.3))\n\n# URL ratio filter\ndataset = dataset.filter(UrlRatioFilter(max_url_ratio=0.2))\n```\n\n### Stage 2: Deduplication\n\n**Exact deduplication**:\n```python\nfrom nemo_curator.modules import ExactDuplicates\n\n# Remove exact duplicates\ndeduped = ExactDuplicates(id_field=\"id\", text_field=\"text\")(dataset)\n```\n\n**Fuzzy deduplication** (16× faster on GPU):\n```python\nfrom nemo_curator.modules import FuzzyDuplicates\n\n# MinHash + LSH deduplication\nfuzzy_dedup = FuzzyDuplicates(\n    id_field=\"id\",\n    text_field=\"text\",\n    num_hashes=260,      # MinHash parameters\n    num_buckets=20,\n    hash_method=\"md5\"\n)\n\ndeduped = fuzzy_dedup(dataset)\n```\n\n**Semantic deduplication**:\n```python\nfrom nemo_curator.modules import SemanticDuplicates\n\n# Embedding-based deduplication\nsemantic_dedup = SemanticDuplicates(\n    id_field=\"id\",\n    text_field=\"text\",\n    embedding_model=\"sentence-transformers\u002Fall-MiniLM-L6-v2\",\n    threshold=0.8  # Cosine similarity threshold\n)\n\ndeduped = semantic_dedup(dataset)\n```\n\n### Stage 3: PII redaction\n\n```python\nfrom nemo_curator.modules import Modify\nfrom nemo_curator.modifiers import PIIRedactor\n\n# Redact personally identifiable information\npii_redactor = PIIRedactor(\n    supported_entities=[\"EMAIL_ADDRESS\", \"PHONE_NUMBER\", \"PERSON\", \"LOCATION\"],\n    anonymize_action=\"replace\"  # or \"redact\"\n)\n\nredacted = Modify(pii_redactor)(dataset)\n```\n\n### Stage 4: Classifier filtering\n\n```python\nfrom nemo_curator.classifiers import QualityClassifier\n\n# Quality classification\nquality_clf = QualityClassifier(\n    model_path=\"nvidia\u002Fquality-classifier-deberta\",\n    batch_size=256,\n    device=\"cuda\"\n)\n\n# Filter low-quality documents\nhigh_quality = dataset.filter(lambda doc: quality_clf(doc[\"text\"]) > 0.5)\n```\n\n## GPU acceleration\n\n### GPU vs CPU performance\n\n| Operation | CPU (16 cores) | GPU (A100) | Speedup |\n|-----------|----------------|------------|---------|\n| Fuzzy dedup (8TB) | 120 hours | 7.5 hours | 16× |\n| Exact dedup (1TB) | 8 hours | 0.5 hours | 16× |\n| Quality filtering | 2 hours | 0.2 hours | 10× |\n\n### Multi-GPU scaling\n\n```python\nfrom nemo_curator import get_client\nimport dask_cuda\n\n# Initialize GPU cluster\nclient = get_client(cluster_type=\"gpu\", n_workers=8)\n\n# Process with 8 GPUs\ndeduped = FuzzyDuplicates(...)(dataset)\n```\n\n## Multi-modal curation\n\n### Image curation\n\n```python\nfrom nemo_curator.image import (\n    AestheticFilter,\n    NSFWFilter,\n    CLIPEmbedder\n)\n\n# Aesthetic scoring\naesthetic_filter = AestheticFilter(threshold=5.0)\nfiltered_images = aesthetic_filter(image_dataset)\n\n# NSFW detection\nnsfw_filter = NSFWFilter(threshold=0.9)\nsafe_images = nsfw_filter(filtered_images)\n\n# Generate CLIP embeddings\nclip_embedder = CLIPEmbedder(model=\"openai\u002Fclip-vit-base-patch32\")\nimage_embeddings = clip_embedder(safe_images)\n```\n\n### Video curation\n\n```python\nfrom nemo_curator.video import (\n    SceneDetector,\n    ClipExtractor,\n    InternVideo2Embedder\n)\n\n# Detect scenes\nscene_detector = SceneDetector(threshold=27.0)\nscenes = scene_detector(video_dataset)\n\n# Extract clips\nclip_extractor = ClipExtractor(min_duration=2.0, max_duration=10.0)\nclips = clip_extractor(scenes)\n\n# Generate embeddings\nvideo_embedder = InternVideo2Embedder()\nvideo_embeddings = video_embedder(clips)\n```\n\n### Audio curation\n\n```python\nfrom nemo_curator.audio import (\n    ASRInference,\n    WERFilter,\n    DurationFilter\n)\n\n# ASR transcription\nasr = ASRInference(model=\"nvidia\u002Fstt_en_fastconformer_hybrid_large_pc\")\ntranscribed = asr(audio_dataset)\n\n# Filter by WER (word error rate)\nwer_filter = WERFilter(max_wer=0.3)\nhigh_quality_audio = wer_filter(transcribed)\n\n# Duration filtering\nduration_filter = DurationFilter(min_duration=1.0, max_duration=30.0)\nfiltered_audio = duration_filter(high_quality_audio)\n```\n\n## Common patterns\n\n### Web scrape curation (Common Crawl)\n\n```python\nfrom nemo_curator import ScoreFilter, Modify\nfrom nemo_curator.filters import *\nfrom nemo_curator.modules import *\nfrom nemo_curator.datasets import DocumentDataset\n\n# Load Common Crawl data\ndataset = DocumentDataset.read_parquet(\"common_crawl\u002F*.parquet\")\n\n# Pipeline\npipeline = [\n    # 1. Quality filtering\n    WordCountFilter(min_words=100, max_words=50000),\n    RepeatedLinesFilter(max_repeated_line_fraction=0.2),\n    SymbolToWordRatioFilter(max_symbol_to_word_ratio=0.3),\n    UrlRatioFilter(max_url_ratio=0.3),\n\n    # 2. Language filtering\n    LanguageIdentificationFilter(target_languages=[\"en\"]),\n\n    # 3. Deduplication\n    ExactDuplicates(id_field=\"id\", text_field=\"text\"),\n    FuzzyDuplicates(id_field=\"id\", text_field=\"text\", num_hashes=260),\n\n    # 4. PII redaction\n    PIIRedactor(),\n\n    # 5. NSFW filtering\n    NSFWClassifier(threshold=0.8)\n]\n\n# Execute\nfor stage in pipeline:\n    dataset = stage(dataset)\n\n# Save\ndataset.to_parquet(\"curated_common_crawl\u002F\")\n```\n\n### Distributed processing\n\n```python\nfrom nemo_curator import get_client\nfrom dask_cuda import LocalCUDACluster\n\n# Multi-GPU cluster\ncluster = LocalCUDACluster(n_workers=8)\nclient = get_client(cluster=cluster)\n\n# Process large dataset\ndataset = DocumentDataset.read_parquet(\"s3:\u002F\u002Flarge_dataset\u002F*.parquet\")\ndeduped = FuzzyDuplicates(...)(dataset)\n\n# Cleanup\nclient.close()\ncluster.close()\n```\n\n## Performance benchmarks\n\n### Fuzzy deduplication (8TB RedPajama v2)\n\n- **CPU (256 cores)**: 120 hours\n- **GPU (8× A100)**: 7.5 hours\n- **Speedup**: 16×\n\n### Exact deduplication (1TB)\n\n- **CPU (64 cores)**: 8 hours\n- **GPU (4× A100)**: 0.5 hours\n- **Speedup**: 16×\n\n### Quality filtering (100GB)\n\n- **CPU (32 cores)**: 2 hours\n- **GPU (2× A100)**: 0.2 hours\n- **Speedup**: 10×\n\n## Cost comparison\n\n**CPU-based curation** (AWS c5.18xlarge × 10):\n- Cost: $3.60\u002Fhour × 10 = $36\u002Fhour\n- Time for 8TB: 120 hours\n- **Total**: $4,320\n\n**GPU-based curation** (AWS p4d.24xlarge × 2):\n- Cost: $32.77\u002Fhour × 2 = $65.54\u002Fhour\n- Time for 8TB: 7.5 hours\n- **Total**: $491.55\n\n**Savings**: 89% reduction ($3,828 saved)\n\n## Supported data formats\n\n- **Input**: Parquet, JSONL, CSV\n- **Output**: Parquet (recommended), JSONL\n- **WebDataset**: TAR archives for multi-modal\n\n## Use cases\n\n**Production deployments**:\n- NVIDIA used NeMo Curator to prepare Nemotron-4 training data\n- Open-source datasets curated: RedPajama v2, The Pile\n\n## References\n\n- **[Filtering Guide](references\u002Ffiltering.md)** - 30+ quality filters, heuristics\n- **[Deduplication Guide](references\u002Fdeduplication.md)** - Exact, fuzzy, semantic methods\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNeMo-Curator ⭐ 500+\n- **Docs**: https:\u002F\u002Fdocs.nvidia.com\u002Fnemo-framework\u002Fuser-guide\u002Flatest\u002Fdatacuration\u002F\n- **Version**: 0.4.0+\n- **License**: Apache 2.0\n\n\n\n",{"name":5,"author":53,"license":14,"version":54,"metadata":55,"description":7,"dependencies":67},"Orchestra Research","1.0.0",{"hermes":56},{"tags":57},[58,59,60,41,61,62,31,63,64,65,66],"Data Processing","NeMo Curator","Data Curation","Deduplication","Quality Filtering","RAPIDS","PII Redaction","Multimodal","LLM Training Data",[5,68,32,69],"cudf","rapids","main","6746276ba18cb774853eb61b3b9399658bc9bc5c","core\u002Fhermes-agent\u002Foptional-skills\u002Fmlops\u002Fnemo-curator","https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002FWORM_V2_SOVEREIGN","https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch","2026-04-28 17:01:03",{"source":8,"skillName":5,"commands":77,"prompt":87,"zipAvailable":88},[78,81,84],{"tool":79,"command":80},"npx","npx skills add gjbvsjjvgm-sketch\u002FWORM_V2_SOVEREIGN --skill nemo-curator",{"tool":82,"command":83},"bunx","bunx skills add gjbvsjjvgm-sketch\u002FWORM_V2_SOVEREIGN --skill nemo-curator",{"tool":85,"command":86},"pnpm","pnpm dlx skills add gjbvsjjvgm-sketch\u002FWORM_V2_SOVEREIGN --skill nemo-curator","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002FWORM_V2_SOVEREIGN\u002Fblob\u002Fmain\u002Fcore\u002Fhermes-agent\u002Foptional-skills\u002Fmlops\u002Fnemo-curator\u002FSKILL.md",true,[90,96],{"path":91,"resourceType":92,"resourceTypeDesc":93,"sizeBytes":94,"sourceHtmlUrl":95},"references\u002Fdeduplication.md","REFERENCE","参考文档",2142,"https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002FWORM_V2_SOVEREIGN\u002Fblob\u002Fmain\u002Fcore\u002Fhermes-agent\u002Foptional-skills\u002Fmlops\u002Fnemo-curator\u002Freferences\u002Fdeduplication.md",{"path":97,"resourceType":92,"resourceTypeDesc":93,"sizeBytes":98,"sourceHtmlUrl":99},"references\u002Ffiltering.md",2351,"https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002FWORM_V2_SOVEREIGN\u002Fblob\u002Fmain\u002Fcore\u002Fhermes-agent\u002Foptional-skills\u002Fmlops\u002Fnemo-curator\u002Freferences\u002Ffiltering.md",[101,140,165,182,201,220],{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":119,"publishedAt":48},"01M71QG27G1R2H2ZWEG4S0WTZT","writing-plans","Write implementation plans: bite-sized tasks, paths, code.","https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002FWORM_V2_SOVEREIGN\u002Fblob\u002Fmain\u002Fcore\u002Fskills\u002Fsoftware-development\u002Fwriting-plans\u002FSKILL.md",[107,110,113,116],{"categoryNo":108,"categoryName":109,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000015","视觉设计",{"categoryNo":111,"categoryName":112,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":114,"categoryName":115,"taxonomyType":25,"taxonomyTypeDesc":26},"01JBSEEDCAT000000000000030","开发工程师",{"categoryNo":117,"categoryName":118,"taxonomyType":25,"taxonomyTypeDesc":26},"01JBSEEDCAT000000000000036","内容创作者",[120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139],"writing","writing plans","plans","test-driven","test-driven-development","superpowers","design","workflow","code","tasks","subagent","code-review","review","documentation","write","paths","planning","development","test","implementation",{"skillNo":141,"name":124,"nameZh":6,"description":142,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":143,"categories":144,"keywords":151,"publishedAt":48},"01M71QFZYRTJXMAE551MRAKZ14","TDD: enforce RED-GREEN-REFACTOR, tests before code.","https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002FWORM_V2_SOVEREIGN\u002Fblob\u002Fmain\u002Fcore\u002Fskills\u002Fsoftware-development\u002Ftest-driven-development\u002FSKILL.md",[145,148],{"categoryNo":146,"categoryName":147,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000007","代码审查",{"categoryNo":149,"categoryName":150,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000008","测试与质量",[152,137,138,153,154,155,156,125,120,157,128,130,158,159,160,161,162,122,163,164],"test driven development","driven","green","red-green-refactor","testing","debugging","systematic","refactor","systematic-debugging","enforce","tests","tdd","quality",{"skillNo":166,"name":160,"nameZh":6,"description":167,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":168,"categories":169,"keywords":171,"publishedAt":48},"01M71QFWR5Q8EYEQ6MCS3SM6MC","4-phase root cause debugging: understand bugs before fixing.","https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002FWORM_V2_SOVEREIGN\u002Fblob\u002Fmain\u002Fcore\u002Fskills\u002Fsoftware-development\u002Fsystematic-debugging\u002FSKILL.md",[170],{"categoryNo":149,"categoryName":150,"taxonomyType":20,"taxonomyTypeDesc":21},[172,157,158,123,173,174,124,125,120,130,175,176,137,177,122,178,138,179,180,181],"systematic debugging","problem","problem-solving","phase","bugs","root cause","root-cause","investigation","understand","hermes",{"skillNo":183,"name":184,"nameZh":6,"description":185,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":186,"categories":187,"keywords":193,"publishedAt":48},"01M71QFS9S04MD8JP747W5WM0N","subagent-driven-development","Execute plans via delegate_task subagents (2-stage review).","https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002FWORM_V2_SOVEREIGN\u002Fblob\u002Fmain\u002Fcore\u002Fskills\u002Fsoftware-development\u002Fsubagent-driven-development\u002FSKILL.md",[188,189,192],{"categoryNo":111,"categoryName":112,"taxonomyType":20,"taxonomyTypeDesc":21},{"categoryNo":190,"categoryName":191,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":117,"categoryName":118,"taxonomyType":25,"taxonomyTypeDesc":26},[194,130,137,153,123,124,195,196,125,120,127,128,131,132,197,198,122,199,200,138],"subagent driven development","delegate","delegation","subagents","parallel","stage","task",{"skillNo":202,"name":203,"nameZh":6,"description":204,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":205,"categories":206,"keywords":209,"publishedAt":48},"01M71QFQ1DBCVAZ511MXFE6T2V","requesting-code-review","Pre-commit review: security scan, quality gates, auto-fix.","https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002FWORM_V2_SOVEREIGN\u002Fblob\u002Fmain\u002Fcore\u002Fskills\u002Fsoftware-development\u002Frequesting-code-review\u002FSKILL.md",[207,208],{"categoryNo":146,"categoryName":147,"taxonomyType":20,"taxonomyTypeDesc":21},{"categoryNo":114,"categoryName":115,"taxonomyType":25,"taxonomyTypeDesc":26},[210,211,128,132,123,124,212,213,125,214,120,215,216,130,131,217,218,137,219,122],"code review","requesting code review","fix","auto-fix","auto","security scan","commit","verification","security","scan",{"skillNo":221,"name":222,"nameZh":6,"description":223,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":224,"categories":225,"keywords":227,"publishedAt":48},"01M71QFM7KWM540YR67FRN807Q","python-debugpy","Debug Python: pdb REPL + debugpy remote (DAP).","https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002FWORM_V2_SOVEREIGN\u002Fblob\u002Fmain\u002Fcore\u002Fskills\u002Fsoftware-development\u002Fpython-debugpy\u002FSKILL.md",[226],{"categoryNo":114,"categoryName":115,"taxonomyType":25,"taxonomyTypeDesc":26},[228,229,230,231,232,233,234,235,157,158,236,237,238,160,239,240,241,242,181,243],"debugpy","python debugpy","python","dap","inspect","post-mortem","pdb","remote","debugger","node","commands","debug","post","breakpoints","node-inspect","tui"]