[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M78ZKWMHAEAS1XJAGM9DA63S":3,"skill-same:01M78ZKWMHAEAS1XJAGM9DA63S":80},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":46,"commitSha":47,"skillDirectory":48,"forksCount":49,"repositoryHtmlUrl":50,"repositoryHomepageUrl":51,"ownerHtmlUrl":52,"ownerBlogUrl":53,"ownerDescription":54,"updatedAt":55,"install":56,"resources":69},"01M78ZKWMHAEAS1XJAGM9DA63S","nemo-curator",null,"GPU-accelerated data curation for LLM training. Supports text\u002Fimage\u002Fvideo\u002Faudio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.","ZDStudios\u002FAIOS","ZDStudios","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F121779558?v=4","01M788P61KMHSJ87R41BH4MSQM","01M788P627Y27KK5MQ1KG3TW5H",7,"MIT","https:\u002F\u002Fgithub.com\u002FZDStudios\u002FAIOS\u002Fblob\u002Fmain\u002Fhermes-agent-main\u002Fhermes-agent-main\u002Foptional-skills\u002Fmlops\u002Fnemo-curator\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: nemo-curator\ndescription: GPU-accelerated data curation for LLM training. Supports text\u002Fimage\u002Fvideo\u002Faudio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ndependencies: [nemo-curator, cudf, dask, rapids]\nplatforms: [linux, macos]\nmetadata:\n  hermes:\n    tags: [Data Processing, NeMo Curator, Data Curation, GPU Acceleration, Deduplication, Quality Filtering, NVIDIA, RAPIDS, PII Redaction, Multimodal, LLM Training Data]\n\n---\n\n# NeMo Curator - GPU-Accelerated Data Curation\n\nNVIDIA's toolkit for preparing high-quality training data for LLMs.\n\n## When to use NeMo Curator\n\n**Use NeMo Curator when:**\n- Preparing LLM training data from web scrapes (Common Crawl)\n- Need fast deduplication (16× faster than CPU)\n- Curating multi-modal datasets (text, images, video, audio)\n- Filtering low-quality or toxic content\n- Scaling data processing across GPU cluster\n\n**Performance**:\n- **16× faster** fuzzy deduplication (8TB RedPajama v2)\n- **40% lower TCO** vs CPU alternatives\n- **Near-linear scaling** across GPU nodes\n\n**Use alternatives instead**:\n- **datatrove**: CPU-based, open-source data processing\n- **dolma**: Allen AI's data toolkit\n- **Ray Data**: General ML data processing (no curation focus)\n\n## Quick start\n\n### Installation\n\n```bash\n# Text curation (CUDA 12)\nuv pip install \"nemo-curator[text_cuda12]\"\n\n# All modalities\nuv pip install \"nemo-curator[all_cuda12]\"\n\n# CPU-only (slower)\nuv pip install \"nemo-curator[cpu]\"\n```\n\n### Basic text curation pipeline\n\n```python\nfrom nemo_curator import ScoreFilter, Modify\nfrom nemo_curator.datasets import DocumentDataset\nimport pandas as pd\n\n# Load data\ndf = pd.DataFrame({\"text\": [\"Good document\", \"Bad doc\", \"Excellent text\"]})\ndataset = DocumentDataset(df)\n\n# Quality filtering\ndef quality_score(doc):\n    return len(doc[\"text\"].split()) > 5  # Filter short docs\n\nfiltered = ScoreFilter(quality_score)(dataset)\n\n# Deduplication\nfrom nemo_curator.modules import ExactDuplicates\ndeduped = ExactDuplicates()(filtered)\n\n# Save\ndeduped.to_parquet(\"curated_data\u002F\")\n```\n\n## Data curation pipeline\n\n### Stage 1: Quality filtering\n\n```python\nfrom nemo_curator.filters import (\n    WordCountFilter,\n    RepeatedLinesFilter,\n    UrlRatioFilter,\n    NonAlphaNumericFilter\n)\n\n# Apply 30+ heuristic filters\nfrom nemo_curator import ScoreFilter\n\n# Word count filter\ndataset = dataset.filter(WordCountFilter(min_words=50, max_words=100000))\n\n# Remove repetitive content\ndataset = dataset.filter(RepeatedLinesFilter(max_repeated_line_fraction=0.3))\n\n# URL ratio filter\ndataset = dataset.filter(UrlRatioFilter(max_url_ratio=0.2))\n```\n\n### Stage 2: Deduplication\n\n**Exact deduplication**:\n```python\nfrom nemo_curator.modules import ExactDuplicates\n\n# Remove exact duplicates\ndeduped = ExactDuplicates(id_field=\"id\", text_field=\"text\")(dataset)\n```\n\n**Fuzzy deduplication** (16× faster on GPU):\n```python\nfrom nemo_curator.modules import FuzzyDuplicates\n\n# MinHash + LSH deduplication\nfuzzy_dedup = FuzzyDuplicates(\n    id_field=\"id\",\n    text_field=\"text\",\n    num_hashes=260,      # MinHash parameters\n    num_buckets=20,\n    hash_method=\"md5\"\n)\n\ndeduped = fuzzy_dedup(dataset)\n```\n\n**Semantic deduplication**:\n```python\nfrom nemo_curator.modules import SemanticDuplicates\n\n# Embedding-based deduplication\nsemantic_dedup = SemanticDuplicates(\n    id_field=\"id\",\n    text_field=\"text\",\n    embedding_model=\"sentence-transformers\u002Fall-MiniLM-L6-v2\",\n    threshold=0.8  # Cosine similarity threshold\n)\n\ndeduped = semantic_dedup(dataset)\n```\n\n### Stage 3: PII redaction\n\n```python\nfrom nemo_curator.modules import Modify\nfrom nemo_curator.modifiers import PIIRedactor\n\n# Redact personally identifiable information\npii_redactor = PIIRedactor(\n    supported_entities=[\"EMAIL_ADDRESS\", \"PHONE_NUMBER\", \"PERSON\", \"LOCATION\"],\n    anonymize_action=\"replace\"  # or \"redact\"\n)\n\nredacted = Modify(pii_redactor)(dataset)\n```\n\n### Stage 4: Classifier filtering\n\n```python\nfrom nemo_curator.classifiers import QualityClassifier\n\n# Quality classification\nquality_clf = QualityClassifier(\n    model_path=\"nvidia\u002Fquality-classifier-deberta\",\n    batch_size=256,\n    device=\"cuda\"\n)\n\n# Filter low-quality documents\nhigh_quality = dataset.filter(lambda doc: quality_clf(doc[\"text\"]) > 0.5)\n```\n\n## GPU acceleration\n\n### GPU vs CPU performance\n\n| Operation | CPU (16 cores) | GPU (A100) | Speedup |\n|-----------|----------------|------------|---------|\n| Fuzzy dedup (8TB) | 120 hours | 7.5 hours | 16× |\n| Exact dedup (1TB) | 8 hours | 0.5 hours | 16× |\n| Quality filtering | 2 hours | 0.2 hours | 10× |\n\n### Multi-GPU scaling\n\n```python\nfrom nemo_curator import get_client\nimport dask_cuda\n\n# Initialize GPU cluster\nclient = get_client(cluster_type=\"gpu\", n_workers=8)\n\n# Process with 8 GPUs\ndeduped = FuzzyDuplicates(...)(dataset)\n```\n\n## Multi-modal curation\n\n### Image curation\n\n```python\nfrom nemo_curator.image import (\n    AestheticFilter,\n    NSFWFilter,\n    CLIPEmbedder\n)\n\n# Aesthetic scoring\naesthetic_filter = AestheticFilter(threshold=5.0)\nfiltered_images = aesthetic_filter(image_dataset)\n\n# NSFW detection\nnsfw_filter = NSFWFilter(threshold=0.9)\nsafe_images = nsfw_filter(filtered_images)\n\n# Generate CLIP embeddings\nclip_embedder = CLIPEmbedder(model=\"openai\u002Fclip-vit-base-patch32\")\nimage_embeddings = clip_embedder(safe_images)\n```\n\n### Video curation\n\n```python\nfrom nemo_curator.video import (\n    SceneDetector,\n    ClipExtractor,\n    InternVideo2Embedder\n)\n\n# Detect scenes\nscene_detector = SceneDetector(threshold=27.0)\nscenes = scene_detector(video_dataset)\n\n# Extract clips\nclip_extractor = ClipExtractor(min_duration=2.0, max_duration=10.0)\nclips = clip_extractor(scenes)\n\n# Generate embeddings\nvideo_embedder = InternVideo2Embedder()\nvideo_embeddings = video_embedder(clips)\n```\n\n### Audio curation\n\n```python\nfrom nemo_curator.audio import (\n    ASRInference,\n    WERFilter,\n    DurationFilter\n)\n\n# ASR transcription\nasr = ASRInference(model=\"nvidia\u002Fstt_en_fastconformer_hybrid_large_pc\")\ntranscribed = asr(audio_dataset)\n\n# Filter by WER (word error rate)\nwer_filter = WERFilter(max_wer=0.3)\nhigh_quality_audio = wer_filter(transcribed)\n\n# Duration filtering\nduration_filter = DurationFilter(min_duration=1.0, max_duration=30.0)\nfiltered_audio = duration_filter(high_quality_audio)\n```\n\n## Common patterns\n\n### Web scrape curation (Common Crawl)\n\n```python\nfrom nemo_curator import ScoreFilter, Modify\nfrom nemo_curator.filters import *\nfrom nemo_curator.modules import *\nfrom nemo_curator.datasets import DocumentDataset\n\n# Load Common Crawl data\ndataset = DocumentDataset.read_parquet(\"common_crawl\u002F*.parquet\")\n\n# Pipeline\npipeline = [\n    # 1. Quality filtering\n    WordCountFilter(min_words=100, max_words=50000),\n    RepeatedLinesFilter(max_repeated_line_fraction=0.2),\n    SymbolToWordRatioFilter(max_symbol_to_word_ratio=0.3),\n    UrlRatioFilter(max_url_ratio=0.3),\n\n    # 2. Language filtering\n    LanguageIdentificationFilter(target_languages=[\"en\"]),\n\n    # 3. Deduplication\n    ExactDuplicates(id_field=\"id\", text_field=\"text\"),\n    FuzzyDuplicates(id_field=\"id\", text_field=\"text\", num_hashes=260),\n\n    # 4. PII redaction\n    PIIRedactor(),\n\n    # 5. NSFW filtering\n    NSFWClassifier(threshold=0.8)\n]\n\n# Execute\nfor stage in pipeline:\n    dataset = stage(dataset)\n\n# Save\ndataset.to_parquet(\"curated_common_crawl\u002F\")\n```\n\n### Distributed processing\n\n```python\nfrom nemo_curator import get_client\nfrom dask_cuda import LocalCUDACluster\n\n# Multi-GPU cluster\ncluster = LocalCUDACluster(n_workers=8)\nclient = get_client(cluster=cluster)\n\n# Process large dataset\ndataset = DocumentDataset.read_parquet(\"s3:\u002F\u002Flarge_dataset\u002F*.parquet\")\ndeduped = FuzzyDuplicates(...)(dataset)\n\n# Cleanup\nclient.close()\ncluster.close()\n```\n\n## Performance benchmarks\n\n### Fuzzy deduplication (8TB RedPajama v2)\n\n- **CPU (256 cores)**: 120 hours\n- **GPU (8× A100)**: 7.5 hours\n- **Speedup**: 16×\n\n### Exact deduplication (1TB)\n\n- **CPU (64 cores)**: 8 hours\n- **GPU (4× A100)**: 0.5 hours\n- **Speedup**: 16×\n\n### Quality filtering (100GB)\n\n- **CPU (32 cores)**: 2 hours\n- **GPU (2× A100)**: 0.2 hours\n- **Speedup**: 10×\n\n## Cost comparison\n\n**CPU-based curation** (AWS c5.18xlarge × 10):\n- Cost: $3.60\u002Fhour × 10 = $36\u002Fhour\n- Time for 8TB: 120 hours\n- **Total**: $4,320\n\n**GPU-based curation** (AWS p4d.24xlarge × 2):\n- Cost: $32.77\u002Fhour × 2 = $65.54\u002Fhour\n- Time for 8TB: 7.5 hours\n- **Total**: $491.55\n\n**Savings**: 89% reduction ($3,828 saved)\n\n## Supported data formats\n\n- **Input**: Parquet, JSONL, CSV\n- **Output**: Parquet (recommended), JSONL\n- **WebDataset**: TAR archives for multi-modal\n\n## Use cases\n\n**Production deployments**:\n- NVIDIA used NeMo Curator to prepare Nemotron-4 training data\n- Open-source datasets curated: RedPajama v2, The Pile\n\n## References\n\n- **[Filtering Guide](references\u002Ffiltering.md)** - 30+ quality filters, heuristics\n- **[Deduplication Guide](references\u002Fdeduplication.md)** - Exact, fuzzy, semantic methods\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNeMo-Curator ⭐ 500+\n- **Docs**: https:\u002F\u002Fdocs.nvidia.com\u002Fnemo-framework\u002Fuser-guide\u002Flatest\u002Fdatacuration\u002F\n- **Version**: 0.4.0+\n- **License**: Apache 2.0\n\n\n\n","# NeMo Curator - GPU-Accelerated Data Curation\n\nNVIDIA's toolkit for preparing high-quality training data for LLMs.\n\n## When to use NeMo Curator\n\n**Use NeMo Curator when:**\n- Preparing LLM training data from web scrapes (Common Crawl)\n- Need fast deduplication (16× faster than CPU)\n- Curating multi-modal datasets (text, images, video, audio)\n- Filtering low-quality or toxic content\n- Scaling data processing across GPU cluster\n\n**Performance**:\n- **16× faster** fuzzy deduplication (8TB RedPajama v2)\n- **40% lower TCO** vs CPU alternatives\n- **Near-linear scaling** across GPU nodes\n\n**Use alternatives instead**:\n- **datatrove**: CPU-based, open-source data processing\n- **dolma**: Allen AI's data toolkit\n- **Ray Data**: General ML data processing (no curation focus)\n\n## Quick start\n\n### Installation\n\n```bash\n# Text curation (CUDA 12)\nuv pip install \"nemo-curator[text_cuda12]\"\n\n# All modalities\nuv pip install \"nemo-curator[all_cuda12]\"\n\n# CPU-only (slower)\nuv pip install \"nemo-curator[cpu]\"\n```\n\n### Basic text curation pipeline\n\n```python\nfrom nemo_curator import ScoreFilter, Modify\nfrom nemo_curator.datasets import DocumentDataset\nimport pandas as pd\n\n# Load data\ndf = pd.DataFrame({\"text\": [\"Good document\", \"Bad doc\", \"Excellent text\"]})\ndataset = DocumentDataset(df)\n\n# Quality filtering\ndef quality_score(doc):\n    return len(doc[\"text\"].split()) > 5  # Filter short docs\n\nfiltered = ScoreFilter(quality_score)(dataset)\n\n# Deduplication\nfrom nemo_curator.modules import ExactDuplicates\ndeduped = ExactDuplicates()(filtered)\n\n# Save\ndeduped.to_parquet(\"curated_data\u002F\")\n```\n\n## Data curation pipeline\n\n### Stage 1: Quality filtering\n\n```python\nfrom nemo_curator.filters import (\n    WordCountFilter,\n    RepeatedLinesFilter,\n    UrlRatioFilter,\n    NonAlphaNumericFilter\n)\n\n# Apply 30+ heuristic filters\nfrom nemo_curator import ScoreFilter\n\n# Word count filter\ndataset = dataset.filter(WordCountFilter(min_words=50, max_words=100000))\n\n# Remove repetitive content\ndataset = dataset.filter(RepeatedLinesFilter(max_repeated_line_fraction=0.3))\n\n# URL ratio filter\ndataset = dataset.filter(UrlRatioFilter(max_url_ratio=0.2))\n```\n\n### Stage 2: Deduplication\n\n**Exact deduplication**:\n```python\nfrom nemo_curator.modules import ExactDuplicates\n\n# Remove exact duplicates\ndeduped = ExactDuplicates(id_field=\"id\", text_field=\"text\")(dataset)\n```\n\n**Fuzzy deduplication** (16× faster on GPU):\n```python\nfrom nemo_curator.modules import FuzzyDuplicates\n\n# MinHash + LSH deduplication\nfuzzy_dedup = FuzzyDuplicates(\n    id_field=\"id\",\n    text_field=\"text\",\n    num_hashes=260,      # MinHash parameters\n    num_buckets=20,\n    hash_method=\"md5\"\n)\n\ndeduped = fuzzy_dedup(dataset)\n```\n\n**Semantic deduplication**:\n```python\nfrom nemo_curator.modules import SemanticDuplicates\n\n# Embedding-based deduplication\nsemantic_dedup = SemanticDuplicates(\n    id_field=\"id\",\n    text_field=\"text\",\n    embedding_model=\"sentence-transformers\u002Fall-MiniLM-L6-v2\",\n    threshold=0.8  # Cosine similarity threshold\n)\n\ndeduped = semantic_dedup(dataset)\n```\n\n### Stage 3: PII redaction\n\n```python\nfrom nemo_curator.modules import Modify\nfrom nemo_curator.modifiers import PIIRedactor\n\n# Redact personally identifiable information\npii_redactor = PIIRedactor(\n    supported_entities=[\"EMAIL_ADDRESS\", \"PHONE_NUMBER\", \"PERSON\", \"LOCATION\"],\n    anonymize_action=\"replace\"  # or \"redact\"\n)\n\nredacted = Modify(pii_redactor)(dataset)\n```\n\n### Stage 4: Classifier filtering\n\n```python\nfrom nemo_curator.classifiers import QualityClassifier\n\n# Quality classification\nquality_clf = QualityClassifier(\n    model_path=\"nvidia\u002Fquality-classifier-deberta\",\n    batch_size=256,\n    device=\"cuda\"\n)\n\n# Filter low-quality documents\nhigh_quality = dataset.filter(lambda doc: quality_clf(doc[\"text\"]) > 0.5)\n```\n\n## GPU acceleration\n\n### GPU vs CPU performance\n\n| Operation | CPU (16 cores) | GPU (A100) | Speedup |\n|-----------|----------------|------------|---------|\n| Fuzzy dedup (8TB) | 120 hours | 7.5 hours | 16× |\n| Exact dedup (1TB) | 8 hours | 0.5 hours | 16× |\n| Quality filtering | 2 hours | 0.2 hours | 10× |\n\n### Multi-GPU scaling\n\n```python\nfrom nemo_curator import get_client\nimport dask_cuda\n\n# Initialize GPU cluster\nclient = get_client(cluster_type=\"gpu\", n_workers=8)\n\n# Process with 8 GPUs\ndeduped = FuzzyDuplicates(...)(dataset)\n```\n\n## Multi-modal curation\n\n### Image curation\n\n```python\nfrom nemo_curator.image import (\n    AestheticFilter,\n    NSFWFilter,\n    CLIPEmbedder\n)\n\n# Aesthetic scoring\naesthetic_filter = AestheticFilter(threshold=5.0)\nfiltered_images = aesthetic_filter(image_dataset)\n\n# NSFW detection\nnsfw_filter = NSFWFilter(threshold=0.9)\nsafe_images = nsfw_filter(filtered_images)\n\n# Generate CLIP embeddings\nclip_embedder = CLIPEmbedder(model=\"openai\u002Fclip-vit-base-patch32\")\nimage_embeddings = clip_embedder(safe_images)\n```\n\n### Video curation\n\n```python\nfrom nemo_curator.video import (\n    SceneDetector,\n    ClipExtractor,\n    InternVideo2Embedder\n)\n\n# Detect scenes\nscene_detector = SceneDetector(threshold=27.0)\nscenes = scene_detector(video_dataset)\n\n# Extract clips\nclip_extractor = ClipExtractor(min_duration=2.0, max_duration=10.0)\nclips = clip_extractor(scenes)\n\n# Generate embeddings\nvideo_embedder = InternVideo2Embedder()\nvideo_embeddings = video_embedder(clips)\n```\n\n### Audio curation\n\n```python\nfrom nemo_curator.audio import (\n    ASRInference,\n    WERFilter,\n    DurationFilter\n)\n\n# ASR transcription\nasr = ASRInference(model=\"nvidia\u002Fstt_en_fastconformer_hybrid_large_pc\")\ntranscribed = asr(audio_dataset)\n\n# Filter by WER (word error rate)\nwer_filter = WERFilter(max_wer=0.3)\nhigh_quality_audio = wer_filter(transcribed)\n\n# Duration filtering\nduration_filter = DurationFilter(min_duration=1.0, max_duration=30.0)\nfiltered_audio = duration_filter(high_quality_audio)\n```\n\n## Common patterns\n\n### Web scrape curation (Common Crawl)\n\n```python\nfrom nemo_curator import ScoreFilter, Modify\nfrom nemo_curator.filters import *\nfrom nemo_curator.modules import *\nfrom nemo_curator.datasets import DocumentDataset\n\n# Load Common Crawl data\ndataset = DocumentDataset.read_parquet(\"common_crawl\u002F*.parquet\")\n\n# Pipeline\npipeline = [\n    # 1. Quality filtering\n    WordCountFilter(min_words=100, max_words=50000),\n    RepeatedLinesFilter(max_repeated_line_fraction=0.2),\n    SymbolToWordRatioFilter(max_symbol_to_word_ratio=0.3),\n    UrlRatioFilter(max_url_ratio=0.3),\n\n    # 2. Language filtering\n    LanguageIdentificationFilter(target_languages=[\"en\"]),\n\n    # 3. Deduplication\n    ExactDuplicates(id_field=\"id\", text_field=\"text\"),\n    FuzzyDuplicates(id_field=\"id\", text_field=\"text\", num_hashes=260),\n\n    # 4. PII redaction\n    PIIRedactor(),\n\n    # 5. NSFW filtering\n    NSFWClassifier(threshold=0.8)\n]\n\n# Execute\nfor stage in pipeline:\n    dataset = stage(dataset)\n\n# Save\ndataset.to_parquet(\"curated_common_crawl\u002F\")\n```\n\n### Distributed processing\n\n```python\nfrom nemo_curator import get_client\nfrom dask_cuda import LocalCUDACluster\n\n# Multi-GPU cluster\ncluster = LocalCUDACluster(n_workers=8)\nclient = get_client(cluster=cluster)\n\n# Process large dataset\ndataset = DocumentDataset.read_parquet(\"s3:\u002F\u002Flarge_dataset\u002F*.parquet\")\ndeduped = FuzzyDuplicates(...)(dataset)\n\n# Cleanup\nclient.close()\ncluster.close()\n```\n\n## Performance benchmarks\n\n### Fuzzy deduplication (8TB RedPajama v2)\n\n- **CPU (256 cores)**: 120 hours\n- **GPU (8× A100)**: 7.5 hours\n- **Speedup**: 16×\n\n### Exact deduplication (1TB)\n\n- **CPU (64 cores)**: 8 hours\n- **GPU (4× A100)**: 0.5 hours\n- **Speedup**: 16×\n\n### Quality filtering (100GB)\n\n- **CPU (32 cores)**: 2 hours\n- **GPU (2× A100)**: 0.2 hours\n- **Speedup**: 10×\n\n## Cost comparison\n\n**CPU-based curation** (AWS c5.18xlarge × 10):\n- Cost: $3.60\u002Fhour × 10 = $36\u002Fhour\n- Time for 8TB: 120 hours\n- **Total**: $4,320\n\n**GPU-based curation** (AWS p4d.24xlarge × 2):\n- Cost: $32.77\u002Fhour × 2 = $65.54\u002Fhour\n- Time for 8TB: 7.5 hours\n- **Total**: $491.55\n\n**Savings**: 89% reduction ($3,828 saved)\n\n## Supported data formats\n\n- **Input**: Parquet, JSONL, CSV\n- **Output**: Parquet (recommended), JSONL\n- **WebDataset**: TAR archives for multi-modal\n\n## Use cases\n\n**Production deployments**:\n- NVIDIA used NeMo Curator to prepare Nemotron-4 training data\n- Open-source datasets curated: RedPajama v2, The Pile\n\n## References\n\n- **[Filtering Guide](references\u002Ffiltering.md)** - 30+ quality filters, heuristics\n- **[Deduplication Guide](references\u002Fdeduplication.md)** - Exact, fuzzy, semantic methods\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNeMo-Curator ⭐ 500+\n- **Docs**: https:\u002F\u002Fdocs.nvidia.com\u002Fnemo-framework\u002Fuser-guide\u002Flatest\u002Fdatacuration\u002F\n- **Version**: 0.4.0+\n- **License**: Apache 2.0\n\n\n\n",{"name":5,"author":23,"license":14,"version":24,"metadata":25,"platforms":39,"description":7,"dependencies":42},"Orchestra Research","1.0.0",{"hermes":26},{"tags":27},[28,29,30,31,32,33,34,35,36,37,38],"Data Processing","NeMo Curator","Data Curation","GPU Acceleration","Deduplication","Quality Filtering","NVIDIA","RAPIDS","PII Redaction","Multimodal","LLM Training Data",[40,41],"linux","macos",[5,43,44,45],"cudf","dask","rapids","main","4faa79f92720631ffd00755d9a1018f8c68d3619","hermes-agent-main\u002Fhermes-agent-main\u002Foptional-skills\u002Fmlops\u002Fnemo-curator",1,"https:\u002F\u002Fgithub.com\u002FZDStudios\u002FAIOS","https:\u002F\u002Fzdstudios.github.io\u002FAIOS\u002F","https:\u002F\u002Fgithub.com\u002FZDStudios","zdstudios.network","Hacker, Coder and Machine Learning Engineer \r\n","2026-10-06 21:45:38",{"source":8,"skillName":5,"commands":57,"prompt":67,"zipAvailable":68},[58,61,64],{"tool":59,"command":60},"npx","npx skills add ZDStudios\u002FAIOS --skill nemo-curator",{"tool":62,"command":63},"bunx","bunx skills add ZDStudios\u002FAIOS --skill nemo-curator",{"tool":65,"command":66},"pnpm","pnpm dlx skills add ZDStudios\u002FAIOS --skill nemo-curator","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FZDStudios\u002FAIOS\u002Fblob\u002Fmain\u002Fhermes-agent-main\u002Fhermes-agent-main\u002Foptional-skills\u002Fmlops\u002Fnemo-curator\u002FSKILL.md",true,[70,76],{"path":71,"resourceType":72,"resourceTypeDesc":73,"sizeBytes":74,"sourceHtmlUrl":75},"references\u002Fdeduplication.md","REFERENCE","参考文档",2142,"https:\u002F\u002Fgithub.com\u002FZDStudios\u002FAIOS\u002Fblob\u002Fmain\u002Fhermes-agent-main\u002Fhermes-agent-main\u002Foptional-skills\u002Fmlops\u002Fnemo-curator\u002Freferences\u002Fdeduplication.md",{"path":77,"resourceType":72,"resourceTypeDesc":73,"sizeBytes":78,"sourceHtmlUrl":79},"references\u002Ffiltering.md",2351,"https:\u002F\u002Fgithub.com\u002FZDStudios\u002FAIOS\u002Fblob\u002Fmain\u002Fhermes-agent-main\u002Fhermes-agent-main\u002Foptional-skills\u002Fmlops\u002Fnemo-curator\u002Freferences\u002Ffiltering.md",[81,89,96,102,109,116],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":85,"sourceHtmlUrl":86,"categories":87,"keywords":88,"publishedAt":18},"01M7910QACPXVF8QY3ZMK6MD2M","ruflo","Ruflo is a multi-agent orchestration platform for AI coding agents (Claude Code, Cursor, Codex, Copilot, Gemini, Amp, +12 more). Use this skill when the user wants to (1) install\u002Finit ruflo in a project, (2) run multi-agent swarms with hierarchical coordination, (3) use ruflo's 314+ MCP tools for memory, routing, hooks, sub-agents, or workflows, (4) check ruflo status\u002Fversion\u002Fdoctor health, or (5) discover which of ruflo's 30+ plugins fits their task.","https:\u002F\u002Fgithub.com\u002FZDStudios\u002FAIOS\u002Fblob\u002Fmain\u002Fruflo-main\u002FLICENSE","https:\u002F\u002Fgithub.com\u002FZDStudios\u002FAIOS\u002Fblob\u002Fmain\u002Fruflo-main\u002FSKILL.md",[],[],{"skillNo":90,"name":91,"nameZh":6,"description":92,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":93,"categories":94,"keywords":95,"publishedAt":18},"01M790G7AX480HX5959531CWBK","ponytail","Forces the laziest solution that actually works, simplest, shortest, most minimal. Channels a senior dev who has seen everything: question whether the task needs to exist at all (YAGNI), reach for the standard library before custom code, native platform features before dependencies, one line before fifty. Supports intensity levels: lite, full (default), ultra. Use on ANY coding task: writing, adding, refactoring, fixing, reviewing, or designing code, and choosing libraries or dependencies. Also use whenever the user says \"ponytail\", \"be lazy\", \"lazy mode\", \"simplest solution\", \"minimal solution\", \"yagni\", \"do less\", or \"shortest path\", or complains about over-engineering, bloat, boilerplate, or unnecessary dependencies. Do NOT use for non-coding requests (general knowledge, prose, translation, summaries, recipes).","https:\u002F\u002Fgithub.com\u002FZDStudios\u002FAIOS\u002Fblob\u002Fmain\u002Fponytail-main\u002Fskills\u002Fponytail\u002FSKILL.md",[],[],{"skillNo":97,"name":91,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":18},"01M790FD2BC49T1M774T6NND33","Lazy senior dev mode for any coding task (write, refactor, fix, review): YAGNI, stdlib first, no unrequested abstractions. Not for non-coding requests.","https:\u002F\u002Fgithub.com\u002FZDStudios\u002FAIOS\u002Fblob\u002Fmain\u002Fponytail-main\u002F.openclaw\u002Fskills\u002Fponytail\u002FSKILL.md",[],[],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":18},"01M790FANKMMJJQK23GQEESEXP","ponytail-review","Review a diff for over-engineering. Finds what to delete: reinvented stdlib, needless deps, speculative abstractions. One line per finding.","https:\u002F\u002Fgithub.com\u002FZDStudios\u002FAIOS\u002Fblob\u002Fmain\u002Fponytail-main\u002F.openclaw\u002Fskills\u002Fponytail-review\u002FSKILL.md",[],[],{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":18},"01M790F7Q3SJZCR6BYQ97TVFEJ","ponytail-help","Quick reference for ponytail's modes, skills, and commands. One-shot display.","https:\u002F\u002Fgithub.com\u002FZDStudios\u002FAIOS\u002Fblob\u002Fmain\u002Fponytail-main\u002F.openclaw\u002Fskills\u002Fponytail-help\u002FSKILL.md",[],[],{"skillNo":117,"name":118,"nameZh":6,"description":119,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":122,"publishedAt":18},"01M790F3TX7VZG0J4VVXE9BSNT","ponytail-gain","Show ponytail measured impact as a scoreboard: less code, less cost, more speed, from the benchmark medians. One-shot display.","https:\u002F\u002Fgithub.com\u002FZDStudios\u002FAIOS\u002Fblob\u002Fmain\u002Fponytail-main\u002F.openclaw\u002Fskills\u002Fponytail-gain\u002FSKILL.md",[],[]]