[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M75JDMSJ3E8VMSJQR8M3BSH0":3,"skill-same:01M75JDMSJ3E8VMSJQR8M3BSH0":46},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":6,"ownerHtmlUrl":29,"ownerBlogUrl":30,"ownerDescription":6,"updatedAt":31,"install":32,"resources":45},"01M75JDMSJ3E8VMSJQR8M3BSH0","fastp-fastq-preprocessing",null,"All-in-one FASTQ QC and adapter trimming. Auto-detects Illumina adapters, filters low-quality reads, corrects paired-end overlaps, emits HTML+JSON QC in one pass. 3-10x faster than Trim Galore\u002FTrimmomatic. First step before STAR, BWA-MEM2, or Salmon.","stanfish06\u002Fskillquarium","stanfish06","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F93099613?v=4","01M75F1CK4ZW109ZK3T26VN52F","Zhiyuan","01M75F1CM53VH6F79DZ8GXKKES",12,"MIT","https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium\u002Fblob\u002Fmaster\u002Fskills\u002Ffastp-fastq-preprocessing\u002FSKILL.md",[],[],"2026-11-12 13:55:43",false,"---\nname: \"fastp-fastq-preprocessing\"\ndescription: \"All-in-one FASTQ QC and adapter trimming. Auto-detects Illumina adapters, filters low-quality reads, corrects paired-end overlaps, emits HTML+JSON QC in one pass. 3-10x faster than Trim Galore\u002FTrimmomatic. First step before STAR, BWA-MEM2, or Salmon.\"\nlicense: \"MIT\"\n---\n\n# fastp — Fast FASTQ Quality Control and Adapter Trimming\n\n## Overview\n\nfastp performs adapter trimming, quality filtering, and QC reporting for Illumina FASTQ files in a single multi-threaded pass. It automatically detects adapter sequences from paired-end read overlaps — eliminating the need to specify adapters manually. fastp corrects mismatches in paired-end overlap regions, filters reads by quality score and length, removes polyX tails (polyA for RNA-seq), and generates interactive HTML and machine-readable JSON QC reports. Being 3–10× faster than Trim Galore and Trimmomatic while providing comparable or better results, fastp has become the standard preprocessing step before alignment in WGS, RNA-seq, and ChIP-seq pipelines.\n\n## When to Use\n\n- Trimming Illumina adapters and low-quality bases before alignment in any NGS pipeline (RNA-seq, WGS, WES, ChIP-seq, ATAC-seq)\n- Generating per-sample QC reports (HTML + JSON) as the first step of a pipeline, before MultiQC aggregation\n- Processing paired-end reads where adapter auto-detection from overlap is preferred over manual adapter specification\n- Removing polyA tails from RNA-seq reads from 3′ end-enriched protocols (Smart-seq, QuantSeq)\n- Splitting a FASTQ file by UMI or by index for demultiplexing workflows\n- Use **Trim Galore** as an alternative when TrimGalore's detailed per-base quality report from FastQC is required alongside trimming\n- Use **Trimmomatic** as an alternative for fine-grained control of sliding-window trimming steps\n\n## Prerequisites\n\n- **Software**: fastp (conda or pre-compiled binary)\n- **Input**: raw Illumina FASTQ files (single-end or paired-end, .fastq or .fastq.gz)\n\n> **Check before installing**: The tool may already be available in the current environment (e.g., inside a `pixi` \u002F `conda` env). Run `command -v fastp` first and skip the install commands below if it returns a path. When running inside a pixi project, invoke the tool via `pixi run fastp` rather than bare `fastp`.\n\n```bash\n# Install with conda (recommended — always resolves to a current release)\nconda install -c bioconda fastp\n\n# Or download a pre-compiled binary (Linux) from the current release —\n# asset naming changes between releases, so check\n# https:\u002F\u002Fgithub.com\u002FOpenGene\u002Ffastp\u002Freleases\u002Flatest for the exact filename\n# rather than hardcoding an old version's asset path.\n\n# Verify\nfastp --version\n# current release as of this writing: fastp 1.3.6\n```\n\n## Quick Start\n\n```bash\n# Paired-end adapter trimming with QC report\nfastp \\\n    -i sample_R1.fastq.gz \\\n    -I sample_R2.fastq.gz \\\n    -o sample_R1.trimmed.fastq.gz \\\n    -O sample_R2.trimmed.fastq.gz \\\n    -h sample_qc.html \\\n    -j sample_qc.json \\\n    --thread 8\n\necho \"Trimmed reads in: sample_R1.trimmed.fastq.gz\"\n```\n\n## Workflow\n\n### Step 1: Single-End Adapter Trimming\n\nRun fastp on single-end FASTQ with automatic adapter detection.\n\n```bash\n# Single-end with auto adapter detection\nfastp \\\n    -i sample.fastq.gz \\\n    -o sample.trimmed.fastq.gz \\\n    -h sample_qc.html \\\n    -j sample_qc.json \\\n    --thread 8 \\\n    --qualified_quality_phred 20 \\\n    --length_required 36\n\necho \"Input reads:   $(zcat sample.fastq.gz | wc -l | awk '{print $1\u002F4}')\"\necho \"Output reads:  $(zcat sample.trimmed.fastq.gz | wc -l | awk '{print $1\u002F4}')\"\n```\n\n### Step 2: Paired-End Adapter Trimming\n\nProcess paired-end FASTQ files with overlap-based adapter detection and correction.\n\n```bash\n# Paired-end with overlap-based adapter auto-detection\nfastp \\\n    -i sample_R1.fastq.gz \\\n    -I sample_R2.fastq.gz \\\n    -o sample_R1.trimmed.fastq.gz \\\n    -O sample_R2.trimmed.fastq.gz \\\n    -h sample_qc.html \\\n    -j sample_qc.json \\\n    --thread 8 \\\n    --correction \\\n    --detect_adapter_for_pe \\\n    --qualified_quality_phred 20 \\\n    --length_required 36\n\n# Specify adapters explicitly (if auto-detection fails)\n# fastp -i R1.fq.gz -I R2.fq.gz \\\n#   --adapter_sequence AGATCGGAAGAGCACACGTCTGAACTCCAGTCA \\\n#   --adapter_sequence_r2 AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGT \\\n#   -o R1.out.fq.gz -O R2.out.fq.gz\n```\n\n### Step 3: Quality Filtering and Read Length Trimming\n\nConfigure quality and length thresholds for stricter or more lenient filtering.\n\n```bash\n# Strict quality filtering (e.g., for variant calling)\nfastp \\\n    -i sample_R1.fastq.gz \\\n    -I sample_R2.fastq.gz \\\n    -o sample_R1.filtered.fastq.gz \\\n    -O sample_R2.filtered.fastq.gz \\\n    -h sample_qc.html \\\n    -j sample_qc.json \\\n    --thread 8 \\\n    --qualified_quality_phred 25 \\\n    --unqualified_percent_limit 20 \\\n    --length_required 50 \\\n    --max_len1 150 \\\n    --max_len2 150 \\\n    --low_complexity_filter \\\n    --complexity_threshold 30\n\necho \"Filtering complete. Check sample_qc.html for pass\u002Ffail rates.\"\n```\n\n### Step 4: RNA-seq polyA Tail Removal\n\nRemove polyA tails from 3′-enriched RNA-seq protocols before alignment.\n\n```bash\n# Remove polyA tails (QuantSeq 3′ mRNA-seq)\nfastp \\\n    -i quantseq_R1.fastq.gz \\\n    -o quantseq_R1.trimmed.fastq.gz \\\n    -h quantseq_qc.html \\\n    -j quantseq_qc.json \\\n    --thread 8 \\\n    --trim_poly_x \\\n    --poly_x_min_len 10 \\\n    --qualified_quality_phred 20 \\\n    --length_required 25\n\n# For Smart-seq2 paired-end with polyA\nfastp \\\n    -i smartseq_R1.fastq.gz \\\n    -I smartseq_R2.fastq.gz \\\n    -o smartseq_R1.trimmed.fastq.gz \\\n    -O smartseq_R2.trimmed.fastq.gz \\\n    --trim_poly_x --poly_x_min_len 10 \\\n    --thread 8 \\\n    -h smartseq_qc.html -j smartseq_qc.json\n```\n\n### Step 5: Parse QC Report JSON for Pipeline Monitoring\n\nExtract key QC metrics from fastp's JSON output for automated quality gates.\n\n```python\nimport json\nfrom pathlib import Path\n\ndef parse_fastp_json(json_path: str) -> dict:\n    with open(json_path) as f:\n        data = json.load(f)\n    \n    before = data[\"summary\"][\"before_filtering\"]\n    after = data[\"summary\"][\"after_filtering\"]\n    \n    return {\n        \"total_reads_in\":  before[\"total_reads\"],\n        \"total_reads_out\": after[\"total_reads\"],\n        \"pct_passed\":      after[\"total_reads\"] \u002F before[\"total_reads\"] * 100,\n        \"q30_rate_before\": before[\"q30_rate\"] * 100,\n        \"q30_rate_after\":  after[\"q30_rate\"] * 100,\n        \"mean_len_before\": before[\"read1_mean_length\"],\n        \"mean_len_after\":  after[\"read1_mean_length\"],\n        \"adapter_trimmed\": data[\"filtering_result\"][\"adapter_trimmed\"],\n    }\n\nmetrics = parse_fastp_json(\"sample_qc.json\")\nfor key, val in metrics.items():\n    print(f\"{key:25s}: {val:.1f}\" if isinstance(val, float) else f\"{key:25s}: {val:,}\")\n\n# Quality gate: fail if \u003C 70% reads pass filter\nif metrics[\"pct_passed\"] \u003C 70:\n    print(\"WARNING: Low pass rate — check raw data quality\")\n```\n\n### Step 6: Batch Preprocessing Pipeline\n\nProcess multiple samples sequentially with per-sample QC summaries.\n\n```bash\n#!\u002Fbin\u002Fbash\n# Batch paired-end preprocessing for multiple samples\nSAMPLES=(ctrl_1 ctrl_2 treat_1 treat_2)\nDATA=\"data\"\nOUT=\"trimmed\"\nQC=\"qc\u002Ffastp\"\nTHREADS=8\n\nmkdir -p \"$OUT\" \"$QC\"\n\nfor sample in \"${SAMPLES[@]}\"; do\n    echo \"=== Processing $sample ===\"\n    fastp \\\n        -i \"$DATA\u002F${sample}_R1.fastq.gz\" \\\n        -I \"$DATA\u002F${sample}_R2.fastq.gz\" \\\n        -o \"$OUT\u002F${sample}_R1.fastq.gz\" \\\n        -O \"$OUT\u002F${sample}_R2.fastq.gz\" \\\n        -h \"$QC\u002F${sample}.html\" \\\n        -j \"$QC\u002F${sample}.json\" \\\n        --thread $THREADS \\\n        --correction \\\n        --detect_adapter_for_pe \\\n        --qualified_quality_phred 20 \\\n        --length_required 36 \\\n        2>&1 | grep -E \"Read[12]|Filtering|Adapter|passed\"\ndone\n\n# Aggregate QC metrics\npython3 - \u003C\u003C 'EOF'\nimport json, pandas as pd\nfrom pathlib import Path\n\nrows = []\nfor jf in sorted(Path(\"qc\u002Ffastp\").glob(\"*.json\")):\n    with open(jf) as f: data = json.load(f)\n    after = data[\"summary\"][\"after_filtering\"]\n    before = data[\"summary\"][\"before_filtering\"]\n    rows.append({\n        \"sample\": jf.stem,\n        \"reads_in\": before[\"total_reads\"],\n        \"reads_out\": after[\"total_reads\"],\n        \"pct_passed\": round(after[\"total_reads\"]\u002Fbefore[\"total_reads\"]*100, 1),\n        \"q30_after\": round(after[\"q30_rate\"]*100, 1),\n    })\ndf = pd.DataFrame(rows)\nprint(df.to_string(index=False))\ndf.to_csv(\"fastp_summary.tsv\", sep=\"\\t\", index=False)\nEOF\n\n# Run MultiQC to aggregate all fastp JSON reports\nmultiqc qc\u002Ffastp\u002F -o qc\u002F -n fastp_multiqc_report\n```\n\n## Key Parameters\n\n| Parameter | Default | Range\u002FOptions | Effect |\n|-----------|---------|---------------|--------|\n| `-i \u002F -I` | required | file path | Input FASTQ (R1 and R2 for paired-end) |\n| `-o \u002F -O` | required | file path | Output trimmed FASTQ (R1 and R2) |\n| `-h \u002F -j` | — | file path | HTML and JSON QC report output paths |\n| `--thread` | `3` | 1–16 | CPU threads; 8 is a good balance |\n| `--qualified_quality_phred` | `15` | 0–40 | Minimum base quality (Phred); 20 = 1% error |\n| `--length_required` | `15` | 1–1000 | Minimum read length after trimming; discard shorter reads |\n| `--correction` | off | flag | Correct mismatches in PE overlap region |\n| `--detect_adapter_for_pe` | off | flag | Enable overlap-based adapter auto-detection for PE data |\n| `--adapter_sequence` | auto | string | Explicit R1 adapter; overrides auto-detection |\n| `--trim_poly_x` | off | flag | Trim polyX (polyA\u002FpolyT) tails; use for 3′-enriched RNA-seq |\n| `--low_complexity_filter` | off | flag | Filter reads with low complexity (\u003C 30% complexity by default) |\n| `--split` | off | integer | Split output into N files per direction (for parallelism) |\n\n## Common Recipes\n\n### Recipe 1: Integrate fastp into a Snakemake Pipeline\n\n```python\n# Snakefile — fastp trimming rule\nconfigfile: \"config.yaml\"\nSAMPLES = config[\"samples\"]\n\nrule fastp_pe:\n    input:\n        r1 = \"data\u002F{sample}_R1.fastq.gz\",\n        r2 = \"data\u002F{sample}_R2.fastq.gz\"\n    output:\n        r1 = \"trimmed\u002F{sample}_R1.fastq.gz\",\n        r2 = \"trimmed\u002F{sample}_R2.fastq.gz\",\n        html = \"qc\u002F{sample}_fastp.html\",\n        json = \"qc\u002F{sample}_fastp.json\"\n    threads: 8\n    shell:\n        \"\"\"\n        fastp -i {input.r1} -I {input.r2} \\\n              -o {output.r1} -O {output.r2} \\\n              -h {output.html} -j {output.json} \\\n              --thread {threads} \\\n              --correction --detect_adapter_for_pe \\\n              --qualified_quality_phred 20 \\\n              --length_required 36\n        \"\"\"\n```\n\n### Recipe 2: Aggregate fastp JSON Reports with Python\n\n```python\nimport json\nimport pandas as pd\nfrom pathlib import Path\n\nqc_dir = Path(\"qc\u002Ffastp\")\nrecords = []\n\nfor jf in sorted(qc_dir.glob(\"*.json\")):\n    with open(jf) as f:\n        d = json.load(f)\n    b = d[\"summary\"][\"before_filtering\"]\n    a = d[\"summary\"][\"after_filtering\"]\n    records.append({\n        \"sample\": jf.stem.replace(\"_fastp\", \"\"),\n        \"reads_in_M\": b[\"total_reads\"] \u002F 1e6,\n        \"reads_out_M\": a[\"total_reads\"] \u002F 1e6,\n        \"pct_passed\": a[\"total_reads\"] \u002F b[\"total_reads\"] * 100,\n        \"q30_pct\": a[\"q30_rate\"] * 100,\n        \"mean_len_bp\": a[\"read1_mean_length\"],\n        \"adapter_pct\": d[\"filtering_result\"][\"adapter_trimmed\"] \u002F b[\"total_reads\"] * 100,\n    })\n\ndf = pd.DataFrame(records).round(2)\nprint(df.to_string(index=False))\n\n# Flag low-quality samples\nlow_q = df[df[\"pct_passed\"] \u003C 80]\nif not low_q.empty:\n    print(f\"\\nSamples with \u003C 80% reads passing: {list(low_q['sample'])}\")\n```\n\n## Expected Outputs\n\n| Output | Format | Description |\n|--------|--------|-------------|\n| `*_R1.trimmed.fastq.gz` | FASTQ.gz | Trimmed R1 reads (adapters and low-quality bases removed) |\n| `*_R2.trimmed.fastq.gz` | FASTQ.gz | Trimmed R2 reads (paired-end only) |\n| `*.html` | HTML | Interactive QC report with per-base quality, GC content, adapter plots |\n| `*.json` | JSON | Machine-readable QC metrics for automation and MultiQC parsing |\n| `fastp.log` | Text | stderr summary with pass\u002Ffail read counts and filtering statistics |\n\n## Troubleshooting\n\n| Problem | Cause | Solution |\n|---------|-------|----------|\n| Adapter not detected in SE mode | SE reads require explicit adapter or `--adapter_sequence` | Use `--detect_adapter_for_pe` only for PE; specify adapter for SE: `--adapter_sequence AGATCGGAAGAGC` |\n| Very high adapter content (> 50%) | Short inserts (small RNA, miRNA) or poor library prep | Check library protocol; use `--overlap_len_require 10` to adjust overlap sensitivity |\n| Too many reads filtered (\u003C 60% pass) | Over-strict quality thresholds or low-quality sequencing run | Relax `--qualified_quality_phred` to 15; lower `--length_required` to 25 |\n| JSON output missing fields | Old fastp version | Upgrade: `conda update fastp` or download latest binary from GitHub |\n| MultiQC not parsing fastp JSON | JSON file not in the scanned directory | Run `multiqc qc\u002F` not `multiqc .`; verify JSON files exist with `ls qc\u002F*.json` |\n| Output FASTQ is empty | All reads filtered (wrong input or extreme thresholds) | Verify input FASTQ with `zcat sample.fq.gz \\| head -8`; run without `--low_complexity_filter` first |\n| Slow performance on large files | Low thread count | Increase `--thread` to 8–12; ensure input is on fast storage (SSD) |\n| polyA not removed | `--trim_poly_x` not set | Add `--trim_poly_x --poly_x_min_len 10` for 3′-enriched protocols |\n\n## References\n\n- [fastp GitHub: OpenGene\u002Ffastp](https:\u002F\u002Fgithub.com\u002FOpenGene\u002Ffastp) — source code, changelog, and usage guide\n- Chen S et al. (2018) \"fastp: an ultra-fast all-in-one FASTQ preprocessor\" — *Bioinformatics* 34(17):i884-i890. [DOI:10.1093\u002Fbioinformatics\u002Fbty560](https:\u002F\u002Fdoi.org\u002F10.1093\u002Fbioinformatics\u002Fbty560)\n- [fastp documentation](https:\u002F\u002Fgithub.com\u002FOpenGene\u002Ffastp#readme) — full parameter reference and recipes\n- [MultiQC fastp module](https:\u002F\u002Fmultiqc.info\u002Fmodules\u002Ffastp\u002F) — aggregating fastp JSON reports across samples\n","# fastp — Fast FASTQ Quality Control and Adapter Trimming\n\n## Overview\n\nfastp performs adapter trimming, quality filtering, and QC reporting for Illumina FASTQ files in a single multi-threaded pass. It automatically detects adapter sequences from paired-end read overlaps — eliminating the need to specify adapters manually. fastp corrects mismatches in paired-end overlap regions, filters reads by quality score and length, removes polyX tails (polyA for RNA-seq), and generates interactive HTML and machine-readable JSON QC reports. Being 3–10× faster than Trim Galore and Trimmomatic while providing comparable or better results, fastp has become the standard preprocessing step before alignment in WGS, RNA-seq, and ChIP-seq pipelines.\n\n## When to Use\n\n- Trimming Illumina adapters and low-quality bases before alignment in any NGS pipeline (RNA-seq, WGS, WES, ChIP-seq, ATAC-seq)\n- Generating per-sample QC reports (HTML + JSON) as the first step of a pipeline, before MultiQC aggregation\n- Processing paired-end reads where adapter auto-detection from overlap is preferred over manual adapter specification\n- Removing polyA tails from RNA-seq reads from 3′ end-enriched protocols (Smart-seq, QuantSeq)\n- Splitting a FASTQ file by UMI or by index for demultiplexing workflows\n- Use **Trim Galore** as an alternative when TrimGalore's detailed per-base quality report from FastQC is required alongside trimming\n- Use **Trimmomatic** as an alternative for fine-grained control of sliding-window trimming steps\n\n## Prerequisites\n\n- **Software**: fastp (conda or pre-compiled binary)\n- **Input**: raw Illumina FASTQ files (single-end or paired-end, .fastq or .fastq.gz)\n\n> **Check before installing**: The tool may already be available in the current environment (e.g., inside a `pixi` \u002F `conda` env). Run `command -v fastp` first and skip the install commands below if it returns a path. When running inside a pixi project, invoke the tool via `pixi run fastp` rather than bare `fastp`.\n\n```bash\n# Install with conda (recommended — always resolves to a current release)\nconda install -c bioconda fastp\n\n# Or download a pre-compiled binary (Linux) from the current release —\n# asset naming changes between releases, so check\n# https:\u002F\u002Fgithub.com\u002FOpenGene\u002Ffastp\u002Freleases\u002Flatest for the exact filename\n# rather than hardcoding an old version's asset path.\n\n# Verify\nfastp --version\n# current release as of this writing: fastp 1.3.6\n```\n\n## Quick Start\n\n```bash\n# Paired-end adapter trimming with QC report\nfastp \\\n    -i sample_R1.fastq.gz \\\n    -I sample_R2.fastq.gz \\\n    -o sample_R1.trimmed.fastq.gz \\\n    -O sample_R2.trimmed.fastq.gz \\\n    -h sample_qc.html \\\n    -j sample_qc.json \\\n    --thread 8\n\necho \"Trimmed reads in: sample_R1.trimmed.fastq.gz\"\n```\n\n## Workflow\n\n### Step 1: Single-End Adapter Trimming\n\nRun fastp on single-end FASTQ with automatic adapter detection.\n\n```bash\n# Single-end with auto adapter detection\nfastp \\\n    -i sample.fastq.gz \\\n    -o sample.trimmed.fastq.gz \\\n    -h sample_qc.html \\\n    -j sample_qc.json \\\n    --thread 8 \\\n    --qualified_quality_phred 20 \\\n    --length_required 36\n\necho \"Input reads:   $(zcat sample.fastq.gz | wc -l | awk '{print $1\u002F4}')\"\necho \"Output reads:  $(zcat sample.trimmed.fastq.gz | wc -l | awk '{print $1\u002F4}')\"\n```\n\n### Step 2: Paired-End Adapter Trimming\n\nProcess paired-end FASTQ files with overlap-based adapter detection and correction.\n\n```bash\n# Paired-end with overlap-based adapter auto-detection\nfastp \\\n    -i sample_R1.fastq.gz \\\n    -I sample_R2.fastq.gz \\\n    -o sample_R1.trimmed.fastq.gz \\\n    -O sample_R2.trimmed.fastq.gz \\\n    -h sample_qc.html \\\n    -j sample_qc.json \\\n    --thread 8 \\\n    --correction \\\n    --detect_adapter_for_pe \\\n    --qualified_quality_phred 20 \\\n    --length_required 36\n\n# Specify adapters explicitly (if auto-detection fails)\n# fastp -i R1.fq.gz -I R2.fq.gz \\\n#   --adapter_sequence AGATCGGAAGAGCACACGTCTGAACTCCAGTCA \\\n#   --adapter_sequence_r2 AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGT \\\n#   -o R1.out.fq.gz -O R2.out.fq.gz\n```\n\n### Step 3: Quality Filtering and Read Length Trimming\n\nConfigure quality and length thresholds for stricter or more lenient filtering.\n\n```bash\n# Strict quality filtering (e.g., for variant calling)\nfastp \\\n    -i sample_R1.fastq.gz \\\n    -I sample_R2.fastq.gz \\\n    -o sample_R1.filtered.fastq.gz \\\n    -O sample_R2.filtered.fastq.gz \\\n    -h sample_qc.html \\\n    -j sample_qc.json \\\n    --thread 8 \\\n    --qualified_quality_phred 25 \\\n    --unqualified_percent_limit 20 \\\n    --length_required 50 \\\n    --max_len1 150 \\\n    --max_len2 150 \\\n    --low_complexity_filter \\\n    --complexity_threshold 30\n\necho \"Filtering complete. Check sample_qc.html for pass\u002Ffail rates.\"\n```\n\n### Step 4: RNA-seq polyA Tail Removal\n\nRemove polyA tails from 3′-enriched RNA-seq protocols before alignment.\n\n```bash\n# Remove polyA tails (QuantSeq 3′ mRNA-seq)\nfastp \\\n    -i quantseq_R1.fastq.gz \\\n    -o quantseq_R1.trimmed.fastq.gz \\\n    -h quantseq_qc.html \\\n    -j quantseq_qc.json \\\n    --thread 8 \\\n    --trim_poly_x \\\n    --poly_x_min_len 10 \\\n    --qualified_quality_phred 20 \\\n    --length_required 25\n\n# For Smart-seq2 paired-end with polyA\nfastp \\\n    -i smartseq_R1.fastq.gz \\\n    -I smartseq_R2.fastq.gz \\\n    -o smartseq_R1.trimmed.fastq.gz \\\n    -O smartseq_R2.trimmed.fastq.gz \\\n    --trim_poly_x --poly_x_min_len 10 \\\n    --thread 8 \\\n    -h smartseq_qc.html -j smartseq_qc.json\n```\n\n### Step 5: Parse QC Report JSON for Pipeline Monitoring\n\nExtract key QC metrics from fastp's JSON output for automated quality gates.\n\n```python\nimport json\nfrom pathlib import Path\n\ndef parse_fastp_json(json_path: str) -> dict:\n    with open(json_path) as f:\n        data = json.load(f)\n    \n    before = data[\"summary\"][\"before_filtering\"]\n    after = data[\"summary\"][\"after_filtering\"]\n    \n    return {\n        \"total_reads_in\":  before[\"total_reads\"],\n        \"total_reads_out\": after[\"total_reads\"],\n        \"pct_passed\":      after[\"total_reads\"] \u002F before[\"total_reads\"] * 100,\n        \"q30_rate_before\": before[\"q30_rate\"] * 100,\n        \"q30_rate_after\":  after[\"q30_rate\"] * 100,\n        \"mean_len_before\": before[\"read1_mean_length\"],\n        \"mean_len_after\":  after[\"read1_mean_length\"],\n        \"adapter_trimmed\": data[\"filtering_result\"][\"adapter_trimmed\"],\n    }\n\nmetrics = parse_fastp_json(\"sample_qc.json\")\nfor key, val in metrics.items():\n    print(f\"{key:25s}: {val:.1f}\" if isinstance(val, float) else f\"{key:25s}: {val:,}\")\n\n# Quality gate: fail if \u003C 70% reads pass filter\nif metrics[\"pct_passed\"] \u003C 70:\n    print(\"WARNING: Low pass rate — check raw data quality\")\n```\n\n### Step 6: Batch Preprocessing Pipeline\n\nProcess multiple samples sequentially with per-sample QC summaries.\n\n```bash\n#!\u002Fbin\u002Fbash\n# Batch paired-end preprocessing for multiple samples\nSAMPLES=(ctrl_1 ctrl_2 treat_1 treat_2)\nDATA=\"data\"\nOUT=\"trimmed\"\nQC=\"qc\u002Ffastp\"\nTHREADS=8\n\nmkdir -p \"$OUT\" \"$QC\"\n\nfor sample in \"${SAMPLES[@]}\"; do\n    echo \"=== Processing $sample ===\"\n    fastp \\\n        -i \"$DATA\u002F${sample}_R1.fastq.gz\" \\\n        -I \"$DATA\u002F${sample}_R2.fastq.gz\" \\\n        -o \"$OUT\u002F${sample}_R1.fastq.gz\" \\\n        -O \"$OUT\u002F${sample}_R2.fastq.gz\" \\\n        -h \"$QC\u002F${sample}.html\" \\\n        -j \"$QC\u002F${sample}.json\" \\\n        --thread $THREADS \\\n        --correction \\\n        --detect_adapter_for_pe \\\n        --qualified_quality_phred 20 \\\n        --length_required 36 \\\n        2>&1 | grep -E \"Read[12]|Filtering|Adapter|passed\"\ndone\n\n# Aggregate QC metrics\npython3 - \u003C\u003C 'EOF'\nimport json, pandas as pd\nfrom pathlib import Path\n\nrows = []\nfor jf in sorted(Path(\"qc\u002Ffastp\").glob(\"*.json\")):\n    with open(jf) as f: data = json.load(f)\n    after = data[\"summary\"][\"after_filtering\"]\n    before = data[\"summary\"][\"before_filtering\"]\n    rows.append({\n        \"sample\": jf.stem,\n        \"reads_in\": before[\"total_reads\"],\n        \"reads_out\": after[\"total_reads\"],\n        \"pct_passed\": round(after[\"total_reads\"]\u002Fbefore[\"total_reads\"]*100, 1),\n        \"q30_after\": round(after[\"q30_rate\"]*100, 1),\n    })\ndf = pd.DataFrame(rows)\nprint(df.to_string(index=False))\ndf.to_csv(\"fastp_summary.tsv\", sep=\"\\t\", index=False)\nEOF\n\n# Run MultiQC to aggregate all fastp JSON reports\nmultiqc qc\u002Ffastp\u002F -o qc\u002F -n fastp_multiqc_report\n```\n\n## Key Parameters\n\n| Parameter | Default | Range\u002FOptions | Effect |\n|-----------|---------|---------------|--------|\n| `-i \u002F -I` | required | file path | Input FASTQ (R1 and R2 for paired-end) |\n| `-o \u002F -O` | required | file path | Output trimmed FASTQ (R1 and R2) |\n| `-h \u002F -j` | — | file path | HTML and JSON QC report output paths |\n| `--thread` | `3` | 1–16 | CPU threads; 8 is a good balance |\n| `--qualified_quality_phred` | `15` | 0–40 | Minimum base quality (Phred); 20 = 1% error |\n| `--length_required` | `15` | 1–1000 | Minimum read length after trimming; discard shorter reads |\n| `--correction` | off | flag | Correct mismatches in PE overlap region |\n| `--detect_adapter_for_pe` | off | flag | Enable overlap-based adapter auto-detection for PE data |\n| `--adapter_sequence` | auto | string | Explicit R1 adapter; overrides auto-detection |\n| `--trim_poly_x` | off | flag | Trim polyX (polyA\u002FpolyT) tails; use for 3′-enriched RNA-seq |\n| `--low_complexity_filter` | off | flag | Filter reads with low complexity (\u003C 30% complexity by default) |\n| `--split` | off | integer | Split output into N files per direction (for parallelism) |\n\n## Common Recipes\n\n### Recipe 1: Integrate fastp into a Snakemake Pipeline\n\n```python\n# Snakefile — fastp trimming rule\nconfigfile: \"config.yaml\"\nSAMPLES = config[\"samples\"]\n\nrule fastp_pe:\n    input:\n        r1 = \"data\u002F{sample}_R1.fastq.gz\",\n        r2 = \"data\u002F{sample}_R2.fastq.gz\"\n    output:\n        r1 = \"trimmed\u002F{sample}_R1.fastq.gz\",\n        r2 = \"trimmed\u002F{sample}_R2.fastq.gz\",\n        html = \"qc\u002F{sample}_fastp.html\",\n        json = \"qc\u002F{sample}_fastp.json\"\n    threads: 8\n    shell:\n        \"\"\"\n        fastp -i {input.r1} -I {input.r2} \\\n              -o {output.r1} -O {output.r2} \\\n              -h {output.html} -j {output.json} \\\n              --thread {threads} \\\n              --correction --detect_adapter_for_pe \\\n              --qualified_quality_phred 20 \\\n              --length_required 36\n        \"\"\"\n```\n\n### Recipe 2: Aggregate fastp JSON Reports with Python\n\n```python\nimport json\nimport pandas as pd\nfrom pathlib import Path\n\nqc_dir = Path(\"qc\u002Ffastp\")\nrecords = []\n\nfor jf in sorted(qc_dir.glob(\"*.json\")):\n    with open(jf) as f:\n        d = json.load(f)\n    b = d[\"summary\"][\"before_filtering\"]\n    a = d[\"summary\"][\"after_filtering\"]\n    records.append({\n        \"sample\": jf.stem.replace(\"_fastp\", \"\"),\n        \"reads_in_M\": b[\"total_reads\"] \u002F 1e6,\n        \"reads_out_M\": a[\"total_reads\"] \u002F 1e6,\n        \"pct_passed\": a[\"total_reads\"] \u002F b[\"total_reads\"] * 100,\n        \"q30_pct\": a[\"q30_rate\"] * 100,\n        \"mean_len_bp\": a[\"read1_mean_length\"],\n        \"adapter_pct\": d[\"filtering_result\"][\"adapter_trimmed\"] \u002F b[\"total_reads\"] * 100,\n    })\n\ndf = pd.DataFrame(records).round(2)\nprint(df.to_string(index=False))\n\n# Flag low-quality samples\nlow_q = df[df[\"pct_passed\"] \u003C 80]\nif not low_q.empty:\n    print(f\"\\nSamples with \u003C 80% reads passing: {list(low_q['sample'])}\")\n```\n\n## Expected Outputs\n\n| Output | Format | Description |\n|--------|--------|-------------|\n| `*_R1.trimmed.fastq.gz` | FASTQ.gz | Trimmed R1 reads (adapters and low-quality bases removed) |\n| `*_R2.trimmed.fastq.gz` | FASTQ.gz | Trimmed R2 reads (paired-end only) |\n| `*.html` | HTML | Interactive QC report with per-base quality, GC content, adapter plots |\n| `*.json` | JSON | Machine-readable QC metrics for automation and MultiQC parsing |\n| `fastp.log` | Text | stderr summary with pass\u002Ffail read counts and filtering statistics |\n\n## Troubleshooting\n\n| Problem | Cause | Solution |\n|---------|-------|----------|\n| Adapter not detected in SE mode | SE reads require explicit adapter or `--adapter_sequence` | Use `--detect_adapter_for_pe` only for PE; specify adapter for SE: `--adapter_sequence AGATCGGAAGAGC` |\n| Very high adapter content (> 50%) | Short inserts (small RNA, miRNA) or poor library prep | Check library protocol; use `--overlap_len_require 10` to adjust overlap sensitivity |\n| Too many reads filtered (\u003C 60% pass) | Over-strict quality thresholds or low-quality sequencing run | Relax `--qualified_quality_phred` to 15; lower `--length_required` to 25 |\n| JSON output missing fields | Old fastp version | Upgrade: `conda update fastp` or download latest binary from GitHub |\n| MultiQC not parsing fastp JSON | JSON file not in the scanned directory | Run `multiqc qc\u002F` not `multiqc .`; verify JSON files exist with `ls qc\u002F*.json` |\n| Output FASTQ is empty | All reads filtered (wrong input or extreme thresholds) | Verify input FASTQ with `zcat sample.fq.gz \\| head -8`; run without `--low_complexity_filter` first |\n| Slow performance on large files | Low thread count | Increase `--thread` to 8–12; ensure input is on fast storage (SSD) |\n| polyA not removed | `--trim_poly_x` not set | Add `--trim_poly_x --poly_x_min_len 10` for 3′-enriched protocols |\n\n## References\n\n- [fastp GitHub: OpenGene\u002Ffastp](https:\u002F\u002Fgithub.com\u002FOpenGene\u002Ffastp) — source code, changelog, and usage guide\n- Chen S et al. (2018) \"fastp: an ultra-fast all-in-one FASTQ preprocessor\" — *Bioinformatics* 34(17):i884-i890. [DOI:10.1093\u002Fbioinformatics\u002Fbty560](https:\u002F\u002Fdoi.org\u002F10.1093\u002Fbioinformatics\u002Fbty560)\n- [fastp documentation](https:\u002F\u002Fgithub.com\u002FOpenGene\u002Ffastp#readme) — full parameter reference and recipes\n- [MultiQC fastp module](https:\u002F\u002Fmultiqc.info\u002Fmodules\u002Ffastp\u002F) — aggregating fastp JSON reports across samples\n",{"name":5,"license":15,"description":7},"master","49e49abf2f670d429e0b7e2c0349739c3fdd5cbe","skills\u002Ffastp-fastq-preprocessing",5,"https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium","https:\u002F\u002Fgithub.com\u002Fstanfish06","https:\u002F\u002Fwww.stanfish.dev\u002F","2026-11-14 21:31:37",{"source":8,"skillName":5,"commands":33,"prompt":43,"zipAvailable":44},[34,37,40],{"tool":35,"command":36},"npx","npx skills add stanfish06\u002Fskillquarium --skill fastp-fastq-preprocessing",{"tool":38,"command":39},"bunx","bunx skills add stanfish06\u002Fskillquarium --skill fastp-fastq-preprocessing",{"tool":41,"command":42},"pnpm","pnpm dlx skills add stanfish06\u002Fskillquarium --skill fastp-fastq-preprocessing","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium\u002Fblob\u002Fmaster\u002Fskills\u002Ffastp-fastq-preprocessing\u002FSKILL.md",true,[],[47,54,63,70,77,84],{"skillNo":48,"name":49,"nameZh":6,"description":50,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":51,"categories":52,"keywords":53,"publishedAt":19},"01M76ZJTRVA456GD7K8Y3EP82H","zarr-python","Stores and queries chunked N-D scientific arrays with Zarr-Python 3, including codecs, sharding, S3\u002FGCS storage, and NumPy\u002FDask\u002FXarray integration. Use for array layout, bounded I\u002FO, format migration, or scientific metadata preservation.","https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium\u002Fblob\u002Fmaster\u002Fskills\u002Fzarr-python\u002FSKILL.md",[],[],{"skillNo":55,"name":56,"nameZh":6,"description":57,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":58,"licenseFileUrl":59,"sourceHtmlUrl":60,"categories":61,"keywords":62,"publishedAt":19},"01M76ZJQZ15Q1HKC5V85NHE325","yeet","Publish local changes to GitHub by confirming scope, committing intentionally, pushing the branch, and opening a draft PR through the GitHub app from this plugin, with `gh` used only as a fallback where connector coverage is insufficient.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium\u002Fblob\u002Fmaster\u002Fskills\u002Fyeet\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium\u002Fblob\u002Fmaster\u002Fskills\u002Fyeet\u002FSKILL.md",[],[],{"skillNo":64,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":58,"licenseFileUrl":6,"sourceHtmlUrl":67,"categories":68,"keywords":69,"publishedAt":19},"01M76ZJ8ASZ4QYQ70Y6SBA4Z4S","xarray","N-dimensional labeled arrays and datasets in Python. Built on top of NumPy and Dask. It introduces labels in the form of dimensions, coordinates, and attributes on top of raw NumPy-like arrays, making data analysis in physical sciences more intuitive and less error-prone. Use for working with multi-dimensional scientific data, NetCDF\u002FGRIB\u002FZarr files, climate\u002Fweather\u002Foceanographic datasets, remote sensing, geospatial imaging, large out-of-memory datasets with Dask, and labeled array operations.","https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium\u002Fblob\u002Fmaster\u002Fskills\u002Fxarray\u002FSKILL.md",[],[],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":19},"01M76ZHFBXD6VXPQ02VCEG202B","writing-mstest-tests","Use when asked to fix, rewrite, update, improve, modernize, show corrected code for, or explain existing MSTest tests or MSTest-specific configuration. Use for \"review\" when corrected code or edits are wanted, even for one pasted assertion or passing tests with bad failure output. Covers expected\u002Factual labels; generic Boolean, collection, string, numeric, null, identity, exception, hard-cast, and object[] checks; TestContext\u002Flifecycle; timeout\u002Fcancellation; OS\u002FCI conditions, retry, cleanup, parallelization, MSTest.Sdk project setup, and MSTESTxxxx. Honor the installed MSTest version. DO NOT USE to design new test cases (code-testing), perform report-only audits, create project files rather than explain MSTest setup, run tests, migrate frameworks, or handle non-MSTest\u002Fnon-.NET code.","https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium\u002Fblob\u002Fmaster\u002Fskills\u002Fwriting-mstest-tests\u002FSKILL.md",[],[],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":19},"01M76ZDFDF4N88RH6HSYGS7GP2","wgs-prs","End-to-end WGS to polygenic risk score pipeline. Takes paired-end FASTQ files (or a pre-existing VCF) through nf-core\u002Fsarek for variant calling, applies VCF QC (normalisation, hard filtering, Ti\u002FTv and Het\u002FHom checks), then computes polygenic risk scores via the PGS Catalog. Fills the FASTQ to VCF gap upstream of the gwas-prs skill.","https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium\u002Fblob\u002Fmaster\u002Fskills\u002Fwgs-prs\u002FSKILL.md",[],[],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":58,"licenseFileUrl":88,"sourceHtmlUrl":89,"categories":90,"keywords":91,"publishedAt":19},"01M76ZCC1SC68J42QK4MDC8RCG","webapp-testing","Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.","https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium\u002Fblob\u002Fmaster\u002Fskills\u002Fwebapp-testing\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium\u002Fblob\u002Fmaster\u002Fskills\u002Fwebapp-testing\u002FSKILL.md",[],[]]