[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M75K8TZPTYNK2R5FF135EZVV":3,"skill-same:01M75K8TZPTYNK2R5FF135EZVV":46},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":6,"ownerHtmlUrl":29,"ownerBlogUrl":6,"ownerDescription":30,"updatedAt":31,"install":32,"resources":45},"01M75K8TZPTYNK2R5FF135EZVV","filter-sequences",null,"Filter FASTA\u002FFASTQ records by length, identity, regex on headers, GC window, complexity, and custom predicates — streaming with Biopython 1.83+.","pradyumnasagar\u002Fopen-research-skills","pradyumnasagar","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F10559871?v=4","01M75K78MYCCVKRTYGT8KKC380","Pradyumna Jayaram","01M75K78NTNE41F2PE1XBVFHAS",8,"MIT","https:\u002F\u002Fgithub.com\u002Fpradyumnasagar\u002Fopen-research-skills\u002Fblob\u002Fmain\u002Fskills\u002Fbioinformatics-sequence\u002Ffilter-sequences\u002FSKILL.md",[],[],"2026-11-12 13:55:43",false,"---\n\nname: filter-sequences\ndescription: \"Filter FASTA\u002FFASTQ records by length, identity, regex on headers, GC window, complexity, and custom predicates — streaming with Biopython 1.83+.\"\nlicense: MIT\n---\n\u003C!-- metadata:\n  category: bioinformatics-sequence\n  oversight: low\n  version: 1.0.0\n  author: Pradyumna Jayaram\n  tags: [bioinformatics-sequence, research]\n  difficulty: intermediate\n-->\n\n# Filter Sequences by Criteria\n## Hard rules\n\n1. **No fabricated citations.** Every cited work must resolve to a verifiable\n2. **No claim without provenance.** Every quantitative or factual claim\n3. **No silent failure.** Every script invocation, API call, or tool use must declare its exit status and what to do on non-zero. A skill that silently swallows errors is a violation.\n\n## When to use\n\n- Selecting reads\u002Fcontigs by length, GC, N count, or complexity.\n- Subsetting FASTA by header pattern (e.g., keep only `chr[1-9XY]`, drop `chrUn_*`).\n- Removing low-complexity reads (e.g., poly-A runs) before assembly.\n- Filtering paired FASTQ — see the paired-end skill for sync filtering.\n\n## When NOT to use\n\n- Adapter\u002Fquality trimming → use `fastp` (single-tool, much faster).\n- Host\u002Fcontaminant screening → use `kraken2`, `minimap2` against a host DB, or `nf-core` modules.\n- For UMI dedup → `umi-tools dedup` or `fab` are better than custom code.\n\n## Prerequisites\n\n- `biopython>=1.83`, `regex>=2024.5` (better Unicode support than `re`).\n- For high-throughput filtering: `seqkit` (C) is ~50x faster.\n\n## Core workflow\n\n1. **Define your predicates** as composable Python functions returning `bool`.\n2. **Wrap them in a generator chain** so the file is read once.\n3. **Stream the survivors to disk** with `SeqIO.write`.\n4. **Always log how many records you kept vs dropped** — for reproducibility.\n\n## Code patterns\n\n### Length filter (FASTA)\n\n```python\nfrom Bio import SeqIO\n\ndef length_filter(records, min_len=500, max_len=10_000_000):\n    for rec in records:\n        if min_len \u003C= len(rec.seq) \u003C= max_len:\n            yield rec\n\nwith open(\"filtered.fasta\", \"w\") as out:\n    n_in = n_out = 0\n    for rec in SeqIO.parse(\"input.fasta\", \"fasta\"):\n        n_in += 1\n        if min_len := 500 \u003C= len(rec.seq):\n            SeqIO.write(rec, out, \"fasta\")\n            n_out += 1\nprint(f\"Kept {n_out}\u002F{n_in} records\")\n```\n\n(Prefer the explicit form for clarity — the example above shows the structural pattern.)\n\n### Identity filter (regex on header)\n\n```python\nimport regex\nfrom Bio import SeqIO\n\nkeep_pattern = regex.compile(r\"^chr(1[0-9]|2[0-2]|[1-9]|X|Y|M)$\")\n\ndef keep_chrom(rec, pat=keep_pattern):\n    return bool(pat.match(rec.id))\n\nwith open(\"primary.fasta\", \"w\") as out:\n    SeqIO.write(r for r in SeqIO.parse(\"genome.fasta\", \"fasta\") if keep_chrom(r)),\n                out, \"fasta\")\n```\n\n### GC window filter\n\n```python\nfrom Bio import SeqIO\nfrom Bio.SeqUtils import gc_fraction\n\ndef gc_window(rec, lo=0.30, hi=0.70):\n    return lo \u003C= gc_fraction(rec.seq) \u003C= hi\n\nSeqIO.write((r for r in SeqIO.parse(\"asm.fasta\", \"fasta\") if gc_window(r)),\n            \"asm_gc_filtered.fasta\", \"fasta\")\n```\n\n### N content filter\n\n```python\ndef max_n(rec, max_frac=0.05):\n    s = str(rec.seq).upper()\n    if not s:\n        return False\n    return (s.count(\"N\") \u002F len(s)) \u003C= max_frac\n```\n\n### Quality filter (FASTQ)\n\n```python\ndef min_mean_quality(rec, min_q=30):\n    q = rec.letter_annotations[\"phred_quality\"]\n    return (sum(q) \u002F len(q)) >= min_q\n```\n\n### Complexity filter (Shannon entropy over a window)\n\nUseful for removing low-complexity reads that escape length filters:\n\n```python\nimport math\nfrom collections import Counter\n\ndef entropy(s, k=4):\n    \"\"\"k-mer Shannon entropy; > ~2 bits usually means 'not junk' for k=4.\"\"\"\n    if len(s) \u003C k:\n        return 0.0\n    counts = Counter(s[i:i+k] for i in range(len(s) - k + 1))\n    total = sum(counts.values())\n    return -sum((c\u002Ftotal) * math.log2(c\u002Ftotal) for c in counts.values())\n\ndef complex_enough(rec, min_entropy=1.8, k=4):\n    return entropy(str(rec.seq).upper(), k=k) >= min_entropy\n```\n\n### Composition predicates (composition)\n\n```python\nfrom functools import reduce\n\ndef keep(rec, preds):\n    return all(p(rec) for p in preds)\n\npredicates = [lambda r: min_length(r, 1000), lambda r: max_n(r, 0.01), gc_window]\nSeqIO.write((r for r in SeqIO.parse(\"in.fasta\", \"fasta\") if keep(r, predicates)),\n            \"out.fasta\", \"fasta\")\n```\n\n### Drop empty records after trim\n\n```python\ndef non_empty(rec):\n    return len(rec.seq) > 0\n```\n\n## Common pitfalls\n\n- **Forgetting the file was modified in place.** Filter chains should always be pure functions of `rec -> bool`.\n- **GC skew from Ns.** Use `Bio.SeqUtils.gc_fraction` (excludes Ns) or normalize manually.\n- **Regex on headers is brittle.** `^chr(\\d+|X|Y|M)$` matches `chr1` but not `CHR1`; decide and be explicit.\n- **Streaming into a list defeats the purpose.** Use generator expressions inside `SeqIO.write`.\n- **Length filter on FASTQ before QC trim** is wrong order. Trim first, then filter.\n\n## Validation\n\n- After every filter, the input count ≥ output count.\n- For paired filtering, mate counts in R1 and R2 match.\n- Spot-check: a few records from the output satisfy every predicate.\n- For complexity, an obvious junk read (e.g., `AAAAAAAAAA...`) has entropy ~0 and is dropped.\n\n## Open alternatives\n\n| Need | Tool |\n|------|------|\n| Sequence filtering at scale | `seqkit seq -m 500 -M 5000 in.fasta` |\n| Regex header subset | `samtools view -b in.bam chr1 chr2 ...` (for BAM) |\n| Complexity filter | `bbduk.sh entropy=0.5` (BBTools) |\n| N-content filter | `seqkit seq -g -G 0.05` (drop GC > 5% — actually means min GC, see docs) |\n\n## References\n\n- Biopython sequence utilities: \u003Chttps:\u002F\u002Fbiopython.org\u002Fdocs\u002Flatest\u002Fapi\u002FBio.SeqUtils.html>\n- `regex` package: \u003Chttps:\u002F\u002Fpypi.org\u002Fproject\u002Fregex\u002F>\n- Companion: `ors-bioinformatics-sequence-fastq-quality-scores`, `ors-bioinformatics-sequence-paired-end-fastq`.\n\n## Changelog\n\n- 1.0.0 (2026-06-10): Initial adaptation by Pradyumna Jayaram from `bio-filter-sequences` (bioSkills-main\u002Fsequence-io\u002Ffilter-sequences).\n\n## Cross-references\n\nOther skills in this category:\n\n- batch-processing\n- bowtie2-alignment\n- bwa-alignment\n- bwa-mem2-alignment\n- codon-usage\n- compressed-sequence-files\n- fastq-quality-scores\n- format-conversion\n- hisat2-alignment\n- motif-search\n- paired-end-fastq\n- pysam-genomics\n- read-write-sequences\n- reverse-complement\n- sam-bam-basics\n- samtools-bam-processing\n- seq-objects\n- sequence-properties\n- sequence-slicing\n- sequence-statistics\n- star-alignment\n- transcription-translation\n","\u003C!-- metadata:\n  category: bioinformatics-sequence\n  oversight: low\n  version: 1.0.0\n  author: Pradyumna Jayaram\n  tags: [bioinformatics-sequence, research]\n  difficulty: intermediate\n-->\n\n# Filter Sequences by Criteria\n## Hard rules\n\n1. **No fabricated citations.** Every cited work must resolve to a verifiable\n2. **No claim without provenance.** Every quantitative or factual claim\n3. **No silent failure.** Every script invocation, API call, or tool use must declare its exit status and what to do on non-zero. A skill that silently swallows errors is a violation.\n\n## When to use\n\n- Selecting reads\u002Fcontigs by length, GC, N count, or complexity.\n- Subsetting FASTA by header pattern (e.g., keep only `chr[1-9XY]`, drop `chrUn_*`).\n- Removing low-complexity reads (e.g., poly-A runs) before assembly.\n- Filtering paired FASTQ — see the paired-end skill for sync filtering.\n\n## When NOT to use\n\n- Adapter\u002Fquality trimming → use `fastp` (single-tool, much faster).\n- Host\u002Fcontaminant screening → use `kraken2`, `minimap2` against a host DB, or `nf-core` modules.\n- For UMI dedup → `umi-tools dedup` or `fab` are better than custom code.\n\n## Prerequisites\n\n- `biopython>=1.83`, `regex>=2024.5` (better Unicode support than `re`).\n- For high-throughput filtering: `seqkit` (C) is ~50x faster.\n\n## Core workflow\n\n1. **Define your predicates** as composable Python functions returning `bool`.\n2. **Wrap them in a generator chain** so the file is read once.\n3. **Stream the survivors to disk** with `SeqIO.write`.\n4. **Always log how many records you kept vs dropped** — for reproducibility.\n\n## Code patterns\n\n### Length filter (FASTA)\n\n```python\nfrom Bio import SeqIO\n\ndef length_filter(records, min_len=500, max_len=10_000_000):\n    for rec in records:\n        if min_len \u003C= len(rec.seq) \u003C= max_len:\n            yield rec\n\nwith open(\"filtered.fasta\", \"w\") as out:\n    n_in = n_out = 0\n    for rec in SeqIO.parse(\"input.fasta\", \"fasta\"):\n        n_in += 1\n        if min_len := 500 \u003C= len(rec.seq):\n            SeqIO.write(rec, out, \"fasta\")\n            n_out += 1\nprint(f\"Kept {n_out}\u002F{n_in} records\")\n```\n\n(Prefer the explicit form for clarity — the example above shows the structural pattern.)\n\n### Identity filter (regex on header)\n\n```python\nimport regex\nfrom Bio import SeqIO\n\nkeep_pattern = regex.compile(r\"^chr(1[0-9]|2[0-2]|[1-9]|X|Y|M)$\")\n\ndef keep_chrom(rec, pat=keep_pattern):\n    return bool(pat.match(rec.id))\n\nwith open(\"primary.fasta\", \"w\") as out:\n    SeqIO.write(r for r in SeqIO.parse(\"genome.fasta\", \"fasta\") if keep_chrom(r)),\n                out, \"fasta\")\n```\n\n### GC window filter\n\n```python\nfrom Bio import SeqIO\nfrom Bio.SeqUtils import gc_fraction\n\ndef gc_window(rec, lo=0.30, hi=0.70):\n    return lo \u003C= gc_fraction(rec.seq) \u003C= hi\n\nSeqIO.write((r for r in SeqIO.parse(\"asm.fasta\", \"fasta\") if gc_window(r)),\n            \"asm_gc_filtered.fasta\", \"fasta\")\n```\n\n### N content filter\n\n```python\ndef max_n(rec, max_frac=0.05):\n    s = str(rec.seq).upper()\n    if not s:\n        return False\n    return (s.count(\"N\") \u002F len(s)) \u003C= max_frac\n```\n\n### Quality filter (FASTQ)\n\n```python\ndef min_mean_quality(rec, min_q=30):\n    q = rec.letter_annotations[\"phred_quality\"]\n    return (sum(q) \u002F len(q)) >= min_q\n```\n\n### Complexity filter (Shannon entropy over a window)\n\nUseful for removing low-complexity reads that escape length filters:\n\n```python\nimport math\nfrom collections import Counter\n\ndef entropy(s, k=4):\n    \"\"\"k-mer Shannon entropy; > ~2 bits usually means 'not junk' for k=4.\"\"\"\n    if len(s) \u003C k:\n        return 0.0\n    counts = Counter(s[i:i+k] for i in range(len(s) - k + 1))\n    total = sum(counts.values())\n    return -sum((c\u002Ftotal) * math.log2(c\u002Ftotal) for c in counts.values())\n\ndef complex_enough(rec, min_entropy=1.8, k=4):\n    return entropy(str(rec.seq).upper(), k=k) >= min_entropy\n```\n\n### Composition predicates (composition)\n\n```python\nfrom functools import reduce\n\ndef keep(rec, preds):\n    return all(p(rec) for p in preds)\n\npredicates = [lambda r: min_length(r, 1000), lambda r: max_n(r, 0.01), gc_window]\nSeqIO.write((r for r in SeqIO.parse(\"in.fasta\", \"fasta\") if keep(r, predicates)),\n            \"out.fasta\", \"fasta\")\n```\n\n### Drop empty records after trim\n\n```python\ndef non_empty(rec):\n    return len(rec.seq) > 0\n```\n\n## Common pitfalls\n\n- **Forgetting the file was modified in place.** Filter chains should always be pure functions of `rec -> bool`.\n- **GC skew from Ns.** Use `Bio.SeqUtils.gc_fraction` (excludes Ns) or normalize manually.\n- **Regex on headers is brittle.** `^chr(\\d+|X|Y|M)$` matches `chr1` but not `CHR1`; decide and be explicit.\n- **Streaming into a list defeats the purpose.** Use generator expressions inside `SeqIO.write`.\n- **Length filter on FASTQ before QC trim** is wrong order. Trim first, then filter.\n\n## Validation\n\n- After every filter, the input count ≥ output count.\n- For paired filtering, mate counts in R1 and R2 match.\n- Spot-check: a few records from the output satisfy every predicate.\n- For complexity, an obvious junk read (e.g., `AAAAAAAAAA...`) has entropy ~0 and is dropped.\n\n## Open alternatives\n\n| Need | Tool |\n|------|------|\n| Sequence filtering at scale | `seqkit seq -m 500 -M 5000 in.fasta` |\n| Regex header subset | `samtools view -b in.bam chr1 chr2 ...` (for BAM) |\n| Complexity filter | `bbduk.sh entropy=0.5` (BBTools) |\n| N-content filter | `seqkit seq -g -G 0.05` (drop GC > 5% — actually means min GC, see docs) |\n\n## References\n\n- Biopython sequence utilities: \u003Chttps:\u002F\u002Fbiopython.org\u002Fdocs\u002Flatest\u002Fapi\u002FBio.SeqUtils.html>\n- `regex` package: \u003Chttps:\u002F\u002Fpypi.org\u002Fproject\u002Fregex\u002F>\n- Companion: `ors-bioinformatics-sequence-fastq-quality-scores`, `ors-bioinformatics-sequence-paired-end-fastq`.\n\n## Changelog\n\n- 1.0.0 (2026-06-10): Initial adaptation by Pradyumna Jayaram from `bio-filter-sequences` (bioSkills-main\u002Fsequence-io\u002Ffilter-sequences).\n\n## Cross-references\n\nOther skills in this category:\n\n- batch-processing\n- bowtie2-alignment\n- bwa-alignment\n- bwa-mem2-alignment\n- codon-usage\n- compressed-sequence-files\n- fastq-quality-scores\n- format-conversion\n- hisat2-alignment\n- motif-search\n- paired-end-fastq\n- pysam-genomics\n- read-write-sequences\n- reverse-complement\n- sam-bam-basics\n- samtools-bam-processing\n- seq-objects\n- sequence-properties\n- sequence-slicing\n- sequence-statistics\n- star-alignment\n- transcription-translation\n",{"name":5,"license":15,"description":7},"main","9c0b41e2036cc5588c699f06e658ea6dc3b9685d","skills\u002Fbioinformatics-sequence\u002Ffilter-sequences",2,"https:\u002F\u002Fgithub.com\u002Fpradyumnasagar\u002Fopen-research-skills","https:\u002F\u002Fgithub.com\u002Fpradyumnasagar","PhD Scholar,\r\nManipal School Of Life Sciences, Manipal\r\n #OpenSourceDeveloper #LifeSciences #ComputationalBiology","2026-10-30 01:30:52",{"source":8,"skillName":5,"commands":33,"prompt":43,"zipAvailable":44},[34,37,40],{"tool":35,"command":36},"npx","npx skills add pradyumnasagar\u002Fopen-research-skills --skill filter-sequences",{"tool":38,"command":39},"bunx","bunx skills add pradyumnasagar\u002Fopen-research-skills --skill filter-sequences",{"tool":41,"command":42},"pnpm","pnpm dlx skills add pradyumnasagar\u002Fopen-research-skills --skill filter-sequences","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fpradyumnasagar\u002Fopen-research-skills\u002Fblob\u002Fmain\u002Fskills\u002Fbioinformatics-sequence\u002Ffilter-sequences\u002FSKILL.md",true,[],[47,54,60,67,74,81],{"skillNo":48,"name":49,"nameZh":6,"description":50,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":51,"categories":52,"keywords":53,"publishedAt":19},"01M75KTS5K9QNX8FQBTJW2GQSH","good-skill","A well-formed test fixture for the validation script.","https:\u002F\u002Fgithub.com\u002Fpradyumnasagar\u002Fopen-research-skills\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fgood-skill\u002FSKILL.md",[],[],{"skillNo":55,"name":56,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":57,"categories":58,"keywords":59,"publishedAt":19},"01M75KTR7FAT0MMMED2EGC9WZZ","BadSkill","https:\u002F\u002Fgithub.com\u002Fpradyumnasagar\u002Fopen-research-skills\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fbad-skill\u002FSKILL.md",[],[],{"skillNo":61,"name":62,"nameZh":6,"description":63,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":64,"categories":65,"keywords":66,"publishedAt":19},"01M75KTN949EDF77GJDS8H6VR8","my-new-skill","One sentence (max 200 chars) — what the skill does and when to use it.","https:\u002F\u002Fgithub.com\u002Fpradyumnasagar\u002Fopen-research-skills\u002Fblob\u002Fmain\u002Ftemplate\u002FSKILL.md",[],[],{"skillNo":68,"name":69,"nameZh":6,"description":70,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":71,"categories":72,"keywords":73,"publishedAt":19},"01M75KTHTFS121SSF1Q5X2FXDA","thesis-structure","Use when planning or checking the architecture of a PhD \u002F MPhil thesis in the life sciences or bioinformatics — monograph vs article-based, chapter sequence, IMRaD-based chapters, abstract-to-conclusion flow, and the examiner expectations that the structure must meet.","https:\u002F\u002Fgithub.com\u002Fpradyumnasagar\u002Fopen-research-skills\u002Fblob\u002Fmain\u002Fskills\u002Fthesis-writing\u002Fthesis-structure\u002FSKILL.md",[],[],{"skillNo":75,"name":76,"nameZh":6,"description":77,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":78,"categories":79,"keywords":80,"publishedAt":19},"01M75KTE3WKW46FXFHQ579H8YP","thesis-methods-reproducibility","Use when writing or checking the Methods, figures, data, and code of a bioinformatics \u002F microbiology \u002F life-science PhD thesis — the FAIR data, versioned code, report-generation, figure-and-table discipline, and the reproducibility expectations that examiners probe at viva.","https:\u002F\u002Fgithub.com\u002Fpradyumnasagar\u002Fopen-research-skills\u002Fblob\u002Fmain\u002Fskills\u002Fthesis-writing\u002Fthesis-methods-reproducibility\u002FSKILL.md",[],[],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":19},"01M75KTA033ZQCJY3DC8P5VQET","thesis-defense-viva","Use when preparing for the viva voce \u002F oral thesis defense — building the examiner's question set, rehearsing the presentation, and the pre-defense polish (slides, summary, the 'what if I'm stuck' responses) for a life-science or bioinformatics PhD.","https:\u002F\u002Fgithub.com\u002Fpradyumnasagar\u002Fopen-research-skills\u002Fblob\u002Fmain\u002Fskills\u002Fthesis-writing\u002Fthesis-defense-viva\u002FSKILL.md",[],[]]