[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7EF0FBYVB7F5MWBKDNPP472":3,"skill-same:01M7EF0FBYVB7F5MWBKDNPP472":62},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":30,"commitSha":31,"skillDirectory":32,"forksCount":14,"repositoryHtmlUrl":33,"repositoryHomepageUrl":33,"ownerHtmlUrl":34,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":35,"install":36,"resources":49},"01M7EF0FBYVB7F5MWBKDNPP472","bio-proteomics-data-import",null,"Loads mass-spectrometry data into Python\u002FR and strips the search engine's bookkeeping before any number is trusted -- removes decoys (REV__\u002FReverse), contaminants (CON__\u002FPotential contaminant), Only-identified-by-site groups, and resolves semicolon razor\u002Fleading protein-ID ambiguity in MaxQuant proteinGroups.txt, DIA-NN report.parquet, and mzML\u002FmzXML. Distinguishes Intensity (raw) vs LFQ intensity (MaxLFQ) vs iBAQ, treats a MaxQuant zero as missing (NaN, not log2(-inf)), and inherits the acquisition mode's missingness contract (DDA MNAR vs DIA MCAR). Use when starting an analysis from raw spectra or a search engine output. Downstream normalization and stats are differential-abundance; reporter-ion\u002FMaxLFQ quant is quantification; protein grouping is protein-inference.","Mr-Nobody-Anonymous\u002FAll-skills","Mr-Nobody-Anonymous","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F184253119?v=4","01M7AEXFJHKTYQW5R0ZCQQTWFV","anonymous","01M7C00M3GQR4QKP9GZA42GFAK",0,"MIT","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fbiology\u002Fbio-proteomics-data-import\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: bio-proteomics-data-import\ndescription: Loads mass-spectrometry data into Python\u002FR and strips the search engine's bookkeeping before any number is trusted -- removes decoys (REV__\u002FReverse), contaminants (CON__\u002FPotential contaminant), Only-identified-by-site groups, and resolves semicolon razor\u002Fleading protein-ID ambiguity in MaxQuant proteinGroups.txt, DIA-NN report.parquet, and mzML\u002FmzXML. Distinguishes Intensity (raw) vs LFQ intensity (MaxLFQ) vs iBAQ, treats a MaxQuant zero as missing (NaN, not log2(-inf)), and inherits the acquisition mode's missingness contract (DDA MNAR vs DIA MCAR). Use when starting an analysis from raw spectra or a search engine output. Downstream normalization and stats are differential-abundance; reporter-ion\u002FMaxLFQ quant is quantification; protein grouping is protein-inference.\ntool_type: mixed\nprimary_tool: pyOpenMS\nsource: \"https:\u002F\u002Fgithub.com\u002FGPTomics\u002FbioSkills\"\nsource_repository: \"GPTomics\u002FbioSkills\"\nsource_path: \"proteomics\u002Fdata-import\u002FSKILL.md\"\nlicense: \"MIT\"\nimported_at: \"2026-09-20\"\n---\n\n## Version Compatibility\n\nReference examples tested with: pyOpenMS 3.1+, pandas 2.2+, numpy 1.26+, MSnbase 2.28+\n\nBefore using code patterns, verify installed versions match. If versions differ:\n- Python: `pip show \u003Cpackage>` then `help(module.function)` to check signatures\n- R: `packageVersion('\u003Cpkg>')` then `?function_name` to verify parameters\n\nIf code throws ImportError, AttributeError, or TypeError, introspect the installed\npackage and adapt the example to match the actual API rather than retrying.\n\n# Mass Spectrometry Data Import -- Inheriting the Acquisition Contract and Stripping the Bookkeeping\n\n**\"Load my mass spec data into Python\"** -> Parse spectra or a search-engine table AND immediately enforce two contracts -- which quant column carries real biology, and which rows are search-engine bookkeeping that must be deleted -- because the same proteinGroups.txt yields different conclusions depending on the column read and the rows kept.\n- Python: `pyopenms.MzMLFile().load(path, exp)` for raw spectra; `pandas.read_csv(sep='\\t')` for MaxQuant; `pandas.read_parquet` for DIA-NN\n- R: `Spectra::Spectra()` \u002F `QFeatures::readQFeatures()` for raw and quantified data (MSnbase still works but is in maintenance mode)\n\nScope: this skill owns reading spectra\u002Fsearch outputs into memory, deleting decoy\u002Fcontaminant\u002Fsite-only rows, picking the correct quant column, and characterizing missingness. Format conversion (RAW -> mzML) -> peptide-identification. MaxLFQ\u002FTMT reporter quant computation -> quantification. Protein-group parsimony -> protein-inference. Normalization and imputation -> differential-abundance and expression-matrix\u002Fnormalization. OUT OF SCOPE: statistical testing, batch correction, and the actual imputation step (this skill only diagnoses the missingness so the right imputer is chosen later).\n\n## The Single Most Important Modern Insight -- Import Is Where Two Contracts Are Read and Enforced\n\n1. **A \"data import\" is never just file parsing -- it is the moment the acquisition mode's quantitative contract and its missingness structure are inherited.** DDA selects the top-N most intense precursors per cycle, and which precursors get picked is partly stochastic and abundance-biased, so the same low-abundance peptide is sampled in run A and missed in run B; this manufactures structured, left-censored MNAR missingness. DIA fragments every precursor in every window every cycle, so its (fewer) missing values are closer to MCAR. The catastrophic error this prevents: imputing a DDA matrix with a mean\u002FKNN method that assumes MCAR, which biases low-abundance proteins upward and manufactures false hits. The mode is born at acquisition and inherited at import; the missingness diagnosis made here dictates which imputation is even legitimate downstream.\n\n2. **The search engine's bookkeeping must be stripped before any number is trusted.** A proteinGroups.txt carries decoy rows (`Reverse == '+'`, `REV__` prefix in the ID) from the target-decoy FDR machinery, contaminant rows (`Potential contaminant == '+'`, `CON__` prefix), and Only-identified-by-site rows (the protein has no unmodified-peptide evidence, only a modified site). Keeping any of these leaks non-biological signal into the intensity matrix and inflates IDs. The catastrophic error: reporting differential abundance on a matrix where decoy or keratin rows survived.\n\n3. **The same proteinGroups.txt yields different biology from different columns, and a zero is not a measurement.** `Intensity` is raw summed precursor signal (not normalized, not comparable across samples for ratios). `LFQ intensity` is MaxLFQ-normalized and is the column for between-sample comparison. `iBAQ` is intensity divided by the number of observable tryptic peptides -- a within-sample molar proxy, not a between-sample quant. MaxQuant writes 0 for \"not quantified\", so log2(0) = -inf; replace 0 -> NaN before any transform. The catastrophic error: log2-transforming raw `Intensity` (or iBAQ) and reading the ratios as biology.\n\n## Tool Taxonomy\n\n| Tool \u002F method | Citation | Mechanism \u002F role | When |\n|---|---|---|---|\n| pyOpenMS `MzMLFile().load` | Chambers 2012 (ProteoWizard lineage) | Loads mzML\u002FmzXML into an MSExperiment in memory; iterate spectra by MS level | Programmatic access to raw peaks, precursor m\u002Fz, isolation windows |\n| pandas `read_csv`\u002F`read_parquet` | -- | Tabular ingest of MaxQuant TSV and DIA-NN parquet | All search-engine output tables |\n| DIA-NN report | Demichev 2020 | Long-format precursor table; `report.parquet` is the default (1.9+) and the only default (2.0) | DIA quant; pivot on `PG.MaxLFQ` after q-filtering |\n| MaxQuant `txt\u002F` outputs | Cox 2014 (MaxLFQ) | `proteinGroups.txt` (group level), `evidence.txt` (per-PSM) | DDA label-free \u002F TMT search results |\n| Spectra + QFeatures (R) | -- | Current Bioconductor raw + quantified-feature containers; `readQFeatures`, `aggregateFeatures` | R pipelines; preferred over MSnbase going forward |\n| MSnbase `readMSData` (R) | -- | On-disk raw reading; maintenance mode (route OUT to Spectra\u002FQFeatures) | Legacy R code only |\n| ThermoRawFileParser \u002F msconvert | Hulstaert 2020 \u002F Chambers 2012 | RAW -> mzML conversion (route OUT) | File conversion is peptide-identification |\n\n## Decision Tree by Scenario\n\n| Scenario | Recommended | Why |\n|---|---|---|\n| MaxQuant DDA label-free, between-sample comparison | Read `LFQ intensity` columns from proteinGroups.txt | MaxLFQ-normalized; the only MaxQuant column valid for cross-sample ratios |\n| MaxQuant, absolute\u002Fmolar abundance within one sample | Read `iBAQ` columns | iBAQ is a within-sample molar proxy; do not use across samples |\n| Need raw uncorrected signal for a custom normalization | Read `Intensity` columns, normalize yourself | `Intensity` is raw summed precursor area, not comparable as-is |\n| DIA-NN output (1.9 or 2.0) | `pd.read_parquet('report.parquet')`, filter q-values, pivot `PG.MaxLFQ` | 2.0 dropped the TSV default; q-filter before pivot or low-confidence rows leak in |\n| Raw spectra, need peaks\u002Fprecursor\u002Fisolation window | pyOpenMS `MzMLFile().load` | Programmatic peak and isolation-window access for QC and co-isolation reasoning |\n| R-based pipeline, quantified features | QFeatures `readQFeatures` + `aggregateFeatures` | Current Bioconductor; MSnbase is maintenance-only |\n| Data came from DDA, planning imputation | Diagnose missingness as MNAR -> route to left-censored imputation | DDA top-N sampling makes missingness abundance-dependent |\n| Data came from DIA, planning imputation | Treat missingness as closer to MCAR | DIA samples every precursor every cycle |\n\nDefault when uncertain: read `LFQ intensity` (MaxQuant) or `PG.MaxLFQ` after q-filtering (DIA-NN), strip Reverse\u002Fcontaminant\u002Fsite-only rows, set 0 -> NaN, then diagnose missingness before choosing an imputer.\n\n## Loading mzML\u002FmzXML with pyOpenMS\n\n**Goal:** Parse raw spectra into memory for QC, peak access, and isolation-window reasoning.\n\n**Approach:** Load into an MSExperiment (filled in place), iterate by MS level; `get_peaks()` returns a tuple of (mz, intensity) numpy arrays, and `getPrecursors()` returns a list.\n\n```python\nfrom pyopenms import MSExperiment, MzMLFile\n\nexp = MSExperiment()\nMzMLFile().load('sample.mzML', exp)  # fills exp in place; returns None\n\nfor spectrum in exp:\n    if spectrum.getMSLevel() == 1:\n        mz, intensity = spectrum.get_peaks()  # tuple of two numpy arrays\n    elif spectrum.getMSLevel() == 2:\n        precursor = spectrum.getPrecursors()[0]  # getPrecursors returns a list\n        precursor_mz = precursor.getMZ()\n        window = precursor.getIsolationWindowLowerOffset() + precursor.getIsolationWindowUpperOffset()\n```\n\n## Loading and Cleaning MaxQuant proteinGroups.txt\n\n**Goal:** Get a trustworthy log2 intensity matrix with bookkeeping rows removed and missing values represented as NaN.\n\n**Approach:** Strip Reverse\u002Fcontaminant\u002Fsite-only rows, resolve the semicolon protein-ID list to a leading ID, pick `LFQ intensity` columns, set 0 -> NaN, then log2-transform.\n\n```python\nimport pandas as pd\nimport numpy as np\n\npg = pd.read_csv('proteinGroups.txt', sep='\\t', low_memory=False)  # mixed-type cols\n\n# Flag columns hold '+' or empty string; all three are proteinGroups-only bookkeeping\nmask = (pg.get('Reverse', '') != '+') & (pg.get('Potential contaminant', '') != '+') & (pg.get('Only identified by site', '') != '+')\npg = pg[mask].copy()\n\n# Protein IDs \u002F Majority protein IDs \u002F Gene names are SEMICOLON lists; take the first (leading\u002Frazor) entry\npg['leading_protein'] = pg['Protein IDs'].str.split(';').str[0]\npg['leading_gene'] = pg['Gene names'].where(pg['Gene names'].notna(), '').str.split(';').str[0]\n\nlfq_cols = [c for c in pg.columns if c.startswith('LFQ intensity ')]  # MaxLFQ-normalized, between-sample comparable\nmatrix = pg[['leading_protein', 'leading_gene'] + lfq_cols].copy()\nmatrix[lfq_cols] = matrix[lfq_cols].replace(0, np.nan)  # MaxQuant writes 0 for missing; log2(0) = -inf\nmatrix[lfq_cols] = np.log2(matrix[lfq_cols])\n```\n\n## Loading DIA-NN report.parquet\n\n**Goal:** Reshape the long DIA-NN report into a confident protein-by-run matrix.\n\n**Approach:** Read the parquet (default since 1.9, only default in 2.0), filter precursor- AND protein-group q-values to 1% FDR BEFORE pivoting on `PG.MaxLFQ`.\n\n```python\nimport pandas as pd\n\nreport = pd.read_parquet('report.parquet')  # report.tsv dropped as default in DIA-NN 2.0\nreport = report[(report['Q.Value'] \u003C= 0.01) & (report['PG.Q.Value'] \u003C= 0.01)]  # 1% FDR before quant\n\nmatrix = report.pivot_table(index='Protein.Group', columns='Run', values='PG.MaxLFQ', aggfunc='first')\n```\n\n## Diagnosing the Missingness Contract\n\n**Goal:** Quantify the missing-value pattern so the legitimate imputation class can be chosen downstream.\n\n**Approach:** Count NaN per protein and per sample; relate the pattern to acquisition mode (DDA -> structured MNAR; DIA -> closer to MCAR). A correlation between missingness and mean abundance is the MNAR signature.\n\n```python\nimport numpy as np\n\ndef assess_missingness(matrix, sample_cols):\n    miss_per_protein = matrix[sample_cols].isna().sum(axis=1)\n    miss_per_sample = matrix[sample_cols].isna().sum(axis=0)\n    total_pct = 100 * matrix[sample_cols].isna().sum().sum() \u002F matrix[sample_cols].size\n    mean_abund = matrix[sample_cols].mean(axis=1)  # negative corr with missingness => MNAR \u002F left-censored\n    mnar_corr = mean_abund.corr(miss_per_protein)\n    return {'per_protein': miss_per_protein, 'per_sample': miss_per_sample, 'total_pct': total_pct, 'abundance_missing_corr': mnar_corr}\n```\n\n## Per-Method Failure Modes\n\n### MaxQuant wrong quant column\n\n**Trigger:** Reading `Intensity` (raw) or `iBAQ` when between-sample ratios are intended.\n**Mechanism:** `Intensity` is un-normalized summed precursor signal; `iBAQ` is a within-sample molar proxy. Neither is comparable across samples the way `LFQ intensity` is.\n**Symptom:** Ratios track total loaded protein \u002F sample depth rather than biology; fold changes shift when one sample's loading changes.\n**Fix:** Use `LFQ intensity` for cross-sample comparison; if computing custom normalization use `Intensity` and normalize explicitly (expression-matrix\u002Fnormalization).\n\n### Zero treated as a measurement\n\n**Trigger:** `np.log2` applied directly to a MaxQuant matrix still containing 0.\n**Mechanism:** MaxQuant encodes \"not quantified\" as 0; log2(0) = -inf, which then propagates into means and tests.\n**Symptom:** -inf values, NaN means, proteins silently dropped or skewed.\n**Fix:** `replace(0, np.nan)` before any transform; then diagnose missingness.\n\n### Bookkeeping rows survive\n\n**Trigger:** Loading proteinGroups.txt without filtering Reverse \u002F Potential contaminant \u002F Only identified by site.\n**Mechanism:** Decoys exist only for FDR estimation; contaminants are keratin\u002Ftrypsin\u002FBSA, not the sample; site-only groups have no unmodified-peptide quant evidence. `Only identified by site` exists only in proteinGroups.txt.\n**Symptom:** Inflated protein counts; a \"hit\" that is a decoy or keratin.\n**Fix:** Filter all three flag columns; cross-check with `REV__`\u002F`CON__` ID prefixes when joining to peptide tables. Caveat: do not delete CON__ rows blindly if a contaminant (e.g. keratin) is the protein of interest.\n\n### Razor \u002F leading protein-ID ambiguity ignored\n\n**Trigger:** Treating `Protein IDs` or `Gene names` as an atomic single value.\n**Mechanism:** These are semicolon-delimited lists; the first entry is the leading (razor) protein for the group, and `Gene names` can be blank while protein IDs are present.\n**Symptom:** Merges fail, NaN gene labels, ambiguous identity downstream.\n**Fix:** Split on `;` and take the first entry; guard `Gene names` with `.notna()`. Group parsimony details -> protein-inference.\n\n### Stale DIA-NN parsing\n\n**Trigger:** Reading `report.tsv` on DIA-NN 2.0, or pivoting before q-filtering.\n**Mechanism:** 2.0 defaults to (and only defaults to) `report.parquet`; pivoting unfiltered rows includes precursors above 1% FDR.\n**Symptom:** FileNotFoundError on report.tsv; or low-confidence quant inflating the matrix.\n**Fix:** `pd.read_parquet('report.parquet')`; filter `Q.Value \u003C= 0.01 & PG.Q.Value \u003C= 0.01` before pivoting `PG.MaxLFQ`.\n\n### MNAR imputed as MCAR\n\n**Trigger:** Mean\u002Fmedian\u002FKNN imputation on a DDA matrix.\n**Mechanism:** DDA missingness is abundance-dependent (left-censored); MCAR imputers fill missing low values with the central tendency, biasing them upward.\n**Symptom:** Low-abundance proteins gain false high values; spurious differential hits.\n**Fix:** Diagnose the abundance-missingness correlation here; route DDA to left-censored imputation (downshifted-Gaussian \u002F QRILC \u002F MinProb) in differential-abundance; DIA tolerates standard imputers.\n\n## Quantitative Thresholds\n\n| Threshold | Source | Rationale |\n|---|---|---|\n| DIA-NN import filter `Q.Value \u003C= 0.01` AND `PG.Q.Value \u003C= 0.01` | Demichev 2020; target-decoy convention | Precursor- and protein-group-level 1% FDR enforced before any quant value is used |\n| Peptide\u002Fprotein FDR 1% (q \u003C= 0.01) | Target-decoy convention | Standard ID confidence at both peptide and protein levels |\n| MaxQuant zero -> NaN | MaxQuant output convention | 0 encodes \"not quantified\"; log2(0) = -inf corrupts every transform |\n| Min peptides per protein for quant >= 2 | Community quant practice | Single-peptide (\"one-hit-wonder\") proteins are ID\u002Fquant-unreliable |\n| Valid-value filter >= 50-70% per group | Modeling choice (document per study) | Caps imputation burden; the exact cutoff is a study decision, not a universal constant |\n| Take FIRST semicolon entry as leading protein\u002Fgene | MaxQuant proteinGroups convention | The leading\u002Frazor protein is the group identifier; trailing entries are shared-peptide members |\n\n## Common Errors\n\n| Error \u002F symptom | Cause | Solution |\n|---|---|---|\n| `-inf` values after log2 | Zeros not converted to NaN | `df.replace(0, np.nan)` before `np.log2` |\n| `FileNotFoundError: report.tsv` (DIA-NN 2.0) | TSV no longer the default output | `pd.read_parquet('report.parquet')` |\n| `KeyError: 'Only identified by site'` | That column exists ONLY in proteinGroups.txt | Use `df.get('Only identified by site', '')` or guard the column lookup |\n| Mixed-type \u002F DtypeWarning on MaxQuant load | Wide TSV with mixed column types | `pd.read_csv(..., low_memory=False)` |\n| NaN gene labels break a merge | `Gene names` is a semicolon list, sometimes blank | `.where(notna(), '').str.split(';').str[0]` |\n| Ratios track loading not biology | Read `Intensity` (raw) instead of `LFQ intensity` | Use `LFQ intensity` for between-sample comparison |\n| `get_peaks()` unpacking error | Expecting a 2D array | It returns a tuple `(mz, intensity)` of two numpy arrays |\n\n## References\n\n- Cox J, Hein MY, Luber CA, Paron I, Nagaraj N, Mann M. 2014. Accurate proteome-wide label-free quantification by delayed normalization and maximal peptide ratio extraction, termed MaxLFQ. *Mol Cell Proteomics* 13(9):2513-2526.\n- Demichev V, Messner CB, Vernardis SI, Lilley KS, Ralser M. 2020. DIA-NN: neural networks and interference correction enable deep proteome coverage in high throughput. *Nat Methods* 17(1):41-44.\n- Chambers MC, Maclean B, Burke R, et al. 2012. A cross-platform toolkit for mass spectrometry and proteomics. *Nat Biotechnol* 30(10):918-920.\n- Hulstaert N, Shofstahl J, Sachsenberg T, et al. 2020. ThermoRawFileParser: modular, scalable, and cross-platform RAW file conversion. *J Proteome Res* 19(1):537-542.\n\n## Related Skills\n\n- peptide-identification - search raw spectra and convert vendor RAW to mzML\n- quantification - compute MaxLFQ and TMT reporter-ion quantities from imported data\n- protein-inference - resolve protein-group parsimony and razor assignment\n- differential-abundance - normalize, impute (per the missingness diagnosis), and test\n- proteomics-qc - assess run-level identification and quant quality\n- dia-analysis - run DIA-NN to produce the report this skill imports\n- expression-matrix\u002Fnormalization - general intensity-matrix normalization patterns\n- workflows\u002Fproteomics-pipeline - end-to-end pipeline that begins with this import step\n","## Version Compatibility\n\nReference examples tested with: pyOpenMS 3.1+, pandas 2.2+, numpy 1.26+, MSnbase 2.28+\n\nBefore using code patterns, verify installed versions match. If versions differ:\n- Python: `pip show \u003Cpackage>` then `help(module.function)` to check signatures\n- R: `packageVersion('\u003Cpkg>')` then `?function_name` to verify parameters\n\nIf code throws ImportError, AttributeError, or TypeError, introspect the installed\npackage and adapt the example to match the actual API rather than retrying.\n\n# Mass Spectrometry Data Import -- Inheriting the Acquisition Contract and Stripping the Bookkeeping\n\n**\"Load my mass spec data into Python\"** -> Parse spectra or a search-engine table AND immediately enforce two contracts -- which quant column carries real biology, and which rows are search-engine bookkeeping that must be deleted -- because the same proteinGroups.txt yields different conclusions depending on the column read and the rows kept.\n- Python: `pyopenms.MzMLFile().load(path, exp)` for raw spectra; `pandas.read_csv(sep='\\t')` for MaxQuant; `pandas.read_parquet` for DIA-NN\n- R: `Spectra::Spectra()` \u002F `QFeatures::readQFeatures()` for raw and quantified data (MSnbase still works but is in maintenance mode)\n\nScope: this skill owns reading spectra\u002Fsearch outputs into memory, deleting decoy\u002Fcontaminant\u002Fsite-only rows, picking the correct quant column, and characterizing missingness. Format conversion (RAW -> mzML) -> peptide-identification. MaxLFQ\u002FTMT reporter quant computation -> quantification. Protein-group parsimony -> protein-inference. Normalization and imputation -> differential-abundance and expression-matrix\u002Fnormalization. OUT OF SCOPE: statistical testing, batch correction, and the actual imputation step (this skill only diagnoses the missingness so the right imputer is chosen later).\n\n## The Single Most Important Modern Insight -- Import Is Where Two Contracts Are Read and Enforced\n\n1. **A \"data import\" is never just file parsing -- it is the moment the acquisition mode's quantitative contract and its missingness structure are inherited.** DDA selects the top-N most intense precursors per cycle, and which precursors get picked is partly stochastic and abundance-biased, so the same low-abundance peptide is sampled in run A and missed in run B; this manufactures structured, left-censored MNAR missingness. DIA fragments every precursor in every window every cycle, so its (fewer) missing values are closer to MCAR. The catastrophic error this prevents: imputing a DDA matrix with a mean\u002FKNN method that assumes MCAR, which biases low-abundance proteins upward and manufactures false hits. The mode is born at acquisition and inherited at import; the missingness diagnosis made here dictates which imputation is even legitimate downstream.\n\n2. **The search engine's bookkeeping must be stripped before any number is trusted.** A proteinGroups.txt carries decoy rows (`Reverse == '+'`, `REV__` prefix in the ID) from the target-decoy FDR machinery, contaminant rows (`Potential contaminant == '+'`, `CON__` prefix), and Only-identified-by-site rows (the protein has no unmodified-peptide evidence, only a modified site). Keeping any of these leaks non-biological signal into the intensity matrix and inflates IDs. The catastrophic error: reporting differential abundance on a matrix where decoy or keratin rows survived.\n\n3. **The same proteinGroups.txt yields different biology from different columns, and a zero is not a measurement.** `Intensity` is raw summed precursor signal (not normalized, not comparable across samples for ratios). `LFQ intensity` is MaxLFQ-normalized and is the column for between-sample comparison. `iBAQ` is intensity divided by the number of observable tryptic peptides -- a within-sample molar proxy, not a between-sample quant. MaxQuant writes 0 for \"not quantified\", so log2(0) = -inf; replace 0 -> NaN before any transform. The catastrophic error: log2-transforming raw `Intensity` (or iBAQ) and reading the ratios as biology.\n\n## Tool Taxonomy\n\n| Tool \u002F method | Citation | Mechanism \u002F role | When |\n|---|---|---|---|\n| pyOpenMS `MzMLFile().load` | Chambers 2012 (ProteoWizard lineage) | Loads mzML\u002FmzXML into an MSExperiment in memory; iterate spectra by MS level | Programmatic access to raw peaks, precursor m\u002Fz, isolation windows |\n| pandas `read_csv`\u002F`read_parquet` | -- | Tabular ingest of MaxQuant TSV and DIA-NN parquet | All search-engine output tables |\n| DIA-NN report | Demichev 2020 | Long-format precursor table; `report.parquet` is the default (1.9+) and the only default (2.0) | DIA quant; pivot on `PG.MaxLFQ` after q-filtering |\n| MaxQuant `txt\u002F` outputs | Cox 2014 (MaxLFQ) | `proteinGroups.txt` (group level), `evidence.txt` (per-PSM) | DDA label-free \u002F TMT search results |\n| Spectra + QFeatures (R) | -- | Current Bioconductor raw + quantified-feature containers; `readQFeatures`, `aggregateFeatures` | R pipelines; preferred over MSnbase going forward |\n| MSnbase `readMSData` (R) | -- | On-disk raw reading; maintenance mode (route OUT to Spectra\u002FQFeatures) | Legacy R code only |\n| ThermoRawFileParser \u002F msconvert | Hulstaert 2020 \u002F Chambers 2012 | RAW -> mzML conversion (route OUT) | File conversion is peptide-identification |\n\n## Decision Tree by Scenario\n\n| Scenario | Recommended | Why |\n|---|---|---|\n| MaxQuant DDA label-free, between-sample comparison | Read `LFQ intensity` columns from proteinGroups.txt | MaxLFQ-normalized; the only MaxQuant column valid for cross-sample ratios |\n| MaxQuant, absolute\u002Fmolar abundance within one sample | Read `iBAQ` columns | iBAQ is a within-sample molar proxy; do not use across samples |\n| Need raw uncorrected signal for a custom normalization | Read `Intensity` columns, normalize yourself | `Intensity` is raw summed precursor area, not comparable as-is |\n| DIA-NN output (1.9 or 2.0) | `pd.read_parquet('report.parquet')`, filter q-values, pivot `PG.MaxLFQ` | 2.0 dropped the TSV default; q-filter before pivot or low-confidence rows leak in |\n| Raw spectra, need peaks\u002Fprecursor\u002Fisolation window | pyOpenMS `MzMLFile().load` | Programmatic peak and isolation-window access for QC and co-isolation reasoning |\n| R-based pipeline, quantified features | QFeatures `readQFeatures` + `aggregateFeatures` | Current Bioconductor; MSnbase is maintenance-only |\n| Data came from DDA, planning imputation | Diagnose missingness as MNAR -> route to left-censored imputation | DDA top-N sampling makes missingness abundance-dependent |\n| Data came from DIA, planning imputation | Treat missingness as closer to MCAR | DIA samples every precursor every cycle |\n\nDefault when uncertain: read `LFQ intensity` (MaxQuant) or `PG.MaxLFQ` after q-filtering (DIA-NN), strip Reverse\u002Fcontaminant\u002Fsite-only rows, set 0 -> NaN, then diagnose missingness before choosing an imputer.\n\n## Loading mzML\u002FmzXML with pyOpenMS\n\n**Goal:** Parse raw spectra into memory for QC, peak access, and isolation-window reasoning.\n\n**Approach:** Load into an MSExperiment (filled in place), iterate by MS level; `get_peaks()` returns a tuple of (mz, intensity) numpy arrays, and `getPrecursors()` returns a list.\n\n```python\nfrom pyopenms import MSExperiment, MzMLFile\n\nexp = MSExperiment()\nMzMLFile().load('sample.mzML', exp)  # fills exp in place; returns None\n\nfor spectrum in exp:\n    if spectrum.getMSLevel() == 1:\n        mz, intensity = spectrum.get_peaks()  # tuple of two numpy arrays\n    elif spectrum.getMSLevel() == 2:\n        precursor = spectrum.getPrecursors()[0]  # getPrecursors returns a list\n        precursor_mz = precursor.getMZ()\n        window = precursor.getIsolationWindowLowerOffset() + precursor.getIsolationWindowUpperOffset()\n```\n\n## Loading and Cleaning MaxQuant proteinGroups.txt\n\n**Goal:** Get a trustworthy log2 intensity matrix with bookkeeping rows removed and missing values represented as NaN.\n\n**Approach:** Strip Reverse\u002Fcontaminant\u002Fsite-only rows, resolve the semicolon protein-ID list to a leading ID, pick `LFQ intensity` columns, set 0 -> NaN, then log2-transform.\n\n```python\nimport pandas as pd\nimport numpy as np\n\npg = pd.read_csv('proteinGroups.txt', sep='\\t', low_memory=False)  # mixed-type cols\n\n# Flag columns hold '+' or empty string; all three are proteinGroups-only bookkeeping\nmask = (pg.get('Reverse', '') != '+') & (pg.get('Potential contaminant', '') != '+') & (pg.get('Only identified by site', '') != '+')\npg = pg[mask].copy()\n\n# Protein IDs \u002F Majority protein IDs \u002F Gene names are SEMICOLON lists; take the first (leading\u002Frazor) entry\npg['leading_protein'] = pg['Protein IDs'].str.split(';').str[0]\npg['leading_gene'] = pg['Gene names'].where(pg['Gene names'].notna(), '').str.split(';').str[0]\n\nlfq_cols = [c for c in pg.columns if c.startswith('LFQ intensity ')]  # MaxLFQ-normalized, between-sample comparable\nmatrix = pg[['leading_protein', 'leading_gene'] + lfq_cols].copy()\nmatrix[lfq_cols] = matrix[lfq_cols].replace(0, np.nan)  # MaxQuant writes 0 for missing; log2(0) = -inf\nmatrix[lfq_cols] = np.log2(matrix[lfq_cols])\n```\n\n## Loading DIA-NN report.parquet\n\n**Goal:** Reshape the long DIA-NN report into a confident protein-by-run matrix.\n\n**Approach:** Read the parquet (default since 1.9, only default in 2.0), filter precursor- AND protein-group q-values to 1% FDR BEFORE pivoting on `PG.MaxLFQ`.\n\n```python\nimport pandas as pd\n\nreport = pd.read_parquet('report.parquet')  # report.tsv dropped as default in DIA-NN 2.0\nreport = report[(report['Q.Value'] \u003C= 0.01) & (report['PG.Q.Value'] \u003C= 0.01)]  # 1% FDR before quant\n\nmatrix = report.pivot_table(index='Protein.Group', columns='Run', values='PG.MaxLFQ', aggfunc='first')\n```\n\n## Diagnosing the Missingness Contract\n\n**Goal:** Quantify the missing-value pattern so the legitimate imputation class can be chosen downstream.\n\n**Approach:** Count NaN per protein and per sample; relate the pattern to acquisition mode (DDA -> structured MNAR; DIA -> closer to MCAR). A correlation between missingness and mean abundance is the MNAR signature.\n\n```python\nimport numpy as np\n\ndef assess_missingness(matrix, sample_cols):\n    miss_per_protein = matrix[sample_cols].isna().sum(axis=1)\n    miss_per_sample = matrix[sample_cols].isna().sum(axis=0)\n    total_pct = 100 * matrix[sample_cols].isna().sum().sum() \u002F matrix[sample_cols].size\n    mean_abund = matrix[sample_cols].mean(axis=1)  # negative corr with missingness => MNAR \u002F left-censored\n    mnar_corr = mean_abund.corr(miss_per_protein)\n    return {'per_protein': miss_per_protein, 'per_sample': miss_per_sample, 'total_pct': total_pct, 'abundance_missing_corr': mnar_corr}\n```\n\n## Per-Method Failure Modes\n\n### MaxQuant wrong quant column\n\n**Trigger:** Reading `Intensity` (raw) or `iBAQ` when between-sample ratios are intended.\n**Mechanism:** `Intensity` is un-normalized summed precursor signal; `iBAQ` is a within-sample molar proxy. Neither is comparable across samples the way `LFQ intensity` is.\n**Symptom:** Ratios track total loaded protein \u002F sample depth rather than biology; fold changes shift when one sample's loading changes.\n**Fix:** Use `LFQ intensity` for cross-sample comparison; if computing custom normalization use `Intensity` and normalize explicitly (expression-matrix\u002Fnormalization).\n\n### Zero treated as a measurement\n\n**Trigger:** `np.log2` applied directly to a MaxQuant matrix still containing 0.\n**Mechanism:** MaxQuant encodes \"not quantified\" as 0; log2(0) = -inf, which then propagates into means and tests.\n**Symptom:** -inf values, NaN means, proteins silently dropped or skewed.\n**Fix:** `replace(0, np.nan)` before any transform; then diagnose missingness.\n\n### Bookkeeping rows survive\n\n**Trigger:** Loading proteinGroups.txt without filtering Reverse \u002F Potential contaminant \u002F Only identified by site.\n**Mechanism:** Decoys exist only for FDR estimation; contaminants are keratin\u002Ftrypsin\u002FBSA, not the sample; site-only groups have no unmodified-peptide quant evidence. `Only identified by site` exists only in proteinGroups.txt.\n**Symptom:** Inflated protein counts; a \"hit\" that is a decoy or keratin.\n**Fix:** Filter all three flag columns; cross-check with `REV__`\u002F`CON__` ID prefixes when joining to peptide tables. Caveat: do not delete CON__ rows blindly if a contaminant (e.g. keratin) is the protein of interest.\n\n### Razor \u002F leading protein-ID ambiguity ignored\n\n**Trigger:** Treating `Protein IDs` or `Gene names` as an atomic single value.\n**Mechanism:** These are semicolon-delimited lists; the first entry is the leading (razor) protein for the group, and `Gene names` can be blank while protein IDs are present.\n**Symptom:** Merges fail, NaN gene labels, ambiguous identity downstream.\n**Fix:** Split on `;` and take the first entry; guard `Gene names` with `.notna()`. Group parsimony details -> protein-inference.\n\n### Stale DIA-NN parsing\n\n**Trigger:** Reading `report.tsv` on DIA-NN 2.0, or pivoting before q-filtering.\n**Mechanism:** 2.0 defaults to (and only defaults to) `report.parquet`; pivoting unfiltered rows includes precursors above 1% FDR.\n**Symptom:** FileNotFoundError on report.tsv; or low-confidence quant inflating the matrix.\n**Fix:** `pd.read_parquet('report.parquet')`; filter `Q.Value \u003C= 0.01 & PG.Q.Value \u003C= 0.01` before pivoting `PG.MaxLFQ`.\n\n### MNAR imputed as MCAR\n\n**Trigger:** Mean\u002Fmedian\u002FKNN imputation on a DDA matrix.\n**Mechanism:** DDA missingness is abundance-dependent (left-censored); MCAR imputers fill missing low values with the central tendency, biasing them upward.\n**Symptom:** Low-abundance proteins gain false high values; spurious differential hits.\n**Fix:** Diagnose the abundance-missingness correlation here; route DDA to left-censored imputation (downshifted-Gaussian \u002F QRILC \u002F MinProb) in differential-abundance; DIA tolerates standard imputers.\n\n## Quantitative Thresholds\n\n| Threshold | Source | Rationale |\n|---|---|---|\n| DIA-NN import filter `Q.Value \u003C= 0.01` AND `PG.Q.Value \u003C= 0.01` | Demichev 2020; target-decoy convention | Precursor- and protein-group-level 1% FDR enforced before any quant value is used |\n| Peptide\u002Fprotein FDR 1% (q \u003C= 0.01) | Target-decoy convention | Standard ID confidence at both peptide and protein levels |\n| MaxQuant zero -> NaN | MaxQuant output convention | 0 encodes \"not quantified\"; log2(0) = -inf corrupts every transform |\n| Min peptides per protein for quant >= 2 | Community quant practice | Single-peptide (\"one-hit-wonder\") proteins are ID\u002Fquant-unreliable |\n| Valid-value filter >= 50-70% per group | Modeling choice (document per study) | Caps imputation burden; the exact cutoff is a study decision, not a universal constant |\n| Take FIRST semicolon entry as leading protein\u002Fgene | MaxQuant proteinGroups convention | The leading\u002Frazor protein is the group identifier; trailing entries are shared-peptide members |\n\n## Common Errors\n\n| Error \u002F symptom | Cause | Solution |\n|---|---|---|\n| `-inf` values after log2 | Zeros not converted to NaN | `df.replace(0, np.nan)` before `np.log2` |\n| `FileNotFoundError: report.tsv` (DIA-NN 2.0) | TSV no longer the default output | `pd.read_parquet('report.parquet')` |\n| `KeyError: 'Only identified by site'` | That column exists ONLY in proteinGroups.txt | Use `df.get('Only identified by site', '')` or guard the column lookup |\n| Mixed-type \u002F DtypeWarning on MaxQuant load | Wide TSV with mixed column types | `pd.read_csv(..., low_memory=False)` |\n| NaN gene labels break a merge | `Gene names` is a semicolon list, sometimes blank | `.where(notna(), '').str.split(';').str[0]` |\n| Ratios track loading not biology | Read `Intensity` (raw) instead of `LFQ intensity` | Use `LFQ intensity` for between-sample comparison |\n| `get_peaks()` unpacking error | Expecting a 2D array | It returns a tuple `(mz, intensity)` of two numpy arrays |\n\n## References\n\n- Cox J, Hein MY, Luber CA, Paron I, Nagaraj N, Mann M. 2014. Accurate proteome-wide label-free quantification by delayed normalization and maximal peptide ratio extraction, termed MaxLFQ. *Mol Cell Proteomics* 13(9):2513-2526.\n- Demichev V, Messner CB, Vernardis SI, Lilley KS, Ralser M. 2020. DIA-NN: neural networks and interference correction enable deep proteome coverage in high throughput. *Nat Methods* 17(1):41-44.\n- Chambers MC, Maclean B, Burke R, et al. 2012. A cross-platform toolkit for mass spectrometry and proteomics. *Nat Biotechnol* 30(10):918-920.\n- Hulstaert N, Shofstahl J, Sachsenberg T, et al. 2020. ThermoRawFileParser: modular, scalable, and cross-platform RAW file conversion. *J Proteome Res* 19(1):537-542.\n\n## Related Skills\n\n- peptide-identification - search raw spectra and convert vendor RAW to mzML\n- quantification - compute MaxLFQ and TMT reporter-ion quantities from imported data\n- protein-inference - resolve protein-group parsimony and razor assignment\n- differential-abundance - normalize, impute (per the missingness diagnosis), and test\n- proteomics-qc - assess run-level identification and quant quality\n- dia-analysis - run DIA-NN to produce the report this skill imports\n- expression-matrix\u002Fnormalization - general intensity-matrix normalization patterns\n- workflows\u002Fproteomics-pipeline - end-to-end pipeline that begins with this import step\n",{"name":5,"source":24,"license":15,"tool_type":25,"description":7,"imported_at":26,"source_path":27,"primary_tool":28,"source_repository":29},"https:\u002F\u002Fgithub.com\u002FGPTomics\u002FbioSkills","mixed","2026-09-20","proteomics\u002Fdata-import\u002FSKILL.md","pyOpenMS","GPTomics\u002FbioSkills","main","2d1750717747097b602acf30bd09a7cbd4b4c6ea","awesome_skills\u002Fbiology\u002Fbio-proteomics-data-import","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous","2026-11-09 11:45:36",{"source":8,"skillName":5,"commands":37,"prompt":47,"zipAvailable":48},[38,41,44],{"tool":39,"command":40},"npx","npx skills add Mr-Nobody-Anonymous\u002FAll-skills --skill bio-proteomics-data-import",{"tool":42,"command":43},"bunx","bunx skills add Mr-Nobody-Anonymous\u002FAll-skills --skill bio-proteomics-data-import",{"tool":45,"command":46},"pnpm","pnpm dlx skills add Mr-Nobody-Anonymous\u002FAll-skills --skill bio-proteomics-data-import","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fbiology\u002Fbio-proteomics-data-import\u002FSKILL.md",true,[50,56],{"path":51,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":54,"sourceHtmlUrl":55},"examples\u002Fload_maxquant.py","SCRIPT","脚本",3864,"https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fbiology\u002Fbio-proteomics-data-import\u002Fexamples\u002Fload_maxquant.py",{"path":57,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":60,"sourceHtmlUrl":61},"usage-guide.md","REFERENCE","参考文档",4802,"https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fbiology\u002Fbio-proteomics-data-import\u002Fusage-guide.md",[63,71,78,85,92,99],{"skillNo":64,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":67,"categories":68,"keywords":69,"publishedAt":70},"01M7FFFZPC2025JQQHPHDX154J","dei-assessment","Assess diversity, equity, and inclusion with metrics. TRIGGERS - Use when user needs help with dei-assessment related tasks.","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fhr-people\u002Fdei-assessment\u002FSKILL.md",[],[],"2026-11-16 10:05:34",{"skillNo":72,"name":73,"nameZh":6,"description":74,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":75,"categories":76,"keywords":77,"publishedAt":70},"01M7FFFWN8H4WA7A6Y398QKB6J","dei-action-plan","Create DEI action plans with goals and initiatives. TRIGGERS - Use when user needs help with dei-action-plan related tasks.","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fhr-people\u002Fdei-action-plan\u002FSKILL.md",[],[],{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":70},"01M7FFFSHTM0QZRVZ2NRJTJ0PY","culture-fit-assessment","Design culture add assessment processes. TRIGGERS - Use when user needs help with culture-fit-assessment related tasks.","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fhr-people\u002Fculture-fit-assessment\u002FSKILL.md",[],[],{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":91,"publishedAt":70},"01M7FFFMA12RKEQA38KHJSXXPS","cowork-to-code-bridge","Use an already-installed, independently verified cowork-to-code bridge to run narrowly approved actions on the user's own macOS, Linux, or WSL2 machine through a local file queue.","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fhr-people\u002Fcowork-to-code-bridge\u002FSKILL.md",[],[],{"skillNo":93,"name":94,"nameZh":6,"description":95,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":70},"01M7FFFH1F6M8YRGW5V487GH1K","counter-offer-response","Draft counter-offer responses for retention. TRIGGERS - Use when user needs help with counter-offer-response related tasks.","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fhr-people\u002Fcounter-offer-response\u002FSKILL.md",[],[],{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":103,"categories":104,"keywords":105,"publishedAt":70},"01M7FFFDGEHV8Y7J9N40HZXW52","continuous-feedback-system","Design continuous feedback systems. TRIGGERS - Use when user needs help with continuous-feedback-system related tasks.","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fhr-people\u002Fcontinuous-feedback-system\u002FSKILL.md",[],[]]