[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7ED75AA8PDQ4A9CDG2J577H":3,"skill-same:01M7ED75AA8PDQ4A9CDG2J577H":62},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":30,"commitSha":31,"skillDirectory":32,"forksCount":14,"repositoryHtmlUrl":33,"repositoryHomepageUrl":33,"ownerHtmlUrl":34,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":35,"install":36,"resources":49},"01M7ED75AA8PDQ4A9CDG2J577H","bio-expression-matrix-sparse-handling",null,"Stores and operates on sparse expression matrices for single-cell and large bulk RNA-seq, covering dgCMatrix\u002FdgRMatrix\u002FdgTMatrix when-each-is-fast, the dgCMatrix (CSC, R) \u003C-> CSR (Python) implicit transpose, AnnData (cells-rows) \u003C-> SingleCellExperiment (cells-cols) orientation flip, HDF5\u002Fh5ad vs Zarr cloud-native shift, HDF5SummarizedExperiment + DelayedArray for out-of-memory bulk, scanpy backed mode for large h5ad, the ~10-15% density crossover where dense beats sparse, 10X format proliferation (MTX vs CellRanger H5 vs h5ad), the dense-conversion memory blow-up, and Dask + Zarr for consortium-scale matrices. Use when choosing sparse format, working with single-cell-sized matrices, importing\u002Fexporting 10X, debugging R\u002FPython interop transposes, processing matrices too large for RAM, or building cloud-native pipelines.","Mr-Nobody-Anonymous\u002FAll-skills","Mr-Nobody-Anonymous","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F184253119?v=4","01M7AEXFJHKTYQW5R0ZCQQTWFV","anonymous","01M7C00M3GQR4QKP9GZA42GFAK",0,"MIT","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fbiology\u002Fbio-expression-matrix-sparse-handling\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: bio-expression-matrix-sparse-handling\ndescription: Stores and operates on sparse expression matrices for single-cell and large bulk RNA-seq, covering dgCMatrix\u002FdgRMatrix\u002FdgTMatrix when-each-is-fast, the dgCMatrix (CSC, R) \u003C-> CSR (Python) implicit transpose, AnnData (cells-rows) \u003C-> SingleCellExperiment (cells-cols) orientation flip, HDF5\u002Fh5ad vs Zarr cloud-native shift, HDF5SummarizedExperiment + DelayedArray for out-of-memory bulk, scanpy backed mode for large h5ad, the ~10-15% density crossover where dense beats sparse, 10X format proliferation (MTX vs CellRanger H5 vs h5ad), the dense-conversion memory blow-up, and Dask + Zarr for consortium-scale matrices. Use when choosing sparse format, working with single-cell-sized matrices, importing\u002Fexporting 10X, debugging R\u002FPython interop transposes, processing matrices too large for RAM, or building cloud-native pipelines.\ntool_type: python\nprimary_tool: scipy.sparse\nsource: \"https:\u002F\u002Fgithub.com\u002FGPTomics\u002FbioSkills\"\nsource_repository: \"GPTomics\u002FbioSkills\"\nsource_path: \"expression-matrix\u002Fsparse-handling\u002FSKILL.md\"\nlicense: \"MIT\"\nimported_at: \"2026-09-20\"\n---\n\n## Version Compatibility\n\nReference examples tested with: numpy 1.26+, scipy 1.12+, pandas 2.2+, anndata 0.10+, scanpy 1.10+, Matrix R package 1.6+, HDF5Array 1.30+ (Bioconductor), DelayedArray 0.28+, zellkonverter 1.12+, zarr-python 2.18+, dask 2024.1+\n\nBefore using code patterns, verify installed versions match. If versions differ:\n- Python: `pip show \u003Cpackage>` then `help(module.function)` to check signatures\n- R: `packageVersion('\u003Cpkg>')` then `?function_name` to verify parameters\n\nIf code throws ImportError, AttributeError, or TypeError, introspect the installed\npackage and adapt the example to match the actual API rather than retrying.\n\n# Sparse Matrix Handling\n\n**\"Store \u002F compute on a single-cell matrix without blowing up memory\"** -> Pick the sparse format that matches the access pattern (CSC for column ops, CSR for row ops), respect the R\u002FPython convention difference (Bioconductor stores cells in columns; AnnData stores cells in rows), and use HDF5\u002FZarr backed mode for matrices too large for RAM.\n\n## The Single Most Important Modern Insight -- The R\u002FPython interop transpose is silent and catastrophic\n\nR Bioconductor (Seurat, SingleCellExperiment) stores cells in COLUMNS and uses dgCMatrix (Column-Compressed Sparse). Python scverse (AnnData, scanpy) stores cells in ROWS and defaults to CSR (Compressed Sparse Row). Round-tripping with `anndata2ri`, `zellkonverter`, or `rpy2` triggers a TRANSPOSE under the hood -- once per direction. Two flips silently cancel. A debugging session that converts back and forth multiple times can end up with mysteriously transposed data and no error.\n\n| Conversion | Implicit transpose |\n|------------|-------------------|\n| Python CSR -> R dgCMatrix | YES (rows \u003C-> cols) |\n| AnnData `.X` (cells x genes) -> SingleCellExperiment counts (genes x cells) | YES |\n| Seurat `@assays$RNA@counts` (genes x cells) -> AnnData `.X` (cells x genes) | YES |\n| `scipy.sparse.csr_matrix(dense)` | None (just format conversion) |\n| `csr.tocsc()` \u002F `csc.tocsr()` | None (semantically same matrix, different layout) |\n\nThe safe pattern for one-shot conversions: file-based intermediate. `adata.write('file.h5ad')` then `zellkonverter::readH5AD('file.h5ad')` -- avoids the in-memory rpy2\u002Freticulate gymnastics, and the file roundtrip makes orientation explicit.\n\nFor a 1M-cell single-cell matrix, the implicit transpose is non-trivial -- minutes of wall time and a temporary memory peak roughly equal to nnz x 12 bytes (CSC) or x 16 bytes (CSR with int64). Avoid unnecessary transposes by aligning the format to the consumer.\n\nTwo adjacent insights:\n\n1. **Sparse becomes inefficient above ~10-15% density.** Sparse iteration has cache-unfriendly indirection; dense iteration is sequential. Above ~10-15% nonzero, dense is often faster for most operations even though it uses more memory.\n2. **AnnData backed mode quietly differs from in-memory in important ways.** `sc.read_h5ad('file.h5ad', backed='r')` returns an AnnData where `.X` is a wrapped HDF5 dataset, read-only. Many scanpy functions silently load to memory; some functions error or hang on backed mode.\n\n## Algorithmic Taxonomy\n\n| Format | Layout | Fast for | Slow for |\n|--------|--------|----------|----------|\n| dgCMatrix (CSC, R) | i (row indices), p (col pointers), x (values) | Column slicing; per-cell ops in single-cell (cells in cols); matrix-vector with col vector | Row slicing |\n| dgRMatrix (CSR, R) | j (col indices), p (row pointers), x (values) | Row slicing; per-gene ops when genes in rows | Column slicing |\n| dgTMatrix (COO, R triplet) | i, j, x | Random insertion when building; reading MTX | Most operations -- convert to dgCMatrix after build |\n| scipy `csc_matrix` | Same as dgCMatrix | Column ops in Python | Row ops |\n| scipy `csr_matrix` | Same as dgRMatrix | Row ops (NumPy convention); matmul; sklearn defaults | Column ops |\n| scipy `coo_matrix` | Triplet (i, j, data) | Construction; MTX I\u002FO | Most ops -- convert after build |\n| HDF5 (h5ad, h5) | Single-file binary chunked | Random access via chunks; compression; widely supported | Cloud \u002F parallel writes |\n| Zarr | Chunked array, per-chunk file (or S3 object) | Cloud-native; parallel writes; Dask integration | Single-file simplicity |\n| DelayedArray + HDF5Array | Bioc lazy evaluation over HDF5 | Out-of-memory bulk ops in R | Speed of in-memory |\n\n## Decision Tree by Scenario\n\n| Scenario | Recommended approach |\n|----------|---------------------|\n| Single-cell (>10k cells), per-cell ops | dgCMatrix in R (Bioconductor); CSR `adata.X` in Python (scanpy) |\n| Bulk RNA-seq (60-80% density) | Dense -- sparse overhead exceeds benefit |\n| Single-cell pseudobulk (after donor aggregation) | Dense -- now 60-80% density typically |\n| 1M+ cells, can't fit in RAM | scanpy backed mode OR HDF5SummarizedExperiment + DelayedArray |\n| TCGA + GTEx + recount3 scale (100k+ samples) | HDF5Array \u002F Zarr + Dask |\n| 10X CellRanger 3.0+ output | `sc.read_10x_h5()` or `Read10X_h5()` -- the .h5 is faster than the .mtx triplet |\n| Cloud-native (anndata on S3, dask compute) | Zarr |\n| Local workstation, single-machine | HDF5 -- faster, more widely supported |\n| Building sparse matrix incrementally | COO (dgTMatrix \u002F coo_matrix); convert to CSC\u002FCSR after |\n| R \u003C-> Python conversion | File-based intermediate (`adata.write` then `zellkonverter::readH5AD`); aware of the transpose |\n\n## Check Sparsity\n\n**Goal:** Decide whether sparse is the right format given the actual data density.\n\n**Approach:** Compute nonzero fraction; rule of thumb is sparse > ~85% (single-cell). Below that, dense often wins.\n\n```python\nimport numpy as np\nimport scipy.sparse as sp\n\ndef sparsity(m):\n    if sp.issparse(m):\n        return 1 - m.nnz \u002F (m.shape[0] * m.shape[1])\n    return (m == 0).mean()\n\ns = sparsity(adata.X)\nprint(f'{s:.1%} sparse')\n```\n\nMemory math:\n\n| Data | Format | Bytes |\n|------|--------|-------|\n| 30k x 100k single-cell matrix, 5% density | dgCMatrix | (5% * 3e9) * 12 bytes ~= 1.8 GB |\n| Same | Dense double | 24 GB |\n| 60k x 100k bulk, 70% density | dgCMatrix | 50 GB (worse than dense!) |\n| Same | Dense double | 48 GB |\n\nFor single-cell (typically 90-95% sparse), sparse is essential. For bulk RNA-seq (typically 60-80% density), dense is faster and not appreciably larger.\n\n## dgCMatrix \u002F scipy CSC \u002F CSR\n\n**Goal:** Construct, query, and convert sparse matrices in the format matching the consumer's expected layout.\n\n**Approach:** `Matrix::sparseMatrix(i, j, x, dims=...)` (R) or `scipy.sparse.csr_matrix((data, (i, j)))` (Python); preserve row\u002Fcolumn names; convert layout (CSC \u003C-> CSR) without changing semantics.\n\n```python\nimport scipy.sparse as sp\nimport pandas as pd\n\ndense_df = pd.read_csv('counts.csv', index_col=0)\nsparse_csr = sp.csr_matrix(dense_df.values)\nsparse_csc = sp.csc_matrix(dense_df.values)\n\ngene_names = dense_df.index.tolist()\nsample_names = dense_df.columns.tolist()\n\nsparse_csr.tocsc()\nsparse_csc.tocsr()\n```\n\n```r\nlibrary(Matrix)\n\ndense_mat \u003C- as.matrix(read.csv('counts.csv', row.names = 1))\nsparse_dgc \u003C- as(dense_mat, 'CsparseMatrix')\n\nclass(sparse_dgc)\nrownames(sparse_dgc) \u003C- rownames(dense_mat)\ncolnames(sparse_dgc) \u003C- colnames(dense_mat)\n```\n\ndgTMatrix is best for building matrices incrementally (reading MTX, parsing per-row); convert to dgCMatrix for downstream ops:\n\n```r\nmat_t \u003C- as(triplet_data, 'TsparseMatrix')\nmat_c \u003C- as(mat_t, 'CsparseMatrix')\n```\n\n## HDF5 vs Zarr -- The Cloud-Native Shift\n\nHDF5 (Hierarchical Data Format 5): hierarchical, single-file binary. Random access via chunks; supports compression (gzip, blosc, lz4). On-disk format for AnnData `.h5ad`, MuData `.h5mu`, 10x Genomics `.h5`, HDF5SummarizedExperiment.\n\nZarr: cloud-native, chunked array storage. Each chunk is a separate file (or S3 object). Parallel-write friendly; splittable by Dask. Format used by recent AnnData (`anndata.write_zarr`), SpatialData, and large-cohort consortia.\n\n| Criterion | HDF5 | Zarr |\n|-----------|------|------|\n| File structure | Single binary file | Directory of chunk files |\n| Parallel writes | Limited (process-level locks) | Native |\n| S3 \u002F cloud object storage | Workarounds (h5cloud); often slow | Native; first-class |\n| Compression options | gzip, blosc, lz4, szip | gzip, blosc, lz4, zstd, custom |\n| Local workstation speed | Faster | Slightly slower (many small files) |\n| Wide ecosystem support | Yes (mature) | Growing; modern scverse |\n\nFor local workstation work, HDF5 is faster and more widely supported. For cloud-mounted analysis (anndata on S3 with dask-distributed compute), Zarr wins because of object-storage friendliness.\n\n## HDF5SummarizedExperiment + DelayedArray (Bioconductor)\n\n**Goal:** Work with bulk SummarizedExperiment objects too large to fit in RAM by keeping the matrix on disk.\n\n**Approach:** `HDF5Array` wraps an HDF5 dataset as a `DelayedArray`. Subsetting builds a delayed operation tree -- no I\u002FO until realization. `DelayedMatrixStats` provides delayed-friendly stat functions.\n\n```r\nlibrary(HDF5Array)\nlibrary(SummarizedExperiment)\nlibrary(DelayedMatrixStats)\n\nse \u003C- loadHDF5SummarizedExperiment('saved_se_dir')\n\ns \u003C- se[1:1000, 1:50]\nrow_means \u003C- rowMeans2(assay(se))\n\nsaveHDF5SummarizedExperiment(se, 'saved_se_dir', replace = TRUE)\n```\n\nThe `assay(se)` returns a DelayedMatrix backed by HDF5. DESeq2, edgeR, limma have varying levels of DelayedArray support; consult package docs before assuming all ops work in delayed mode.\n\nFor TCGA + GTEx + recount3 scale (100k+ samples, 60k genes), a dense matrix is ~48 GB (double); dgCMatrix at 70% density is ~50 GB (sparse loses). HDF5Array + chunk-aware ops keeps memory at whatever-fits-in-RAM.\n\n## scanpy Backed Mode\n\n**Goal:** Work with h5ad files too large for memory by loading only accessed slices on demand.\n\n**Approach:** `sc.read_h5ad(..., backed='r')` returns an AnnData with `.X` as a wrapped HDF5 dataset. Subset operations are lazy; `.to_memory()` realizes.\n\n```python\nimport scanpy as sc\n\nadata = sc.read_h5ad('large_dataset.h5ad', backed='r')\nprint(f'Shape: {adata.shape}, X type: {type(adata.X)}')\n\nt_cells = adata[adata.obs['cell_type'] == 'T_cell', :].to_memory()\n```\n\nLimitations:\n- `.X` is read-only in `backed='r'`. Use `backed='r+'` for in-place updates, but only `.X` updates supported.\n- `.obs` and `.var` are fully loaded -- only `.X` supports backed access.\n- Very large sparse h5ad (>35 GB) can still cause memory issues even in backed mode (anndata library overhead).\n- Many scanpy functions internally load to memory; check `?function` docs for backed compatibility.\n- Functions like `sc.tl.pca`, `sc.pp.neighbors` typically require in-memory; subset first with `.to_memory()`.\n\nFor datasets too large for backed mode, process in chunks:\n\n```python\nimport anndata as ad\n\ndef process_in_chunks(h5ad_path, chunk_size=10000, func=None):\n    adata = sc.read_h5ad(h5ad_path, backed='r')\n    n_cells = adata.shape[0]\n    results = []\n    for start in range(0, n_cells, chunk_size):\n        end = min(start + chunk_size, n_cells)\n        chunk = adata[start:end].to_memory()\n        if func:\n            chunk = func(chunk)\n        results.append(chunk)\n    return ad.concat(results)\n```\n\n## 10X Genomics Format Proliferation\n\n| Format | Files | Notes |\n|--------|-------|-------|\n| MTX (pre-CellRanger 3.0) | `matrix.mtx` + `barcodes.tsv` + `features.tsv` (or `genes.tsv`) | Triplet format; slow to read for large matrices |\n| H5 (CellRanger 3.0+) | `filtered_feature_bc_matrix.h5` | HDF5 with `\u002Fmatrix\u002Fdata`, `\u002Fmatrix\u002Findices`, `\u002Fmatrix\u002Findptr`, `\u002Fmatrix\u002Fshape`; single file, fast |\n| H5AD | `data.h5ad` | AnnData; convert on import |\n| kallisto|bustools output | `output.bus` + barcode and gene mappings | `BUSpaRse` \u002F `kb-python` |\n\n```python\nimport scanpy as sc\n\nadata = sc.read_10x_h5('filtered_feature_bc_matrix.h5')\nadata = sc.read_10x_mtx('filtered_feature_bc_matrix\u002F')\n```\n\n```r\nlibrary(Seurat)\nmat \u003C- Read10X_h5('filtered_feature_bc_matrix.h5')\nmat \u003C- Read10X(data.dir = 'filtered_feature_bc_matrix\u002F')\n\nlibrary(DropletUtils)\nsce \u003C- read10xCounts('filtered_feature_bc_matrix\u002F')\n```\n\nFor 10X output, prefer the `.h5` over the MTX triplet -- typically 5-10x faster for large matrices.\n\n## Dense Conversion -- The Memory Blow-Up\n\n`adata.X.toarray()` (Python) or `as.matrix(seurat_obj@assays$RNA@counts)` (R) on a 30k x 100k single-cell matrix instantiates a ~24 GB dense double array. Common triggers:\n\n- Passing sparse to a function that internally calls `as.matrix()` (older R `cor()` implementations).\n- Heatmap functions (`pheatmap`, `ComplexHeatmap`) that require dense.\n- ML libraries with no sparse support (some sklearn models; XGBoost requires specific sparse API).\n- Plotting functions (`plot()`, `ggplot2`) called on the full matrix.\n\nDefensive pattern: subset to a manageable gene\u002Fcell set BEFORE dense conversion.\n\nFor per-cell PCA-style ops, use sparse-aware solvers:\n\n```python\nfrom scipy.sparse.linalg import svds\nU, s, Vt = svds(adata.X, k=50)\n```\n\n```r\nlibrary(irlba)\nsvd_res \u003C- irlba(sparse_mat, nv = 50)\n```\n\n`irlba` (R) and `scipy.sparse.linalg.svds` (Python) compute truncated SVD without densifying.\n\n## SCE vs AnnData vs MuData -- Where Bulk Fits\n\n| Container | Library | Cells\u002Fsamples | Multi-modal | Bulk fit |\n|-----------|---------|---------------|-------------|----------|\n| SummarizedExperiment \u002F RangedSummarizedExperiment | Bioconductor | n\u002Fa; bulk | No | YES -- standard for DESeq2\u002FedgeR\u002Flimma bulk |\n| SingleCellExperiment (Amezquita 2020) | Bioconductor | cells in cols | Via `altExps` | scRNA-seq with spike-ins, ADT |\n| AnnData | scverse\u002FPython | cells in rows | Via layers | scRNA-seq; bulk is unusual |\n| MuData | scverse\u002FPython | cells in rows | Yes, multiple AnnData | Multi-modal scRNA + ATAC + protein |\n| MultiAssayExperiment | Bioconductor | samples | Yes | R-side multi-modal analog |\n\nBulk RNA-seq rarely uses AnnData -- it shines on the single-cell dimensionality reduction \u002F neighbors \u002F clustering machinery. For bulk in R, use SummarizedExperiment. For bulk in Python, a tidy DataFrame + numpy array is usually sufficient.\n\n## Dask + Zarr for Consortium-Scale Matrices\n\nFor TCGA + GTEx + recount3 (Wilks 2021 *Genome Biol* 22:323) or pancancer assemblies:\n\n```python\nimport zarr\nimport dask.array as da\n\nz = zarr.open('counts.zarr', mode='r')\nda_arr = da.from_zarr(z)\n\ncol_sums = da_arr.sum(axis=0).compute()\nfiltered = da_arr[da_arr.sum(axis=1) > 100, :]\n```\n\n```python\nimport anndata as ad\nadata_disk = ad.read_zarr('large_data.zarr')\n```\n\nFor R: `HDF5Array` + `DelayedArray` is the equivalent, but R doesn't have a true Dask analog. BiocParallel can parallelize chunks, but lazy planning is more manual.\n\n## Sparse Operations\n\n```python\nimport numpy as np\nimport scipy.sparse as sp\n\nrow_sums = np.array(sparse_matrix.sum(axis=1)).flatten()\ncol_sums = np.array(sparse_matrix.sum(axis=0)).flatten()\n\nkeep_rows = row_sums > 10\nsparse_filt = sparse_matrix[keep_rows, :]\n\nsparse_log = sparse_matrix.copy()\nsparse_log.data = np.log1p(sparse_log.data)\n```\n\nSubsetting: select genes (rows) or samples (cols) by index:\n\n```python\ngene_idx = [gene_names.index(g) for g in ['TP53', 'BRCA1', 'MYC'] if g in gene_names]\nsubset = sparse_matrix[gene_idx, :]\n```\n\n## CPM Normalization on Sparse\n\n**Goal:** Apply CPM normalization without densifying.\n\n**Approach:** Compute library sizes from column sums; broadcast scaling factors with sparse multiply for CPM; transform only the nonzero data array in-place with log1p.\n\n```python\nimport numpy as np\nimport scipy.sparse as sp\n\ndef normalize_sparse_cpm(sparse_matrix):\n    lib_sizes = np.array(sparse_matrix.sum(axis=0)).flatten()\n    scaling = 1e6 \u002F lib_sizes\n    return sparse_matrix.multiply(scaling)\n\ndef log1p_inplace(sparse_matrix):\n    out = sparse_matrix.copy()\n    out.data = np.log1p(out.data)\n    return out\n\ncpm = normalize_sparse_cpm(adata.X)\nlog_cpm = log1p_inplace(cpm)\n```\n\nAfter log-transformation, sparsity is PRESERVED (log1p(0) = 0). After CPM with pseudocount, zeros become nonzero -- check sparsity and convert to dense if density drops below ~15%.\n\n## Save \u002F Load Sparse Matrices\n\n```python\nimport scipy.sparse as sp\nimport numpy as np\n\nsp.save_npz('counts_sparse.npz', sparse_matrix)\nloaded = sp.load_npz('counts_sparse.npz')\n\nnp.savez('counts_with_meta.npz',\n    data    = sparse_matrix.data,\n    indices = sparse_matrix.indices,\n    indptr  = sparse_matrix.indptr,\n    shape   = sparse_matrix.shape,\n    genes   = np.array(gene_names),\n    samples = np.array(sample_names))\n```\n\nFor interop and durable storage, prefer h5ad or zarr:\n\n```python\nadata.write_h5ad('counts.h5ad')\nadata.write_zarr('counts.zarr')\n```\n\n## Per-Method Failure Modes\n\n### Implicit transpose in R\u002FPython conversion\n\n**Trigger:** AnnData with cells in rows passed to a SingleCellExperiment workflow that expects cells in cols; downstream `colSums` returns gene-level totals.\n\n**Mechanism:** AnnData stores cells in rows; SCE in cols. The conversion auto-transposes ONCE per direction; two roundtrips silently restore.\n\n**Symptom:** Per-cell stats look like per-gene stats; QC plots have wrong axes.\n\n**Fix:** Use file-based intermediate (`adata.write('file.h5ad')`; `zellkonverter::readH5AD('file.h5ad')`). Always verify dimensions and orientation after conversion.\n\n### Dense conversion blew up memory\n\n**Trigger:** `as.matrix(seurat_obj@assays$RNA@counts)` on a 100k-cell dataset; R session crashes with OOM.\n\n**Mechanism:** 100k cells x 30k genes = 3e9 entries; double precision = 24 GB.\n\n**Symptom:** R session killed; \"cannot allocate vector of size N GB\".\n\n**Fix:** Don't densify the full matrix. Subset to genes\u002Fcells of interest first. For dimensionality reduction, use `irlba::irlba()` (sparse SVD).\n\n### scanpy backed mode silently loaded to memory\n\n**Trigger:** `adata = sc.read_h5ad(path, backed='r')` then `sc.tl.pca(adata)`; memory spikes to dense-equivalent.\n\n**Mechanism:** Many scanpy functions internally call `.to_memory()` because they cannot operate on backed mode. `sc.tl.pca`, `sc.pp.neighbors`, `sc.tl.umap` all materialize.\n\n**Symptom:** OOM despite backed mode.\n\n**Fix:** Subset first (`adata[mask].to_memory()`), then operate. Or use a streaming-aware alternative (Dask + Zarr).\n\n### Sparse stored where dense would be faster\n\n**Trigger:** Bulk RNA-seq with 70% density stored as dgCMatrix; per-gene `rowVars` is slow.\n\n**Mechanism:** Sparse iteration has cache-unfriendly indirection; above ~10-15% density, dense wins.\n\n**Symptom:** Operations notably slower than expected; profiler shows time in sparse indexing.\n\n**Fix:** Convert to dense for the hot path: `as.matrix(sparse_mat)` (R) or `sparse_matrix.toarray()` (Python). Memory may go up but speed improves substantially.\n\n### 10X MTX read is slow\n\n**Trigger:** Reading a 100k-cell 10X dataset via the MTX three-file format; takes 10+ minutes.\n\n**Mechanism:** MTX is a text format; parsing is slow for large matrices.\n\n**Symptom:** Long load times; user kills the process before completion.\n\n**Fix:** Use the CellRanger H5 (`.h5`) instead -- typically 5-10x faster.\n\n## Common errors\n\n| Error \u002F symptom | Cause | Fix |\n|-----------------|-------|-----|\n| `cannot allocate vector of size N GB` | Implicit dense conversion | Subset first; use sparse-aware solver (irlba) |\n| Sparse-dense arithmetic returns `numpy.matrix` | Deprecated NumPy type from sparse+dense | `np.asarray(sparse + dense)` to force ndarray |\n| `KeyError: '_index'` reading h5ad | anndata version mismatch | Update anndata; or `sc.read_h5ad(..., backed=None)` |\n| Empty rows\u002Fcols after sparse subset | Subset removed all data | Verify the index list; cross-check sample\u002Fgene names |\n| Backed mode AnnData crash on `sc.tl.umap` | Function not backed-compatible | `.to_memory()` on the subset first |\n| Per-cell totals look wrong after R\u003C->Python conversion | Implicit transpose | Verify dimensions; use file-based intermediate |\n| CSR (Python) \u003C-> dgCMatrix (R) treated as same | Convention difference | They're transposes of each other; verify shape and a known cell-gene pair |\n\n## References\n\n- Amezquita RA, Lun ATL, Becht E et al. 2020. Orchestrating single-cell analysis with Bioconductor. *Nat Methods* 17:137-145. doi:10.1038\u002Fs41592-019-0654-x\n- Wolf FA, Angerer P, Theis FJ. 2018. SCANPY: large-scale single-cell gene expression data analysis. *Genome Biol* 19:15. doi:10.1186\u002Fs13059-017-1382-0\n- Wilks C et al. 2021. recount3: summaries and queries for large-scale RNA-seq expression and splicing. *Genome Biol* 22:323. doi:10.1186\u002Fs13059-021-02533-6\n- Bates D, Maechler M. 2023. Matrix: Sparse and Dense Matrix Classes and Methods. R package version 1.6-x.\n- Pages H et al. 2020. HDF5Array: HDF5 backend for DelayedArray objects. Bioconductor package.\n- Miles A et al. 2020. zarr-python. Python package documentation.\n- Rocklin M. 2015. Dask: Parallel Computation with Blocked algorithms and Task Scheduling. *Proc Python Sci Conf.* (canonical Dask reference)\n- Lachmann A et al. 2018. Massive mining of publicly available RNA-seq data from human and mouse. *Nat Commun* 9:1366. doi:10.1038\u002Fs41467-018-03751-6\n\n## Related Skills\n\n- counts-ingest - Reading 10X formats; building sparse matrices from quantification output\n- gene-id-mapping - Var (gene) metadata in AnnData\n- metadata-joins - Obs (sample) metadata in AnnData\n- normalization - log1p and CPM patterns on sparse\n- differential-expression\u002Fdeseq2-basics - Pseudobulk aggregation makes dense\n- single-cell\u002Fdata-io - Single-cell file format ecosystem\n- single-cell\u002Fpreprocessing - Standard single-cell sparse pipeline\n","## Version Compatibility\n\nReference examples tested with: numpy 1.26+, scipy 1.12+, pandas 2.2+, anndata 0.10+, scanpy 1.10+, Matrix R package 1.6+, HDF5Array 1.30+ (Bioconductor), DelayedArray 0.28+, zellkonverter 1.12+, zarr-python 2.18+, dask 2024.1+\n\nBefore using code patterns, verify installed versions match. If versions differ:\n- Python: `pip show \u003Cpackage>` then `help(module.function)` to check signatures\n- R: `packageVersion('\u003Cpkg>')` then `?function_name` to verify parameters\n\nIf code throws ImportError, AttributeError, or TypeError, introspect the installed\npackage and adapt the example to match the actual API rather than retrying.\n\n# Sparse Matrix Handling\n\n**\"Store \u002F compute on a single-cell matrix without blowing up memory\"** -> Pick the sparse format that matches the access pattern (CSC for column ops, CSR for row ops), respect the R\u002FPython convention difference (Bioconductor stores cells in columns; AnnData stores cells in rows), and use HDF5\u002FZarr backed mode for matrices too large for RAM.\n\n## The Single Most Important Modern Insight -- The R\u002FPython interop transpose is silent and catastrophic\n\nR Bioconductor (Seurat, SingleCellExperiment) stores cells in COLUMNS and uses dgCMatrix (Column-Compressed Sparse). Python scverse (AnnData, scanpy) stores cells in ROWS and defaults to CSR (Compressed Sparse Row). Round-tripping with `anndata2ri`, `zellkonverter`, or `rpy2` triggers a TRANSPOSE under the hood -- once per direction. Two flips silently cancel. A debugging session that converts back and forth multiple times can end up with mysteriously transposed data and no error.\n\n| Conversion | Implicit transpose |\n|------------|-------------------|\n| Python CSR -> R dgCMatrix | YES (rows \u003C-> cols) |\n| AnnData `.X` (cells x genes) -> SingleCellExperiment counts (genes x cells) | YES |\n| Seurat `@assays$RNA@counts` (genes x cells) -> AnnData `.X` (cells x genes) | YES |\n| `scipy.sparse.csr_matrix(dense)` | None (just format conversion) |\n| `csr.tocsc()` \u002F `csc.tocsr()` | None (semantically same matrix, different layout) |\n\nThe safe pattern for one-shot conversions: file-based intermediate. `adata.write('file.h5ad')` then `zellkonverter::readH5AD('file.h5ad')` -- avoids the in-memory rpy2\u002Freticulate gymnastics, and the file roundtrip makes orientation explicit.\n\nFor a 1M-cell single-cell matrix, the implicit transpose is non-trivial -- minutes of wall time and a temporary memory peak roughly equal to nnz x 12 bytes (CSC) or x 16 bytes (CSR with int64). Avoid unnecessary transposes by aligning the format to the consumer.\n\nTwo adjacent insights:\n\n1. **Sparse becomes inefficient above ~10-15% density.** Sparse iteration has cache-unfriendly indirection; dense iteration is sequential. Above ~10-15% nonzero, dense is often faster for most operations even though it uses more memory.\n2. **AnnData backed mode quietly differs from in-memory in important ways.** `sc.read_h5ad('file.h5ad', backed='r')` returns an AnnData where `.X` is a wrapped HDF5 dataset, read-only. Many scanpy functions silently load to memory; some functions error or hang on backed mode.\n\n## Algorithmic Taxonomy\n\n| Format | Layout | Fast for | Slow for |\n|--------|--------|----------|----------|\n| dgCMatrix (CSC, R) | i (row indices), p (col pointers), x (values) | Column slicing; per-cell ops in single-cell (cells in cols); matrix-vector with col vector | Row slicing |\n| dgRMatrix (CSR, R) | j (col indices), p (row pointers), x (values) | Row slicing; per-gene ops when genes in rows | Column slicing |\n| dgTMatrix (COO, R triplet) | i, j, x | Random insertion when building; reading MTX | Most operations -- convert to dgCMatrix after build |\n| scipy `csc_matrix` | Same as dgCMatrix | Column ops in Python | Row ops |\n| scipy `csr_matrix` | Same as dgRMatrix | Row ops (NumPy convention); matmul; sklearn defaults | Column ops |\n| scipy `coo_matrix` | Triplet (i, j, data) | Construction; MTX I\u002FO | Most ops -- convert after build |\n| HDF5 (h5ad, h5) | Single-file binary chunked | Random access via chunks; compression; widely supported | Cloud \u002F parallel writes |\n| Zarr | Chunked array, per-chunk file (or S3 object) | Cloud-native; parallel writes; Dask integration | Single-file simplicity |\n| DelayedArray + HDF5Array | Bioc lazy evaluation over HDF5 | Out-of-memory bulk ops in R | Speed of in-memory |\n\n## Decision Tree by Scenario\n\n| Scenario | Recommended approach |\n|----------|---------------------|\n| Single-cell (>10k cells), per-cell ops | dgCMatrix in R (Bioconductor); CSR `adata.X` in Python (scanpy) |\n| Bulk RNA-seq (60-80% density) | Dense -- sparse overhead exceeds benefit |\n| Single-cell pseudobulk (after donor aggregation) | Dense -- now 60-80% density typically |\n| 1M+ cells, can't fit in RAM | scanpy backed mode OR HDF5SummarizedExperiment + DelayedArray |\n| TCGA + GTEx + recount3 scale (100k+ samples) | HDF5Array \u002F Zarr + Dask |\n| 10X CellRanger 3.0+ output | `sc.read_10x_h5()` or `Read10X_h5()` -- the .h5 is faster than the .mtx triplet |\n| Cloud-native (anndata on S3, dask compute) | Zarr |\n| Local workstation, single-machine | HDF5 -- faster, more widely supported |\n| Building sparse matrix incrementally | COO (dgTMatrix \u002F coo_matrix); convert to CSC\u002FCSR after |\n| R \u003C-> Python conversion | File-based intermediate (`adata.write` then `zellkonverter::readH5AD`); aware of the transpose |\n\n## Check Sparsity\n\n**Goal:** Decide whether sparse is the right format given the actual data density.\n\n**Approach:** Compute nonzero fraction; rule of thumb is sparse > ~85% (single-cell). Below that, dense often wins.\n\n```python\nimport numpy as np\nimport scipy.sparse as sp\n\ndef sparsity(m):\n    if sp.issparse(m):\n        return 1 - m.nnz \u002F (m.shape[0] * m.shape[1])\n    return (m == 0).mean()\n\ns = sparsity(adata.X)\nprint(f'{s:.1%} sparse')\n```\n\nMemory math:\n\n| Data | Format | Bytes |\n|------|--------|-------|\n| 30k x 100k single-cell matrix, 5% density | dgCMatrix | (5% * 3e9) * 12 bytes ~= 1.8 GB |\n| Same | Dense double | 24 GB |\n| 60k x 100k bulk, 70% density | dgCMatrix | 50 GB (worse than dense!) |\n| Same | Dense double | 48 GB |\n\nFor single-cell (typically 90-95% sparse), sparse is essential. For bulk RNA-seq (typically 60-80% density), dense is faster and not appreciably larger.\n\n## dgCMatrix \u002F scipy CSC \u002F CSR\n\n**Goal:** Construct, query, and convert sparse matrices in the format matching the consumer's expected layout.\n\n**Approach:** `Matrix::sparseMatrix(i, j, x, dims=...)` (R) or `scipy.sparse.csr_matrix((data, (i, j)))` (Python); preserve row\u002Fcolumn names; convert layout (CSC \u003C-> CSR) without changing semantics.\n\n```python\nimport scipy.sparse as sp\nimport pandas as pd\n\ndense_df = pd.read_csv('counts.csv', index_col=0)\nsparse_csr = sp.csr_matrix(dense_df.values)\nsparse_csc = sp.csc_matrix(dense_df.values)\n\ngene_names = dense_df.index.tolist()\nsample_names = dense_df.columns.tolist()\n\nsparse_csr.tocsc()\nsparse_csc.tocsr()\n```\n\n```r\nlibrary(Matrix)\n\ndense_mat \u003C- as.matrix(read.csv('counts.csv', row.names = 1))\nsparse_dgc \u003C- as(dense_mat, 'CsparseMatrix')\n\nclass(sparse_dgc)\nrownames(sparse_dgc) \u003C- rownames(dense_mat)\ncolnames(sparse_dgc) \u003C- colnames(dense_mat)\n```\n\ndgTMatrix is best for building matrices incrementally (reading MTX, parsing per-row); convert to dgCMatrix for downstream ops:\n\n```r\nmat_t \u003C- as(triplet_data, 'TsparseMatrix')\nmat_c \u003C- as(mat_t, 'CsparseMatrix')\n```\n\n## HDF5 vs Zarr -- The Cloud-Native Shift\n\nHDF5 (Hierarchical Data Format 5): hierarchical, single-file binary. Random access via chunks; supports compression (gzip, blosc, lz4). On-disk format for AnnData `.h5ad`, MuData `.h5mu`, 10x Genomics `.h5`, HDF5SummarizedExperiment.\n\nZarr: cloud-native, chunked array storage. Each chunk is a separate file (or S3 object). Parallel-write friendly; splittable by Dask. Format used by recent AnnData (`anndata.write_zarr`), SpatialData, and large-cohort consortia.\n\n| Criterion | HDF5 | Zarr |\n|-----------|------|------|\n| File structure | Single binary file | Directory of chunk files |\n| Parallel writes | Limited (process-level locks) | Native |\n| S3 \u002F cloud object storage | Workarounds (h5cloud); often slow | Native; first-class |\n| Compression options | gzip, blosc, lz4, szip | gzip, blosc, lz4, zstd, custom |\n| Local workstation speed | Faster | Slightly slower (many small files) |\n| Wide ecosystem support | Yes (mature) | Growing; modern scverse |\n\nFor local workstation work, HDF5 is faster and more widely supported. For cloud-mounted analysis (anndata on S3 with dask-distributed compute), Zarr wins because of object-storage friendliness.\n\n## HDF5SummarizedExperiment + DelayedArray (Bioconductor)\n\n**Goal:** Work with bulk SummarizedExperiment objects too large to fit in RAM by keeping the matrix on disk.\n\n**Approach:** `HDF5Array` wraps an HDF5 dataset as a `DelayedArray`. Subsetting builds a delayed operation tree -- no I\u002FO until realization. `DelayedMatrixStats` provides delayed-friendly stat functions.\n\n```r\nlibrary(HDF5Array)\nlibrary(SummarizedExperiment)\nlibrary(DelayedMatrixStats)\n\nse \u003C- loadHDF5SummarizedExperiment('saved_se_dir')\n\ns \u003C- se[1:1000, 1:50]\nrow_means \u003C- rowMeans2(assay(se))\n\nsaveHDF5SummarizedExperiment(se, 'saved_se_dir', replace = TRUE)\n```\n\nThe `assay(se)` returns a DelayedMatrix backed by HDF5. DESeq2, edgeR, limma have varying levels of DelayedArray support; consult package docs before assuming all ops work in delayed mode.\n\nFor TCGA + GTEx + recount3 scale (100k+ samples, 60k genes), a dense matrix is ~48 GB (double); dgCMatrix at 70% density is ~50 GB (sparse loses). HDF5Array + chunk-aware ops keeps memory at whatever-fits-in-RAM.\n\n## scanpy Backed Mode\n\n**Goal:** Work with h5ad files too large for memory by loading only accessed slices on demand.\n\n**Approach:** `sc.read_h5ad(..., backed='r')` returns an AnnData with `.X` as a wrapped HDF5 dataset. Subset operations are lazy; `.to_memory()` realizes.\n\n```python\nimport scanpy as sc\n\nadata = sc.read_h5ad('large_dataset.h5ad', backed='r')\nprint(f'Shape: {adata.shape}, X type: {type(adata.X)}')\n\nt_cells = adata[adata.obs['cell_type'] == 'T_cell', :].to_memory()\n```\n\nLimitations:\n- `.X` is read-only in `backed='r'`. Use `backed='r+'` for in-place updates, but only `.X` updates supported.\n- `.obs` and `.var` are fully loaded -- only `.X` supports backed access.\n- Very large sparse h5ad (>35 GB) can still cause memory issues even in backed mode (anndata library overhead).\n- Many scanpy functions internally load to memory; check `?function` docs for backed compatibility.\n- Functions like `sc.tl.pca`, `sc.pp.neighbors` typically require in-memory; subset first with `.to_memory()`.\n\nFor datasets too large for backed mode, process in chunks:\n\n```python\nimport anndata as ad\n\ndef process_in_chunks(h5ad_path, chunk_size=10000, func=None):\n    adata = sc.read_h5ad(h5ad_path, backed='r')\n    n_cells = adata.shape[0]\n    results = []\n    for start in range(0, n_cells, chunk_size):\n        end = min(start + chunk_size, n_cells)\n        chunk = adata[start:end].to_memory()\n        if func:\n            chunk = func(chunk)\n        results.append(chunk)\n    return ad.concat(results)\n```\n\n## 10X Genomics Format Proliferation\n\n| Format | Files | Notes |\n|--------|-------|-------|\n| MTX (pre-CellRanger 3.0) | `matrix.mtx` + `barcodes.tsv` + `features.tsv` (or `genes.tsv`) | Triplet format; slow to read for large matrices |\n| H5 (CellRanger 3.0+) | `filtered_feature_bc_matrix.h5` | HDF5 with `\u002Fmatrix\u002Fdata`, `\u002Fmatrix\u002Findices`, `\u002Fmatrix\u002Findptr`, `\u002Fmatrix\u002Fshape`; single file, fast |\n| H5AD | `data.h5ad` | AnnData; convert on import |\n| kallisto|bustools output | `output.bus` + barcode and gene mappings | `BUSpaRse` \u002F `kb-python` |\n\n```python\nimport scanpy as sc\n\nadata = sc.read_10x_h5('filtered_feature_bc_matrix.h5')\nadata = sc.read_10x_mtx('filtered_feature_bc_matrix\u002F')\n```\n\n```r\nlibrary(Seurat)\nmat \u003C- Read10X_h5('filtered_feature_bc_matrix.h5')\nmat \u003C- Read10X(data.dir = 'filtered_feature_bc_matrix\u002F')\n\nlibrary(DropletUtils)\nsce \u003C- read10xCounts('filtered_feature_bc_matrix\u002F')\n```\n\nFor 10X output, prefer the `.h5` over the MTX triplet -- typically 5-10x faster for large matrices.\n\n## Dense Conversion -- The Memory Blow-Up\n\n`adata.X.toarray()` (Python) or `as.matrix(seurat_obj@assays$RNA@counts)` (R) on a 30k x 100k single-cell matrix instantiates a ~24 GB dense double array. Common triggers:\n\n- Passing sparse to a function that internally calls `as.matrix()` (older R `cor()` implementations).\n- Heatmap functions (`pheatmap`, `ComplexHeatmap`) that require dense.\n- ML libraries with no sparse support (some sklearn models; XGBoost requires specific sparse API).\n- Plotting functions (`plot()`, `ggplot2`) called on the full matrix.\n\nDefensive pattern: subset to a manageable gene\u002Fcell set BEFORE dense conversion.\n\nFor per-cell PCA-style ops, use sparse-aware solvers:\n\n```python\nfrom scipy.sparse.linalg import svds\nU, s, Vt = svds(adata.X, k=50)\n```\n\n```r\nlibrary(irlba)\nsvd_res \u003C- irlba(sparse_mat, nv = 50)\n```\n\n`irlba` (R) and `scipy.sparse.linalg.svds` (Python) compute truncated SVD without densifying.\n\n## SCE vs AnnData vs MuData -- Where Bulk Fits\n\n| Container | Library | Cells\u002Fsamples | Multi-modal | Bulk fit |\n|-----------|---------|---------------|-------------|----------|\n| SummarizedExperiment \u002F RangedSummarizedExperiment | Bioconductor | n\u002Fa; bulk | No | YES -- standard for DESeq2\u002FedgeR\u002Flimma bulk |\n| SingleCellExperiment (Amezquita 2020) | Bioconductor | cells in cols | Via `altExps` | scRNA-seq with spike-ins, ADT |\n| AnnData | scverse\u002FPython | cells in rows | Via layers | scRNA-seq; bulk is unusual |\n| MuData | scverse\u002FPython | cells in rows | Yes, multiple AnnData | Multi-modal scRNA + ATAC + protein |\n| MultiAssayExperiment | Bioconductor | samples | Yes | R-side multi-modal analog |\n\nBulk RNA-seq rarely uses AnnData -- it shines on the single-cell dimensionality reduction \u002F neighbors \u002F clustering machinery. For bulk in R, use SummarizedExperiment. For bulk in Python, a tidy DataFrame + numpy array is usually sufficient.\n\n## Dask + Zarr for Consortium-Scale Matrices\n\nFor TCGA + GTEx + recount3 (Wilks 2021 *Genome Biol* 22:323) or pancancer assemblies:\n\n```python\nimport zarr\nimport dask.array as da\n\nz = zarr.open('counts.zarr', mode='r')\nda_arr = da.from_zarr(z)\n\ncol_sums = da_arr.sum(axis=0).compute()\nfiltered = da_arr[da_arr.sum(axis=1) > 100, :]\n```\n\n```python\nimport anndata as ad\nadata_disk = ad.read_zarr('large_data.zarr')\n```\n\nFor R: `HDF5Array` + `DelayedArray` is the equivalent, but R doesn't have a true Dask analog. BiocParallel can parallelize chunks, but lazy planning is more manual.\n\n## Sparse Operations\n\n```python\nimport numpy as np\nimport scipy.sparse as sp\n\nrow_sums = np.array(sparse_matrix.sum(axis=1)).flatten()\ncol_sums = np.array(sparse_matrix.sum(axis=0)).flatten()\n\nkeep_rows = row_sums > 10\nsparse_filt = sparse_matrix[keep_rows, :]\n\nsparse_log = sparse_matrix.copy()\nsparse_log.data = np.log1p(sparse_log.data)\n```\n\nSubsetting: select genes (rows) or samples (cols) by index:\n\n```python\ngene_idx = [gene_names.index(g) for g in ['TP53', 'BRCA1', 'MYC'] if g in gene_names]\nsubset = sparse_matrix[gene_idx, :]\n```\n\n## CPM Normalization on Sparse\n\n**Goal:** Apply CPM normalization without densifying.\n\n**Approach:** Compute library sizes from column sums; broadcast scaling factors with sparse multiply for CPM; transform only the nonzero data array in-place with log1p.\n\n```python\nimport numpy as np\nimport scipy.sparse as sp\n\ndef normalize_sparse_cpm(sparse_matrix):\n    lib_sizes = np.array(sparse_matrix.sum(axis=0)).flatten()\n    scaling = 1e6 \u002F lib_sizes\n    return sparse_matrix.multiply(scaling)\n\ndef log1p_inplace(sparse_matrix):\n    out = sparse_matrix.copy()\n    out.data = np.log1p(out.data)\n    return out\n\ncpm = normalize_sparse_cpm(adata.X)\nlog_cpm = log1p_inplace(cpm)\n```\n\nAfter log-transformation, sparsity is PRESERVED (log1p(0) = 0). After CPM with pseudocount, zeros become nonzero -- check sparsity and convert to dense if density drops below ~15%.\n\n## Save \u002F Load Sparse Matrices\n\n```python\nimport scipy.sparse as sp\nimport numpy as np\n\nsp.save_npz('counts_sparse.npz', sparse_matrix)\nloaded = sp.load_npz('counts_sparse.npz')\n\nnp.savez('counts_with_meta.npz',\n    data    = sparse_matrix.data,\n    indices = sparse_matrix.indices,\n    indptr  = sparse_matrix.indptr,\n    shape   = sparse_matrix.shape,\n    genes   = np.array(gene_names),\n    samples = np.array(sample_names))\n```\n\nFor interop and durable storage, prefer h5ad or zarr:\n\n```python\nadata.write_h5ad('counts.h5ad')\nadata.write_zarr('counts.zarr')\n```\n\n## Per-Method Failure Modes\n\n### Implicit transpose in R\u002FPython conversion\n\n**Trigger:** AnnData with cells in rows passed to a SingleCellExperiment workflow that expects cells in cols; downstream `colSums` returns gene-level totals.\n\n**Mechanism:** AnnData stores cells in rows; SCE in cols. The conversion auto-transposes ONCE per direction; two roundtrips silently restore.\n\n**Symptom:** Per-cell stats look like per-gene stats; QC plots have wrong axes.\n\n**Fix:** Use file-based intermediate (`adata.write('file.h5ad')`; `zellkonverter::readH5AD('file.h5ad')`). Always verify dimensions and orientation after conversion.\n\n### Dense conversion blew up memory\n\n**Trigger:** `as.matrix(seurat_obj@assays$RNA@counts)` on a 100k-cell dataset; R session crashes with OOM.\n\n**Mechanism:** 100k cells x 30k genes = 3e9 entries; double precision = 24 GB.\n\n**Symptom:** R session killed; \"cannot allocate vector of size N GB\".\n\n**Fix:** Don't densify the full matrix. Subset to genes\u002Fcells of interest first. For dimensionality reduction, use `irlba::irlba()` (sparse SVD).\n\n### scanpy backed mode silently loaded to memory\n\n**Trigger:** `adata = sc.read_h5ad(path, backed='r')` then `sc.tl.pca(adata)`; memory spikes to dense-equivalent.\n\n**Mechanism:** Many scanpy functions internally call `.to_memory()` because they cannot operate on backed mode. `sc.tl.pca`, `sc.pp.neighbors`, `sc.tl.umap` all materialize.\n\n**Symptom:** OOM despite backed mode.\n\n**Fix:** Subset first (`adata[mask].to_memory()`), then operate. Or use a streaming-aware alternative (Dask + Zarr).\n\n### Sparse stored where dense would be faster\n\n**Trigger:** Bulk RNA-seq with 70% density stored as dgCMatrix; per-gene `rowVars` is slow.\n\n**Mechanism:** Sparse iteration has cache-unfriendly indirection; above ~10-15% density, dense wins.\n\n**Symptom:** Operations notably slower than expected; profiler shows time in sparse indexing.\n\n**Fix:** Convert to dense for the hot path: `as.matrix(sparse_mat)` (R) or `sparse_matrix.toarray()` (Python). Memory may go up but speed improves substantially.\n\n### 10X MTX read is slow\n\n**Trigger:** Reading a 100k-cell 10X dataset via the MTX three-file format; takes 10+ minutes.\n\n**Mechanism:** MTX is a text format; parsing is slow for large matrices.\n\n**Symptom:** Long load times; user kills the process before completion.\n\n**Fix:** Use the CellRanger H5 (`.h5`) instead -- typically 5-10x faster.\n\n## Common errors\n\n| Error \u002F symptom | Cause | Fix |\n|-----------------|-------|-----|\n| `cannot allocate vector of size N GB` | Implicit dense conversion | Subset first; use sparse-aware solver (irlba) |\n| Sparse-dense arithmetic returns `numpy.matrix` | Deprecated NumPy type from sparse+dense | `np.asarray(sparse + dense)` to force ndarray |\n| `KeyError: '_index'` reading h5ad | anndata version mismatch | Update anndata; or `sc.read_h5ad(..., backed=None)` |\n| Empty rows\u002Fcols after sparse subset | Subset removed all data | Verify the index list; cross-check sample\u002Fgene names |\n| Backed mode AnnData crash on `sc.tl.umap` | Function not backed-compatible | `.to_memory()` on the subset first |\n| Per-cell totals look wrong after R\u003C->Python conversion | Implicit transpose | Verify dimensions; use file-based intermediate |\n| CSR (Python) \u003C-> dgCMatrix (R) treated as same | Convention difference | They're transposes of each other; verify shape and a known cell-gene pair |\n\n## References\n\n- Amezquita RA, Lun ATL, Becht E et al. 2020. Orchestrating single-cell analysis with Bioconductor. *Nat Methods* 17:137-145. doi:10.1038\u002Fs41592-019-0654-x\n- Wolf FA, Angerer P, Theis FJ. 2018. SCANPY: large-scale single-cell gene expression data analysis. *Genome Biol* 19:15. doi:10.1186\u002Fs13059-017-1382-0\n- Wilks C et al. 2021. recount3: summaries and queries for large-scale RNA-seq expression and splicing. *Genome Biol* 22:323. doi:10.1186\u002Fs13059-021-02533-6\n- Bates D, Maechler M. 2023. Matrix: Sparse and Dense Matrix Classes and Methods. R package version 1.6-x.\n- Pages H et al. 2020. HDF5Array: HDF5 backend for DelayedArray objects. Bioconductor package.\n- Miles A et al. 2020. zarr-python. Python package documentation.\n- Rocklin M. 2015. Dask: Parallel Computation with Blocked algorithms and Task Scheduling. *Proc Python Sci Conf.* (canonical Dask reference)\n- Lachmann A et al. 2018. Massive mining of publicly available RNA-seq data from human and mouse. *Nat Commun* 9:1366. doi:10.1038\u002Fs41467-018-03751-6\n\n## Related Skills\n\n- counts-ingest - Reading 10X formats; building sparse matrices from quantification output\n- gene-id-mapping - Var (gene) metadata in AnnData\n- metadata-joins - Obs (sample) metadata in AnnData\n- normalization - log1p and CPM patterns on sparse\n- differential-expression\u002Fdeseq2-basics - Pseudobulk aggregation makes dense\n- single-cell\u002Fdata-io - Single-cell file format ecosystem\n- single-cell\u002Fpreprocessing - Standard single-cell sparse pipeline\n",{"name":5,"source":24,"license":15,"tool_type":25,"description":7,"imported_at":26,"source_path":27,"primary_tool":28,"source_repository":29},"https:\u002F\u002Fgithub.com\u002FGPTomics\u002FbioSkills","python","2026-09-20","expression-matrix\u002Fsparse-handling\u002FSKILL.md","scipy.sparse","GPTomics\u002FbioSkills","main","2d1750717747097b602acf30bd09a7cbd4b4c6ea","awesome_skills\u002Fbiology\u002Fbio-expression-matrix-sparse-handling","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous","2026-11-09 11:45:36",{"source":8,"skillName":5,"commands":37,"prompt":47,"zipAvailable":48},[38,41,44],{"tool":39,"command":40},"npx","npx skills add Mr-Nobody-Anonymous\u002FAll-skills --skill bio-expression-matrix-sparse-handling",{"tool":42,"command":43},"bunx","bunx skills add Mr-Nobody-Anonymous\u002FAll-skills --skill bio-expression-matrix-sparse-handling",{"tool":45,"command":46},"pnpm","pnpm dlx skills add Mr-Nobody-Anonymous\u002FAll-skills --skill bio-expression-matrix-sparse-handling","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fbiology\u002Fbio-expression-matrix-sparse-handling\u002FSKILL.md",true,[50,56],{"path":51,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":54,"sourceHtmlUrl":55},"examples\u002Fsparse_operations.py","SCRIPT","脚本",1928,"https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fbiology\u002Fbio-expression-matrix-sparse-handling\u002Fexamples\u002Fsparse_operations.py",{"path":57,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":60,"sourceHtmlUrl":61},"usage-guide.md","REFERENCE","参考文档",6287,"https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fbiology\u002Fbio-expression-matrix-sparse-handling\u002Fusage-guide.md",[63,71,78,85,92,99],{"skillNo":64,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":67,"categories":68,"keywords":69,"publishedAt":70},"01M7FFFZPC2025JQQHPHDX154J","dei-assessment","Assess diversity, equity, and inclusion with metrics. TRIGGERS - Use when user needs help with dei-assessment related tasks.","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fhr-people\u002Fdei-assessment\u002FSKILL.md",[],[],"2026-11-16 10:05:34",{"skillNo":72,"name":73,"nameZh":6,"description":74,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":75,"categories":76,"keywords":77,"publishedAt":70},"01M7FFFWN8H4WA7A6Y398QKB6J","dei-action-plan","Create DEI action plans with goals and initiatives. TRIGGERS - Use when user needs help with dei-action-plan related tasks.","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fhr-people\u002Fdei-action-plan\u002FSKILL.md",[],[],{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":70},"01M7FFFSHTM0QZRVZ2NRJTJ0PY","culture-fit-assessment","Design culture add assessment processes. TRIGGERS - Use when user needs help with culture-fit-assessment related tasks.","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fhr-people\u002Fculture-fit-assessment\u002FSKILL.md",[],[],{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":91,"publishedAt":70},"01M7FFFMA12RKEQA38KHJSXXPS","cowork-to-code-bridge","Use an already-installed, independently verified cowork-to-code bridge to run narrowly approved actions on the user's own macOS, Linux, or WSL2 machine through a local file queue.","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fhr-people\u002Fcowork-to-code-bridge\u002FSKILL.md",[],[],{"skillNo":93,"name":94,"nameZh":6,"description":95,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":70},"01M7FFFH1F6M8YRGW5V487GH1K","counter-offer-response","Draft counter-offer responses for retention. TRIGGERS - Use when user needs help with counter-offer-response related tasks.","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fhr-people\u002Fcounter-offer-response\u002FSKILL.md",[],[],{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":103,"categories":104,"keywords":105,"publishedAt":70},"01M7FFFDGEHV8Y7J9N40HZXW52","continuous-feedback-system","Design continuous feedback systems. TRIGGERS - Use when user needs help with continuous-feedback-system related tasks.","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fhr-people\u002Fcontinuous-feedback-system\u002FSKILL.md",[],[]]