rank-perturbations-by-edistance
Rank the perturbations of a pooled CRISPR screen with single-cell readout (Perturb-seq, CROP-seq, CRISPRi or CRISPRa) by E-distance (energy distance) to the non-targeting control in PCA space, with a label-permutation test and Benjamini-Hochberg correction, from an AnnData with a guide or perturbation column to a ranked CSV, the list of hits to carry into differential expression, and an optional perturbation-by-perturbation distance matrix. Use when asking which guides actually changed the transcriptome, when separating real hits from silent guides before per-perturbation DE, or when grouping perturbations with similar effects.
- License 许可
- MIT
- 最近更新
- 2026-11-14
- Stars
- ★ 0
- Forks
- 2
01 前言 Front Matter
- name
- rank-perturbations-by-edistance
- license
- MIT
- metadata
- { "tags": "crispr, perturb-seq, crop-seq, scrna-seq, e-distance, energy-distance, permutation-test, scperturb", "author": "Yijun Sun", "status": "community", "aliases": "perturbation_edistance", "compute": "1 CPU, under 0.5 GB RAM, seconds for 270 cells x 60 genes with 1000 permutations; about 0.1 s per perturbation per 1000 permutations with 2000 control cells and 30 PCs", "version": "1.0.0", "category": "crispr-screens", "scheduler": "none", "tested-with": "scanpy 1.12.1, anndata 0.12.16, pandas 2.3.3, numpy 2.4.6, scikit-learn 1.9.0, Python 3.13.9", "requires-gpu": "false", "requires-network": "install", "requires-secrets": "none" }
- description
- Rank the perturbations of a pooled CRISPR screen with single-cell readout (Perturb-seq, CROP-seq, CRISPRi or CRISPRa) by E-distance (energy distance) to the non-targeting control in PCA space, with a label-permutation test and Benjamini-Hochberg correction, from an AnnData with a guide or perturbation column to a ranked CSV, the list of hits to carry into differential expression, and an optional perturbation-by-perturbation distance matrix. Use when asking which guides actually changed the transcriptome, when separating real hits from silent guides before per-perturbation DE, or when grouping perturbations with similar effects.
- compatibility
- Python 3.12+ with scanpy and anndata (conda or pip). No GPU, no scheduler.
02 资源文件 2
03 SKILL.md 正文
Rank perturbations by E-distance to the control
In a pooled screen most guides are silent: the guide did not edit, or the gene is dispensable in
this context. scripts/reference.py answers "which perturbations actually did something?" before
any differential expression. It measures how far each perturbation moved its cells away from the
non-targeting control in PCA space (the scPerturb E-distance, Peidli et al., Nat. Methods 2024),
tests the shift with a label permutation, corrects with Benjamini-Hochberg, and writes a ranked
table, the hits for the DE step (compare-perturbations-to-control, ONLY_PERTURBATIONS) and,
optionally, a perturbation-by-perturbation distance matrix.
The most likely mistake is keeping N_PERM = 1000 for a large screen. The smallest permutation
p-value is 1/(N_PERM + 1), so after Benjamini-Hochberg over m tested perturbations a lone hit
needs N_PERM + 1 > m / FDR. With 1000 permutations and FDR 0.05 that holds only up to 50 tested
perturbations; for 1000 perturbations use at least 20 000. The second most likely is reading the
ranking by distance alone: under no effect the distance is larger for smaller groups (see
section 5), so read it with the permutation p-values.
Tested on synthetic data only (tests/: 270 cells, 60 genes, a non-targeting control of 80
cells, two guides that switch on the same 5-gene program, one silent guide and one 10-cell
guide; plus a 30-cell and a 300-cell silent guide, and a check against the pairwise definition)
with scanpy 1.12.1, anndata 0.12.16, pandas 2.3.3, numpy 2.4.6, scikit-learn 1.9.0 and Python
3.13.9. It has been the AiScientist template for this step of the Perturb-seq pipeline; no
real-screen run is recorded here.
What the statistic is
E(P,C) = 2·δ(P,C) − δ(P,P) − δ(C,C), where δ(X,Y) is the mean squared Euclidean distance over all pairs of a cell of X and a cell of Y. The script computes it in closed form from group means and variances (O(n·d) memory, no n×n matrix), so it is exact and scales to large screens. The means include each cell paired with itself, and then E(P,C) = 2·‖mean(P) − mean(C)‖²: with squared Euclidean cost the statistic measures a shift of the centroid, not a change of shape.
| Implementation | Same number as this script? |
|---|---|
scPerturb edist(..., dist="sqeuclidean", sample_correct=False) |
yes |
scPerturb edist defaults (sample_correct=True, within-group means over n(n−1) pairs) |
no: smaller by 2·Var(P)/(n_P − 1) + 2·Var(C)/(n_C − 1) |
pertpy Distance(metric="edistance") (checked in pertpy 1.4.0) |
no: it uses plain Euclidean distances |
A variant sensitive to the shape of the distribution needs non-squared Euclidean distances and subsampled pairwise distances. Say which definition you used when you compare numbers across tools.
Significance: for each perturbation the script pools its cells with the control cells, shuffles
the two labels N_PERM times, counts how often the shuffled E-distance reaches the observed one
(plus one for the observed, Davison-Hinkley), and applies Benjamini-Hochberg across the tested
perturbations. A perturbation not significant at FDR is silent.
1. Environment setup
mamba create -y -n scanpy_1.12 -c conda-forge python=3.13 scanpy=1.12.1 anndata=0.12.16
source "$(conda info --base)/etc/profile.d/conda.sh"
conda activate scanpy_1.12
python -c "import scanpy; print(scanpy.__version__)"
pip install scanpy==1.12.1 anndata==0.12.16 in a Python 3.12+ venv works too. Inside an
AiScientist run, the analysis image already has scanpy.
2. Prepare inputs
- An AnnData file with a guide or perturbation label per cell in an
obscolumn. The script reads, in this order,adata_clustered.h5adoradata_qc.h5adfrom$AISCIENTIST_WORK, else the file named by$AISCIENTIST_DATASET. - It uses the first
N_PCScolumns ofobsm["X_pca"]when present (a clustered checkpoint has it). Otherwise it runssc.pp.pca(adata, n_comps=N_PCS)onX, which should then be log-normalized (HVG-subset if you like). $AISCIENTIST_ARTIFACTSis where the tables go (tables/is created).
Identify the perturbation column and the non-targeting control level from the data (column names
such as perturbation, guide, sgRNA, target_gene, feature_call; control levels such as
NT, NTC, non-targeting, control, sgLacZ, sgGFP, scramble), state which you chose,
and ask if it is ambiguous. Count cells per perturbation. If guide identity is not in obs,
assign cells to guides first; never invent the column. The PCA must include the control cells.
3. Configure
Edit the values at the top of the script:
| Variable | Meaning | Default |
|---|---|---|
PERT_KEY |
the obs column naming the guide or perturbation |
perturbation |
CONTROL |
the shared non-targeting level of PERT_KEY |
NT |
N_PCS |
PCA dimensions the distance is measured in | 30 |
MIN_CELLS |
perturbations with fewer cells are counted but not tested | 30 |
N_PERM |
label permutations per perturbation; 0 gives distances only, no calls |
1000 |
FDR |
Benjamini-Hochberg threshold on the permutation p-values | 0.05 |
PAIRWISE |
also write the perturbation-by-perturbation E-distance matrix | False |
SEED |
seed of the permutations | 0 |
Set N_PERM from the number of perturbations you will test (above). The cost grows linearly
with N_PERM, the number of perturbations, and the cells in each perturbation plus the control.
The script works at whatever level PERT_KEY holds; to work at target-gene level, collapse the
guide labels into a new column first and point PERT_KEY at it.
4. Run
export AISCIENTIST_WORK=/path/to/work AISCIENTIST_ARTIFACTS=/path/to/results
# or, with no checkpoint in the work directory:
export AISCIENTIST_DATASET=/path/to/screen.h5ad
python scripts/reference.py
In AiScientist: read the file with read_skill_reference, set the values for this dataset, and
run it with run_code, which sets the variables.
5. Validate before using the output
Outputs under $AISCIENTIST_ARTIFACTS/tables/:
| Path | Contents |
|---|---|
perturbation_edistance.csv |
tested perturbations sorted by edistance, with n_cells, perm_pvalue, perm_padj, significant |
perturbation_edistance_pairwise.csv |
with PAIRWISE = True: symmetric matrix of E-distances between perturbations with at least MIN_CELLS cells |
Check, in this order:
- The JSON summary:
n_perturbations,n_tested,n_skipped_low_cells,n_significant. Perturbations belowMIN_CELLSare counted but not in the table; list them yourself from the cell counts, because the report must name them. - The permutation floor: if every hit has
perm_pvalue= 1/(N_PERM + 1) and nothing passesFDR, the permutations, not the data, limited the call. RaiseN_PERM. - Distance against group size: with no effect the expected E-distance is about
2·V·(1/n_P + 1/n_C), V being the total variance of the PCs used. A 30-cell silent guide
therefore sits higher in the ranking than a 300-cell silent guide (the tests show it). The
permutation null has the same group sizes, so
perm_padjis the call; the distance is the effect size among the called hits. - Silent perturbations: not significant means no detectable transcriptional effect at this depth. It is a result: report it, and do not run downstream biology on it.
real_hits_for_DE: the significant perturbations, forONLY_PERTURBATIONS. WithN_PERM = 0it is every tested perturbation ranked by distance, with nothing called.- Pairwise matrix: a small distance between two perturbations means a similar shift, a candidate shared pathway or module. Frame it as a hypothesis.
Report
The perturbation column, the control level, guide or target level, N_PCS, MIN_CELLS,
N_PERM, FDR and the E-distance definition (squared Euclidean, self-pairs included). The
ranked perturbations with E-distance, permutation p-value and adjusted p-value, never names
alone. The silent perturbations, and every perturbation skipped for low cell count. Keep "the
ranking ran" separate from "the hits are real": a hit that also fails its self-knockdown check in
the DE step needs a second look.
04 同仓库更多 Skills
verb-object-with-tool
One or two sentences on what the skill does and when to use it. Name the tool, the input it takes and the output it produces, then list the situations that should trigger it (for example "Use when setting up X, writing a Y config, or diagnosing a failed Z run").
write-variant-tables
Write the five standard variant-annotation result tables - consequence, impact and ClinVar-significance distributions, a ClinVar pathogenic and likely-pathogenic list, and a shortlist of rare, damaging variants not in ClinVar - plus a JSON summary, from an annotated per-variant TSV (VEP consequence and impact, SIFT, PolyPhen, max population AF, rsID, ClinVar significance), in pure Python without re-parsing the VCF or re-running VEP. Use when a study needs these tables with a different rarity cutoff, impact set or damaging-predictor rule, or when rebuilding them from an existing annotation table. In AiScientist, annotate_variants already writes the default tables, so use this only for a custom cut.
rank-phenotype-differentials-with-mcp
Map clinical phenotype text to grounded HPO terms and rank a research differential through an already configured phenotype-diagnosis MCP service with LIRICAL and optional literature adjudication. Use for explicit service-based analysis while preserving negation, uncertainty and original probabilities; retain the established workflow when the service is unavailable.
qc-vcf-callset
Judge whether a germline VCF callset is trustworthy before interpreting any variant - Ti/Tv, Het/Hom, SNP, indel and multiallelic counts, sample count and genotype call rate from bcftools stats (falling back to cyvcf2, then to a plain-Python Ti/Tv scan) - and flag values outside the expected human WGS or WES ranges. Writes tables/vcf_qc_summary.csv and data/vcf_qc.json with a qc_flags list. Use when a VCF (one sample or a cohort) arrives for annotation or prioritization, when a Ti/Tv or Het/Hom value needs interpreting, or when deciding whether downstream variant tables must carry a low-confidence caveat.
prioritize-clinical-variants
Rank annotated variants into research-triage tiers (ClinVar pathogenic, likely pathogenic, VUS favouring pathogenic, VUS, likely benign, benign) from ClinVar significance used verbatim, gnomAD allele frequency against a dominant or recessive disease-model cutoff and the ACMG BA1 5 percent rule, and in-silico predictors (SIFT, PolyPhen, REVEL, CADD, AlphaMissense) as supporting evidence only. Pure Python; reads a per-variant annotation TSV and writes a tier-sorted CSV with the evidence for each tier plus a JSON of tier counts. Use when a rare-variant list from VEP, ClinVar and gnomAD annotation must be ordered for review (which variants to look at first) or when choosing the disease model and cutoffs for that ordering. Research triage only, NOT a clinical ACMG/AMP classification.
normalize-vcf-with-bcftools
Normalize a VCF with bcftools norm before annotation or ClinVar/gnomAD matching - split multiallelic sites into biallelic records, atomize MNPs into SNPs, then left-align and trim indels against the reference FASTA of the same build - writing an indexed normalized.vcf.gz and a JSON of record counts before and after. Use when a VCF not known to be normalized (especially an indel-heavy one) will be matched against ClinVar, gnomAD or dbSNP, compared between callers or loaded into PLINK, or when a known pathogenic indel comes back as not in ClinVar. In AiScientist, the offline annotate_variants path already normalizes, so do not plan it as a separate step there.