evaluation-and-export
Evaluate MedMNIST predictions and export 2D or 3D datasets safely, including metric calculation, standard result filenames, montages, CSV/image output, and the Fire CLI.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- name
- evaluation-and-export
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Evaluate MedMNIST predictions and export 2D or 3D datasets safely, including metric calculation, standard result filenames, montages, CSV/image output, and the Fire CLI.
- disable-model-invocation
- true
02 资源文件 5
- 参考文档references/api-reference.md4.3 KB
- 参考文档references/cli-reference.md3.2 KB
- 参考文档references/export-recipes.md5.0 KB
- 参考文档references/troubleshooting.md6.1 KB
- 脚本scripts/medmnist_smoke.py10.4 KB
03 SKILL.md 正文
MedMNIST evaluation and export
Use this skill when the task concerns Evaluator, getAUC, getACC, score
shapes, metric interpretation, standard result CSV names, parse_and_evaluate,
MedMNIST2D/3D.save, montage, PNG/GIF/CSV export, or the public save and
evaluate CLI commands. This skill is for evaluation and output handling, not
for loading arbitrary datasets or training models.
Route and safety boundary
- First identify the flag, split, task, score shape, and root. The split
must be
train,val, ortest;y_score.shape[0]must equal the label count. UseINFO[flag]["task"]to choose binary, multilabel, multiclass, or ordinal handling. For registry and NPZ-loading questions, route to the siblingdataset-loadingskill instead. - Use an explicit, already-created
rootfor programmatic evaluation. The constructor never creates a missing custom root and never downloads unless the dataset class is constructed withdownload=True. - Keep result files and exports in a new, isolated output directory. The
evaluator does not create
save_folder; create it first. Do not use bulk download, default-root cleanup, broad developmenttest(), or network downloads as part of a smoke check. - Scores are probabilities/scores, not class labels. Binary and multilabel
scores are thresholded only for ACC; multiclass and ordinal predictions use
argmax. Preserve row order and do not add a CSV header to an evaluator input file.
For a compact API contract, see references/api-reference.md. For CLI syntax and network/destructive boundaries, see references/cli-reference.md. For reproducible evaluation and export procedures, see references/export-recipes.md. For failures and recovery, see references/troubleshooting.md. Run the safe local check with scripts/medmnist_smoke.py.
Evaluation procedure
Direct Evaluator use
Use the installed API, with the dataset NPZ already present in root:
import numpy as np
from medmnist import Evaluator
evaluator = Evaluator("pneumoniamnist", "test", size=64, root="./data")
y_score = np.asarray(...) # N or N x 2 for binary
assert y_score.shape[0] == evaluator.labels.shape[0]
metrics = evaluator.evaluate(y_score, save_folder="./results", run="run1")
print(metrics.AUC, metrics.ACC)
Evaluator(flag, split, size=None, root="~/.medmnist") maps None and 28
to the unsuffixed NPZ. A non-28 size is written as flag_size.npz; the
available 2D sizes are 28/64/128/224 and 3D sizes are 28/64. The root must
exist and contain the exact file. evaluate computes a Metrics(AUC, ACC)
namedtuple and, when save_folder is supplied, writes the score matrix with
pandas using no header and an index column. Pass a stable run value when the
filename must be reproducible.
Shape and metric decision table
INFO[flag]["task"] |
Accepted score shape | AUC | ACC |
|---|---|---|---|
binary-class |
(N,) or (N, 2) |
ROC one-vs-rest on the positive score; for 2-D input the last column is used | threshold the selected score at 0.5 |
multi-label, binary-class |
(N, L) |
compute ROC AUC independently for each label and average | threshold each label at 0.5, compute per-label accuracy, then average |
multi-class |
(N, C) |
one-vs-rest AUC for each class column and average | argmax over columns |
ordinal-regression |
(N, C) |
same one-vs-rest loop as multiclass | argmax over columns |
Labels should be class indices 0..C-1 for multiclass/ordinal and binary
0/1 for binary. Every one-vs-rest target must contain both positive and
negative examples; otherwise scikit-learn cannot define ROC AUC. For
multilabel, retain the 2-D (N, L) shape even when inspecting one label.
getAUC(y_true, y_score, task) and getACC(y_true, y_score, task, threshold=0.5) are the low-level functions. They call squeeze(), so an
unexpected singleton axis can change dimensionality; assert shapes before
calling them. Do not interpret ACC as a probability metric, and do not pass
already-argmaxed class labels in place of scores.
get_dummy_prediction() is useful for discovering a rough output shape, but
it is random and is not a reproducible test fixture. In the inspected version,
its multiclass branch derives the column count from labels.max(); construct
explicit (N, C) scores when all classes must be represented.
Standard result filenames and round trips
With metrics = Metrics(AUC, ACC), the standard name is:
{flag}{size_flag}_{split}_[AUC]{auc:.3f}_[ACC]{acc:.3f}@{run}.csv
size_flag is empty for 28/default and _64, _128, or _224 for the
corresponding 2-D size (or _64 for a 3-D size). For example:
pneumoniamnist_64_test_[AUC]1.000_[ACC]1.000@run1.csv.
run=None uses the current time, so use a string or number for stable output.
Evaluator.parse_and_evaluate(path, run=None) parses the flag, optional size,
and split from the underscore-separated filename, reads the CSV with
index_col=0, header=None, sorts by that index, evaluates the resulting score
matrix, prints Metrics(...), and writes a standardized evaluated CSV beside
it. If run is omitted, the filename must contain @. The parser constructs
an evaluator with the default root; direct evaluation with an explicit root is
safer for isolated fixtures. Put parsed inputs in an isolated output directory
because a matching standard name can be rewritten.
Export and montage procedure
2-D
from medmnist import PathMNIST
dataset = PathMNIST(split="test", root="./data", size=28)
dataset.save("./export", postfix="png", write_csv=True)
montage = dataset.montage(length=2, replace=True, save_folder="./export")
MedMNIST2D.save calls save2d, creates export/{flag}{size_flag}/, and
writes one image per sample plus {flag}{size_flag}.csv when write_csv=True.
The default postfix is png; the implementation can pass another image
extension accepted by Pillow. Image names are {SPLIT}{index}_{labels}.{postfix}
where SPLIT is TRAIN, VALIDATION, or TEST. The CSV is append-mode,
has no generated header, and contains split, filename, and label values;
re-running against the same CSV can append duplicates.
montage(length=20, replace=False, save_folder=None) randomly selects
length * length samples. For a tiny fixture, either use replace=True or
ensure the dataset has at least that many samples. A saved 2-D montage is
{flag}{size_flag}_{split}_montage.jpg; the method returns a PIL image.
Random selection is not deterministic unless the caller seeds NumPy before
calling it.
3-D
from medmnist import OrganMNIST3D
dataset = OrganMNIST3D(split="test", root="./data", size=28)
dataset.save("./export", postfix="gif", write_csv=True)
frames = dataset.montage(length=2, replace=True, save_folder="./export")
MedMNIST3D.save requires postfix="gif"; it emits one GIF per volume and a
CSV under the same naming layout. A 3-D montage returns a list of frames and,
when saved, writes {flag}{size_flag}_{split}_montage.gif. The current 3-D
montage path requires a single channel. GIF output requires a nonempty volume
and compatible PIL frames.
CLI procedure
The package exposes Fire commands. With data already local, use:
python -m medmnist available
python -m medmnist info --flag=pneumoniamnist
python -m medmnist save --flag=pneumoniamnist --folder=./export --postfix=png --download=False --size=28 --root=./data
python -m medmnist save --flag=organmnist3d --folder=./export --postfix=gif --download=False --size=28 --root=./data
python -m medmnist evaluate --path=./results/pneumoniamnist_test_[AUC]1.000_[ACC]1.000@run1.csv
save loops over all three splits, so all three corresponding NPZ arrays are
required. --download=True is a network operation and is intentionally not
part of a safe smoke test. evaluate has only --path; its parser resolves
the dataset through the default root rather than exposing a custom-root flag.
For a custom root, use the direct Evaluator API (or arrange the data in the
expected default root only with explicit user approval). available and
info are read-only metadata commands. Do not use clean in this skill: it
deletes downloaded *mnist*.npz files.
Verification and return handoff
Run the bundled helper before relying on a new environment:
python scripts/medmnist_smoke.py --help
python scripts/medmnist_smoke.py
A successful run reports deterministic binary, multilabel, and multiclass
AUC=1.000 ACC=1.000 checks, a _64/@run CSV parse round trip in a temporary
root, and 2-D/3-D export checks without network access. For a concise handoff,
report: files used or emitted; evidence (task, split, root, score
shapes); checks and metric values; difficult cases (singleton axes,
missing ROC classes, _64/@run, tiny montage sampling, 3-D GIF); gaps such
as default-root-only parsing or untested optional backends; and intentional
omissions (training, bulk downloads, default-root clean, and broad
test()).
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).