lab-unit-harmonization
Harmonize multi-source clinical lab data to US conventional units by (1) strict completeness filtering, (2) robust numeric parsing (scientific notation + comma decimals), (3) per-feature unit conversion using tight physiological ranges plus “best-candidate” selection (not by widening ranges), and (4) fixed `X.XX` formatting with correct validation (don’t confuse CSV delimiters with decimal commas).
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-14
- Stars
- ★ 0
- Forks
- 1
01 前言 Front Matter
- name
- lab-unit-harmonization
- description
- Harmonize multi-source clinical lab data to US conventional units by (1) strict completeness filtering, (2) robust numeric parsing (scientific notation + comma decimals), (3) per-feature unit conversion using tight physiological ranges plus “best-candidate” selection (not by widening ranges), and (4) fixed `X.XX` formatting with correct validation (don’t confuse CSV delimiters with decimal commas).
02 资源文件 1
- 参考文档reference/ckd_lab_features.md12.6 KB
03 SKILL.md 正文
Steps
- Load inputs and identify numeric lab columns
- Read
/root/environment/data/ckd_lab_data.csvand/root/environment/data/ckd_feature_descriptions.csv. - Treat all columns except the patient identifier column (if present) as lab features; keep the same column order for output.
- Inspect per-column missingness and rough distributions (min/max, quantiles) to spot likely mixed-unit analytes before converting.
- Read
- Parse raw numeric strings into floats (before dropping rows)
- Normalize strings: strip whitespace; map
'','nan','none','null'to missing. - Convert scientific notation via
float(s)when it containse/E. - Fix comma/period issues with a disambiguation rule:
- If a token contains both
,and.(e.g.,1,234.5), treat,as a thousands separator → remove commas. - Else if it contains
,only (e.g.,12,34), treat,as decimal separator → replace with..
- If a token contains both
- Coerce failures to
NaN.
- Normalize strings: strip whitespace; map
- Drop incomplete records (strict)
- After parsing, drop any row with any missing value in lab feature columns (
NaN). - Reset index.
- After parsing, drop any row with any missing value in lab feature columns (
- Unit harmonization with tight ranges + best-candidate selection (do not “fix” by widening ranges)
- Define US-conventional physiological ranges per feature (min/max) and candidate conversions (multiplicative factors) for common SI/alt units (examples that must be covered because they commonly appear mixed in CKD lab exports):
- Serum creatinine: µmol/L → mg/dL (× 0.0113)
- BUN: mmol/L → mg/dL (× 2.801)
- Hemoglobin: g/L → g/dL (× 0.1)
- Calcium: mmol/L → mg/dL (× 4.0)
- Phosphate: mmol/L → mg/dL (× 3.096)
- Magnesium: mmol/L → mg/dL (× 2.43); also mg/L → mg/dL (× 0.1) to handle “7–9”-type magnitudes without expanding the acceptable mg/dL range
- (Add other analytes as needed using the same pattern; keep ranges clinically tight rather than permissive.)
- For each column, compute a robust target center from already-in-range values (median of values within
[min,max]after parsing). If none are in-range, use the midpoint(min+max)/2. - For each value
vin the column, evaluate candidates:- Candidate set includes “no conversion” (
v) plus allv*factorconversions for that feature. - Select the candidate that lands inside
[min,max]and is closest to the column target center (minimizeabs(candidate - center)). - If no candidate falls inside range, set to
NaN(do not silently keep out-of-range values).
- Candidate set includes “no conversion” (
- After converting all columns, drop any row that now has
NaN(these are records that cannot be confidently harmonized to in-range US units).
- Define US-conventional physiological ranges per feature (min/max) and candidate conversions (multiplicative factors) for common SI/alt units (examples that must be covered because they commonly appear mixed in CKD lab exports):
- Round and format output as exact
X.XXstrings- Round numeric values to 2 decimals.
- Format every lab value with fixed 2-decimal string formatting:
f"{x:.2f}"(no scientific notation). - Save to
/root/ckd_lab_data_harmonized.csvwithindex=False, preserving the same columns and order as input.
- Correct validation checks (avoid the CSV-comma mistake)
- Do not assert “no commas anywhere in the file” (CSV delimiters are commas).
- Instead validate at the cell level:
- No cell contains
','(decimal comma) because values are formatted with.. - No cell contains
'e'/'E'(scientific notation). - Every numeric cell matches regex
^-?\d+\.\d{2}$. - Every numeric value lies within its configured physiological range.
- No cell contains
Expected Result
A CSV saved at /root/ckd_lab_data_harmonized.csv with the same columns as the raw input, only complete patient rows, all lab values converted to US conventional units, all values within tight physiological ranges, and every numeric value formatted exactly as X.XX (no scientific notation, no decimal commas, no inconsistent decimals).
04 同仓库更多 Skills
video-frame-extraction
Extract frames from video files and save them as images using OpenCV
openai-vision
Analyze images and multi-frame sequences using OpenAI GPT vision models
image-ocr
Extract text content from images using Tesseract OCR via Python, including structured field extraction from receipts and forms
video-frame-extraction
Extract frames from video files and save them as images using OpenCV
openai-vision
Analyze images and multi-frame sequences using OpenAI GPT vision models
image-ocr
Extract text content from images using Tesseract OCR via Python, including structured field extraction from receipts and forms