[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6EC99VBAHZ07SZPXS81WN41":3,"skill-same:01M6EC99VBAHZ07SZPXS81WN41":53},{"skillNo":4,"name":5,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":22,"publishedAt":26,"linkOnly":27,"markdownContent":28,"markdownBody":28,"frontMatter":29,"refName":30,"commitSha":31,"skillDirectory":32,"forksCount":33,"repositoryHtmlUrl":34,"repositoryHomepageUrl":35,"ownerHtmlUrl":36,"ownerBlogUrl":6,"ownerDescription":37,"updatedAt":38,"install":39,"resources":52},"01M6EC99VBAHZ07SZPXS81WN41","imaging-data-commons",null,"ComeOnOliver\u002Fskillshub","ComeOnOliver","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F43215937?v=4","01M6E5JNFECJAAVFE0W7SMWWD2","Haoran Shu","01M6E5JNNDQS7AJZDYFQ8TPZCK",66,"MIT","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002FK-Dense-AI\u002Fclaude-scientific-skills\u002Fimaging-data-commons\u002FSKILL.md",[17],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000033","数据分析师","PROFESSION","适用职业",[23,24,25],"data","imaging data commons","imaging","2026-11-10 10:12:30",false,"# Imaging Data Commons\n\n## Overview\n\nUse the `idc-index` Python package to query and download public cancer imaging data from the National Cancer Institute Imaging Data Commons (IDC). No authentication required for data access.\n\n**Current IDC Data Version: v23** (always verify with `IDCClient().get_idc_version()`)\n\n**Primary tool:** `idc-index` ([GitHub](https:\u002F\u002Fgithub.com\u002Fimagingdatacommons\u002Fidc-index))\n\n**CRITICAL - Check package version and upgrade if needed (run this FIRST):**\n\n```python\nimport idc_index\n\nREQUIRED_VERSION = \"0.11.10\"  # Must match metadata.idc-index in this file\ninstalled = idc_index.__version__\n\nif installed \u003C REQUIRED_VERSION:\n    print(f\"Upgrading idc-index from {installed} to {REQUIRED_VERSION}...\")\n    import subprocess\n    subprocess.run([\"pip3\", \"install\", \"--upgrade\", \"--break-system-packages\", \"idc-index\"], check=True)\n    print(\"Upgrade complete. Restart Python to use new version.\")\nelse:\n    print(f\"idc-index {installed} meets requirement ({REQUIRED_VERSION})\")\n```\n\n**Verify IDC data version and check current data scale:**\n\n```python\nfrom idc_index import IDCClient\nclient = IDCClient()\n\n# Verify IDC data version (should be \"v23\")\nprint(f\"IDC data version: {client.get_idc_version()}\")\n\n# Get collection count and total series\nstats = client.sql_query(\"\"\"\n    SELECT\n        COUNT(DISTINCT collection_id) as collections,\n        COUNT(DISTINCT analysis_result_id) as analysis_results,\n        COUNT(DISTINCT PatientID) as patients,\n        COUNT(DISTINCT StudyInstanceUID) as studies,\n        COUNT(DISTINCT SeriesInstanceUID) as series,\n        SUM(instanceCount) as instances,\n        SUM(series_size_MB)\u002F1000000 as size_TB\n    FROM index\n\"\"\")\nprint(stats)\n```\n\n**Core workflow:**\n1. Query metadata → `client.sql_query()`\n2. Download DICOM files → `client.download_from_selection()`\n3. Visualize in browser → `client.get_viewer_URL(seriesInstanceUID=...)`\n\n## When to Use This Skill\n\n- Finding publicly available radiology (CT, MR, PET) or pathology (slide microscopy) images\n- Selecting image subsets by cancer type, modality, anatomical site, or other metadata\n- Downloading DICOM data from IDC\n- Checking data licenses before use in research or commercial applications\n- Visualizing medical images in a browser without local DICOM viewer software\n\n## Quick Navigation\n\n**Core Sections (inline):**\n- IDC Data Model - Collection and analysis result hierarchy\n- Index Tables - Available tables and joining patterns\n- Installation - Package setup and version verification\n- Core Capabilities - Essential API patterns (query, download, visualize, license, citations, batch)\n- Best Practices - Usage guidelines\n- Troubleshooting - Common issues and solutions\n\n**Reference Guides (load on demand):**\n\n| Guide | When to Load |\n|-------|--------------|\n| `index_tables_guide.md` | Complex JOINs, schema discovery, DataFrame access |\n| `use_cases.md` | End-to-end workflow examples (training datasets, batch downloads) |\n| `sql_patterns.md` | Quick SQL patterns for filter discovery, annotations, size estimation |\n| `clinical_data_guide.md` | Clinical\u002Ftabular data, imaging+clinical joins, value mapping |\n| `cloud_storage_guide.md` | Direct S3\u002FGCS access, versioning, UUID mapping |\n| `dicomweb_guide.md` | DICOMweb endpoints, PACS integration |\n| `digital_pathology_guide.md` | Slide microscopy (SM), annotations (ANN), pathology workflows |\n| `bigquery_guide.md` | Full DICOM metadata, private elements (requires GCP) |\n| `cli_guide.md` | Command-line tools (`idc download`, manifest files) |\n\n## IDC Data Model\n\nIDC adds two grouping levels above the standard DICOM hierarchy (Patient → Study → Series → Instance):\n\n- **collection_id**: Groups patients by disease, modality, or research focus (e.g., `tcga_luad`, `nlst`). A patient belongs to exactly one collection.\n- **analysis_result_id**: Identifies derived objects (segmentations, annotations, radiomics features) across one or more original collections.\n\nUse `collection_id` to find original imaging data, may include annotations deposited along with the images; use `analysis_result_id` to find AI-generated or expert annotations.\n\n**Key identifiers for queries:**\n| Identifier | Scope | Use for |\n|------------|-------|---------|\n| `collection_id` | Dataset grouping | Filtering by project\u002Fstudy |\n| `PatientID` | Patient | Grouping images by patient |\n| `StudyInstanceUID` | DICOM study | Grouping of related series, visualization |\n| `SeriesInstanceUID` | DICOM series | Grouping of related series, visualization |\n\n## Index Tables\n\nThe `idc-index` package provides multiple metadata index tables, accessible via SQL or as pandas DataFrames.\n\n**Complete index table documentation:** Use https:\u002F\u002Fidc-index.readthedocs.io\u002Fen\u002Flatest\u002Findices_reference.html for quick check of available tables and columns without executing any code.\n\n**Important:** Use `client.indices_overview` to get current table descriptions and column schemas. This is the authoritative source for available columns and their types — always query it when writing SQL or exploring data structure.\n\n### Available Tables\n\n| Table | Row Granularity | Loaded | Description |\n|-------|-----------------|--------|-------------|\n| `index` | 1 row = 1 DICOM series | Auto | Primary metadata for all current IDC data |\n| `prior_versions_index` | 1 row = 1 DICOM series | Auto | Series from previous IDC releases; for downloading deprecated data |\n| `collections_index` | 1 row = 1 collection | fetch_index() | Collection-level metadata and descriptions |\n| `analysis_results_index` | 1 row = 1 analysis result collection | fetch_index() | Metadata about derived datasets (annotations, segmentations) |\n| `clinical_index` | 1 row = 1 clinical data column | fetch_index() | Dictionary mapping clinical table columns to collections |\n| `sm_index` | 1 row = 1 slide microscopy series | fetch_index() | Slide Microscopy (pathology) series metadata |\n| `sm_instance_index` | 1 row = 1 slide microscopy instance | fetch_index() | Instance-level (SOPInstanceUID) metadata for slide microscopy |\n| `seg_index` | 1 row = 1 DICOM Segmentation series | fetch_index() | Segmentation metadata: algorithm, segment count, reference to source image series |\n| `ann_index` | 1 row = 1 DICOM ANN series | fetch_index() | Microscopy Bulk Simple Annotations series metadata; references annotated image series |\n| `ann_group_index` | 1 row = 1 annotation group | fetch_index() | Detailed annotation group metadata: graphic type, annotation count, property codes, algorithm |\n| `contrast_index` | 1 row = 1 series with contrast info | fetch_index() | Contrast agent metadata: agent name, ingredient, administration route (CT, MR, PT, XA, RF) |\n\n**Auto** = loaded automatically when `IDCClient()` is instantiated\n**fetch_index()** = requires `client.fetch_index(\"table_name\")` to load\n\n### Joining Tables\n\n**Key columns are not explicitly labeled, the following is a subset that can be used in joins.**\n\n| Join Column | Tables | Use Case |\n|-------------|--------|----------|\n| `collection_id` | index, prior_versions_index, collections_index, clinical_index | Link series to collection metadata or clinical data |\n| `SeriesInstanceUID` | index, prior_versions_index, sm_index, sm_instance_index | Link series across tables; connect to slide microscopy details |\n| `StudyInstanceUID` | index, prior_versions_index | Link studies across current and historical data |\n| `PatientID` | index, prior_versions_index | Link patients across current and historical data |\n| `analysis_result_id` | index, analysis_results_index | Link series to analysis result metadata (annotations, segmentations) |\n| `source_DOI` | index, analysis_results_index | Link by publication DOI |\n| `crdc_series_uuid` | index, prior_versions_index | Link by CRDC unique identifier |\n| `Modality` | index, prior_versions_index | Filter by imaging modality |\n| `SeriesInstanceUID` | index, seg_index, ann_index, ann_group_index, contrast_index | Link segmentation\u002Fannotation\u002Fcontrast series to its index metadata |\n| `segmented_SeriesInstanceUID` | seg_index → index | Link segmentation to its source image series (join seg_index.segmented_SeriesInstanceUID = index.SeriesInstanceUID) |\n| `referenced_SeriesInstanceUID` | ann_index → index | Link annotation to its source image series (join ann_index.referenced_SeriesInstanceUID = index.SeriesInstanceUID) |\n\n**Note:** `Subjects`, `Updated`, and `Description` appear in multiple tables but have different meanings (counts vs identifiers, different update contexts).\n\nFor detailed join examples, schema discovery patterns, key columns reference, and DataFrame access, see `references\u002Findex_tables_guide.md`.\n\n### Clinical Data Access\n\n```python\n# Fetch clinical index (also downloads clinical data tables)\nclient.fetch_index(\"clinical_index\")\n\n# Query clinical index to find available tables and their columns\ntables = client.sql_query(\"SELECT DISTINCT table_name, column_label FROM clinical_index\")\n\n# Load a specific clinical table as DataFrame\nclinical_df = client.get_clinical_table(\"table_name\")\n```\n\nSee `references\u002Fclinical_data_guide.md` for detailed workflows including value mapping patterns and joining clinical data with imaging.\n\n## Data Access Options\n\n| Method | Auth Required | Best For |\n|--------|---------------|----------|\n| `idc-index` | No | Key queries and downloads (recommended) |\n| IDC Portal | No | Interactive exploration, manual selection, browser-based download |\n| BigQuery | Yes (GCP account) | Complex queries, full DICOM metadata |\n| DICOMweb proxy | No | Tool integration via DICOMweb API |\n| Cloud storage (S3\u002FGCS) | No | Direct file access, bulk downloads, custom pipelines |\n\n**Cloud storage organization**\n\nIDC maintains all DICOM files in public cloud storage buckets mirrored between AWS S3 and Google Cloud Storage. Files are organized by CRDC UUIDs (not DICOM UIDs) to support versioning.\n\n| Bucket (AWS \u002F GCS) | License | Content |\n|--------------------|---------|---------|\n| `idc-open-data` \u002F `idc-open-data` | No commercial restriction | >90% of IDC data |\n| `idc-open-data-two` \u002F `idc-open-idc1` | No commercial restriction | Collections with potential head scans |\n| `idc-open-data-cr` \u002F `idc-open-cr` | Commercial use restricted (CC BY-NC) | ~4% of data |\n\nFiles are stored as `\u003Ccrdc_series_uuid>\u002F\u003Ccrdc_instance_uuid>.dcm`. Access is free (no egress fees) via AWS CLI, gsutil, or s5cmd with anonymous access. Use `series_aws_url` column from the index for S3 URLs; GCS uses the same path structure.\n\nSee `references\u002Fcloud_storage_guide.md` for bucket details, access commands, UUID mapping, and versioning.\n\n**DICOMweb access**\n\nIDC data is available via DICOMweb interface (Google Cloud Healthcare API implementation) for integration with PACS systems and DICOMweb-compatible tools.\n\n| Endpoint | Auth | Use Case |\n|----------|------|----------|\n| Public proxy | No | Testing, moderate queries, daily quota |\n| Google Healthcare | Yes (GCP) | Production use, higher quotas |\n\nSee `references\u002Fdicomweb_guide.md` for endpoint URLs, code examples, supported operations, and implementation details.\n\n## Installation and Setup\n\n**Required (for basic access):**\n```bash\npip install --upgrade idc-index\n```\n\n**Important:** New IDC data release will always trigger a new version of `idc-index`. Always use `--upgrade` flag while installing, unless an older version is needed for reproducibility.\n\n**IMPORTANT:** IDC data version v23 is current. Always verify your version:\n```python\nprint(client.get_idc_version())  # Should return \"v23\"\n```\nIf you see an older version, upgrade with: `pip install --upgrade idc-index`\n\n**Tested with:** idc-index 0.11.10 (IDC data version v23)\n\n**Optional (for data analysis):**\n```bash\npip install pandas numpy pydicom\n```\n\n## Core Capabilities\n\n### 1. Data Discovery and Exploration\n\nDiscover what imaging collections and data are available in IDC:\n\n```python\nfrom idc_index import IDCClient\n\nclient = IDCClient()\n\n# Get summary statistics from primary index\nquery = \"\"\"\nSELECT\n  collection_id,\n  COUNT(DISTINCT PatientID) as patients,\n  COUNT(DISTINCT SeriesInstanceUID) as series,\n  SUM(series_size_MB) as size_mb\nFROM index\nGROUP BY collection_id\nORDER BY patients DESC\n\"\"\"\ncollections_summary = client.sql_query(query)\n\n# For richer collection metadata, use collections_index\nclient.fetch_index(\"collections_index\")\ncollections_info = client.sql_query(\"\"\"\n    SELECT collection_id, CancerTypes, TumorLocations, Species, Subjects, SupportingData\n    FROM collections_index\n\"\"\")\n\n# For analysis results (annotations, segmentations), use analysis_results_index\nclient.fetch_index(\"analysis_results_index\")\nanalysis_info = client.sql_query(\"\"\"\n    SELECT analysis_result_id, analysis_result_title, Subjects, Collections, Modalities\n    FROM analysis_results_index\n\"\"\")\n```\n\n**`collections_index`** provides curated metadata per collection: cancer types, tumor locations, species, subject counts, and supporting data types — without needing to aggregate from the primary index.\n\n**`analysis_results_index`** lists derived datasets (AI segmentations, expert annotations, radiomics features) with their source collections and modalities.\n\n### 2. Querying Metadata with SQL\n\nQuery the IDC mini-index using SQL to find specific datasets.\n\n**First, explore available values for filter columns:**\n```python\nfrom idc_index import IDCClient\n\nclient = IDCClient()\n\n# Check what Modality values exist\nmodalities = client.sql_query(\"\"\"\n    SELECT DISTINCT Modality, COUNT(*) as series_count\n    FROM index\n    GROUP BY Modality\n    ORDER BY series_count DESC\n\"\"\")\nprint(modalities)\n\n# Check what BodyPartExamined values exist for MR modality\nbody_parts = client.sql_query(\"\"\"\n    SELECT DISTINCT BodyPartExamined, COUNT(*) as series_count\n    FROM index\n    WHERE Modality = 'MR' AND BodyPartExamined IS NOT NULL\n    GROUP BY BodyPartExamined\n    ORDER BY series_count DESC\n    LIMIT 20\n\"\"\")\nprint(body_parts)\n```\n\n**Then query with validated filter values:**\n```python\n# Find breast MRI scans (use actual values from exploration above)\nresults = client.sql_query(\"\"\"\n    SELECT\n      collection_id,\n      PatientID,\n      SeriesInstanceUID,\n      Modality,\n      SeriesDescription,\n      license_short_name\n    FROM index\n    WHERE Modality = 'MR'\n      AND BodyPartExamined = 'BREAST'\n    LIMIT 20\n\"\"\")\n\n# Access results as pandas DataFrame\nfor idx, row in results.iterrows():\n    print(f\"Patient: {row['PatientID']}, Series: {row['SeriesInstanceUID']}\")\n```\n\n**To filter by cancer type, join with `collections_index`:**\n```python\nclient.fetch_index(\"collections_index\")\nresults = client.sql_query(\"\"\"\n    SELECT i.collection_id, i.PatientID, i.SeriesInstanceUID, i.Modality\n    FROM index i\n    JOIN collections_index c ON i.collection_id = c.collection_id\n    WHERE c.CancerTypes LIKE '%Breast%'\n      AND i.Modality = 'MR'\n    LIMIT 20\n\"\"\")\n```\n\n**Available metadata fields** (use `client.indices_overview` for complete list):\n- Identifiers: collection_id, PatientID, StudyInstanceUID, SeriesInstanceUID\n- Imaging: Modality, BodyPartExamined, Manufacturer, ManufacturerModelName\n- Clinical: PatientAge, PatientSex, StudyDate\n- Descriptions: StudyDescription, SeriesDescription\n- Licensing: license_short_name\n\n**Note:** Cancer type is in `collections_index.CancerTypes`, not in the primary `index` table.\n\n### 3. Downloading DICOM Files\n\nDownload imaging data efficiently from IDC's cloud storage:\n\n**Download entire collection:**\n```python\nfrom idc_index import IDCClient\n\nclient = IDCClient()\n\n# Download small collection (RIDER Pilot ~1GB)\nclient.download_from_selection(\n    collection_id=\"rider_pilot\",\n    downloadDir=\".\u002Fdata\u002Frider\"\n)\n```\n\n**Download specific series:**\n```python\n# First, query for series UIDs\nseries_df = client.sql_query(\"\"\"\n    SELECT SeriesInstanceUID\n    FROM index\n    WHERE Modality = 'CT'\n      AND BodyPartExamined = 'CHEST'\n      AND collection_id = 'nlst'\n    LIMIT 5\n\"\"\")\n\n# Download only those series\nclient.download_from_selection(\n    seriesInstanceUID=list(series_df['SeriesInstanceUID'].values),\n    downloadDir=\".\u002Fdata\u002Flung_ct\"\n)\n```\n\n**Custom directory structure:**\n\nDefault `dirTemplate`: `%collection_id\u002F%PatientID\u002F%StudyInstanceUID\u002F%Modality_%SeriesInstanceUID`\n\n```python\n# Simplified hierarchy (omit StudyInstanceUID level)\nclient.download_from_selection(\n    collection_id=\"tcga_luad\",\n    downloadDir=\".\u002Fdata\",\n    dirTemplate=\"%collection_id\u002F%PatientID\u002F%Modality\"\n)\n# Results in: .\u002Fdata\u002Ftcga_luad\u002FTCGA-05-4244\u002FCT\u002F\n\n# Flat structure (all files in one directory)\nclient.download_from_selection(\n    seriesInstanceUID=list(series_df['SeriesInstanceUID'].values),\n    downloadDir=\".\u002Fdata\u002Fflat\",\n    dirTemplate=\"\"\n)\n# Results in: .\u002Fdata\u002Fflat\u002F*.dcm\n```\n\n**Downloaded file names:**\n\nIndividual DICOM files are named using their CRDC instance UUID: `\u003Ccrdc_instance_uuid>.dcm` (e.g., `0d73f84e-70ae-4eeb-96a0-1c613b5d9229.dcm`). This UUID-based naming:\n- Enables version tracking (UUIDs change when file content changes)\n- Matches cloud storage organization (`s3:\u002F\u002Fidc-open-data\u002F\u003Ccrdc_series_uuid>\u002F\u003Ccrdc_instance_uuid>.dcm`)\n- Differs from DICOM UIDs (SOPInstanceUID) which are preserved inside the file metadata\n\nTo identify files, use the `crdc_instance_uuid` column in queries or read DICOM metadata (SOPInstanceUID) from the files.\n\n### Command-Line Download\n\nThe `idc download` command provides command-line access to download functionality without writing Python code. Available after installing `idc-index`.\n\n**Auto-detects input type:** manifest file path, or identifiers (collection_id, PatientID, StudyInstanceUID, SeriesInstanceUID, crdc_series_uuid).\n\n```bash\n# Download entire collection\nidc download rider_pilot --download-dir .\u002Fdata\n\n# Download specific series by UID\nidc download \"1.3.6.1.4.1.9328.50.1.69736\" --download-dir .\u002Fdata\n\n# Download multiple items (comma-separated)\nidc download \"tcga_luad,tcga_lusc\" --download-dir .\u002Fdata\n\n# Download from manifest file (auto-detected)\nidc download manifest.txt --download-dir .\u002Fdata\n```\n\n**Options:**\n\n| Option | Description |\n|--------|-------------|\n| `--download-dir` | Output directory (default: current directory) |\n| `--dir-template` | Directory hierarchy template (default: `%collection_id\u002F%PatientID\u002F%StudyInstanceUID\u002F%Modality_%SeriesInstanceUID`) |\n| `--log-level` | Verbosity: debug, info, warning, error, critical |\n\n**Manifest files:**\n\nManifest files contain S3 URLs (one per line) and can be:\n- Exported from the IDC Portal after cohort selection\n- Shared by collaborators for reproducible data access\n- Generated programmatically from query results\n\nFormat (one S3 URL per line):\n```\ns3:\u002F\u002Fidc-open-data\u002Fcb09464a-c5cc-4428-9339-d7fa87cfe837\u002F*\ns3:\u002F\u002Fidc-open-data\u002F88f3990d-bdef-49cd-9b2b-4787767240f2\u002F*\n```\n\n**Example: Generate manifest from Python query:**\n\n```python\nfrom idc_index import IDCClient\n\nclient = IDCClient()\n\n# Query for series URLs\nresults = client.sql_query(\"\"\"\n    SELECT series_aws_url\n    FROM index\n    WHERE collection_id = 'rider_pilot' AND Modality = 'CT'\n\"\"\")\n\n# Save as manifest file\nwith open('ct_manifest.txt', 'w') as f:\n    for url in results['series_aws_url']:\n        f.write(url + '\\n')\n```\n\nThen download:\n```bash\nidc download ct_manifest.txt --download-dir .\u002Fct_data\n```\n\n### 4. Visualizing IDC Images\n\nView DICOM data in browser without downloading:\n\n```python\nfrom idc_index import IDCClient\nimport webbrowser\n\nclient = IDCClient()\n\n# First query to get valid UIDs\nresults = client.sql_query(\"\"\"\n    SELECT SeriesInstanceUID, StudyInstanceUID\n    FROM index\n    WHERE collection_id = 'rider_pilot' AND Modality = 'CT'\n    LIMIT 1\n\"\"\")\n\n# View single series\nviewer_url = client.get_viewer_URL(seriesInstanceUID=results.iloc[0]['SeriesInstanceUID'])\nwebbrowser.open(viewer_url)\n\n# View all series in a study (useful for multi-series exams like MRI protocols)\nviewer_url = client.get_viewer_URL(studyInstanceUID=results.iloc[0]['StudyInstanceUID'])\nwebbrowser.open(viewer_url)\n```\n\nThe method automatically selects OHIF v3 for radiology or SLIM for slide microscopy. Viewing by study is useful when a DICOM Study contains multiple Series (e.g., T1, T2, DWI sequences from a single MRI session).\n\n### 5. Understanding and Checking Licenses\n\nCheck data licensing before use (critical for commercial applications):\n\n```python\nfrom idc_index import IDCClient\n\nclient = IDCClient()\n\n# Check licenses for all collections\nquery = \"\"\"\nSELECT DISTINCT\n  collection_id,\n  license_short_name,\n  COUNT(DISTINCT SeriesInstanceUID) as series_count\nFROM index\nGROUP BY collection_id, license_short_name\nORDER BY collection_id\n\"\"\"\n\nlicenses = client.sql_query(query)\nprint(licenses)\n```\n\n**License types in IDC:**\n- **CC BY 4.0** \u002F **CC BY 3.0** (~97% of data) - Allows commercial use with attribution\n- **CC BY-NC 4.0** \u002F **CC BY-NC 3.0** (~3% of data) - Non-commercial use only\n- **Custom licenses** (rare) - Some collections have specific terms (e.g., NLM Terms and Conditions)\n\n**Important:** Always check the license before using IDC data in publications or commercial applications. Each DICOM file is tagged with its specific license in metadata.\n\n### Generating Citations for Attribution\n\nThe `source_DOI` column contains DOIs linking to publications describing how the data was generated. To satisfy attribution requirements, use `citations_from_selection()` to generate properly formatted citations:\n\n```python\nfrom idc_index import IDCClient\n\nclient = IDCClient()\n\n# Get citations for a collection (APA format by default)\ncitations = client.citations_from_selection(collection_id=\"rider_pilot\")\nfor citation in citations:\n    print(citation)\n\n# Get citations for specific series\nresults = client.sql_query(\"\"\"\n    SELECT SeriesInstanceUID FROM index\n    WHERE collection_id = 'tcga_luad' LIMIT 5\n\"\"\")\ncitations = client.citations_from_selection(\n    seriesInstanceUID=list(results['SeriesInstanceUID'].values)\n)\n\n# Alternative format: BibTeX (for LaTeX documents)\nbibtex_citations = client.citations_from_selection(\n    collection_id=\"tcga_luad\",\n    citation_format=IDCClient.CITATION_FORMAT_BIBTEX\n)\n```\n\n**Parameters:**\n- `collection_id`: Filter by collection(s)\n- `patientId`: Filter by patient ID(s)\n- `studyInstanceUID`: Filter by study UID(s)\n- `seriesInstanceUID`: Filter by series UID(s)\n- `citation_format`: Use `IDCClient.CITATION_FORMAT_*` constants:\n  - `CITATION_FORMAT_APA` (default) - APA style\n  - `CITATION_FORMAT_BIBTEX` - BibTeX for LaTeX\n  - `CITATION_FORMAT_JSON` - CSL JSON\n  - `CITATION_FORMAT_TURTLE` - RDF Turtle\n\n**Best practice:** When publishing results using IDC data, include the generated citations to properly attribute the data sources and satisfy license requirements.\n\n### 6. Batch Processing and Filtering\n\nProcess large datasets efficiently with filtering:\n\n```python\nfrom idc_index import IDCClient\nimport pandas as pd\n\nclient = IDCClient()\n\n# Find chest CT scans from GE scanners\nquery = \"\"\"\nSELECT\n  SeriesInstanceUID,\n  PatientID,\n  collection_id,\n  ManufacturerModelName\nFROM index\nWHERE Modality = 'CT'\n  AND BodyPartExamined = 'CHEST'\n  AND Manufacturer = 'GE MEDICAL SYSTEMS'\n  AND license_short_name = 'CC BY 4.0'\nLIMIT 100\n\"\"\"\n\nresults = client.sql_query(query)\n\n# Save manifest for later\nresults.to_csv('lung_ct_manifest.csv', index=False)\n\n# Download in batches to avoid timeout\nbatch_size = 10\nfor i in range(0, len(results), batch_size):\n    batch = results.iloc[i:i+batch_size]\n    client.download_from_selection(\n        seriesInstanceUID=list(batch['SeriesInstanceUID'].values),\n        downloadDir=f\".\u002Fdata\u002Fbatch_{i\u002F\u002Fbatch_size}\"\n    )\n```\n\n### 7. Advanced Queries with BigQuery\n\nFor queries requiring full DICOM metadata, complex JOINs, clinical data tables, or private DICOM elements, use Google BigQuery. Requires GCP account with billing enabled.\n\n**Quick reference:**\n- Dataset: `bigquery-public-data.idc_current.*`\n- Main table: `dicom_all` (combined metadata)\n- Full metadata: `dicom_metadata` (all DICOM tags)\n- Private elements: `OtherElements` column (vendor-specific tags like diffusion b-values)\n\nSee `references\u002Fbigquery_guide.md` for setup, table schemas, query patterns, private element access, and cost optimization.\n\n**Before using BigQuery**, always check if a specialized index table already has the metadata you need:\n1. Use `client.indices_overview` or the [idc-index indices reference](https:\u002F\u002Fidc-index.readthedocs.io\u002Fen\u002Flatest\u002Findices_reference.html) to discover all available tables and their columns\n2. Fetch the relevant index: `client.fetch_index(\"table_name\")`\n3. Query locally with `client.sql_query()` (free, no GCP account needed)\n\nCommon specialized indices: `seg_index` (segmentations), `ann_index` \u002F `ann_group_index` (microscopy annotations), `sm_index` (slide microscopy), `collections_index` (collection metadata). Only use BigQuery if you need private DICOM elements or attributes not in any index.\n\n### 8. Tool Selection Guide\n\n| Task | Tool | Reference |\n|------|------|-----------|\n| Programmatic queries & downloads | `idc-index` | This document |\n| Interactive exploration | IDC Portal | https:\u002F\u002Fportal.imaging.datacommons.cancer.gov\u002F |\n| Complex metadata queries | BigQuery | `references\u002Fbigquery_guide.md` |\n| 3D visualization & analysis | SlicerIDCBrowser | https:\u002F\u002Fgithub.com\u002FImagingDataCommons\u002FSlicerIDCBrowser |\n\n**Default choice:** Use `idc-index` for most tasks (no auth, easy API, batch downloads).\n\n### 9. Integration with Analysis Pipelines\n\nIntegrate IDC data into imaging analysis workflows:\n\n**Read downloaded DICOM files:**\n```python\nimport pydicom\nimport os\n\n# Read DICOM files from downloaded series\nseries_dir = \".\u002Fdata\u002Frider\u002Frider_pilot\u002FRIDER-1007893286\u002FCT_1.3.6.1...\"\n\ndicom_files = [os.path.join(series_dir, f) for f in os.listdir(series_dir)\n               if f.endswith('.dcm')]\n\n# Load first image\nds = pydicom.dcmread(dicom_files[0])\nprint(f\"Patient ID: {ds.PatientID}\")\nprint(f\"Modality: {ds.Modality}\")\nprint(f\"Image shape: {ds.pixel_array.shape}\")\n```\n\n**Build 3D volume from CT series:**\n```python\nimport pydicom\nimport numpy as np\nfrom pathlib import Path\n\ndef load_ct_series(series_path):\n    \"\"\"Load CT series as 3D numpy array\"\"\"\n    files = sorted(Path(series_path).glob('*.dcm'))\n    slices = [pydicom.dcmread(str(f)) for f in files]\n\n    # Sort by slice location\n    slices.sort(key=lambda x: float(x.ImagePositionPatient[2]))\n\n    # Stack into 3D array\n    volume = np.stack([s.pixel_array for s in slices])\n\n    return volume, slices[0]  # Return volume and first slice for metadata\n\nvolume, metadata = load_ct_series(\".\u002Fdata\u002Flung_ct\u002Fseries_dir\")\nprint(f\"Volume shape: {volume.shape}\")  # (z, y, x)\n```\n\n**Integrate with SimpleITK:**\n```python\nimport SimpleITK as sitk\nfrom pathlib import Path\n\n# Read DICOM series\nseries_path = \".\u002Fdata\u002Fct_series\"\nreader = sitk.ImageSeriesReader()\ndicom_names = reader.GetGDCMSeriesFileNames(series_path)\nreader.SetFileNames(dicom_names)\nimage = reader.Execute()\n\n# Apply processing\nsmoothed = sitk.CurvatureFlow(image1=image, timeStep=0.125, numberOfIterations=5)\n\n# Save as NIfTI\nsitk.WriteImage(smoothed, \"processed_volume.nii.gz\")\n```\n\n## Common Use Cases\n\nSee `references\u002Fuse_cases.md` for complete end-to-end workflow examples including:\n- Building deep learning training datasets from lung CT scans\n- Comparing image quality across scanner manufacturers\n- Previewing data in browser before downloading\n- License-aware batch downloads for commercial use\n\n## Best Practices\n\n- **Verify IDC version before generating responses** - Always call `client.get_idc_version()` at the start of a session to confirm you're using the expected data version (currently v23). If using an older version, recommend `pip install --upgrade idc-index`\n- **Check licenses before use** - Always query the `license_short_name` field and respect licensing terms (CC BY vs CC BY-NC)\n- **Generate citations for attribution** - Use `citations_from_selection()` to get properly formatted citations from `source_DOI` values; include these in publications\n- **Start with small queries** - Use `LIMIT` clause when exploring to avoid long downloads and understand data structure\n- **Use mini-index for simple queries** - Only use BigQuery when you need comprehensive metadata or complex JOINs\n- **Organize downloads with dirTemplate** - Use meaningful directory structures like `%collection_id\u002F%PatientID\u002F%Modality`\n- **Cache query results** - Save DataFrames to CSV files to avoid re-querying and ensure reproducibility\n- **Estimate size first** - Check collection size before downloading - some collection sizes are in terabytes!\n- **Save manifests** - Always save query results with Series UIDs for reproducibility and data provenance\n- **Read documentation** - IDC data structure and metadata fields are documented at https:\u002F\u002Flearn.canceridc.dev\u002F\n- **Use IDC forum** - Search for questons\u002Fanswers and ask your questions to the IDC maintainers and users at https:\u002F\u002Fdiscourse.canceridc.dev\u002F\n\n## Troubleshooting\n\n**Issue: `ModuleNotFoundError: No module named 'idc_index'`**\n- **Cause:** idc-index package not installed\n- **Solution:** Install with `pip install --upgrade idc-index`\n\n**Issue: Download fails with connection timeout**\n- **Cause:** Network instability or large download size\n- **Solution:**\n  - Download smaller batches (e.g., 10-20 series at a time)\n  - Check network connection\n  - Use `dirTemplate` to organize downloads by batch\n  - Implement retry logic with delays\n\n**Issue: `BigQuery quota exceeded` or billing errors**\n- **Cause:** BigQuery requires billing-enabled GCP project\n- **Solution:** Use idc-index mini-index for simple queries (no billing required), or see `references\u002Fbigquery_guide.md` for cost optimization tips\n\n**Issue: Series UID not found or no data returned**\n- **Cause:** Typo in UID, data not in current IDC version, or wrong field name\n- **Solution:**\n  - Check if data is in current IDC version (some old data may be deprecated)\n  - Use `LIMIT 5` to test query first\n  - Check field names against metadata schema documentation\n\n**Issue: Downloaded DICOM files won't open**\n- **Cause:** Corrupted download or incompatible viewer\n- **Solution:**\n  - Check DICOM object type (Modality and SOPClassUID attributes) - some object types require specialized tools\n  - Verify file integrity (check file sizes)\n  - Use pydicom to validate: `pydicom.dcmread(file, force=True)`\n  - Try different DICOM viewer (3D Slicer, Horos, RadiAnt, QuPath)\n  - Re-download the series\n\n## Common SQL Query Patterns\n\nSee `references\u002Fsql_patterns.md` for quick-reference SQL patterns including:\n- Filter value discovery (modalities, body parts, manufacturers)\n- Annotation and segmentation queries (including seg_index, ann_index joins)\n- Slide microscopy queries (sm_index patterns)\n- Download size estimation\n- Clinical data linking\n\nFor segmentation and annotation details, also see `references\u002Fdigital_pathology_guide.md`.\n\n## Related Skills\n\nThe following skills complement IDC workflows for downstream analysis and visualization:\n\n### DICOM Processing\n- **pydicom** - Read, write, and manipulate downloaded DICOM files. Use for extracting pixel data, reading metadata, anonymization, and format conversion. Essential for working with IDC radiology data (CT, MR, PET).\n\n### Pathology and Slide Microscopy\nSee `references\u002Fdigital_pathology_guide.md` for DICOM-compatible tools (highdicom, wsidicom, TIA-Toolbox, Slim viewer).\n\n### Metadata Visualization\n- **matplotlib** - Low-level plotting for full customization. Use for creating static figures summarizing IDC query results (bar charts of modalities, histograms of series counts, etc.).\n- **seaborn** - Statistical visualization with pandas integration. Use for quick exploration of IDC metadata distributions, relationships between variables, and categorical comparisons with attractive defaults.\n- **plotly** - Interactive visualization. Use when you need hover info, zoom, and pan for exploring IDC metadata, or for creating web-embeddable dashboards of collection statistics.\n\n### Data Exploration\n- **exploratory-data-analysis** - Comprehensive EDA on scientific data files. Use after downloading IDC data to understand file structure, quality, and characteristics before analysis.\n\n## Resources\n\n### Schema Reference (Primary Source)\n\n**Always use `client.indices_overview` for current column schemas.** This ensures accuracy with the installed idc-index version:\n\n```python\n# Get all column names and types for any table\nschema = client.indices_overview[\"index\"][\"schema\"]\ncolumns = [(c['name'], c['type'], c.get('description', '')) for c in schema['columns']]\n```\n\n### Reference Documentation\n\nSee the Quick Navigation section at the top for the full list of reference guides with decision triggers.\n\n- **[indices_reference](https:\u002F\u002Fidc-index.readthedocs.io\u002Fen\u002Flatest\u002Findices_reference.html)** - External documentation for index tables (may be ahead of the installed version)\n\n### External Links\n\n- **IDC Portal**: https:\u002F\u002Fportal.imaging.datacommons.cancer.gov\u002Fexplore\u002F\n- **Documentation**: https:\u002F\u002Flearn.canceridc.dev\u002F\n- **Tutorials**: https:\u002F\u002Fgithub.com\u002FImagingDataCommons\u002FIDC-Tutorials\n- **User Forum**: https:\u002F\u002Fdiscourse.canceridc.dev\u002F\n- **idc-index GitHub**: https:\u002F\u002Fgithub.com\u002FImagingDataCommons\u002Fidc-index\n- **Citation**: Fedorov, A., et al. \"National Cancer Institute Imaging Data Commons: Toward Transparency, Reproducibility, and Scalability in Imaging Artificial Intelligence.\" RadioGraphics 43.12 (2023). https:\u002F\u002Fdoi.org\u002F10.1148\u002Frg.230180\n\n### Skill Updates\n\nThis skill version is available in skill metadata. To check for updates:\n- Visit the [releases page](https:\u002F\u002Fgithub.com\u002FImagingDataCommons\u002Fidc-claude-skill\u002Freleases)\n- Watch the repository on GitHub (Watch → Custom → Releases)\n",{},"main","def8531e65114c0fca8fb8551c1871ee0eed705c","skills\u002FK-Dense-AI\u002Fclaude-scientific-skills\u002Fimaging-data-commons",24,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub","https:\u002F\u002Fskillshub.wtf","https:\u002F\u002Fgithub.com\u002FComeOnOliver","AI & Blockchain Engineer\r\n","2026-06-24 22:07:19",{"source":7,"skillName":5,"commands":40,"prompt":50,"zipAvailable":51},[41,44,47],{"tool":42,"command":43},"npx","npx skills add ComeOnOliver\u002Fskillshub --skill imaging-data-commons",{"tool":45,"command":46},"bunx","bunx skills add ComeOnOliver\u002Fskillshub --skill imaging-data-commons",{"tool":48,"command":49},"pnpm","pnpm dlx skills add ComeOnOliver\u002Fskillshub --skill imaging-data-commons","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002FK-Dense-AI\u002Fclaude-scientific-skills\u002Fimaging-data-commons\u002FSKILL.md",true,[],[54,68,78,87,93,100],{"skillNo":55,"name":56,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":57,"categories":58,"keywords":64,"publishedAt":26},"01M6FKJN13GMG4KY57J3QD8GY5","model-hierarchy-skill","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzscole\u002Fmodel-hierarchy-skill\u002Fmodel-hierarchy-skill\u002FSKILL.md",[59],{"categoryNo":60,"categoryName":61,"taxonomyType":62,"taxonomyTypeDesc":63},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[65,66,67],"model","model hierarchy","model hierarchy skill",{"skillNo":69,"name":70,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":71,"categories":72,"keywords":74,"publishedAt":26},"01M6FKJJJ6GCAMR4PPPRD12M5X","zoom-mcp","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-mcp\u002FSKILL.md",[73],{"categoryNo":60,"categoryName":61,"taxonomyType":62,"taxonomyTypeDesc":63},[75,76,77],"MCP","zoom","zoom mcp",{"skillNo":79,"name":80,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":26},"01M6FKJE3KYT3KPEZMEEHYKA6K","zoom-apps-sdk","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-apps-sdk\u002FSKILL.md",[],[84,85,86,76],"sdk","zoom apps sdk","apps",{"skillNo":88,"name":89,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":26},"01M6FKJ9EDH9Q5VWDF8HAJRSP0","windows","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwindows\u002FSKILL.md",[],[89],{"skillNo":94,"name":95,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":99,"publishedAt":26},"01M6FKJ666BEEFC3AE9GJ72Q93","whiteboard","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwhiteboard\u002FSKILL.md",[98],{"categoryNo":60,"categoryName":61,"taxonomyType":62,"taxonomyTypeDesc":63},[95],{"skillNo":101,"name":102,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":103,"categories":104,"keywords":105,"publishedAt":26},"01M6FKJ21WFCDGTJQ2Q91T9DVW","websockets","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwebsockets\u002FSKILL.md",[],[102]]