[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M74ZP97Q08ZCGZT46DP46ZNC":3,"skill-same:01M74ZP97Q08ZCGZT46DP46ZNC":54},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":30,"commitSha":31,"skillDirectory":32,"forksCount":33,"repositoryHtmlUrl":34,"repositoryHomepageUrl":35,"ownerHtmlUrl":36,"ownerBlogUrl":37,"ownerDescription":38,"updatedAt":39,"install":40,"resources":53},"01M74ZP97Q08ZCGZT46DP46ZNC","tiledbvcf",null,"Stores and queries genomic variant data in TileDB-VCF datasets using the tiledbvcf Python API and CLI (create, store, export, list, stat). Covers ingesting single-sample VCF\u002FBCF files with .csi or .tbi indexes, adding samples incrementally, querying regions and samples in parallel, and exporting to VCF\u002FBCF or TSV, on local disk or S3, Azure, and GCS. Use when building a variant database for a cohort, adding new samples to an existing dataset, querying specific regions across many samples, exporting subsets of a large VCF collection, or preparing population genomics data such as allele frequency or GWAS inputs. Not for multi-sample VCFs, which are unsupported, or for one-off parsing of a single VCF file.","KalarisLabs\u002Fresearch-agent-skills","KalarisLabs","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F300878568?v=4","01M74Y7MT23GXXC56VM6TVSXZY","Kalarislabs","01M74Y7MTQM9REXZX4SNXWX95C",132,"MIT","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ftiledbvcf\u002FSKILL.md",[],[],"2026-11-11 17:47:46",false,"---\nname: tiledbvcf\ndescription: Stores and queries genomic variant data in TileDB-VCF datasets using the tiledbvcf Python API and CLI (create, store, export, list, stat). Covers ingesting single-sample VCF\u002FBCF files with .csi or .tbi indexes, adding samples incrementally, querying regions and samples in parallel, and exporting to VCF\u002FBCF or TSV, on local disk or S3, Azure, and GCS. Use when building a variant database for a cohort, adding new samples to an existing dataset, querying specific regions across many samples, exporting subsets of a large VCF collection, or preparing population genomics data such as allele frequency or GWAS inputs. Not for multi-sample VCFs, which are unsupported, or for one-off parsing of a single VCF file.\nlicense: MIT\nmetadata:\n  version: '1.1'\n  category: life-sciences\n  maintainer: Kalaris Labs\n  contributor: Jeremy Leipzig\n  tags: genomic variants, VCF, variant storage, population genomics\n---\n\n# TileDB-VCF\n\n## Overview\n\nTileDB-VCF is a high-performance C++ library with Python and CLI interfaces for efficient storage and retrieval of genomic variant-call data. Built on TileDB's sparse array technology, it enables scalable ingestion of VCF\u002FBCF files, incremental sample addition without expensive merging operations, and efficient parallel queries of variant data stored locally or in the cloud.\n\n## When to Use This Skill\n\nThis skill should be used when:\n- Learning TileDB-VCF concepts and workflows\n- Prototyping genomics analyses and pipelines\n- Working with small-to-medium datasets (\u003C 1000 samples)\n- Need incremental addition of new samples to existing datasets\n- Require efficient querying of specific genomic regions across many samples\n- Working with cloud-stored variant data (S3, Azure, GCS)\n- Need to export subsets of large VCF datasets\n- Building variant databases for cohort studies\n- Educational projects and method development\n- Performance is critical for variant data operations\n\n## Quick Start\n\n### Installation\n\n**Preferred Method: Conda\u002FMamba**\n```bash\n# Enter the following two lines if you are on a M1 Mac\nCONDA_SUBDIR=osx-64\nconda config --env --set subdir osx-64\n\n# Create the conda environment\nconda create -n tiledb-vcf \"python\u003C3.10\"\nconda activate tiledb-vcf\n\n# Mamba is a faster and more reliable alternative to conda\nconda install -c conda-forge mamba\n\n# Install TileDB-Py and TileDB-VCF, align with other useful libraries\nmamba install -y -c conda-forge -c bioconda -c tiledb tiledb-py tiledbvcf-py pandas pyarrow numpy\n```\n\n**Alternative: Docker Images**\n```bash\ndocker pull tiledb\u002Ftiledbvcf-py     # Python interface\ndocker pull tiledb\u002Ftiledbvcf-cli    # Command-line interface\n```\n\n### Basic Examples\n\n**Create and populate a dataset:**\n```python\nimport tiledbvcf\n\n# Create a new dataset\nds = tiledbvcf.Dataset(uri=\"my_dataset\", mode=\"w\",\n                      cfg=tiledbvcf.ReadConfig(memory_budget=1024))\n\n# Ingest VCF files (must be single-sample with indexes)\n# Requirements:\n# - VCFs must be single-sample (not multi-sample)\n# - Must have indexes: .csi (bcftools) or .tbi (tabix)\nds.ingest_samples([\"sample1.vcf.gz\", \"sample2.vcf.gz\"])\n```\n\n**Query variant data:**\n```python\n# Open existing dataset for reading\nds = tiledbvcf.Dataset(uri=\"my_dataset\", mode=\"r\")\n\n# Query specific regions and samples\ndf = ds.read(\n    attrs=[\"sample_name\", \"pos_start\", \"pos_end\", \"alleles\", \"fmt_GT\"],\n    regions=[\"chr1:1000000-2000000\", \"chr2:500000-1500000\"],\n    samples=[\"sample1\", \"sample2\", \"sample3\"]\n)\nprint(df.head())\n```\n\n**Export to VCF:**\n```python\nimport os\n\n# Export two VCF samples\nds.export(\n    regions=[\"chr21:8220186-8405573\"],\n    samples=[\"HG00101\", \"HG00097\"],\n    output_format=\"v\",\n    output_dir=os.path.expanduser(\"~\"),\n)\n```\n\n## Core Capabilities\n\n### 1. Dataset Creation and Ingestion\n\nCreate TileDB-VCF datasets and incrementally ingest variant data from multiple VCF\u002FBCF files. This is appropriate for building population genomics databases and cohort studies.\n\n**Requirements:**\n- **Single-sample VCFs only**: Multi-sample VCFs are not supported\n- **Index files required**: VCF\u002FBCF files must have indexes (.csi or .tbi)\n\n**Common operations:**\n- Create new datasets with optimized array schemas\n- Ingest single or multiple VCF\u002FBCF files in parallel\n- Add new samples incrementally without re-processing existing data\n- Configure memory usage and compression settings\n- Handle various VCF formats and INFO\u002FFORMAT fields\n- Resume interrupted ingestion processes\n- Validate data integrity during ingestion\n\n\n### 2. Efficient Querying and Filtering\n\nQuery variant data with high performance across genomic regions, samples, and variant attributes. This is appropriate for association studies, variant discovery, and population analysis.\n\n**Common operations:**\n- Query specific genomic regions (single or multiple)\n- Filter by sample names or sample groups\n- Extract specific variant attributes (position, alleles, genotypes, quality)\n- Access INFO and FORMAT fields efficiently\n- Combine spatial and attribute-based filtering\n- Stream large query results\n- Perform aggregations across samples or regions\n\n\n### 3. Data Export and Interoperability\n\nExport data in various formats for downstream analysis or integration with other genomics tools. This is appropriate for sharing datasets, creating analysis subsets, or feeding other pipelines.\n\n**Common operations:**\n- Export to standard VCF\u002FBCF formats\n- Generate TSV files with selected fields\n- Create sample\u002Fregion-specific subsets\n- Maintain data provenance and metadata\n- Lossless data export preserving all annotations\n- Compressed output formats\n- Streaming exports for large datasets\n\n\n### 4. Population Genomics Workflows\n\nTileDB-VCF excels at large-scale population genomics analyses requiring efficient access to variant data across many samples and genomic regions.\n\n**Common workflows:**\n- Genome-wide association studies (GWAS) data preparation\n- Rare variant burden testing\n- Population stratification analysis\n- Allele frequency calculations across populations\n- Quality control across large cohorts\n- Variant annotation and filtering\n- Cross-population comparative analysis\n\n\n## Key Concepts\n\n### Array Schema and Data Model\n\n**TileDB-VCF Data Model:**\n- Variants stored as sparse arrays with genomic coordinates as dimensions\n- Samples stored as attributes allowing efficient sample-specific queries\n- INFO and FORMAT fields preserved with original data types\n- Automatic compression and chunking for optimal storage\n\n**Schema Configuration:**\n```python\n# Custom schema with specific tile extents\nconfig = tiledbvcf.ReadConfig(\n    memory_budget=2048,  # MB\n    region_partition=(0, 3095677412),  # Full genome\n    sample_partition=(0, 10000)  # Up to 10k samples\n)\n```\n\n### Coordinate Systems and Regions\n\n**Critical:** TileDB-VCF uses **1-based genomic coordinates** following VCF standard:\n- Positions are 1-based (first base is position 1)\n- Ranges are inclusive on both ends\n- Region \"chr1:1000-2000\" includes positions 1000-2000 (1001 bases total)\n\n**Region specification formats:**\n```python\n# Single region\nregions = [\"chr1:1000000-2000000\"]\n\n# Multiple regions\nregions = [\"chr1:1000000-2000000\", \"chr2:500000-1500000\"]\n\n# Whole chromosome\nregions = [\"chr1\"]\n\n# BED-style (0-based, half-open converted internally)\nregions = [\"chr1:999999-2000000\"]  # Equivalent to 1-based chr1:1000000-2000000\n```\n\n### Memory Management\n\n**Performance considerations:**\n1. **Set appropriate memory budget** based on available system memory\n2. **Use streaming queries** for very large result sets\n3. **Partition large ingestions** to avoid memory exhaustion\n4. **Configure tile cache** for repeated region access\n5. **Use parallel ingestion** for multiple files\n6. **Optimize region queries** by combining nearby regions\n\n### Cloud Storage Integration\n\nTileDB-VCF seamlessly works with cloud storage:\n```python\n# S3 dataset\nds = tiledbvcf.Dataset(uri=\"s3:\u002F\u002Fbucket\u002Fdataset\", mode=\"r\")\n\n# Azure Blob Storage\nds = tiledbvcf.Dataset(uri=\"azure:\u002F\u002Fcontainer\u002Fdataset\", mode=\"r\")\n\n# Google Cloud Storage\nds = tiledbvcf.Dataset(uri=\"gcs:\u002F\u002Fbucket\u002Fdataset\", mode=\"r\")\n```\n\n## Common Pitfalls\n\n1. **Memory exhaustion during ingestion:** Use appropriate memory budget and batch processing for large VCF files\n2. **Inefficient region queries:** Combine nearby regions instead of many separate queries\n3. **Missing sample names:** Ensure sample names in VCF headers match query sample specifications\n4. **Coordinate system confusion:** Remember TileDB-VCF uses 1-based coordinates like VCF standard\n5. **Large result sets:** Use streaming or pagination for queries returning millions of variants\n6. **Cloud permissions:** Ensure proper authentication for cloud storage access\n7. **Concurrent access:** Multiple writers to the same dataset can cause corruption—use appropriate locking\n\n## CLI Usage\n\nTileDB-VCF provides a command-line interface with the following subcommands:\n\n**Available Subcommands:**\n- `create` - Creates an empty TileDB-VCF dataset\n- `store` - Ingests samples into a TileDB-VCF dataset\n- `export` - Exports data from a TileDB-VCF dataset\n- `list` - Lists all sample names present in a TileDB-VCF dataset\n- `stat` - Prints high-level statistics about a TileDB-VCF dataset\n- `utils` - Utils for working with a TileDB-VCF dataset\n- `version` - Print the version information and exit\n\n```bash\n# Create empty dataset\ntiledbvcf create --uri my_dataset\n\n# Ingest samples (requires single-sample VCFs with indexes)\ntiledbvcf store --uri my_dataset --samples sample1.vcf.gz,sample2.vcf.gz\n\n# Export data\ntiledbvcf export --uri my_dataset \\\n  --regions \"chr1:1000000-2000000\" \\\n  --sample-names \"sample1,sample2\"\n\n# List all samples\ntiledbvcf list --uri my_dataset\n\n# Show dataset statistics\ntiledbvcf stat --uri my_dataset\n```\n\n## Advanced Features\n\n### Allele Frequency Analysis\n```python\n# Calculate allele frequencies\naf_df = tiledbvcf.read_allele_frequency(\n    uri=\"my_dataset\",\n    regions=[\"chr1:1000000-2000000\"],\n    samples=[\"sample1\", \"sample2\", \"sample3\"]\n)\n```\n\n### Sample Quality Control\n```python\n# Perform sample QC\nqc_results = tiledbvcf.sample_qc(\n    uri=\"my_dataset\",\n    samples=[\"sample1\", \"sample2\"]\n)\n```\n\n### Custom Configurations\n```python\n# Advanced configuration\nconfig = tiledbvcf.ReadConfig(\n    memory_budget=4096,\n    tiledb_config={\n        \"sm.tile_cache_size\": \"1000000000\",\n        \"vfs.s3.region\": \"us-east-1\"\n    }\n)\n```\n\n\n## Resources\n\n## Getting Help\n\n### Open Source TileDB-VCF Resources\n\n**Open Source Documentation:**\n- TileDB Academy: https:\u002F\u002Fcloud.tiledb.com\u002Facademy\u002F\n- Population Genomics Guide: https:\u002F\u002Fcloud.tiledb.com\u002Facademy\u002Fstructure\u002Flife-sciences\u002Fpopulation-genomics\u002F\n- TileDB-VCF GitHub: https:\u002F\u002Fgithub.com\u002FTileDB-Inc\u002FTileDB-VCF\n\n### TileDB-Cloud Resources\n\n**For Large-Scale\u002FProduction Genomics:**\n- TileDB-Cloud Platform: https:\u002F\u002Fcloud.tiledb.com\n- TileDB Academy (All Documentation): https:\u002F\u002Fcloud.tiledb.com\u002Facademy\u002F\n\n**Getting Started:**\n- Free account signup: https:\u002F\u002Fcloud.tiledb.com\n- Contact: sales@tiledb.com for enterprise needs\n\n## Scaling to TileDB-Cloud\n\nWhen your genomics workloads outgrow single-node processing, TileDB-Cloud provides enterprise-scale capabilities for production genomics pipelines.\n\n**Note**: This section covers TileDB-Cloud capabilities based on available documentation. For complete API details and current functionality, consult the official TileDB-Cloud documentation and API reference.\n\n### Setting Up TileDB-Cloud\n\n**1. Create Account and Get API Token**\n```bash\n# Sign up at https:\u002F\u002Fcloud.tiledb.com\n# Generate API token in your account settings\n```\n\n**2. Install TileDB-Cloud Python Client**\n```bash\n# Base installation\nuv pip install tiledb-cloud\n\n# With genomics-specific functionality\nuv pip install tiledb-cloud[life-sciences]\n```\n\n**3. Configure Authentication**\n```bash\n# Set environment variable with your API token\nexport TILEDB_REST_TOKEN=\"your_api_token\"\n```\n\n```python\nimport tiledb.cloud\n\n# Authentication is automatic via TILEDB_REST_TOKEN\n# No explicit login required in code\n```\n\n### Migrating from Open Source to TileDB-Cloud\n\n**Large-Scale Ingestion**\n```python\n# TileDB-Cloud: Distributed VCF ingestion\nimport tiledb.cloud.vcf\n\n# Use specialized VCF ingestion module\n# Note: Exact API requires TileDB-Cloud documentation\n# This represents the available functionality structure\ntiledb.cloud.vcf.ingestion.ingest_vcf_dataset(\n    source=\"s3:\u002F\u002Fmy-bucket\u002Fvcf-files\u002F\",\n    output=\"tiledb:\u002F\u002Fmy-namespace\u002Flarge-dataset\",\n    namespace=\"my-namespace\",\n    acn=\"my-s3-credentials\",\n    ingest_resources={\"cpu\": \"16\", \"memory\": \"64Gi\"}\n)\n```\n\n**Distributed Query Processing**\n```python\n# TileDB-Cloud: VCF querying across distributed storage\nimport tiledb.cloud.vcf\nimport tiledbvcf\n\n# Define the dataset URI\ndataset_uri = \"tiledb:\u002F\u002FTileDB-Inc\u002Fgvcf-1kg-dragen-v376\"\n\n# Get all samples from the dataset\nds = tiledbvcf.Dataset(dataset_uri, tiledb_config=cfg)\nsamples = ds.samples()\n\n# Define attributes and ranges to query on\nattrs = [\"sample_name\", \"fmt_GT\", \"fmt_AD\", \"fmt_DP\"]\nregions = [\"chr13:32396898-32397044\", \"chr13:32398162-32400268\"]\n\n# Perform the read, which is executed in a distributed fashion\ndf = tiledb.cloud.vcf.read(\n    dataset_uri=dataset_uri,\n    regions=regions,\n    samples=samples,\n    attrs=attrs,\n    namespace=\"my-namespace\",  # specifies which account to charge\n)\ndf.to_pandas()\n```\n\n### Enterprise Features\n\n**Data Sharing and Collaboration**\n```python\n# TileDB-Cloud provides enterprise data sharing capabilities\n# through namespace-based permissions and group management\n\n# Access shared datasets via TileDB-Cloud URIs\ndataset_uri = \"tiledb:\u002F\u002Fshared-namespace\u002Fpopulation-study\"\n\n# Collaborate through shared notebooks and compute resources\n# (Specific API requires TileDB-Cloud documentation)\n```\n\n**Cost Optimization**\n- **Serverless Compute**: Pay only for actual compute time\n- **Auto-scaling**: Automatically scale up\u002Fdown based on workload\n- **Spot Instances**: Use cost-optimized compute for batch jobs\n- **Data Tiering**: Automatic hot\u002Fcold storage management\n\n**Security and Compliance**\n- **End-to-end Encryption**: Data encrypted in transit and at rest\n- **Access Controls**: Fine-grained permissions and audit logs\n- **HIPAA\u002FSOC2 Compliance**: Enterprise security standards\n- **VPC Support**: Deploy in private cloud environments\n\n### When to Migrate Checklist\n\n✅ **Migrate to TileDB-Cloud if you have:**\n- [ ] Datasets > 1000 samples\n- [ ] Need to process > 100GB of VCF data\n- [ ] Require distributed computing\n- [ ] Multiple team members need access\n- [ ] Need enterprise security\u002Fcompliance\n- [ ] Want cost-optimized serverless compute\n- [ ] Require 24\u002F7 production uptime\n\n### Getting Started with TileDB-Cloud\n\n1. **Start Free**: TileDB-Cloud offers free tier for evaluation\n2. **Migration Support**: TileDB team provides migration assistance\n3. **Training**: Access to genomics-specific tutorials and examples\n4. **Professional Services**: Custom deployment and optimization\n\n**Next Steps:**\n- Visit https:\u002F\u002Fcloud.tiledb.com to create account\n- Review documentation at https:\u002F\u002Fcloud.tiledb.com\u002Facademy\u002F\n- Contact sales@tiledb.com for enterprise needs\n\n## Related skills\n\n- `pysam`: Python\u002FHTSlib workflows for genomic files.\n- `polars-bio`: Python library polars-bio for genomic interval operations and bioinformatics file I\u002FO on Polars DataFrames, built on Arrow and DataFusion.\n- `pinecone`: Guides use of Pinecone, a managed serverless vector database, through its Python client and the LangChain and LlamaIndex integrations.\n","# TileDB-VCF\n\n## Overview\n\nTileDB-VCF is a high-performance C++ library with Python and CLI interfaces for efficient storage and retrieval of genomic variant-call data. Built on TileDB's sparse array technology, it enables scalable ingestion of VCF\u002FBCF files, incremental sample addition without expensive merging operations, and efficient parallel queries of variant data stored locally or in the cloud.\n\n## When to Use This Skill\n\nThis skill should be used when:\n- Learning TileDB-VCF concepts and workflows\n- Prototyping genomics analyses and pipelines\n- Working with small-to-medium datasets (\u003C 1000 samples)\n- Need incremental addition of new samples to existing datasets\n- Require efficient querying of specific genomic regions across many samples\n- Working with cloud-stored variant data (S3, Azure, GCS)\n- Need to export subsets of large VCF datasets\n- Building variant databases for cohort studies\n- Educational projects and method development\n- Performance is critical for variant data operations\n\n## Quick Start\n\n### Installation\n\n**Preferred Method: Conda\u002FMamba**\n```bash\n# Enter the following two lines if you are on a M1 Mac\nCONDA_SUBDIR=osx-64\nconda config --env --set subdir osx-64\n\n# Create the conda environment\nconda create -n tiledb-vcf \"python\u003C3.10\"\nconda activate tiledb-vcf\n\n# Mamba is a faster and more reliable alternative to conda\nconda install -c conda-forge mamba\n\n# Install TileDB-Py and TileDB-VCF, align with other useful libraries\nmamba install -y -c conda-forge -c bioconda -c tiledb tiledb-py tiledbvcf-py pandas pyarrow numpy\n```\n\n**Alternative: Docker Images**\n```bash\ndocker pull tiledb\u002Ftiledbvcf-py     # Python interface\ndocker pull tiledb\u002Ftiledbvcf-cli    # Command-line interface\n```\n\n### Basic Examples\n\n**Create and populate a dataset:**\n```python\nimport tiledbvcf\n\n# Create a new dataset\nds = tiledbvcf.Dataset(uri=\"my_dataset\", mode=\"w\",\n                      cfg=tiledbvcf.ReadConfig(memory_budget=1024))\n\n# Ingest VCF files (must be single-sample with indexes)\n# Requirements:\n# - VCFs must be single-sample (not multi-sample)\n# - Must have indexes: .csi (bcftools) or .tbi (tabix)\nds.ingest_samples([\"sample1.vcf.gz\", \"sample2.vcf.gz\"])\n```\n\n**Query variant data:**\n```python\n# Open existing dataset for reading\nds = tiledbvcf.Dataset(uri=\"my_dataset\", mode=\"r\")\n\n# Query specific regions and samples\ndf = ds.read(\n    attrs=[\"sample_name\", \"pos_start\", \"pos_end\", \"alleles\", \"fmt_GT\"],\n    regions=[\"chr1:1000000-2000000\", \"chr2:500000-1500000\"],\n    samples=[\"sample1\", \"sample2\", \"sample3\"]\n)\nprint(df.head())\n```\n\n**Export to VCF:**\n```python\nimport os\n\n# Export two VCF samples\nds.export(\n    regions=[\"chr21:8220186-8405573\"],\n    samples=[\"HG00101\", \"HG00097\"],\n    output_format=\"v\",\n    output_dir=os.path.expanduser(\"~\"),\n)\n```\n\n## Core Capabilities\n\n### 1. Dataset Creation and Ingestion\n\nCreate TileDB-VCF datasets and incrementally ingest variant data from multiple VCF\u002FBCF files. This is appropriate for building population genomics databases and cohort studies.\n\n**Requirements:**\n- **Single-sample VCFs only**: Multi-sample VCFs are not supported\n- **Index files required**: VCF\u002FBCF files must have indexes (.csi or .tbi)\n\n**Common operations:**\n- Create new datasets with optimized array schemas\n- Ingest single or multiple VCF\u002FBCF files in parallel\n- Add new samples incrementally without re-processing existing data\n- Configure memory usage and compression settings\n- Handle various VCF formats and INFO\u002FFORMAT fields\n- Resume interrupted ingestion processes\n- Validate data integrity during ingestion\n\n\n### 2. Efficient Querying and Filtering\n\nQuery variant data with high performance across genomic regions, samples, and variant attributes. This is appropriate for association studies, variant discovery, and population analysis.\n\n**Common operations:**\n- Query specific genomic regions (single or multiple)\n- Filter by sample names or sample groups\n- Extract specific variant attributes (position, alleles, genotypes, quality)\n- Access INFO and FORMAT fields efficiently\n- Combine spatial and attribute-based filtering\n- Stream large query results\n- Perform aggregations across samples or regions\n\n\n### 3. Data Export and Interoperability\n\nExport data in various formats for downstream analysis or integration with other genomics tools. This is appropriate for sharing datasets, creating analysis subsets, or feeding other pipelines.\n\n**Common operations:**\n- Export to standard VCF\u002FBCF formats\n- Generate TSV files with selected fields\n- Create sample\u002Fregion-specific subsets\n- Maintain data provenance and metadata\n- Lossless data export preserving all annotations\n- Compressed output formats\n- Streaming exports for large datasets\n\n\n### 4. Population Genomics Workflows\n\nTileDB-VCF excels at large-scale population genomics analyses requiring efficient access to variant data across many samples and genomic regions.\n\n**Common workflows:**\n- Genome-wide association studies (GWAS) data preparation\n- Rare variant burden testing\n- Population stratification analysis\n- Allele frequency calculations across populations\n- Quality control across large cohorts\n- Variant annotation and filtering\n- Cross-population comparative analysis\n\n\n## Key Concepts\n\n### Array Schema and Data Model\n\n**TileDB-VCF Data Model:**\n- Variants stored as sparse arrays with genomic coordinates as dimensions\n- Samples stored as attributes allowing efficient sample-specific queries\n- INFO and FORMAT fields preserved with original data types\n- Automatic compression and chunking for optimal storage\n\n**Schema Configuration:**\n```python\n# Custom schema with specific tile extents\nconfig = tiledbvcf.ReadConfig(\n    memory_budget=2048,  # MB\n    region_partition=(0, 3095677412),  # Full genome\n    sample_partition=(0, 10000)  # Up to 10k samples\n)\n```\n\n### Coordinate Systems and Regions\n\n**Critical:** TileDB-VCF uses **1-based genomic coordinates** following VCF standard:\n- Positions are 1-based (first base is position 1)\n- Ranges are inclusive on both ends\n- Region \"chr1:1000-2000\" includes positions 1000-2000 (1001 bases total)\n\n**Region specification formats:**\n```python\n# Single region\nregions = [\"chr1:1000000-2000000\"]\n\n# Multiple regions\nregions = [\"chr1:1000000-2000000\", \"chr2:500000-1500000\"]\n\n# Whole chromosome\nregions = [\"chr1\"]\n\n# BED-style (0-based, half-open converted internally)\nregions = [\"chr1:999999-2000000\"]  # Equivalent to 1-based chr1:1000000-2000000\n```\n\n### Memory Management\n\n**Performance considerations:**\n1. **Set appropriate memory budget** based on available system memory\n2. **Use streaming queries** for very large result sets\n3. **Partition large ingestions** to avoid memory exhaustion\n4. **Configure tile cache** for repeated region access\n5. **Use parallel ingestion** for multiple files\n6. **Optimize region queries** by combining nearby regions\n\n### Cloud Storage Integration\n\nTileDB-VCF seamlessly works with cloud storage:\n```python\n# S3 dataset\nds = tiledbvcf.Dataset(uri=\"s3:\u002F\u002Fbucket\u002Fdataset\", mode=\"r\")\n\n# Azure Blob Storage\nds = tiledbvcf.Dataset(uri=\"azure:\u002F\u002Fcontainer\u002Fdataset\", mode=\"r\")\n\n# Google Cloud Storage\nds = tiledbvcf.Dataset(uri=\"gcs:\u002F\u002Fbucket\u002Fdataset\", mode=\"r\")\n```\n\n## Common Pitfalls\n\n1. **Memory exhaustion during ingestion:** Use appropriate memory budget and batch processing for large VCF files\n2. **Inefficient region queries:** Combine nearby regions instead of many separate queries\n3. **Missing sample names:** Ensure sample names in VCF headers match query sample specifications\n4. **Coordinate system confusion:** Remember TileDB-VCF uses 1-based coordinates like VCF standard\n5. **Large result sets:** Use streaming or pagination for queries returning millions of variants\n6. **Cloud permissions:** Ensure proper authentication for cloud storage access\n7. **Concurrent access:** Multiple writers to the same dataset can cause corruption—use appropriate locking\n\n## CLI Usage\n\nTileDB-VCF provides a command-line interface with the following subcommands:\n\n**Available Subcommands:**\n- `create` - Creates an empty TileDB-VCF dataset\n- `store` - Ingests samples into a TileDB-VCF dataset\n- `export` - Exports data from a TileDB-VCF dataset\n- `list` - Lists all sample names present in a TileDB-VCF dataset\n- `stat` - Prints high-level statistics about a TileDB-VCF dataset\n- `utils` - Utils for working with a TileDB-VCF dataset\n- `version` - Print the version information and exit\n\n```bash\n# Create empty dataset\ntiledbvcf create --uri my_dataset\n\n# Ingest samples (requires single-sample VCFs with indexes)\ntiledbvcf store --uri my_dataset --samples sample1.vcf.gz,sample2.vcf.gz\n\n# Export data\ntiledbvcf export --uri my_dataset \\\n  --regions \"chr1:1000000-2000000\" \\\n  --sample-names \"sample1,sample2\"\n\n# List all samples\ntiledbvcf list --uri my_dataset\n\n# Show dataset statistics\ntiledbvcf stat --uri my_dataset\n```\n\n## Advanced Features\n\n### Allele Frequency Analysis\n```python\n# Calculate allele frequencies\naf_df = tiledbvcf.read_allele_frequency(\n    uri=\"my_dataset\",\n    regions=[\"chr1:1000000-2000000\"],\n    samples=[\"sample1\", \"sample2\", \"sample3\"]\n)\n```\n\n### Sample Quality Control\n```python\n# Perform sample QC\nqc_results = tiledbvcf.sample_qc(\n    uri=\"my_dataset\",\n    samples=[\"sample1\", \"sample2\"]\n)\n```\n\n### Custom Configurations\n```python\n# Advanced configuration\nconfig = tiledbvcf.ReadConfig(\n    memory_budget=4096,\n    tiledb_config={\n        \"sm.tile_cache_size\": \"1000000000\",\n        \"vfs.s3.region\": \"us-east-1\"\n    }\n)\n```\n\n\n## Resources\n\n## Getting Help\n\n### Open Source TileDB-VCF Resources\n\n**Open Source Documentation:**\n- TileDB Academy: https:\u002F\u002Fcloud.tiledb.com\u002Facademy\u002F\n- Population Genomics Guide: https:\u002F\u002Fcloud.tiledb.com\u002Facademy\u002Fstructure\u002Flife-sciences\u002Fpopulation-genomics\u002F\n- TileDB-VCF GitHub: https:\u002F\u002Fgithub.com\u002FTileDB-Inc\u002FTileDB-VCF\n\n### TileDB-Cloud Resources\n\n**For Large-Scale\u002FProduction Genomics:**\n- TileDB-Cloud Platform: https:\u002F\u002Fcloud.tiledb.com\n- TileDB Academy (All Documentation): https:\u002F\u002Fcloud.tiledb.com\u002Facademy\u002F\n\n**Getting Started:**\n- Free account signup: https:\u002F\u002Fcloud.tiledb.com\n- Contact: sales@tiledb.com for enterprise needs\n\n## Scaling to TileDB-Cloud\n\nWhen your genomics workloads outgrow single-node processing, TileDB-Cloud provides enterprise-scale capabilities for production genomics pipelines.\n\n**Note**: This section covers TileDB-Cloud capabilities based on available documentation. For complete API details and current functionality, consult the official TileDB-Cloud documentation and API reference.\n\n### Setting Up TileDB-Cloud\n\n**1. Create Account and Get API Token**\n```bash\n# Sign up at https:\u002F\u002Fcloud.tiledb.com\n# Generate API token in your account settings\n```\n\n**2. Install TileDB-Cloud Python Client**\n```bash\n# Base installation\nuv pip install tiledb-cloud\n\n# With genomics-specific functionality\nuv pip install tiledb-cloud[life-sciences]\n```\n\n**3. Configure Authentication**\n```bash\n# Set environment variable with your API token\nexport TILEDB_REST_TOKEN=\"your_api_token\"\n```\n\n```python\nimport tiledb.cloud\n\n# Authentication is automatic via TILEDB_REST_TOKEN\n# No explicit login required in code\n```\n\n### Migrating from Open Source to TileDB-Cloud\n\n**Large-Scale Ingestion**\n```python\n# TileDB-Cloud: Distributed VCF ingestion\nimport tiledb.cloud.vcf\n\n# Use specialized VCF ingestion module\n# Note: Exact API requires TileDB-Cloud documentation\n# This represents the available functionality structure\ntiledb.cloud.vcf.ingestion.ingest_vcf_dataset(\n    source=\"s3:\u002F\u002Fmy-bucket\u002Fvcf-files\u002F\",\n    output=\"tiledb:\u002F\u002Fmy-namespace\u002Flarge-dataset\",\n    namespace=\"my-namespace\",\n    acn=\"my-s3-credentials\",\n    ingest_resources={\"cpu\": \"16\", \"memory\": \"64Gi\"}\n)\n```\n\n**Distributed Query Processing**\n```python\n# TileDB-Cloud: VCF querying across distributed storage\nimport tiledb.cloud.vcf\nimport tiledbvcf\n\n# Define the dataset URI\ndataset_uri = \"tiledb:\u002F\u002FTileDB-Inc\u002Fgvcf-1kg-dragen-v376\"\n\n# Get all samples from the dataset\nds = tiledbvcf.Dataset(dataset_uri, tiledb_config=cfg)\nsamples = ds.samples()\n\n# Define attributes and ranges to query on\nattrs = [\"sample_name\", \"fmt_GT\", \"fmt_AD\", \"fmt_DP\"]\nregions = [\"chr13:32396898-32397044\", \"chr13:32398162-32400268\"]\n\n# Perform the read, which is executed in a distributed fashion\ndf = tiledb.cloud.vcf.read(\n    dataset_uri=dataset_uri,\n    regions=regions,\n    samples=samples,\n    attrs=attrs,\n    namespace=\"my-namespace\",  # specifies which account to charge\n)\ndf.to_pandas()\n```\n\n### Enterprise Features\n\n**Data Sharing and Collaboration**\n```python\n# TileDB-Cloud provides enterprise data sharing capabilities\n# through namespace-based permissions and group management\n\n# Access shared datasets via TileDB-Cloud URIs\ndataset_uri = \"tiledb:\u002F\u002Fshared-namespace\u002Fpopulation-study\"\n\n# Collaborate through shared notebooks and compute resources\n# (Specific API requires TileDB-Cloud documentation)\n```\n\n**Cost Optimization**\n- **Serverless Compute**: Pay only for actual compute time\n- **Auto-scaling**: Automatically scale up\u002Fdown based on workload\n- **Spot Instances**: Use cost-optimized compute for batch jobs\n- **Data Tiering**: Automatic hot\u002Fcold storage management\n\n**Security and Compliance**\n- **End-to-end Encryption**: Data encrypted in transit and at rest\n- **Access Controls**: Fine-grained permissions and audit logs\n- **HIPAA\u002FSOC2 Compliance**: Enterprise security standards\n- **VPC Support**: Deploy in private cloud environments\n\n### When to Migrate Checklist\n\n✅ **Migrate to TileDB-Cloud if you have:**\n- [ ] Datasets > 1000 samples\n- [ ] Need to process > 100GB of VCF data\n- [ ] Require distributed computing\n- [ ] Multiple team members need access\n- [ ] Need enterprise security\u002Fcompliance\n- [ ] Want cost-optimized serverless compute\n- [ ] Require 24\u002F7 production uptime\n\n### Getting Started with TileDB-Cloud\n\n1. **Start Free**: TileDB-Cloud offers free tier for evaluation\n2. **Migration Support**: TileDB team provides migration assistance\n3. **Training**: Access to genomics-specific tutorials and examples\n4. **Professional Services**: Custom deployment and optimization\n\n**Next Steps:**\n- Visit https:\u002F\u002Fcloud.tiledb.com to create account\n- Review documentation at https:\u002F\u002Fcloud.tiledb.com\u002Facademy\u002F\n- Contact sales@tiledb.com for enterprise needs\n\n## Related skills\n\n- `pysam`: Python\u002FHTSlib workflows for genomic files.\n- `polars-bio`: Python library polars-bio for genomic interval operations and bioinformatics file I\u002FO on Polars DataFrames, built on Arrow and DataFusion.\n- `pinecone`: Guides use of Pinecone, a managed serverless vector database, through its Python client and the LangChain and LlamaIndex integrations.\n",{"name":5,"license":15,"metadata":24,"description":7},{"tags":25,"version":26,"category":27,"maintainer":28,"contributor":29},"genomic variants, VCF, variant storage, population genomics","1.1","life-sciences","Kalaris Labs","Jeremy Leipzig","main","29fe36d26127205057d5e0ed8425f6b0e9a8c13a","skills\u002Ftiledbvcf",16,"https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills","https:\u002F\u002Fdocs.kalarislabs.com\u002Fresearch-agent-skills","https:\u002F\u002Fgithub.com\u002FKalarisLabs","kalarislabs.com","Applied AI lab building the infrastructure for the agentic era","2026-11-14 17:49:08",{"source":8,"skillName":5,"commands":41,"prompt":51,"zipAvailable":52},[42,45,48],{"tool":43,"command":44},"npx","npx skills add KalarisLabs\u002Fresearch-agent-skills --skill tiledbvcf",{"tool":46,"command":47},"bunx","bunx skills add KalarisLabs\u002Fresearch-agent-skills --skill tiledbvcf",{"tool":49,"command":50},"pnpm","pnpm dlx skills add KalarisLabs\u002Fresearch-agent-skills --skill tiledbvcf","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ftiledbvcf\u002FSKILL.md",true,[],[55,62,69,76,83,90],{"skillNo":56,"name":57,"nameZh":6,"description":58,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":59,"categories":60,"keywords":61,"publishedAt":19},"01M74ZSRSWD92NYM8TNJRH35DK","zarr-python","Guides use of Zarr-Python 3 for storing chunked, compressed N-dimensional arrays and groups, with local, in-memory, ZIP, and fsspec-backed S3\u002FGCS\u002FHTTP stores, plus NumPy, Dask, and Xarray integration. Covers array creation, resizing and appending, attributes, chunk and shard sizing, codecs, consolidated metadata, and v2-to-v3 migration. Use when creating or opening Zarr arrays or groups, choosing chunk sizes or compression for large datasets, reading or writing arrays on S3 or GCS, appending to time-series arrays, or migrating code from Zarr-Python 2 to 3. Use when setting up parallel I\u002FO with Dask or Xarray. For labeled multi-dimensional datasets, use Xarray instead.","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fzarr-python\u002FSKILL.md",[],[],{"skillNo":63,"name":64,"nameZh":6,"description":65,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":66,"categories":67,"keywords":68,"publishedAt":19},"01M74ZSK6JG5HKNC1AHF4D97G9","whisper","Transcribes and translates audio with OpenAI's Whisper (openai-whisper Python package and whisper CLI), covering model sizes from tiny to large plus turbo, language specification, initial prompts, word and segment timestamps, temperature fallback, batch processing, and subtitle generation. Use when transcribing speech, podcasts, meetings, or video audio to text. Use when translating non-English speech to English. Use when working with noisy or multilingual audio in any of 99 languages. Use when choosing a Whisper model size for available VRAM. Use when generating subtitles from audio. Not for speaker diarization or live captioning; use AssemblyAI or Deepgram instead.","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fwhisper\u002FSKILL.md",[],[],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":19},"01M74ZSBYQ2EGDFTTA4ZK6K5TR","weights-and-biases","Logs and tracks machine learning experiments with Weights & Biases (W&B, wandb): metrics, hyperparameters, checkpoints, sweeps, artifacts with lineage, model registry, custom charts and shareable reports. Includes integrations for PyTorch, HuggingFace Transformers, PyTorch Lightning and Keras\u002FTensorFlow. Use when logging training runs and comparing them across configurations, running automated hyperparameter sweeps, versioning datasets and models as artifacts, registering model versions, or sharing results with a team workspace. Use when setting up wandb.init for offline or unstable-connection training. Not for general data versioning outside ML runs.","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fweights-and-biases\u002FSKILL.md",[],[],{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":80,"categories":81,"keywords":82,"publishedAt":19},"01M74ZS5MZTK84WHEQXCVDZ6GB","waypoint-bio","Use when working with Outpost Bio's open microbiome foundation models - the Waypoint checkpoints (Waypoint-6m, Waypoint-45m, Waypoint-170m), the Atlas pretraining corpus, the Compass eight-task benchmark, or the `waypoint` CLI from the `waypoint-bio` package. Covers embedding microbiome samples, fine-tuning on taxonomic abundance data, benchmarking a checkpoint on Compass, pretraining a GPT-2 model on taxonomic abundance profiles, and converting MetaPhlAn, Kraken2, QIIME 2, or MGnify abundance tables into waypoint format.","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fwaypoint-bio\u002FSKILL.md",[],[],{"skillNo":84,"name":85,"nameZh":6,"description":86,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":87,"categories":88,"keywords":89,"publishedAt":19},"01M74ZS0TCW96WJAKSG07C5Z7A","verl-rl-training","Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL). Use when implementing RLHF, GRPO, PPO, or other RL algorithms for LLM post-training at scale with flexible infrastructure backends.","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fverl-rl-training\u002FSKILL.md",[],[],{"skillNo":91,"name":92,"nameZh":6,"description":93,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":94,"categories":95,"keywords":96,"publishedAt":19},"01M74ZRVG6SSWGT1N3Q55134KA","venue-templates","Prepare journal manuscripts, conference papers, research posters, and grant documents using venue-specific formatting guidance and bundled LaTeX scaffolds. Use when selecting an official template, checking current page or anonymity rules, adapting academic writing to a venue, or inspecting a submission PDF.","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fvenue-templates\u002FSKILL.md",[],[]]