data-storage
Use ClearML Dataset, StorageManager, clearml-data CLI, storage backends, cache behavior, and HyperDataset/DataView routing for dataset versioning and retrieval.
- License 许可
- Apache-2.0
- 最近更新
- 2026-10-08
- Stars
- ★ 11
- Forks
- 3
01 前言 Front Matter
- name
- data-storage
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Use ClearML Dataset, StorageManager, clearml-data CLI, storage backends, cache behavior, and HyperDataset/DataView routing for dataset versioning and retrieval.
- disable-model-invocation
- true
02 资源文件 4
- 参考文档references/api-reference.md9.4 KB
- 参考文档references/cli-reference.md7.2 KB
- 参考文档references/troubleshooting.md6.8 KB
- 参考文档references/workflows.md7.1 KB
03 SKILL.md 正文
ClearML Data and Storage
Use this sub-skill when the user needs to create, version, upload, finalize, retrieve, verify, or troubleshoot ClearML datasets, build clearml-data commands, manage remote storage URIs, or reason about StorageManager caching and HyperDataset/DataView routing.
Route First
- Use this sub-skill for
Dataset,StorageManager,clearml-data, dataset parents/versions, local copies, external links, storage credentials/extras, cache limits, and HyperDataset/DataView selection. - Route experiment task setup, scalar/image reporting, and non-dataset
Task.upload_artifact()usage to../experiment-tracking/SKILL.md. - Route queued task execution, agents,
clearml-task,Task.execute_remotely(), or remote job launch to../remote-execution-cli/SKILL.md. - Route pipelines that consume datasets as pipeline inputs or steps to
../automation-pipelines/SKILL.md.
Core Dataset Pattern
Use a two-stage mental model: a dataset version is mutable until finalized, and after finalization it should be retrieved or used as a parent for a new version.
from clearml import Dataset
base = Dataset.create(
dataset_project="datasets/examples",
dataset_name="images",
dataset_version="1.0.0",
output_uri="s3://bucket/clearml/images",
)
base.add_files(path="data/train", dataset_path="train")
base.upload(output_url="s3://bucket/clearml/images", verbose=True)
base.finalize()
child = Dataset.create(
dataset_project="datasets/examples",
dataset_name="images",
dataset_version="1.1.0",
parent_datasets=[base.id],
)
child.sync_folder(local_path="data/train", dataset_path="train", verbose=True)
child.upload(verbose=True)
child.finalize()
Prefer explicit IDs after creation. Dataset.get() can search by project/name/version, but dataset_id avoids ambiguity when multiple semantic versions or recent updates match.
CLI Planning
Use clearml-data when the user wants shell commands or quick dataset lifecycle operations. The CLI stores the most recent dataset ID in ~/.clearml_data.json; commands without --id may silently target the last accessed dataset, so production instructions should include --id after create unless the user intentionally wants one-shell stateful use.
clearml-data create --project datasets/examples --name images --version 1.0.0 --storage s3://bucket/clearml/images
clearml-data add --id DATASET_ID --dataset-folder train --files data/train
clearml-data upload --id DATASET_ID --storage s3://bucket/clearml/images --verbose
clearml-data close --id DATASET_ID --verbose
Use scripts/dataset_cli_plan.py to generate safe command sequences without running ClearML calls. Use scripts/validate_dataset_plan.py to check JSON plan invariants before translating a plan to Python or CLI.
Storage Guidance
Dataset.create(..., output_uri=...),Dataset.upload(output_url=...), and CLI--storageaccepts3://...,gs://...,azure://..., a shared filesystem path, or the default ClearML files server.- Cloud storage support requires the matching package extras: install ClearML with
s3,gs, orazureextras when those schemes are used. - Credentials are read from ClearML configuration and provider defaults; do not put secrets in generated commands, scripts, dataset names, or skill content.
- Use
StorageManager.get_local_copy()for remote file caching/downloading andStorageManager.upload_file()orupload_folder()for direct storage operations that are not dataset-versioning workflows.
References
references/api-reference.mddocuments Dataset, StorageManager, and HyperDataset/DataView signatures and behavior notes.references/cli-reference.mddocumentsclearml-datasubcommands, required flags, state behavior, and safe command construction.references/workflows.mdgives Python and CLI workflows for create/add/sync/upload/finalize/get/parents/external links.references/troubleshooting.mdcovers credentials, storage extras, stale CLI state, missing IDs, pending uploads, parent mistakes, and cache/local-copy pitfalls.scripts/dataset_cli_plan.pybuilds offlineclearml-datacommand plans.scripts/validate_dataset_plan.pyvalidates JSON dataset plan invariants with the Python standard library only.
04 同仓库更多 Skills
obsidian-clipper-template-creator
Guide for creating templates for the Obsidian Web Clipper. Use when you want to create a new clipping template, understand available variables, or format clipped content.
obsidian-clipper-template-creator
Guide for creating templates for the Obsidian Web Clipper. Use when you want to create a new clipping template, understand available variables, or format clipped content.
python-api
Programmatic Object-Detection-Metrics API guidance for BoundingBox, BoundingBoxes, Evaluator, enums, coordinate conversion, AP/mAP semantics, and source-style import diagnostics.
file-evaluation
Folder-based PASCAL VOC AP/mAP evaluation from ground-truth and detection text files.
object-detection-metrics
Guides legacy Object-Detection-Metrics PASCAL VOC AP/mAP evaluation from detection text folders and direct Python API classes.
nx-workspace-patterns
Configure and optimize Nx monorepo workspaces. Use when setting up Nx, configuring project boundaries, optimizing build caching, or implementing affected commands.