domino-distributed-computing
Work with Domino on-demand compute clusters - Apache Spark, Ray, Dask and Open MPI - attached to a Workspace or Job. Covers choosing a framework, the cluster and execution environments, attaching a cluster from the UI or the Jobs API (computeCluster), connecting from code, autoscaling, cluster web UIs and lifecycle, and troubleshooting. Use when processing large datasets, running distributed or multi-GPU training, or scaling Python workloads. Slurm is domino-hpc-slurm.
- License 许可
- MIT
- 最近更新
- 2026-11-14
- Stars
- ★ 5
- Forks
- 4
01 前言 Front Matter
- name
- domino-distributed-computing
- description
- Work with Domino on-demand compute clusters - Apache Spark, Ray, Dask and Open MPI - attached to a Workspace or Job. Covers choosing a framework, the cluster and execution environments, attaching a cluster from the UI or the Jobs API (computeCluster), connecting from code, autoscaling, cluster web UIs and lifecycle, and troubleshooting. Use when processing large datasets, running distributed or multi-GPU training, or scaling Python workloads. Slurm is domino-hpc-slurm.
- compatibility
- Domino 6.3 and Domino Cloud. Requires DOMINO_API_HOST; in a run use DOMINO_API_PROXY or the localhost:8899 access-token endpoint, outside a run a Personal Access Token.
02 资源文件 1
- 参考文档FRAMEWORK-EXAMPLES.md9.7 KB
03 SKILL.md 正文
Domino distributed computing
Applies to Domino 6.3 and Domino Cloud.
Domino provisions an on-demand Spark, Ray, Dask or Open MPI cluster when a Workspace or Job starts and destroys it when the execution stops. It belongs to that execution alone and its nodes run a separate cluster environment. Slurm: domino-hpc-slurm.
Docs: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc (same paths under /6.3/).
Choose a framework
| Framework | Use for | Not for |
|---|---|---|
| Spark | SQL/ETL over large data, MLlib, PySpark | streaming, HDFS-style storage, strict-SLA pipelines (a cluster starts per execution) |
| Ray | Ray Train, Ray Tune, RLlib, parallel Python | serving (clusters are ephemeral; use a Domino endpoint) |
| Dask | pandas/NumPy/scikit-learn at scale, custom task graphs | |
| Open MPI | multi-node, multi-GPU mpirun training (Horovod, PyTorch, TF) |
Apps, Launchers, endpoints (not supported) |
| Slurm | existing sbatch scripts: domino-hpc-slurm |
Jobs (Workspace only) |
Environments
Each cluster type needs two environments with the same Python version:
- Cluster environment (head/master/scheduler and workers): create it from the catalog image (
quay.io/domino/{spark,ray,dask,mpi}-cluster-environment, plusray-cluster-gpu-environment; tags change per release), tick the framework under Supported Clusters, leave Pluggable Workspace Tools empty. A DSE cannot be a worker environment. Pre/post setup and run scripts never run on cluster nodes. - Execution environment for the Workspace or Job (driver/client): the matching
domino-{spark,ray,dask,mpi}-environmentimage with workspace tools. MPI needs the same Open MPI version in both.
Add packages to both with Dockerfile instructions (RUN pip install torch==2.14.1; domino-environments).
Attach a cluster
UI: Workspaces > + Create New Workspace or Jobs > Run, then the Compute Cluster step. Settings: Min / Max workers (Spark: executors), Auto-scale workers, Worker hardware tier, Head / Master / Scheduler hardware tier (none for MPI), Cluster compute environment, Dedicated local storage per executor (/tmp, discarded at shutdown). Cluster size is capped by the per-user execution limit (master and execution each take a slot). Scheduled Jobs take the same settings (domino-jobs).
REST: computeCluster on POST /api/jobs/v1/jobs. Clustered Workspaces: the UI, or computeClusterConfig on the internal create route (domino-workspaces API.md; non-public route; may change in a future Domino version, see route types).
body = {
"projectId": os.environ["DOMINO_PROJECT_ID"],
"runCommand": "python train.py",
"computeCluster": {
"clusterType": "ray", # spark | ray | dask | mpi
"computeEnvironmentId": "<cluster environment id>",
"workerCount": 2, # min workers when maxWorkerCount is set
"maxWorkerCount": 6, # present = autoscaling on
"workerHardwareTier": "<hardware tier id>",
"masterHardwareTierId": "<hardware tier id>", # omit for mpi
"workerStorageMB": 10240,
"computeEnvironmentRevisionSpec": "ActiveRevision",
},
}
requests.post(f"{os.environ['DOMINO_API_PROXY']}/api/jobs/v1/jobs", json=body).raise_for_status()
Required in computeCluster: clusterType, computeEnvironmentId, workerCount, workerHardwareTier. python-domino: job_start(command, compute_cluster_properties={...}) (workerHardwareTierId keys; domino-python-sdk).
Connect from your code
Domino injects connection details; never hard-code hosts.
Spark: spark.master and spark.driver.host are preset (do not override); spark.executor.instances, cores and memory follow the cluster settings.
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("my-app").config("spark.sql.shuffle.partitions", "200").getOrCreate()
spark-submit works in a Job if wrapped in a shell script (spark-submit "$@"). Project-wide extra config: Settings > Integrations > Domino managed on-demand cluster, or PATCH /v4/projects/{projectId}/sparkConfig (non-public route; may change in a future Domino version; FRAMEWORK-EXAMPLES.md).
Ray:
import os, ray
if not ray.is_initialized():
ray.init(f"ray://{os.environ['RAY_HEAD_SERVICE_HOST']}:{os.environ['RAY_HEAD_SERVICE_PORT']}")
Dask:
import os
from dask.distributed import Client
client = Client(f"{os.environ['DASK_SCHEDULER_SERVICE_HOST']}:{os.environ['DASK_SCHEDULER_SERVICE_PORT']}")
client.wait_for_workers(2) # optional; client.close() when done, never client.shutdown()
Client() with no address starts a local cluster, not the Domino one. client.dashboard_link does not route through Domino; use the Dask tab.
MPI: workers are listed in /domino/mpi/hosts. In a Workspace run File Changes > MPI Cluster File Sync > Sync first (copies /mnt code; Datasets are already on the workers), then mpirun python train.py (slots = worker CPU cores; --map-by, --bind-to, -x VAR as usual). In a Job, Domino wraps the command in mpirun itself; tune with an mpi.conf in the Project root, one name = value per line (rmaps_base_mapping_policy = core, hwloc_base_binding_policy = socket, mca_base_env_list = VAR, orte_hostfile = /mnt/hosts).
Datasets are mounted on every cluster node at the same path, so a Dataset, NetApp Volume or External Data Volume is the default for output. Object storage (S3 and similar) is fine where enabled for the cluster: client libraries in both environments, propagated credentials (domino-data-connectivity).
Web UI, autoscaling, lifecycle
- Spark, Ray and Dask web UIs: a dedicated tab in the Workspace, or the Job Details tab (needs a data plane enabled for Workspaces on Nexus).
- Autoscaling adds workers at 80% average CPU and removes them after 5 minutes of low use (tunable); scale-down can discard workers holding intermediate results.
- The cluster, its
/tmpand anything not written to a Dataset, Project files or external storage die with the execution.
Troubleshooting
- Cluster never ready: the cluster environment lacks the Supported Clusters flag, Python versions differ, tier capacity or cluster-type restriction, or the per-user execution limit.
KeyErroronRAY_HEAD_SERVICE_HOST/DASK_SCHEDULER_SERVICE_HOST: no cluster was attached to this execution.- Worker OOM: bigger worker tier, more workers, fewer shuffles; Spark adds 10% (min 384 MB) overhead per worker on top of the tier memory.
- MPI changes not seen on workers: sync files again; do not run
mpirunwhile a sync is in progress.
Reference files
- FRAMEWORK-EXAMPLES.md: Spark I/O and MLlib, Ray tasks/Train/Tune, Dask DataFrames and Dask-ML, GPU patterns.
04 同仓库更多 Skills
tags-and-properties
Manage Domino tags and properties via the Tags (Taxonomy) API at /api/taxonomy/v1. Covers namespaces and tags (create, list, update, deprecate, delete; tag entities such as projects, models, datasets, apps, project templates, NetApp volumes, environments; query by tag; autocomplete; merge; CSV import/export) and properties (typed metadata fields - text, number, date, boolean, url, user, organization, select, multi_select - with per-entity values, groups and versioned entities like model_version/app_version). Use when organizing entities with tags, building hierarchical namespaces, defining typed metadata fields, setting property values, finding entities by tag, bulk-tagging during onboarding, or migrating a taxonomy across environments.
netapp-volumes
Work with Domino Volumes for NetApp ONTAP (NetApp Volumes) - shared, multi-terabyte storage with near-instant snapshots. Covers creating volumes, attaching them to projects, mount paths (/mnt/netapp-volumes/ or /domino/netapp-volumes/), snapshots and tags, roles, deletion, mounting volumes in jobs, and the NetApp Volumes REST API (/remotefs/v1). Use when managing large-scale shared data, taking fast no-copy snapshots, or integrating NetApp ONTAP storage with Domino.
domino-workspaces
Work with Domino Workspaces - interactive sessions running Jupyter, JupyterLab, VS Code or RStudio on a hardware tier with a compute environment. Covers launching and configuring a Workspace, what persists across stop and resume, file paths and syncing for DFS and Git-based Projects, package and home-directory persistence, SSH access from a local IDE, the Workspace API routes, and troubleshooting. Use when setting up or debugging an interactive development session in Domino.
domino-ui-bootstrap
Bootstrap or retrofit a Vite + React 18 + TypeScript project so it uses the Domino design system (`@dominodatalab/extensions-tools`). Scoped to a **fully standalone SPA frontend for a Domino application or extension**, not snippets, library additions, or work inside an existing Domino monorepo package. Use whenever the user wants to create, scaffold, refactor, retrofit, or set up such an SPA (a React app, web app, frontend, UI, or extension) that should "look like Domino", use Domino components, follow the Domino design system, or integrate with the Domino platform, including when they mention DominoThemeProviderDecorator, extensions-tools, or base-components without saying "Domino". Also use to make an existing standalone React/Vite project "Domino-styled" or add Domino theming. Handles React 18 / react-router 5 pinning, Storybook MCP registration, theme provider wiring, and build verification.
domino-trace-setup
Set up GenAI tracing for an LLM or agent application in Domino by checking the Domino SDK and MLflow requirements (the DSE needs nothing installed), then adding a tracing_setup.py helper, span evaluators, a config.yaml, and @add_tracing / DominoAgentContext wiring. Use when the user asks to add tracing, observability, or evaluation to an agent or LLM app running on Domino.
domino-rust-apps
Build a Rust web app (axum) in Domino and ship it as a Domino App that starts from a prebuilt binary. Covers installing a pinned Rust toolchain into a Compute Environment with rustup, Cargo project defaults (edition 2024, rust-toolchain.toml, Cargo.lock, Clippy pedantic, rustfmt, cargo-nextest, cargo-deny), rust-analyzer for the IDE and coding agents, check hooks, calling Domino as the viewer with extended identity propagation, dev and release build profiles, keeping target/ out of Git and Domino storage, and an app.sh that runs the binary in App Preview and deployments without compiling. Use when the user wants a Rust app, API, dashboard or Proa site on Domino, a performance-sensitive or correctness-critical App, or asks to set up Rust, cargo, Clippy, nextest or rust-analyzer in a Domino Workspace or environment.