tensorflow-on-spark
Operate TensorFlowOnSpark workflows that combine TensorFlow training, inference, TFRecords, and Spark clusters or Spark ML pipelines.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- name
- tensorflow-on-spark
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Operate TensorFlowOnSpark workflows that combine TensorFlow training, inference, TFRecords, and Spark clusters or Spark ML pipelines.
- disable-model-invocation
- true
02 资源文件 5
- 参考文档references/api-map.md2.8 KB
- 参考文档references/repo-provenance.md2.1 KB
- 参考文档references/repo-routing-metadata.json354 B
- 参考文档references/troubleshooting.md5.0 KB
- 脚本scripts/check_environment.py6.6 KB
03 SKILL.md 正文
TensorFlowOnSpark
Use this repo skill when a task involves the tensorflowonspark package, Yahoo TensorFlowOnSpark, distributed TensorFlow on Apache Spark, TFCluster, TFNode.DataFeed, Spark ML TFEstimator / TFModel, TFRecord DataFrame conversion, or converting TensorFlow examples to spark-submit workflows.
TensorFlowOnSpark lets a Spark application start TensorFlow functions on executors, coordinate node reservations, feed Spark RDD or DataFrame data into TensorFlow workers, and shut the cluster down after training or inference. Most practical tasks need both Python-package checks and Spark runtime checks.
Install and first checks
For a normal Python environment, install the public package and the runtime dependencies needed by the workflow:
pip install tensorflowonspark
# Add TensorFlow, PySpark, NumPy/SciPy/HDF5, and Spark/Java according to your cluster policy.
# For current TensorFlow 2.x CPU-only inspection, a package such as tensorflow-cpu can be sufficient.
For a repository development checkout, use an editable install only in a private development environment. TensorFlowOnSpark 2.2.5 uses legacy packaging and pkg_resources; if modern packaging tools reject its metadata or remove pkg_resources, apply that repair in the development environment rather than baking local paths into runtime instructions.
-
Confirm the package imports:
python - <<'PY' import tensorflowonspark from tensorflowonspark import TFCluster, TFNode, TFParallel, dfutil from tensorflowonspark.pipeline import TFEstimator, TFModel print(tensorflowonspark.__version__) PY -
Confirm Spark and Java before running native Spark jobs:
python scripts/check_environment.py --json -
For Spark-fed training/inference, confirm the Spark application has one task per executor, enough executor slots for the TensorFlow cluster,
spark.python.worker.reusefor background mode, and the TensorFlow Hadoop jar on the Spark classpath when TFRecords are used.
Route by task
| User task | Read |
|---|---|
Start, stop, or debug a TFCluster.run(...) job; reservation waits; TensorBoard URL; TF_CONFIG; GPU allocation; TFParallel.run(...) |
cluster-lifecycle |
Write a map_fun(args, ctx) using InputMode.SPARK; feed RDD/DStream rows; return inference results; handle DataFeed timeouts or early termination |
datafeed-inputmode |
Train or infer with Spark ML DataFrames using TFEstimator or TFModel; map DataFrame columns to TensorFlow tensors; load SavedModels |
spark-ml-pipelines |
Save/load TFRecords as Spark DataFrames; reason about tf.train.Example schemas; handle binary_features; prepare MNIST-shaped rows |
dataframes-tfrecords |
Convert TensorFlow/Keras/Estimator examples to TensorFlowOnSpark; build spark-submit command plans for MNIST, ResNet, segmentation, serving, or batch inference |
examples-conversion |
Shared references and scripts
- Repository provenance records the source commit, package version, evidence paths, and refresh baseline.
- API map shows which sub-skill owns each public module and workflow family.
- Troubleshooting covers cross-cutting install/import, Java/Spark, TensorFlow/PySpark, Hadoop jar, Python metadata, and GPU issues.
- check_environment.py is a safe diagnostic helper; it does not start training, download datasets, mutate clusters, or allocate GPUs.
Operating rules
- Do not run long training examples, TF Serving Docker containers, cloud installers, or dataset downloads unless the user explicitly asks and the runtime is suitable.
- Do not assume Spark local mode is equivalent to TensorFlowOnSpark's distributed executor behavior. For cluster semantics, prefer Spark Standalone, YARN, Kubernetes, or
local-cluster[...]style tests that isolate executor processes. - For
InputMode.SPARK, the TensorFlow function must consume rows fromTFNode.DataFeed; Spark actions are still required to trigger lazy RDD inference. - For Spark ML Pipelines, the DataFrame API is Spark-fed (
InputMode.SPARK); route TensorFlow-native file-reading workflows toexamples-conversionorcluster-lifecycleinstead. - For TFRecords, keep records flat enough for Spark SQL types and supply
binary_featuresfor raw bytes. - Treat GPU use as optional unless the user explicitly requires it. CPU can validate many workflows, but it does not prove CUDA allocation, driver compatibility, or GPU TensorFlow execution.
Common validation signals
TFCluster.run(...)returns a cluster object after every executor reserves a node.cluster.train(...),cluster.inference(...), andcluster.shutdown(...)finish without late worker exceptions.- Spark ML
TFEstimator.fit(...)writes the expected checkpoint or SavedModel artifact, andTFModel.transform(...)returns a DataFrame with the requested output columns. - TFRecord conversion preserves simple scalar/array fields and treats string versus binary features as intended.
- Render-only helper scripts output reviewable commands/templates without running Spark, Docker, downloads, or destructive cleanup.
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).