AGENT SKILL

tensorflow-on-spark

Operate TensorFlowOnSpark workflows that combine TensorFlow training, inference, TFRecords, and Spark clusters or Spark ML pipelines.

License 许可
Apache-2.0
最近更新
2026-09-03
Stars
★ 351
Forks
27

01 前言 Front Matter

name
tensorflow-on-spark
license
Apache 2.0
metadata
{ "disco-role": "operating" }
description
Operate TensorFlowOnSpark workflows that combine TensorFlow training, inference, TFRecords, and Spark clusters or Spark ML pipelines.
disable-model-invocation
true

02 资源文件 5

03 SKILL.md 正文

TensorFlowOnSpark

Use this repo skill when a task involves the tensorflowonspark package, Yahoo TensorFlowOnSpark, distributed TensorFlow on Apache Spark, TFCluster, TFNode.DataFeed, Spark ML TFEstimator / TFModel, TFRecord DataFrame conversion, or converting TensorFlow examples to spark-submit workflows.

TensorFlowOnSpark lets a Spark application start TensorFlow functions on executors, coordinate node reservations, feed Spark RDD or DataFrame data into TensorFlow workers, and shut the cluster down after training or inference. Most practical tasks need both Python-package checks and Spark runtime checks.

Install and first checks

For a normal Python environment, install the public package and the runtime dependencies needed by the workflow:

pip install tensorflowonspark
# Add TensorFlow, PySpark, NumPy/SciPy/HDF5, and Spark/Java according to your cluster policy.
# For current TensorFlow 2.x CPU-only inspection, a package such as tensorflow-cpu can be sufficient.

For a repository development checkout, use an editable install only in a private development environment. TensorFlowOnSpark 2.2.5 uses legacy packaging and pkg_resources; if modern packaging tools reject its metadata or remove pkg_resources, apply that repair in the development environment rather than baking local paths into runtime instructions.

  1. Confirm the package imports:

    python - <<'PY'
    import tensorflowonspark
    from tensorflowonspark import TFCluster, TFNode, TFParallel, dfutil
    from tensorflowonspark.pipeline import TFEstimator, TFModel
    print(tensorflowonspark.__version__)
    PY
    
  2. Confirm Spark and Java before running native Spark jobs:

    python scripts/check_environment.py --json
    
  3. For Spark-fed training/inference, confirm the Spark application has one task per executor, enough executor slots for the TensorFlow cluster, spark.python.worker.reuse for background mode, and the TensorFlow Hadoop jar on the Spark classpath when TFRecords are used.

Route by task

User task Read
Start, stop, or debug a TFCluster.run(...) job; reservation waits; TensorBoard URL; TF_CONFIG; GPU allocation; TFParallel.run(...) cluster-lifecycle
Write a map_fun(args, ctx) using InputMode.SPARK; feed RDD/DStream rows; return inference results; handle DataFeed timeouts or early termination datafeed-inputmode
Train or infer with Spark ML DataFrames using TFEstimator or TFModel; map DataFrame columns to TensorFlow tensors; load SavedModels spark-ml-pipelines
Save/load TFRecords as Spark DataFrames; reason about tf.train.Example schemas; handle binary_features; prepare MNIST-shaped rows dataframes-tfrecords
Convert TensorFlow/Keras/Estimator examples to TensorFlowOnSpark; build spark-submit command plans for MNIST, ResNet, segmentation, serving, or batch inference examples-conversion

Shared references and scripts

  • Repository provenance records the source commit, package version, evidence paths, and refresh baseline.
  • API map shows which sub-skill owns each public module and workflow family.
  • Troubleshooting covers cross-cutting install/import, Java/Spark, TensorFlow/PySpark, Hadoop jar, Python metadata, and GPU issues.
  • check_environment.py is a safe diagnostic helper; it does not start training, download datasets, mutate clusters, or allocate GPUs.

Operating rules

  • Do not run long training examples, TF Serving Docker containers, cloud installers, or dataset downloads unless the user explicitly asks and the runtime is suitable.
  • Do not assume Spark local mode is equivalent to TensorFlowOnSpark's distributed executor behavior. For cluster semantics, prefer Spark Standalone, YARN, Kubernetes, or local-cluster[...] style tests that isolate executor processes.
  • For InputMode.SPARK, the TensorFlow function must consume rows from TFNode.DataFeed; Spark actions are still required to trigger lazy RDD inference.
  • For Spark ML Pipelines, the DataFrame API is Spark-fed (InputMode.SPARK); route TensorFlow-native file-reading workflows to examples-conversion or cluster-lifecycle instead.
  • For TFRecords, keep records flat enough for Spark SQL types and supply binary_features for raw bytes.
  • Treat GPU use as optional unless the user explicitly requires it. CPU can validate many workflows, but it does not prove CUDA allocation, driver compatibility, or GPU TensorFlow execution.

Common validation signals

  • TFCluster.run(...) returns a cluster object after every executor reserves a node.
  • cluster.train(...), cluster.inference(...), and cluster.shutdown(...) finish without late worker exceptions.
  • Spark ML TFEstimator.fit(...) writes the expected checkpoint or SavedModel artifact, and TFModel.transform(...) returns a DataFrame with the requested output columns.
  • TFRecord conversion preserves simple scalar/array fields and treats string versus binary features as intended.
  • Render-only helper scripts output reviewable commands/templates without running Spark, Docker, downloads, or destructive cleanup.

04 同仓库更多 Skills

VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-triton-opt

Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-solve

END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-skill

PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-pattern-fusion

Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-orchestrate

Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-feedback

Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).

★ 351