[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7960DXADM677M3V5JDWRHM9":3,"skill-same:01M7960DXADM677M3V5JDWRHM9":71},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":27,"commitSha":28,"skillDirectory":29,"forksCount":30,"repositoryHtmlUrl":31,"repositoryHomepageUrl":6,"ownerHtmlUrl":32,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":33,"install":34,"resources":46},"01M7960DXADM677M3V5JDWRHM9","tensorflow-on-spark",null,"Operate TensorFlowOnSpark workflows that combine TensorFlow training, inference, TFRecords, and Spark clusters or Spark ML pipelines.","VectorSpaceLab\u002FAREX-Skill","VectorSpaceLab","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F180637102?v=4","01M7538C5WPSJGWF7KHR4EFZ9F","01M7538C6R2GFP53NPXTCZNBRT",351,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FVectorSpaceLab\u002FAREX-Skill\u002Fblob\u002Fmain\u002Fskills\u002Frepositories\u002Frepo-skills\u002Ftensorflow-on-spark\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: tensorflow-on-spark\ndescription: \"Operate TensorFlowOnSpark workflows that combine TensorFlow\n  training, inference, TFRecords, and Spark clusters or Spark ML pipelines.\"\ndisable-model-invocation: true\nmetadata:\n  disco-role: operating\nlicense: Apache 2.0\n---\n\n# TensorFlowOnSpark\n\nUse this repo skill when a task involves the `tensorflowonspark` package, Yahoo TensorFlowOnSpark, distributed TensorFlow on Apache Spark, `TFCluster`, `TFNode.DataFeed`, Spark ML `TFEstimator` \u002F `TFModel`, TFRecord DataFrame conversion, or converting TensorFlow examples to `spark-submit` workflows.\n\nTensorFlowOnSpark lets a Spark application start TensorFlow functions on executors, coordinate node reservations, feed Spark RDD or DataFrame data into TensorFlow workers, and shut the cluster down after training or inference. Most practical tasks need both Python-package checks and Spark runtime checks.\n\n## Install and first checks\n\nFor a normal Python environment, install the public package and the runtime dependencies needed by the workflow:\n\n```bash\npip install tensorflowonspark\n# Add TensorFlow, PySpark, NumPy\u002FSciPy\u002FHDF5, and Spark\u002FJava according to your cluster policy.\n# For current TensorFlow 2.x CPU-only inspection, a package such as tensorflow-cpu can be sufficient.\n```\n\nFor a repository development checkout, use an editable install only in a private development environment. TensorFlowOnSpark 2.2.5 uses legacy packaging and `pkg_resources`; if modern packaging tools reject its metadata or remove `pkg_resources`, apply that repair in the development environment rather than baking local paths into runtime instructions.\n\n1. Confirm the package imports:\n\n   ```bash\n   python - \u003C\u003C'PY'\n   import tensorflowonspark\n   from tensorflowonspark import TFCluster, TFNode, TFParallel, dfutil\n   from tensorflowonspark.pipeline import TFEstimator, TFModel\n   print(tensorflowonspark.__version__)\n   PY\n   ```\n\n2. Confirm Spark and Java before running native Spark jobs:\n\n   ```bash\n   python scripts\u002Fcheck_environment.py --json\n   ```\n\n3. For Spark-fed training\u002Finference, confirm the Spark application has one task per executor, enough executor slots for the TensorFlow cluster, `spark.python.worker.reuse` for background mode, and the TensorFlow Hadoop jar on the Spark classpath when TFRecords are used.\n\n## Route by task\n\n| User task | Read |\n|---|---|\n| Start, stop, or debug a `TFCluster.run(...)` job; reservation waits; TensorBoard URL; TF_CONFIG; GPU allocation; `TFParallel.run(...)` | [cluster-lifecycle](sub-skills\u002Fcluster-lifecycle\u002FSKILL.md) |\n| Write a `map_fun(args, ctx)` using `InputMode.SPARK`; feed RDD\u002FDStream rows; return inference results; handle `DataFeed` timeouts or early termination | [datafeed-inputmode](sub-skills\u002Fdatafeed-inputmode\u002FSKILL.md) |\n| Train or infer with Spark ML DataFrames using `TFEstimator` or `TFModel`; map DataFrame columns to TensorFlow tensors; load SavedModels | [spark-ml-pipelines](sub-skills\u002Fspark-ml-pipelines\u002FSKILL.md) |\n| Save\u002Fload TFRecords as Spark DataFrames; reason about `tf.train.Example` schemas; handle `binary_features`; prepare MNIST-shaped rows | [dataframes-tfrecords](sub-skills\u002Fdataframes-tfrecords\u002FSKILL.md) |\n| Convert TensorFlow\u002FKeras\u002FEstimator examples to TensorFlowOnSpark; build `spark-submit` command plans for MNIST, ResNet, segmentation, serving, or batch inference | [examples-conversion](sub-skills\u002Fexamples-conversion\u002FSKILL.md) |\n\n## Shared references and scripts\n\n- [Repository provenance](references\u002Frepo-provenance.md) records the source commit, package version, evidence paths, and refresh baseline.\n- [API map](references\u002Fapi-map.md) shows which sub-skill owns each public module and workflow family.\n- [Troubleshooting](references\u002Ftroubleshooting.md) covers cross-cutting install\u002Fimport, Java\u002FSpark, TensorFlow\u002FPySpark, Hadoop jar, Python metadata, and GPU issues.\n- [check_environment.py](scripts\u002Fcheck_environment.py) is a safe diagnostic helper; it does not start training, download datasets, mutate clusters, or allocate GPUs.\n\n## Operating rules\n\n- Do not run long training examples, TF Serving Docker containers, cloud installers, or dataset downloads unless the user explicitly asks and the runtime is suitable.\n- Do not assume Spark local mode is equivalent to TensorFlowOnSpark's distributed executor behavior. For cluster semantics, prefer Spark Standalone, YARN, Kubernetes, or `local-cluster[...]` style tests that isolate executor processes.\n- For `InputMode.SPARK`, the TensorFlow function must consume rows from `TFNode.DataFeed`; Spark actions are still required to trigger lazy RDD inference.\n- For Spark ML Pipelines, the DataFrame API is Spark-fed (`InputMode.SPARK`); route TensorFlow-native file-reading workflows to `examples-conversion` or `cluster-lifecycle` instead.\n- For TFRecords, keep records flat enough for Spark SQL types and supply `binary_features` for raw bytes.\n- Treat GPU use as optional unless the user explicitly requires it. CPU can validate many workflows, but it does not prove CUDA allocation, driver compatibility, or GPU TensorFlow execution.\n\n## Common validation signals\n\n- `TFCluster.run(...)` returns a cluster object after every executor reserves a node.\n- `cluster.train(...)`, `cluster.inference(...)`, and `cluster.shutdown(...)` finish without late worker exceptions.\n- Spark ML `TFEstimator.fit(...)` writes the expected checkpoint or SavedModel artifact, and `TFModel.transform(...)` returns a DataFrame with the requested output columns.\n- TFRecord conversion preserves simple scalar\u002Farray fields and treats string versus binary features as intended.\n- Render-only helper scripts output reviewable commands\u002Ftemplates without running Spark, Docker, downloads, or destructive cleanup.\n","# TensorFlowOnSpark\n\nUse this repo skill when a task involves the `tensorflowonspark` package, Yahoo TensorFlowOnSpark, distributed TensorFlow on Apache Spark, `TFCluster`, `TFNode.DataFeed`, Spark ML `TFEstimator` \u002F `TFModel`, TFRecord DataFrame conversion, or converting TensorFlow examples to `spark-submit` workflows.\n\nTensorFlowOnSpark lets a Spark application start TensorFlow functions on executors, coordinate node reservations, feed Spark RDD or DataFrame data into TensorFlow workers, and shut the cluster down after training or inference. Most practical tasks need both Python-package checks and Spark runtime checks.\n\n## Install and first checks\n\nFor a normal Python environment, install the public package and the runtime dependencies needed by the workflow:\n\n```bash\npip install tensorflowonspark\n# Add TensorFlow, PySpark, NumPy\u002FSciPy\u002FHDF5, and Spark\u002FJava according to your cluster policy.\n# For current TensorFlow 2.x CPU-only inspection, a package such as tensorflow-cpu can be sufficient.\n```\n\nFor a repository development checkout, use an editable install only in a private development environment. TensorFlowOnSpark 2.2.5 uses legacy packaging and `pkg_resources`; if modern packaging tools reject its metadata or remove `pkg_resources`, apply that repair in the development environment rather than baking local paths into runtime instructions.\n\n1. Confirm the package imports:\n\n   ```bash\n   python - \u003C\u003C'PY'\n   import tensorflowonspark\n   from tensorflowonspark import TFCluster, TFNode, TFParallel, dfutil\n   from tensorflowonspark.pipeline import TFEstimator, TFModel\n   print(tensorflowonspark.__version__)\n   PY\n   ```\n\n2. Confirm Spark and Java before running native Spark jobs:\n\n   ```bash\n   python scripts\u002Fcheck_environment.py --json\n   ```\n\n3. For Spark-fed training\u002Finference, confirm the Spark application has one task per executor, enough executor slots for the TensorFlow cluster, `spark.python.worker.reuse` for background mode, and the TensorFlow Hadoop jar on the Spark classpath when TFRecords are used.\n\n## Route by task\n\n| User task | Read |\n|---|---|\n| Start, stop, or debug a `TFCluster.run(...)` job; reservation waits; TensorBoard URL; TF_CONFIG; GPU allocation; `TFParallel.run(...)` | [cluster-lifecycle](sub-skills\u002Fcluster-lifecycle\u002FSKILL.md) |\n| Write a `map_fun(args, ctx)` using `InputMode.SPARK`; feed RDD\u002FDStream rows; return inference results; handle `DataFeed` timeouts or early termination | [datafeed-inputmode](sub-skills\u002Fdatafeed-inputmode\u002FSKILL.md) |\n| Train or infer with Spark ML DataFrames using `TFEstimator` or `TFModel`; map DataFrame columns to TensorFlow tensors; load SavedModels | [spark-ml-pipelines](sub-skills\u002Fspark-ml-pipelines\u002FSKILL.md) |\n| Save\u002Fload TFRecords as Spark DataFrames; reason about `tf.train.Example` schemas; handle `binary_features`; prepare MNIST-shaped rows | [dataframes-tfrecords](sub-skills\u002Fdataframes-tfrecords\u002FSKILL.md) |\n| Convert TensorFlow\u002FKeras\u002FEstimator examples to TensorFlowOnSpark; build `spark-submit` command plans for MNIST, ResNet, segmentation, serving, or batch inference | [examples-conversion](sub-skills\u002Fexamples-conversion\u002FSKILL.md) |\n\n## Shared references and scripts\n\n- [Repository provenance](references\u002Frepo-provenance.md) records the source commit, package version, evidence paths, and refresh baseline.\n- [API map](references\u002Fapi-map.md) shows which sub-skill owns each public module and workflow family.\n- [Troubleshooting](references\u002Ftroubleshooting.md) covers cross-cutting install\u002Fimport, Java\u002FSpark, TensorFlow\u002FPySpark, Hadoop jar, Python metadata, and GPU issues.\n- [check_environment.py](scripts\u002Fcheck_environment.py) is a safe diagnostic helper; it does not start training, download datasets, mutate clusters, or allocate GPUs.\n\n## Operating rules\n\n- Do not run long training examples, TF Serving Docker containers, cloud installers, or dataset downloads unless the user explicitly asks and the runtime is suitable.\n- Do not assume Spark local mode is equivalent to TensorFlowOnSpark's distributed executor behavior. For cluster semantics, prefer Spark Standalone, YARN, Kubernetes, or `local-cluster[...]` style tests that isolate executor processes.\n- For `InputMode.SPARK`, the TensorFlow function must consume rows from `TFNode.DataFeed`; Spark actions are still required to trigger lazy RDD inference.\n- For Spark ML Pipelines, the DataFrame API is Spark-fed (`InputMode.SPARK`); route TensorFlow-native file-reading workflows to `examples-conversion` or `cluster-lifecycle` instead.\n- For TFRecords, keep records flat enough for Spark SQL types and supply `binary_features` for raw bytes.\n- Treat GPU use as optional unless the user explicitly requires it. CPU can validate many workflows, but it does not prove CUDA allocation, driver compatibility, or GPU TensorFlow execution.\n\n## Common validation signals\n\n- `TFCluster.run(...)` returns a cluster object after every executor reserves a node.\n- `cluster.train(...)`, `cluster.inference(...)`, and `cluster.shutdown(...)` finish without late worker exceptions.\n- Spark ML `TFEstimator.fit(...)` writes the expected checkpoint or SavedModel artifact, and `TFModel.transform(...)` returns a DataFrame with the requested output columns.\n- TFRecord conversion preserves simple scalar\u002Farray fields and treats string versus binary features as intended.\n- Render-only helper scripts output reviewable commands\u002Ftemplates without running Spark, Docker, downloads, or destructive cleanup.\n",{"name":5,"license":23,"metadata":24,"description":7,"disable-model-invocation":26},"Apache 2.0",{"disco-role":25},"operating",true,"main","ac3fe1afa80fb9a09775ecfb2b6cc3ba850a2db6","skills\u002Frepositories\u002Frepo-skills\u002Ftensorflow-on-spark",27,"https:\u002F\u002Fgithub.com\u002FVectorSpaceLab\u002FAREX-Skill","https:\u002F\u002Fgithub.com\u002FVectorSpaceLab","2026-09-03 10:25:21",{"source":8,"skillName":5,"commands":35,"prompt":45,"zipAvailable":26},[36,39,42],{"tool":37,"command":38},"npx","npx skills add VectorSpaceLab\u002FAREX-Skill --skill tensorflow-on-spark",{"tool":40,"command":41},"bunx","bunx skills add VectorSpaceLab\u002FAREX-Skill --skill tensorflow-on-spark",{"tool":43,"command":44},"pnpm","pnpm dlx skills add VectorSpaceLab\u002FAREX-Skill --skill tensorflow-on-spark","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FVectorSpaceLab\u002FAREX-Skill\u002Fblob\u002Fmain\u002Fskills\u002Frepositories\u002Frepo-skills\u002Ftensorflow-on-spark\u002FSKILL.md",[47,53,57,61,65],{"path":48,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":51,"sourceHtmlUrl":52},"references\u002Fapi-map.md","REFERENCE","参考文档",2825,"https:\u002F\u002Fgithub.com\u002FVectorSpaceLab\u002FAREX-Skill\u002Fblob\u002Fmain\u002Fskills\u002Frepositories\u002Frepo-skills\u002Ftensorflow-on-spark\u002Freferences\u002Fapi-map.md",{"path":54,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":55,"sourceHtmlUrl":56},"references\u002Frepo-provenance.md",2151,"https:\u002F\u002Fgithub.com\u002FVectorSpaceLab\u002FAREX-Skill\u002Fblob\u002Fmain\u002Fskills\u002Frepositories\u002Frepo-skills\u002Ftensorflow-on-spark\u002Freferences\u002Frepo-provenance.md",{"path":58,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":59,"sourceHtmlUrl":60},"references\u002Frepo-routing-metadata.json",354,"https:\u002F\u002Fgithub.com\u002FVectorSpaceLab\u002FAREX-Skill\u002Fblob\u002Fmain\u002Fskills\u002Frepositories\u002Frepo-skills\u002Ftensorflow-on-spark\u002Freferences\u002Frepo-routing-metadata.json",{"path":62,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":63,"sourceHtmlUrl":64},"references\u002Ftroubleshooting.md",5125,"https:\u002F\u002Fgithub.com\u002FVectorSpaceLab\u002FAREX-Skill\u002Fblob\u002Fmain\u002Fskills\u002Frepositories\u002Frepo-skills\u002Ftensorflow-on-spark\u002Freferences\u002Ftroubleshooting.md",{"path":66,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":69,"sourceHtmlUrl":70},"scripts\u002Fcheck_environment.py","SCRIPT","脚本",6789,"https:\u002F\u002Fgithub.com\u002FVectorSpaceLab\u002FAREX-Skill\u002Fblob\u002Fmain\u002Fskills\u002Frepositories\u002Frepo-skills\u002Ftensorflow-on-spark\u002Fscripts\u002Fcheck_environment.py",[72,79,86,93,100,107],{"skillNo":73,"name":74,"nameZh":6,"description":75,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":76,"categories":77,"keywords":78,"publishedAt":18},"01M799NYSH1R3RVAHSM2D88Y84","passnet-triton-opt","Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block\u002Fgrid\u002Fwarp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.","https:\u002F\u002Fgithub.com\u002FVectorSpaceLab\u002FAREX-Skill\u002Fblob\u002Fmain\u002Fskills\u002Ftask-oriented\u002FPassNet\u002Fskills\u002Fpassnet-triton-opt\u002FSKILL.md",[],[],{"skillNo":80,"name":81,"nameZh":6,"description":82,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":18},"01M799NS5APEC0N1FZ7XZVW6BN","passnet-solve","END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.","https:\u002F\u002Fgithub.com\u002FVectorSpaceLab\u002FAREX-Skill\u002Fblob\u002Fmain\u002Fskills\u002Ftask-oriented\u002FPassNet\u002Fskills\u002Fpassnet-solve\u002FSKILL.md",[],[],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":18},"01M799NNY9BKSM21V5R7QVTBJX","passnet-skill","PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under .\u002Fpass_dir\u002F, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.","https:\u002F\u002Fgithub.com\u002FVectorSpaceLab\u002FAREX-Skill\u002Fblob\u002Fmain\u002Fskills\u002Ftask-oriented\u002FPassNet\u002Fskills\u002Fpassnet-skill\u002FSKILL.md",[],[],{"skillNo":94,"name":95,"nameZh":6,"description":96,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":97,"categories":98,"keywords":99,"publishedAt":18},"01M799NHH6NP7Z1RK77KGXAN6S","passnet-pattern-fusion","Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()\u002Freplacement_args()\u002Freplacement_func() files, when a pass \"failed to match\", or when deciding how to fuse multiple kernels\u002Fops.","https:\u002F\u002Fgithub.com\u002FVectorSpaceLab\u002FAREX-Skill\u002Fblob\u002Fmain\u002Fskills\u002Ftask-oriented\u002FPassNet\u002Fskills\u002Fpassnet-pattern-fusion\u002FSKILL.md",[],[],{"skillNo":101,"name":102,"nameZh":6,"description":103,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":18},"01M799ND2J4WNC2PGAZ0A06NDF","passnet-orchestrate","Round\u002Fbatch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep\u002Frevert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.","https:\u002F\u002Fgithub.com\u002FVectorSpaceLab\u002FAREX-Skill\u002Fblob\u002Fmain\u002Fskills\u002Ftask-oriented\u002FPassNet\u002Fskills\u002Fpassnet-orchestrate\u002FSKILL.md",[],[],{"skillNo":108,"name":109,"nameZh":6,"description":110,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":111,"categories":112,"keywords":113,"publishedAt":18},"01M799N8SBTMY02ZDG2AZF5ZWE","passnet-feedback","Fast, deterministic feedback for PassNet work: pre-flight pattern\u002Fmatch verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).","https:\u002F\u002Fgithub.com\u002FVectorSpaceLab\u002FAREX-Skill\u002Fblob\u002Fmain\u002Fskills\u002Ftask-oriented\u002FPassNet\u002Fskills\u002Fpassnet-feedback\u002FSKILL.md",[],[]]