[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6CDQKAMC7GGMPKS93NTPPQR":3,"skill-same:01M6CDQKAMC7GGMPKS93NTPPQR":97},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":45,"linkOnly":46,"markdownContent":47,"markdownBody":48,"frontMatter":49,"refName":27,"commitSha":59,"skillDirectory":60,"forksCount":61,"repositoryHtmlUrl":62,"repositoryHomepageUrl":6,"ownerHtmlUrl":63,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":64,"install":65,"resources":78},"01M6CDQKAMC7GGMPKS93NTPPQR","ai-pipeline-orchestration",null,"Orchestrate AI\u002FML pipelines for data ingestion, model training, batch inference, and RAG indexing using Prefect, Airflow, or Dagster.","boisenoise\u002Fskills-collections","boisenoise","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F193180?v=4","01M6CDBDQK6BDN55ZMA6B6PKBM","01M6CDBDVSJ3JPTBYN26FZ7TZK",12,"MIT","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-ai-pipeline-orchestration\u002FSKILL.md",[],[18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44],"orchestration","ai pipeline","ai pipeline orchestration","pipeline","relevant","github","dagster","critical","indexing","main","community","agent-skills","ingestion","only","environment","model","access","helm","devops-security","ci","orchestrate","runners","batch","license","airflow","training","ml","2026-11-10 10:12:30",false,"---\nname: ai-pipeline-orchestration\ndescription: Orchestrate AI\u002FML pipelines for data ingestion, model training, batch\n  inference, and RAG indexing using Prefect, Airflow, or Dagster.\ncategory: devops\nrisk: critical\nsource: https:\u002F\u002Fgithub.com\u002FBagelHole\u002FDevOps-Security-Agent-Skills\nsource_repo: BagelHole\u002FDevOps-Security-Agent-Skills\nsource_type: community\ndate_added: '2026-09-20'\nlicense: MIT\nlicense_source: https:\u002F\u002Fgithub.com\u002FBagelHole\u002FDevOps-Security-Agent-Skills\u002Fblob\u002Fmain\u002FLICENSE\ncompatibility: Requires the relevant platform CLIs (kubectl, helm, terraform, git,\n  CI runners) and authorized access to the target environment. Docs-only; helper scripts\n  and templates not bundled.\nmetadata:\n  author: devops-skills\n  version: '1.0'\n---\n\n# AI Pipeline Orchestration\n\nBuild reliable, observable AI workflows — from document ingestion to batch inference to model training pipelines.\n\n## When to Use This Skill\n\nUse this skill when:\n- Scheduling recurring RAG document ingestion and re-indexing\n- Orchestrating multi-step batch LLM processing workflows\n- Running nightly model evaluation and fine-tuning jobs\n- Building ETL pipelines that feed into AI models\n- Managing dependencies between data preparation and model serving\n\n## Execution contract\n\nThese are application design fragments: adapters, datasets, model files, secrets, vector schemas and deployment handlers are not bundled. Pin supported framework\u002Fprovider versions, select the actual target and permissions, and authorize schedules, provider charges and writes separately. Use synthetic fixtures first; preserve private data, validate fetched sources, and persist versioned checkpoints to avoid repeating side effects.\n\n## Tool Selection\n\n| Tool | Best For | Complexity | GPU Jobs |\n|------|----------|------------|----------|\n| **Prefect** | Modern Python-first; easy to adopt | Low | Good |\n| **Airflow** | Complex DAGs; large teams; existing usage | High | Good |\n| **Dagster** | Asset-centric; strong data lineage | Medium | Excellent |\n| **Temporal** | Long-running workflows; reliability-first | Medium | Good |\n\n## Prefect — Quick Start\n\n```bash\npip install prefect prefect-kubernetes\n\n# Start Prefect server (or use Prefect Cloud)\nprefect server start\n\n# In another terminal\nprefect worker start --pool default-agent-pool\n```\n\n## Prefect: RAG ingestion design sketch (not executable)\n\n```text\nfrom prefect import flow, task, get_run_logger\nfrom prefect.tasks import task_input_hash\nfrom datetime import timedelta\nimport hashlib\n\n@task(cache_key_fn=task_input_hash, cache_expiration=timedelta(hours=24))\ndef fetch_documents(source_url: str) -> list[dict]:\n    \"\"\"Fetch documents from source; cached to avoid re-fetching.\"\"\"\n    logger = get_run_logger()\n    logger.info(f\"Fetching from {source_url}\")\n    # ... fetch logic\n    return documents\n\n@task(retries=3, retry_delay_seconds=30)\ndef chunk_and_embed(documents: list[dict]) -> list[dict]:\n    \"\"\"Chunk documents and generate embeddings with retry on failure.\"\"\"\n    from sentence_transformers import SentenceTransformer\n    model = SentenceTransformer(\"BAAI\u002Fbge-large-en-v1.5\")\n    chunks = []\n    for doc in documents:\n        doc_chunks = chunk_text(doc[\"content\"])\n        embeddings = model.encode(doc_chunks, batch_size=64)\n        for chunk, emb in zip(doc_chunks, embeddings):\n            chunks.append({\"text\": chunk, \"embedding\": emb.tolist(),\n                           \"source\": doc[\"url\"], \"doc_hash\": doc[\"hash\"]})\n    return chunks\n\n@task(retries=2)\ndef upsert_to_vector_store(chunks: list[dict]) -> int:\n    \"\"\"Upsert embeddings to Qdrant, skip unchanged documents.\"\"\"\n    from qdrant_client import QdrantClient\n    client = QdrantClient(\"http:\u002F\u002Fqdrant:6333\")\n    client.upsert(collection_name=\"knowledge-base\", points=[...])\n    return len(chunks)\n\n@flow(name=\"rag-ingestion\", log_prints=True)\ndef rag_ingestion_pipeline(sources: list[str]):\n    \"\"\"Full RAG ingestion flow — runs daily.\"\"\"\n    logger = get_run_logger()\n    total = 0\n    for source in sources:\n        docs = fetch_documents(source)\n        chunks = chunk_and_embed(docs)\n        count = upsert_to_vector_store(chunks)\n        total += count\n        logger.info(f\"Ingested {count} chunks from {source}\")\n    logger.info(f\"Pipeline complete: {total} total chunks indexed\")\n\nif __name__ == \"__main__\":\n    rag_ingestion_pipeline.serve(\n        name=\"daily-rag-ingestion\",\n        cron=\"0 2 * * *\",          # 2 AM daily\n        parameters={\"sources\": [\"https:\u002F\u002Fdocs.myapp.com\", \"https:\u002F\u002Fapi.myapp.com\u002Fkb\"]},\n    )\n```\n\n## Prefect: Batch LLM Inference Pipeline\n\n```python\nfrom prefect import flow, task\nfrom prefect.concurrency.asyncio import concurrency\nimport asyncio\nfrom openai import AsyncOpenAI\n\n@task(retries=0)  # A batch retry can repeat successful billable calls\nasync def process_batch(items: list[dict], model: str) -> list[dict]:\n    \"\"\"Process a batch of items through LLM with rate limit protection.\"\"\"\n    client = AsyncOpenAI(timeout=60.0, max_retries=0)\n    async with concurrency(\"openai-api\", occupy=len(items), timeout_seconds=30, strict=True):  # Concurrent slots, not requests per minute\n        tasks = [\n            client.chat.completions.create(\n                model=model,\n                messages=[{\"role\": \"user\", \"content\": item[\"prompt\"]}],\n                max_tokens=256,\n            )\n            for item in items\n        ]\n        responses = await asyncio.gather(*tasks, return_exceptions=True)\n\n    await client.close()\n    results = []\n    for item, response in zip(items, responses):\n        if isinstance(response, Exception):\n            results.append({**item, \"error\": type(response).__name__, \"output\": None})\n        else:\n            results.append({**item, \"output\": response.choices[0].message.content})\n    return results\n\n@flow(name=\"batch-llm-inference\")\nasync def batch_inference_flow(input_file: str, output_file: str, model: str, batch_size: int = 10):\n    import json\n    if not 1 \u003C= batch_size \u003C= 10:\n        raise ValueError(\"batch_size must be between 1 and 10\")\n    with open(input_file, encoding=\"utf-8\") as source:\n        items = []\n        for line in source:\n            if len(line.encode(\"utf-8\")) > 32768 or len(items) >= 10000:\n                raise ValueError(\"Input exceeds reviewed fixture limits\")\n            item = json.loads(line)\n            if not isinstance(item, dict) or not isinstance(item.get(\"prompt\"), str):\n                raise ValueError(\"Invalid input item\")\n            items.append(item)\n    batches = [items[i:i+batch_size] for i in range(0, len(items), batch_size)]\n\n    all_results = []\n    for batch in batches:\n        results = await process_batch(batch, model)\n        all_results.extend(results)\n\n    # A full implementation reserves the output\u002Fcheckpoint before any billed calls;\n    # late file creation can otherwise fail after the whole batch has been charged.\n    with open(output_file, \"x\", encoding=\"utf-8\") as f:\n        for result in all_results:\n            f.write(json.dumps(result) + \"\\n\")\n    failed = sum(result.get(\"output\") is None for result in all_results)\n    return {\"total\": len(all_results), \"failed_or_empty\": failed, \"completed\": len(all_results) - failed}\n```\n\nThe async batch fragment requires a pre-created limit large enough for each batch; it fails if absent. Store outputs in an authorized private directory, handle existing\u002Fpartial files explicitly, and reconcile every provider call on timeout before retrying. Close the client in a `finally` block in a full implementation. Semaphore slots do not alone enforce token\u002FRPM or currency budgets.\n\n## Airflow: training DAG design sketch (not executable)\n\n```text\nfrom airflow.decorators import dag, task\nfrom airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator\nfrom datetime import datetime\nfrom kubernetes.client import models as k8s\n\n@dag(\n    dag_id=\"llm_fine_tuning\",\n    schedule=\"@weekly\",\n    start_date=datetime(2025, 1, 1),\n    catchup=False,\n    tags=[\"ai\", \"training\"],\n)\ndef llm_fine_tuning_dag():\n\n    @task\n    def prepare_dataset() -> str:\n        \"\"\"Download and preprocess training data.\"\"\"\n        # ... data prep logic\n        return \"s3:\u002F\u002Fmy-bucket\u002Ftraining-data\u002F2025-03-01\u002F\"\n\n    train = KubernetesPodOperator(\n        task_id=\"train_model\",\n        name=\"llm-training-job\",\n        namespace=\"ml\",\n        image=\"nvcr.io\u002Fnvidia\u002Fpytorch:24.05-py3\",\n        cmds=[\"accelerate\", \"launch\", \"-m\", \"axolotl.cli.train\", \"\u002Fconfig\u002Fconfig.yaml\"],\n        container_resources=k8s.V1ResourceRequirements(\n            limits={\"nvidia.com\u002Fgpu\": \"4\", \"memory\": \"320Gi\"},\n            requests={\"nvidia.com\u002Fgpu\": \"4\"},\n        ),\n        node_selector={\"nvidia.com\u002Fgpu.product\": \"A100-SXM4-80GB\"},\n        volumes=[...],\n        volume_mounts=[...],\n        get_logs=True,\n        is_delete_operator_pod=True,\n    )\n\n    @task\n    def evaluate_model(dataset_path: str) -> dict:\n        \"\"\"Run evals; fail pipeline if quality drops.\"\"\"\n        metrics = run_evals()\n        if metrics[\"accuracy\"] \u003C 0.85:\n            raise ValueError(f\"Model quality too low: {metrics}\")\n        return metrics\n\n    @task\n    def deploy_model(metrics: dict):\n        \"\"\"Push merged model to HF Hub and update vLLM config.\"\"\"\n        update_serving_config(new_model=\"org\u002Ffine-tuned-v2\")\n\n    dataset = prepare_dataset()\n    train.set_upstream(dataset)\n    eval_result = evaluate_model(dataset)\n    eval_result.set_upstream(train)\n    deploy_model(eval_result)\n\nllm_fine_tuning_dag()\n```\n\n## Dagster: asset pipeline design sketch (not executable)\n\n```text\nfrom dagster import asset, AssetExecutionContext, define_asset_job, ScheduleDefinition\n\n@asset(description=\"Raw documents fetched from knowledge sources\")\ndef raw_documents(context: AssetExecutionContext) -> list[dict]:\n    context.log.info(\"Fetching documents...\")\n    return fetch_all_documents()\n\n@asset(\n    deps=[raw_documents],\n    description=\"Chunked and embedded document vectors\",\n)\ndef document_embeddings(context: AssetExecutionContext, raw_documents) -> int:\n    chunks = process_and_embed(raw_documents)\n    context.log.info(f\"Generated {len(chunks)} embeddings\")\n    upsert_to_qdrant(chunks)\n    return len(chunks)\n\n@asset(\n    deps=[document_embeddings],\n    description=\"RAG system quality metrics\",\n)\ndef rag_quality_metrics(context: AssetExecutionContext) -> dict:\n    metrics = evaluate_rag_system()\n    context.add_output_metadata({\"ragas_score\": metrics[\"ragas_score\"]})\n    return metrics\n\n# Schedule: refresh embeddings nightly\nnightly_refresh = ScheduleDefinition(\n    job=define_asset_job(\"rag_refresh_job\", [raw_documents, document_embeddings]),\n    cron_schedule=\"0 1 * * *\",\n)\n```\n\n## Best Practices\n\n- Use task-level retries for API calls; use flow-level retries for transient infra failures.\n- Cache expensive steps (embedding generation, data fetching) to speed up reruns.\n- Emit custom metrics from pipelines (chunk count, error rate, cost) to your observability stack.\n- Use `concurrency` limits in Prefect or `pool` slots in Airflow to respect external rate limits.\n- Separate ingestion, training, and deployment pipelines — don't couple them in one giant DAG.\n\n## Related Skills\n\n- rag-infrastructure (`rag-infrastructure`) - RAG system setup\n- llm-fine-tuning (`llm-fine-tuning`) - Training jobs\n- agent-observability (`agent-observability`) - Pipeline monitoring\n- kubernetes-ops (`kubernetes-ops`) - Running pipeline pods on K8s\n\n## Limitations\n\n- Guidance executes against real environments: confirm target, blast radius, and rollback plan before applying anything.\n- Never deploy to production without explicit approval. Docs-only import: upstream scripts and templates not bundled.\n\n### Example\n\n```bash\ngit status && git diff --stat\nkubectl diff -f manifest.yaml\n```\n\n> Adapted from [BagelHole\u002FDevOps-Security-Agent-Skills](https:\u002F\u002Fgithub.com\u002FBagelHole\u002FDevOps-Security-Agent-Skills) (MIT); frontmatter, When to Use\u002FLimitations, and safety boundaries added for upstream compliance. Docs-only import: helper scripts and templates not bundled.\n","# AI Pipeline Orchestration\n\nBuild reliable, observable AI workflows — from document ingestion to batch inference to model training pipelines.\n\n## When to Use This Skill\n\nUse this skill when:\n- Scheduling recurring RAG document ingestion and re-indexing\n- Orchestrating multi-step batch LLM processing workflows\n- Running nightly model evaluation and fine-tuning jobs\n- Building ETL pipelines that feed into AI models\n- Managing dependencies between data preparation and model serving\n\n## Execution contract\n\nThese are application design fragments: adapters, datasets, model files, secrets, vector schemas and deployment handlers are not bundled. Pin supported framework\u002Fprovider versions, select the actual target and permissions, and authorize schedules, provider charges and writes separately. Use synthetic fixtures first; preserve private data, validate fetched sources, and persist versioned checkpoints to avoid repeating side effects.\n\n## Tool Selection\n\n| Tool | Best For | Complexity | GPU Jobs |\n|------|----------|------------|----------|\n| **Prefect** | Modern Python-first; easy to adopt | Low | Good |\n| **Airflow** | Complex DAGs; large teams; existing usage | High | Good |\n| **Dagster** | Asset-centric; strong data lineage | Medium | Excellent |\n| **Temporal** | Long-running workflows; reliability-first | Medium | Good |\n\n## Prefect — Quick Start\n\n```bash\npip install prefect prefect-kubernetes\n\n# Start Prefect server (or use Prefect Cloud)\nprefect server start\n\n# In another terminal\nprefect worker start --pool default-agent-pool\n```\n\n## Prefect: RAG ingestion design sketch (not executable)\n\n```text\nfrom prefect import flow, task, get_run_logger\nfrom prefect.tasks import task_input_hash\nfrom datetime import timedelta\nimport hashlib\n\n@task(cache_key_fn=task_input_hash, cache_expiration=timedelta(hours=24))\ndef fetch_documents(source_url: str) -> list[dict]:\n    \"\"\"Fetch documents from source; cached to avoid re-fetching.\"\"\"\n    logger = get_run_logger()\n    logger.info(f\"Fetching from {source_url}\")\n    # ... fetch logic\n    return documents\n\n@task(retries=3, retry_delay_seconds=30)\ndef chunk_and_embed(documents: list[dict]) -> list[dict]:\n    \"\"\"Chunk documents and generate embeddings with retry on failure.\"\"\"\n    from sentence_transformers import SentenceTransformer\n    model = SentenceTransformer(\"BAAI\u002Fbge-large-en-v1.5\")\n    chunks = []\n    for doc in documents:\n        doc_chunks = chunk_text(doc[\"content\"])\n        embeddings = model.encode(doc_chunks, batch_size=64)\n        for chunk, emb in zip(doc_chunks, embeddings):\n            chunks.append({\"text\": chunk, \"embedding\": emb.tolist(),\n                           \"source\": doc[\"url\"], \"doc_hash\": doc[\"hash\"]})\n    return chunks\n\n@task(retries=2)\ndef upsert_to_vector_store(chunks: list[dict]) -> int:\n    \"\"\"Upsert embeddings to Qdrant, skip unchanged documents.\"\"\"\n    from qdrant_client import QdrantClient\n    client = QdrantClient(\"http:\u002F\u002Fqdrant:6333\")\n    client.upsert(collection_name=\"knowledge-base\", points=[...])\n    return len(chunks)\n\n@flow(name=\"rag-ingestion\", log_prints=True)\ndef rag_ingestion_pipeline(sources: list[str]):\n    \"\"\"Full RAG ingestion flow — runs daily.\"\"\"\n    logger = get_run_logger()\n    total = 0\n    for source in sources:\n        docs = fetch_documents(source)\n        chunks = chunk_and_embed(docs)\n        count = upsert_to_vector_store(chunks)\n        total += count\n        logger.info(f\"Ingested {count} chunks from {source}\")\n    logger.info(f\"Pipeline complete: {total} total chunks indexed\")\n\nif __name__ == \"__main__\":\n    rag_ingestion_pipeline.serve(\n        name=\"daily-rag-ingestion\",\n        cron=\"0 2 * * *\",          # 2 AM daily\n        parameters={\"sources\": [\"https:\u002F\u002Fdocs.myapp.com\", \"https:\u002F\u002Fapi.myapp.com\u002Fkb\"]},\n    )\n```\n\n## Prefect: Batch LLM Inference Pipeline\n\n```python\nfrom prefect import flow, task\nfrom prefect.concurrency.asyncio import concurrency\nimport asyncio\nfrom openai import AsyncOpenAI\n\n@task(retries=0)  # A batch retry can repeat successful billable calls\nasync def process_batch(items: list[dict], model: str) -> list[dict]:\n    \"\"\"Process a batch of items through LLM with rate limit protection.\"\"\"\n    client = AsyncOpenAI(timeout=60.0, max_retries=0)\n    async with concurrency(\"openai-api\", occupy=len(items), timeout_seconds=30, strict=True):  # Concurrent slots, not requests per minute\n        tasks = [\n            client.chat.completions.create(\n                model=model,\n                messages=[{\"role\": \"user\", \"content\": item[\"prompt\"]}],\n                max_tokens=256,\n            )\n            for item in items\n        ]\n        responses = await asyncio.gather(*tasks, return_exceptions=True)\n\n    await client.close()\n    results = []\n    for item, response in zip(items, responses):\n        if isinstance(response, Exception):\n            results.append({**item, \"error\": type(response).__name__, \"output\": None})\n        else:\n            results.append({**item, \"output\": response.choices[0].message.content})\n    return results\n\n@flow(name=\"batch-llm-inference\")\nasync def batch_inference_flow(input_file: str, output_file: str, model: str, batch_size: int = 10):\n    import json\n    if not 1 \u003C= batch_size \u003C= 10:\n        raise ValueError(\"batch_size must be between 1 and 10\")\n    with open(input_file, encoding=\"utf-8\") as source:\n        items = []\n        for line in source:\n            if len(line.encode(\"utf-8\")) > 32768 or len(items) >= 10000:\n                raise ValueError(\"Input exceeds reviewed fixture limits\")\n            item = json.loads(line)\n            if not isinstance(item, dict) or not isinstance(item.get(\"prompt\"), str):\n                raise ValueError(\"Invalid input item\")\n            items.append(item)\n    batches = [items[i:i+batch_size] for i in range(0, len(items), batch_size)]\n\n    all_results = []\n    for batch in batches:\n        results = await process_batch(batch, model)\n        all_results.extend(results)\n\n    # A full implementation reserves the output\u002Fcheckpoint before any billed calls;\n    # late file creation can otherwise fail after the whole batch has been charged.\n    with open(output_file, \"x\", encoding=\"utf-8\") as f:\n        for result in all_results:\n            f.write(json.dumps(result) + \"\\n\")\n    failed = sum(result.get(\"output\") is None for result in all_results)\n    return {\"total\": len(all_results), \"failed_or_empty\": failed, \"completed\": len(all_results) - failed}\n```\n\nThe async batch fragment requires a pre-created limit large enough for each batch; it fails if absent. Store outputs in an authorized private directory, handle existing\u002Fpartial files explicitly, and reconcile every provider call on timeout before retrying. Close the client in a `finally` block in a full implementation. Semaphore slots do not alone enforce token\u002FRPM or currency budgets.\n\n## Airflow: training DAG design sketch (not executable)\n\n```text\nfrom airflow.decorators import dag, task\nfrom airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator\nfrom datetime import datetime\nfrom kubernetes.client import models as k8s\n\n@dag(\n    dag_id=\"llm_fine_tuning\",\n    schedule=\"@weekly\",\n    start_date=datetime(2025, 1, 1),\n    catchup=False,\n    tags=[\"ai\", \"training\"],\n)\ndef llm_fine_tuning_dag():\n\n    @task\n    def prepare_dataset() -> str:\n        \"\"\"Download and preprocess training data.\"\"\"\n        # ... data prep logic\n        return \"s3:\u002F\u002Fmy-bucket\u002Ftraining-data\u002F2025-03-01\u002F\"\n\n    train = KubernetesPodOperator(\n        task_id=\"train_model\",\n        name=\"llm-training-job\",\n        namespace=\"ml\",\n        image=\"nvcr.io\u002Fnvidia\u002Fpytorch:24.05-py3\",\n        cmds=[\"accelerate\", \"launch\", \"-m\", \"axolotl.cli.train\", \"\u002Fconfig\u002Fconfig.yaml\"],\n        container_resources=k8s.V1ResourceRequirements(\n            limits={\"nvidia.com\u002Fgpu\": \"4\", \"memory\": \"320Gi\"},\n            requests={\"nvidia.com\u002Fgpu\": \"4\"},\n        ),\n        node_selector={\"nvidia.com\u002Fgpu.product\": \"A100-SXM4-80GB\"},\n        volumes=[...],\n        volume_mounts=[...],\n        get_logs=True,\n        is_delete_operator_pod=True,\n    )\n\n    @task\n    def evaluate_model(dataset_path: str) -> dict:\n        \"\"\"Run evals; fail pipeline if quality drops.\"\"\"\n        metrics = run_evals()\n        if metrics[\"accuracy\"] \u003C 0.85:\n            raise ValueError(f\"Model quality too low: {metrics}\")\n        return metrics\n\n    @task\n    def deploy_model(metrics: dict):\n        \"\"\"Push merged model to HF Hub and update vLLM config.\"\"\"\n        update_serving_config(new_model=\"org\u002Ffine-tuned-v2\")\n\n    dataset = prepare_dataset()\n    train.set_upstream(dataset)\n    eval_result = evaluate_model(dataset)\n    eval_result.set_upstream(train)\n    deploy_model(eval_result)\n\nllm_fine_tuning_dag()\n```\n\n## Dagster: asset pipeline design sketch (not executable)\n\n```text\nfrom dagster import asset, AssetExecutionContext, define_asset_job, ScheduleDefinition\n\n@asset(description=\"Raw documents fetched from knowledge sources\")\ndef raw_documents(context: AssetExecutionContext) -> list[dict]:\n    context.log.info(\"Fetching documents...\")\n    return fetch_all_documents()\n\n@asset(\n    deps=[raw_documents],\n    description=\"Chunked and embedded document vectors\",\n)\ndef document_embeddings(context: AssetExecutionContext, raw_documents) -> int:\n    chunks = process_and_embed(raw_documents)\n    context.log.info(f\"Generated {len(chunks)} embeddings\")\n    upsert_to_qdrant(chunks)\n    return len(chunks)\n\n@asset(\n    deps=[document_embeddings],\n    description=\"RAG system quality metrics\",\n)\ndef rag_quality_metrics(context: AssetExecutionContext) -> dict:\n    metrics = evaluate_rag_system()\n    context.add_output_metadata({\"ragas_score\": metrics[\"ragas_score\"]})\n    return metrics\n\n# Schedule: refresh embeddings nightly\nnightly_refresh = ScheduleDefinition(\n    job=define_asset_job(\"rag_refresh_job\", [raw_documents, document_embeddings]),\n    cron_schedule=\"0 1 * * *\",\n)\n```\n\n## Best Practices\n\n- Use task-level retries for API calls; use flow-level retries for transient infra failures.\n- Cache expensive steps (embedding generation, data fetching) to speed up reruns.\n- Emit custom metrics from pipelines (chunk count, error rate, cost) to your observability stack.\n- Use `concurrency` limits in Prefect or `pool` slots in Airflow to respect external rate limits.\n- Separate ingestion, training, and deployment pipelines — don't couple them in one giant DAG.\n\n## Related Skills\n\n- rag-infrastructure (`rag-infrastructure`) - RAG system setup\n- llm-fine-tuning (`llm-fine-tuning`) - Training jobs\n- agent-observability (`agent-observability`) - Pipeline monitoring\n- kubernetes-ops (`kubernetes-ops`) - Running pipeline pods on K8s\n\n## Limitations\n\n- Guidance executes against real environments: confirm target, blast radius, and rollback plan before applying anything.\n- Never deploy to production without explicit approval. Docs-only import: upstream scripts and templates not bundled.\n\n### Example\n\n```bash\ngit status && git diff --stat\nkubectl diff -f manifest.yaml\n```\n\n> Adapted from [BagelHole\u002FDevOps-Security-Agent-Skills](https:\u002F\u002Fgithub.com\u002FBagelHole\u002FDevOps-Security-Agent-Skills) (MIT); frontmatter, When to Use\u002FLimitations, and safety boundaries added for upstream compliance. Docs-only import: helper scripts and templates not bundled.\n",{"name":5,"risk":25,"source":50,"license":14,"category":51,"metadata":52,"date_added":55,"description":7,"source_repo":56,"source_type":28,"compatibility":57,"license_source":58},"https:\u002F\u002Fgithub.com\u002FBagelHole\u002FDevOps-Security-Agent-Skills","devops",{"author":53,"version":54},"devops-skills","1.0","2026-09-20","BagelHole\u002FDevOps-Security-Agent-Skills","Requires the relevant platform CLIs (kubectl, helm, terraform, git, CI runners) and authorized access to the target environment. Docs-only; helper scripts and templates not bundled.","https:\u002F\u002Fgithub.com\u002FBagelHole\u002FDevOps-Security-Agent-Skills\u002Fblob\u002Fmain\u002FLICENSE","e8c4cb5cbafb600fab08ffdf4b23359c9f608d9d","skills\u002Fantigravity-ai-pipeline-orchestration",3,"https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections","https:\u002F\u002Fgithub.com\u002Fboisenoise","2026-11-10 14:20:15",{"source":8,"skillName":5,"commands":66,"prompt":76,"zipAvailable":77},[67,70,73],{"tool":68,"command":69},"npx","npx skills add boisenoise\u002Fskills-collections --skill ai-pipeline-orchestration",{"tool":71,"command":72},"bunx","bunx skills add boisenoise\u002Fskills-collections --skill ai-pipeline-orchestration",{"tool":74,"command":75},"pnpm","pnpm dlx skills add boisenoise\u002Fskills-collections --skill ai-pipeline-orchestration","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-ai-pipeline-orchestration\u002FSKILL.md",true,[79,85,89,93],{"path":80,"resourceType":81,"resourceTypeDesc":82,"sizeBytes":83,"sourceHtmlUrl":84},"references\u002Ffiltering.md","REFERENCE","参考文档",10038,"https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fexternal_plugins\u002Fscientific-skills\u002Fmatchms\u002Freferences\u002Ffiltering.md",{"path":86,"resourceType":81,"resourceTypeDesc":82,"sizeBytes":87,"sourceHtmlUrl":88},"references\u002Fimporting_exporting.md",10894,"https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fexternal_plugins\u002Fscientific-skills\u002Fmatchms\u002Freferences\u002Fimporting_exporting.md",{"path":90,"resourceType":81,"resourceTypeDesc":82,"sizeBytes":91,"sourceHtmlUrl":92},"references\u002Fsimilarity.md",12974,"https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fexternal_plugins\u002Fscientific-skills\u002Fmatchms\u002Freferences\u002Fsimilarity.md",{"path":94,"resourceType":81,"resourceTypeDesc":82,"sizeBytes":95,"sourceHtmlUrl":96},"references\u002Fworkflows.md",22230,"https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fexternal_plugins\u002Fscientific-skills\u002Fmatchms\u002Freferences\u002Fworkflows.md",[98,125,151,177,196,219],{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":102,"licenseFileUrl":103,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":45},"01M6WNH92BHVFXHGPDZCPJ0AG0","web-access","Use Treg for public web or platform data only when existing search, browser and connected tools cannot cover the task, with explicit cost and data boundaries.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-web-access\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-web-access\u002FSKILL.md",[],[107,34,108,109,110,111,112,113,114,115,116,31,117,118,119,25,120,121,122,123,124],"web","web access","official","development","browser","search","data","Tools","read","fetch","youtube","free tools","platform","free","cover","normal","when","2026",{"skillNo":126,"name":127,"nameZh":6,"description":128,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":102,"licenseFileUrl":129,"sourceHtmlUrl":130,"categories":131,"keywords":137,"publishedAt":45},"01M6WNF9XAYAB2CZDDJBAKNM93","treg","Use the Treg CLI or MCP catalog to discover, price and call authorized external tools with team-scoped credentials and a user-approved budget.","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-treg\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-treg\u002FSKILL.md",[132],{"categoryNo":133,"categoryName":134,"taxonomyType":135,"taxonomyTypeDesc":136},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[127,138,139,23,109,110,41,114,140,141,142,143,144,145,146,25,147,148,149,150,124],"MCP","cli","Budget","credentials","team","catalog","use","call","discover","external tools","with","scoped","blob",{"skillNo":152,"name":153,"nameZh":6,"description":154,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":102,"licenseFileUrl":155,"sourceHtmlUrl":156,"categories":157,"keywords":163,"publishedAt":45},"01M6WN93MEAZ4E41DYAHJRPHQT","recruit","Turn a job description into an evidence-linked candidate shortlist through Treg, with job-related ranking and optional authorized business contact enrichment.","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-recruit\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-recruit\u002FSKILL.md",[158],{"categoryNo":159,"categoryName":160,"taxonomyType":161,"taxonomyTypeDesc":162},"01JBSEEDCAT000000000000038","人力资源","PROFESSION","适用职业",[153,23,109,41,164,165,166,167,168,169,25,170,171,172,148,173,174,175,176,150,127,124],"https","email","enrichment","business","evidence","phone","ranking","post","job description","for","person","candidate","reason",{"skillNo":178,"name":179,"nameZh":6,"description":180,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":102,"licenseFileUrl":181,"sourceHtmlUrl":182,"categories":183,"keywords":184,"publishedAt":45},"01M6WN8R5RXSMHM9B4D9FAFX42","rea-growth","Analyze a company’s public growth and content through Treg, distinguish observations from hypotheses, and produce an evidence-linked playbook within an approved budget.","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-rea-growth\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-rea-growth\u002FSKILL.md",[],[185,186,187,23,109,41,188,189,164,190,140,191,192,193,168,25,194,195,150,127,124],"growth","rea","rea growth","produce","content","playbook","marketing","analyze","public","observations","company",{"skillNo":197,"name":198,"nameZh":6,"description":199,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":102,"licenseFileUrl":200,"sourceHtmlUrl":201,"categories":202,"keywords":203,"publishedAt":45},"01M6WN2EVE3RQRGWAKBT4S7YT2","make-ugc","Create original AI UGC-style product videos through Treg: research hooks, approve a licensed presenter and voice, generate clips, add captions and report costs.","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-make-ugc\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-make-ugc\u002FSKILL.md",[],[204,205,23,206,109,207,41,208,209,210,211,212,25,213,214,215,216,217,218,150],"make","make ugc","research","create","Video","Captions","ads","short","hooks","head","style","niche","trending","talking-head","approve",{"skillNo":220,"name":221,"nameZh":6,"description":222,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":102,"licenseFileUrl":223,"sourceHtmlUrl":224,"categories":225,"keywords":226,"publishedAt":45},"01M6WN10ARAA63Y8K1YNGXJ2RM","lead-signals","Find and qualify public buying signals through Treg, rank fitting business accounts and optionally enrich authorized business contact information.","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-lead-signals\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-lead-signals\u002FSKILL.md",[],[227,228,229,23,109,41,230,231,167,232,233,25,234,235,236,237,238,239,240,150],"signals","lead","lead signals","how","find","ready","as","how to","qualify","people","information","enrich","changes","funding"]