AGENT SKILL

senior-ml-engineer

ML engineering skill for productionizing models, building MLOps pipelines, and integrating LLMs. Covers model deployment, feature stores, drift monitoring, RAG systems, and cost optimization. Use when the user asks about deploying ML models to production, setting up MLOps infrastructure (MLflow, Kubeflow, Kubernetes, Docker), monitoring model performance or drift, building RAG pipelines, or integrating LLM APIs with retry logic and cost controls. Focused on production and operational concerns rather than model research or initial training.

License 许可
MIT
最近更新
2026-06-04
Stars
★ 2
Forks
0

01 前言 Front Matter

name
senior-ml-engineer
triggers
[ "MLOps pipeline", "model deployment", "feature store", "model monitoring", "drift detection", "RAG system", "LLM integration", "model serving", "A/B testing ML", "automated retraining" ]
description
ML engineering skill for productionizing models, building MLOps pipelines, and integrating LLMs. Covers model deployment, feature stores, drift monitoring, RAG systems, and cost optimization. Use when the user asks about deploying ML models to production, setting up MLOps infrastructure (MLflow, Kubeflow, Kubernetes, Docker), monitoring model performance or drift, building RAG pipelines, or integrating LLM APIs with retry logic and cost controls. Focused on production and operational concerns rather than model research or initial training.

02 资源文件 6

03 SKILL.md 正文

Senior ML Engineer

Production ML engineering patterns for model deployment, MLOps infrastructure, and LLM integration.


Table of Contents


Model Deployment Workflow

Deploy a trained model to production with monitoring:

  1. Export model to standardized format (ONNX, TorchScript, SavedModel)
  2. Package model with dependencies in Docker container
  3. Deploy to staging environment
  4. Run integration tests against staging
  5. Deploy canary (5% traffic) to production
  6. Monitor latency and error rates for 1 hour
  7. Promote to full production if metrics pass
  8. Validation: p95 latency < 100ms, error rate < 0.1%

Container Template

FROM python:3.11-slim

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY model/ /app/model/
COPY src/ /app/src/

HEALTHCHECK CMD curl -f http://localhost:8080/health || exit 1

EXPOSE 8080
CMD ["uvicorn", "src.server:app", "--host", "0.0.0.0", "--port", "8080"]

Serving Options

Option Latency Throughput Use Case
FastAPI + Uvicorn Low Medium REST APIs, small models
Triton Inference Server Very Low Very High GPU inference, batching
TensorFlow Serving Low High TensorFlow models
TorchServe Low High PyTorch models
Ray Serve Medium High Complex pipelines, multi-model

MLOps Pipeline Setup

Establish automated training and deployment:

  1. Configure feature store (Feast, Tecton) for training data
  2. Set up experiment tracking (MLflow, Weights & Biases)
  3. Create training pipeline with hyperparameter logging
  4. Register model in model registry with version metadata
  5. Configure staging deployment triggered by registry events
  6. Set up A/B testing infrastructure for model comparison
  7. Enable drift monitoring with alerting
  8. Validation: New models automatically evaluated against baseline

Feature Store Pattern

from feast import Entity, Feature, FeatureView, FileSource

user = Entity(name="user_id", value_type=ValueType.INT64)

user_features = FeatureView(
    name="user_features",
    entities=["user_id"],
    ttl=timedelta(days=1),
    features=[
        Feature(name="purchase_count_30d", dtype=ValueType.INT64),
        Feature(name="avg_order_value", dtype=ValueType.FLOAT),
    ],
    online=True,
    source=FileSource(path="data/user_features.parquet"),
)

Retraining Triggers

Trigger Detection Action
Scheduled Cron (weekly/monthly) Full retrain
Performance drop Accuracy < threshold Immediate retrain
Data drift PSI > 0.2 Evaluate, then retrain
New data volume X new samples Incremental update

LLM Integration Workflow

Integrate LLM APIs into production applications:

  1. Create provider abstraction layer for vendor flexibility
  2. Implement retry logic with exponential backoff
  3. Configure fallback to secondary provider
  4. Set up token counting and context truncation
  5. Add response caching for repeated queries
  6. Implement cost tracking per request
  7. Add structured output validation with Pydantic
  8. Validation: Response parses correctly, cost within budget

Provider Abstraction

from abc import ABC, abstractmethod
from tenacity import retry, stop_after_attempt, wait_exponential

class LLMProvider(ABC):
    @abstractmethod
    def complete(self, prompt: str, **kwargs) -> str:
        pass

@retry(stop=stop_after_attempt(3), wait=wait_exponential(min=1, max=10))
def call_llm_with_retry(provider: LLMProvider, prompt: str) -> str:
    return provider.complete(prompt)

Cost Management

Provider Input Cost Output Cost
GPT-4 $0.03/1K $0.06/1K
GPT-3.5 $0.0005/1K $0.0015/1K
Claude 3 Opus $0.015/1K $0.075/1K
Claude 3 Haiku $0.00025/1K $0.00125/1K

RAG System Implementation

Build retrieval-augmented generation pipeline:

  1. Choose vector database (Pinecone, Qdrant, Weaviate)
  2. Select embedding model based on quality/cost tradeoff
  3. Implement document chunking strategy
  4. Create ingestion pipeline with metadata extraction
  5. Build retrieval with query embedding
  6. Add reranking for relevance improvement
  7. Format context and send to LLM
  8. Validation: Response references retrieved context, no hallucinations

Vector Database Selection

Database Hosting Scale Latency Best For
Pinecone Managed High Low Production, managed
Qdrant Both High Very Low Performance-critical
Weaviate Both High Low Hybrid search
Chroma Self-hosted Medium Low Prototyping
pgvector Self-hosted Medium Medium Existing Postgres

Chunking Strategies

Strategy Chunk Size Overlap Best For
Fixed 500-1000 tokens 50-100 General text
Sentence 3-5 sentences 1 sentence Structured text
Semantic Variable Based on meaning Research papers
Recursive Hierarchical Parent-child Long documents

Model Monitoring

Monitor production models for drift and degradation:

  1. Set up latency tracking (p50, p95, p99)
  2. Configure error rate alerting
  3. Implement input data drift detection
  4. Track prediction distribution shifts
  5. Log ground truth when available
  6. Compare model versions with A/B metrics
  7. Set up automated retraining triggers
  8. Validation: Alerts fire before user-visible degradation

Drift Detection

from scipy.stats import ks_2samp

def detect_drift(reference, current, threshold=0.05):
    statistic, p_value = ks_2samp(reference, current)
    return {
        "drift_detected": p_value < threshold,
        "ks_statistic": statistic,
        "p_value": p_value
    }

Alert Thresholds

Metric Warning Critical
p95 latency > 100ms > 200ms
Error rate > 0.1% > 1%
PSI (drift) > 0.1 > 0.2
Accuracy drop > 2% > 5%

Reference Documentation

MLOps Production Patterns

references/mlops_production_patterns.md contains:

  • Model deployment pipeline with Kubernetes manifests
  • Feature store architecture with Feast examples
  • Model monitoring with drift detection code
  • A/B testing infrastructure with traffic splitting
  • Automated retraining pipeline with MLflow

LLM Integration Guide

references/llm_integration_guide.md contains:

  • Provider abstraction layer pattern
  • Retry and fallback strategies with tenacity
  • Prompt engineering templates (few-shot, CoT)
  • Token optimization with tiktoken
  • Cost calculation and tracking

RAG System Architecture

references/rag_system_architecture.md contains:

  • RAG pipeline implementation with code
  • Vector database comparison and integration
  • Chunking strategies (fixed, semantic, recursive)
  • Embedding model selection guide
  • Hybrid search and reranking patterns

Tools

Model Deployment Pipeline

python scripts/model_deployment_pipeline.py --model model.pkl --target staging

Generates deployment artifacts: Dockerfile, Kubernetes manifests, health checks.

RAG System Builder

python scripts/rag_system_builder.py --config rag_config.yaml --analyze

Scaffolds RAG pipeline with vector store integration and retrieval logic.

ML Monitoring Suite

python scripts/ml_monitoring_suite.py --config monitoring.yaml --deploy

Sets up drift detection, alerting, and performance dashboards.


Tech Stack

Category Tools
ML Frameworks PyTorch, TensorFlow, Scikit-learn, XGBoost
LLM Frameworks LangChain, LlamaIndex, DSPy
MLOps MLflow, Weights & Biases, Kubeflow
Data Spark, Airflow, dbt, Kafka
Deployment Docker, Kubernetes, Triton
Databases PostgreSQL, BigQuery, Pinecone, Redis

04 同仓库更多 Skills

kruxshnx
kruxshnx/claude-skills-devin

syllabus

Generates a curated supplementary reading list from any course syllabus using Consensus academic search. Grill-me intake (syllabus input format + course audience + year range) plus a grouping forcing-options checkpoint before any search runs — so the reading list matches the course's level and recency need. Parses the syllabus to extract topics and learning outcomes, searches Consensus for recent peer-reviewed papers per topic, and produces a professionally formatted .docx with clickable Consensus links, plain-language summaries calibrated to audience level, and Bloom-higher-order discussion questions tied to course learning goals. Triggers whenever a user uploads a syllabus, course outline, or curriculum document and wants supplementary readings. Also triggers on: 'syllabus reading list', 'find papers for my course', 'create a reading list from this syllabus', 'recent research for my class', 'supplementary readings', 'find journal articles for these topics', 'what recent papers cover this material', 'any new research on these course topics', 'update my syllabus with recent papers'. Even casual mentions when a syllabus is attached should trigger this skill.

★ 2
kruxshnx
kruxshnx/claude-skills-devin

research

Default entry point for any research request — a hybrid router that classifies the question deterministically and either delegates to a specialist research skill (pulse for trends/sentiment, grants for NIH funding, litreview for academic literature, syllabus for course reading, patent for prior-art + IP landscape, dossier for entity research) or runs its own plan-decompose-multi-source-search-synthesize-cite fallback workflow when no specialist matches. Always surfaces the routing decision so users can override. Triggers — "research [topic]", "look into [topic]", "what do we know about [topic]", "investigate [topic]", "find me information on [topic]", "do some research on [topic]", "I need to understand [topic]", or any research request that doesn't obviously match a more-specific specialist skill. Output is a markdown briefing (default) or .docx document (on request) with full citations and an audit log.

★ 2
kruxshnx
kruxshnx/claude-skills-devin

pulse

Multi-source recency research skill that takes the pulse of any topic across Reddit, Hacker News, the open web, and optionally X/Twitter within a configurable recent window (default 30 days). Forcing intake clarifies topic specificity, angle (trend/sentiment/problems/opportunities/comparison), time window, and platform scope before searching. Returns a synthesized briefing with citations, engagement metrics, and cross-platform pattern analysis. Triggers: 'pulse on [topic]', 'what's happening with [topic]', 'what are people saying about [topic]', 'current conversation about [topic]', 'take the pulse of [topic]', 'trending: [topic]', 'find me info on [topic]', or any variation requesting multi-source recency intelligence on a topic. Also use for competitor research, trend discovery, tool comparisons, and audience sentiment analysis.

★ 2
kruxshnx
kruxshnx/claude-skills-devin

patent

Patent prior-art and landscape intelligence skill — not generic patent help. Commits to one of five sub-use-cases via forcing intake (novelty search / freedom-to-operate / competitive landscape / acquisition diligence / litigation prior-art) before any search runs. Searches Google Patents, Espacenet, USPTO, and optionally Lens.org for citation-graph signals. Output is an editable Word document (.docx) with verdict, ranked closest art (claim-text extracted), CPC-class-aware landscape, family-resolved hits, geographic coverage, FTO flags where applicable, strategy recommendations, and full audit log. Triggers: 'prior art search for [invention]', 'patent search on [topic]', 'freedom to operate analysis', 'FTO for [product]', 'patent landscape for [field]', 'is [invention] novel', 'patents on [topic]', 'competitive patent analysis', 'prior art for litigation', 'patent diligence on [company]'. Produces search signal, not legal advice — always recommends consulting a patent attorney before filing or licensing decisions. Trademark, copyright, and trade-secret questions are out of scope.

★ 2
kruxshnx
kruxshnx/claude-skills-devin

notebooklm

Browser automation skill for controlling Google's NotebookLM. Handles reading and querying notebooks, adding sources (URLs, text, files, YouTube links, synthesized content), generating Studio outputs (Audio Overview, infographics, slide decks, study guides, briefing docs, mind maps, timelines, FAQs), and creating new notebooks. Triggers on any phrase involving NotebookLM — 'open NotebookLM', 'check my [name] notebook', 'pull info from NotebookLM', 'ask my notebook about X', 'add [source] to NotebookLM', 'create an infographic in NotebookLM', 'use NotebookLM Studio', 'generate a slide deck from my notebook', or any variation where the goal involves NotebookLM. Requires browser automation environment — fails gracefully when unavailable.

★ 2
kruxshnx
kruxshnx/claude-skills-devin

litreview

Academic literature orientation skill that searches papers via Consensus, builds a strategic search plan using PICO (default) or SPIDER / Decomposition / hybrid as fallbacks, and synthesizes findings into a professionally formatted Word document (.docx) research guide. Grill-me intake (research question specificity + framework hint + tentative depth) before the recon search; a second forcing checkpoint after Phase 2 confirms framework + sub-areas + depth before searches consume budget. Configurable depth (5/10/20 queries) controls coverage vs. speed. Output is a 'launching pad' — not a finished review, but an orientation guide that lets a researcher dive in confidently. Triggers: 'litreview on [topic]', 'literature review on [topic]', 'I'm starting a literature review on X', 'I'm writing a paper on X', 'help me research X', 'I'm doing research on X', 'can you help me research X'. Do NOT trigger for single one-off paper searches where the user just wants a quick list — that's a plain Consensus search.

★ 2