AGENT SKILL

mlperf-inference-benchmarks

暂无简介

License 许可
Apache-2.0
最近更新
2026-11-14
Stars
★ 13
Forks
2

03 SKILL.md 正文

/mlperf-inference-benchmarks

When to Use

  • Measure inference system performance with industry-standard reproducible metrics
  • Compare GPU/CPU hardware choices for a given model/task
  • Validate SLA targets: p99 latency < 100ms at 1000 QPS
  • Pre-deployment stress test: flood the system to find breaking point

Do NOT use for

  • Training benchmarks (use MLPerf Training)
  • Quick sanity checks (use [[llm-eval-framework]])

MLPerf scenarios

SingleStream: one query at a time, measure latency distribution
  Target: p90 latency (e.g., < 10ms for image classification)
  Use: edge devices, latency-sensitive APIs

MultiStream: N queries in parallel, measure all-complete latency
  Target: p99 latency across all N queries
  Use: camera pipelines, fixed batch size hardware

Server: Poisson-distributed query arrival, measure sustained throughput
  Target: max QPS while keeping p99 < SLA threshold
  Use: production cloud inference

Offline: all queries available upfront, measure total throughput
  Target: max queries per second (no latency constraint)
  Use: batch processing, overnight jobs

Custom Loadgen harness (Python)

import mlperf_loadgen as lg

class YamtamSUT:
  """System Under Test — wraps your inference engine"""

  def issue_queries(self, query_samples):
    for qs in query_samples:
      # Run inference
      result = run_inference(qs.index)
      # Report response
      response = lg.QuerySampleResponse(
        id     = qs.id,
        data   = result.ctypes.data,
        size   = result.nbytes,
      )
      lg.QuerySamplesComplete([response])

  def flush_queries(self): pass


class YamtamQSL:
  """Query Sample Library — dataset loader"""

  def __init__(self, dataset, total_count, perf_count):
    self.qsl = lg.ConstructQSL(
      total_count, perf_count,
      self.load_query_samples, self.unload_query_samples
    )
    self.dataset = dataset

  def load_query_samples(self, sample_list):
    for s in sample_list: self.dataset.load(s)

  def unload_query_samples(self, sample_list):
    for s in sample_list: self.dataset.unload(s)


# Run benchmark
settings = lg.TestSettings()
settings.scenario    = lg.TestScenario.Server
settings.mode        = lg.TestMode.PerformanceOnly
settings.server_target_qps = 500

sut = YamtamSUT()
qsl = YamtamQSL(dataset, total_count=5000, perf_count=1024)

lg.StartTest(sut, qsl.qsl, settings)
lg.DestroyQSL(qsl.qsl)

Key metrics to track

# After loadgen run, parse mlperf_log_summary.txt
grep -E "(Samples per second|Min latency|Max latency|Mean latency|p99.00 latency)" \
  mlperf_log_summary.txt

# Target thresholds for yamtam tool calls:
# Mean latency:   < 200ms  (acceptable for interactive agent)
# p99 latency:    < 500ms  (hard SLA)
# Throughput:     > 50 QPS (for 50 concurrent agents)

Quick server scenario stress test (without full MLPerf)

# Use wrk2 for HTTP inference endpoint load testing
wrk2 -t 4 -c 100 -d 60s -R 500 \
  --script post.lua \
  http://inference-server:8000/infer

# post.lua
wrk.method  = "POST"
wrk.headers = {["Content-Type"] = "application/json"}
wrk.body    = '{"input": "benchmark query"}'

Anti-Fake-Pass Checklist

❌ Using accuracy mode results for performance reporting → different sample count, not valid perf number
❌ Warmup not included → first queries hit cold cache; p99 inflated
❌ p50 latency meets SLA but p99 does not → tail latency the real constraint; always report p99
❌ QPS measured on a single sample → not representative; use Server scenario with realistic load
❌ Not isolating benchmark host → background processes inflate latency; dedicate hardware
❌ Comparing results across different MLPerf versions → LoadGen version affects timing methodology

04 同仓库更多 Skills

Horonomos-laps
Horonomos-laps/Yana-AI

openai--sentry--sentry

Use when the user asks to inspect Sentry issues or events, summarize recent production errors, or pull basic Sentry health data via the Sentry API; perform read-only queries with the bundled script and require `SENTRY_AUTH_TOKEN`.

★ 13
Horonomos-laps
Horonomos-laps/Yana-AI

ito-basket-compare

Compare Itô prediction-market baskets against a user's knowledge base, portfolio notes, financial context, watchlist, or research thesis. Use for read-only basket comparison and gap analysis without investment advice or live trading.

★ 13
Horonomos-laps
Horonomos-laps/Yana-AI

implementing-network-traffic-baselining

Build network traffic baselines from NetFlow/IPFIX data using Python pandas for statistical analysis, z-score anomaly detection, and hourly/daily traffic pattern profiling

★ 13
Horonomos-laps
Horonomos-laps/Yana-AI

detecting-evasion-techniques-in-endpoint-logs

Detects defense evasion techniques used by adversaries in endpoint logs including log tampering, timestomping, process injection, and security tool disabling. Use when investigating suspicious endpoint behavior, building detection rules for evasion tactics, or conducting threat hunting for stealthy adversary activity. Activates for requests involving evasion detection, defense evasion analysis, log tampering detection, or MITRE ATT&CK TA0005.

★ 13
Horonomos-laps
Horonomos-laps/Yana-AI

implementing-microsegmentation-with-guardicore

Implementing microsegmentation using Akamai Guardicore Segmentation to map application dependencies, create granular network policies, visualize east-west traffic flows, and enforce least-privilege communication between workloads across data centers and cloud.

★ 13
Horonomos-laps
Horonomos-laps/Yana-AI

ramdisk-tmpfs-patterns

暂无简介

★ 13