[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6C2BN9514NJWDK8W4TYZAV0":3,"skill-same:01M6C2BN9514NJWDK8W4TYZAV0":69},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":26,"publishedAt":37,"linkOnly":38,"markdownContent":39,"markdownBody":40,"frontMatter":41,"refName":45,"commitSha":46,"skillDirectory":47,"forksCount":48,"repositoryHtmlUrl":49,"repositoryHomepageUrl":50,"ownerHtmlUrl":51,"ownerBlogUrl":52,"ownerDescription":53,"updatedAt":54,"install":55,"resources":68},"01M6C2BN9514NJWDK8W4TYZAV0","regression-performance",null,"Detect performance regressions by comparing benchmarks across versions with latency, throughput, and statistical significance analysis","jmagly\u002Faiwg","jmagly","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1159087?v=4","01M6C10AK8N2WCSQN11FYGTJWN","Joseph Magly","01M6C10AQ8A2RMED1Y1BSVY7NF",229,"MIT","https:\u002F\u002Fgithub.com\u002Fjmagly\u002Faiwg\u002Fblob\u002Fmain\u002Fagentic\u002Fcode\u002Fframeworks\u002Fsdlc-complete\u002Fskills\u002Fregression-performance\u002FSKILL.md",[18,23],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000008","测试与质量","SCENARIO","使用场景",{"categoryNo":24,"categoryName":25,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发",[27,28,29,30,31,32,33,34,35,36],"performance","regression","regression performance","analysis","benchmarks","across","latency","detect","throughput","with","2026-11-10 10:12:30",false,"---\nnamespace: aiwg\nname: regression-performance\nplatforms: [all]\ndescription: Detect performance regressions by comparing benchmarks across versions with latency, throughput, and statistical significance analysis\n\n---\n\n# regression-performance\n\nDetect performance regressions by comparing benchmarks across versions, analyzing latency\u002Fthroughput degradation, and providing statistical significance testing.\n\n## Triggers\n\n\nAlternate expressions and non-obvious activations (primary phrases are matched automatically from the skill description):\n\n- \"latency regression\" → performance benchmark comparison\n- \"p99\" \u002F \"p95\" → percentile-based performance metrics\n- \"benchmark diff\" → performance baseline comparison\n\n## Purpose\n\nThis skill detects performance regressions across software versions by:\n- Comparing latency metrics (p50, p95, p99) between baseline and current versions\n- Detecting throughput regressions (requests\u002Fsec, transactions\u002Fsec)\n- Identifying memory regressions (heap growth, memory leaks)\n- Analyzing resource utilization (CPU, disk I\u002FO, network)\n- Running benchmark comparisons with statistical significance testing\n- Generating performance regression reports with visualizations\n\n## Behavior\n\nWhen triggered, this skill:\n\n1. **Identifies baseline version**:\n   - Detect last known good version from git tags\n   - Load baseline benchmark results\n   - Extract performance metrics from monitoring\n\n2. **Runs performance benchmarks**:\n   - Execute load tests using k6, Artillery, or wrk\n   - Capture latency distributions (p50, p95, p99)\n   - Measure throughput (req\u002Fs, TPS)\n   - Profile memory usage and heap growth\n   - Monitor CPU and I\u002FO utilization\n\n3. **Performs statistical comparison**:\n   - Calculate delta and percentage change\n   - Apply statistical significance tests (t-test, Mann-Whitney U)\n   - Determine if degradation exceeds threshold\n   - Account for variance and noise\n\n4. **Detects regression patterns**:\n   - Latency spikes at specific percentiles\n   - Throughput capacity reduction\n   - Memory leak indicators (growing heap)\n   - CPU saturation points\n   - I\u002FO bottlenecks\n\n5. **Generates regression report**:\n   - Performance comparison tables\n   - Percentile distribution graphs\n   - Time-series trend analysis\n   - Root cause indicators\n   - Recommendations\n\n6. **Logs regression findings**:\n   - Create regression register entry\n   - Tag commits with performance impact\n   - Alert on threshold violations\n\n## Performance Metrics Model\n\n```\n┌─────────────────────┐\n│   BASELINE v2.3.0   │\n├─────────────────────┤\n│ p50:  45ms          │\n│ p95: 120ms          │\n│ p99: 180ms          │\n│ RPS: 2500           │\n│ Mem: 256MB          │\n└─────────────────────┘\n         │\n         ▼ Compare\n┌─────────────────────┐\n│   CURRENT v2.4.0    │\n├─────────────────────┤\n│ p50:  52ms (+15%)   │  ⚠️ REGRESSION\n│ p95: 145ms (+21%)   │  ⚠️ REGRESSION\n│ p99: 220ms (+22%)   │  ⚠️ REGRESSION\n│ RPS: 2100 (-16%)    │  ⚠️ REGRESSION\n│ Mem: 312MB (+22%)   │  ⚠️ REGRESSION\n└─────────────────────┘\n         │\n         ▼\n┌─────────────────────┐\n│ REGRESSION REPORT   │\n│                     │\n│ Type: Latency       │\n│ Severity: HIGH      │\n│ Confidence: 99.5%   │\n│ Root Cause: TBD     │\n└─────────────────────┘\n```\n\n## Metric Categories\n\n### Latency Metrics\n\n| Metric | Description | Threshold | Tool |\n|--------|-------------|-----------|------|\n| p50 (median) | 50th percentile latency | +10% | k6, Artillery, wrk |\n| p95 | 95th percentile latency | +15% | k6, Artillery, wrk |\n| p99 | 99th percentile latency | +20% | k6, Artillery, wrk |\n| max | Maximum observed latency | +30% | k6, Artillery, wrk |\n\n### Throughput Metrics\n\n| Metric | Description | Threshold | Tool |\n|--------|-------------|-----------|------|\n| Requests\u002Fsec | HTTP requests per second | -10% | k6, wrk, ab |\n| Transactions\u002Fsec | Business transactions per second | -10% | Custom |\n| Bytes\u002Fsec | Network throughput | -15% | iperf3, iftop |\n| Queries\u002Fsec | Database query throughput | -10% | pgbench, sysbench |\n\n### Memory Metrics\n\n| Metric | Description | Threshold | Tool |\n|--------|-------------|-----------|------|\n| Heap size | JavaScript heap usage | +20% | Node.js heap snapshot |\n| RSS | Resident set size | +20% | ps, top |\n| Memory growth rate | MB\u002Fhour increase | >10 MB\u002Fhour | Continuous profiling |\n| GC pressure | Garbage collection frequency | +30% | Node.js --trace-gc |\n\n### Resource Metrics\n\n| Metric | Description | Threshold | Tool |\n|--------|-------------|-----------|------|\n| CPU utilization | Average CPU usage | +20% | mpstat, top |\n| Disk I\u002FO wait | I\u002FO wait percentage | +25% | iostat |\n| Network bandwidth | Network utilization | +15% | iftop, nethogs |\n| File descriptors | Open file handles | +30% | lsof |\n\n## Benchmark Tools Integration\n\n### k6 Load Testing\n\n```javascript\n\u002F\u002F benchmark.k6.js\nimport http from 'k6\u002Fhttp';\nimport { check, sleep } from 'k6';\n\nexport let options = {\n  stages: [\n    { duration: '2m', target: 100 },   \u002F\u002F Ramp up\n    { duration: '5m', target: 100 },   \u002F\u002F Steady state\n    { duration: '2m', target: 0 },     \u002F\u002F Ramp down\n  ],\n  thresholds: {\n    'http_req_duration': ['p(50)\u003C100', 'p(95)\u003C200', 'p(99)\u003C300'],\n    'http_req_failed': ['rate\u003C0.01'],\n  },\n};\n\nexport default function () {\n  const res = http.get('https:\u002F\u002Fapi.example.com\u002Fendpoint');\n  check(res, {\n    'status is 200': (r) => r.status === 200,\n    'response time \u003C 200ms': (r) => r.timings.duration \u003C 200,\n  });\n  sleep(1);\n}\n```\n\n**Running comparison**:\n```bash\n# Baseline\nk6 run --out json=baseline-results.json benchmark.k6.js\n\n# Current version\nk6 run --out json=current-results.json benchmark.k6.js\n\n# Compare\n.\u002Fcompare-k6-results.sh baseline-results.json current-results.json\n```\n\n### Artillery Load Testing\n\n```yaml\n# artillery-config.yml\nconfig:\n  target: 'https:\u002F\u002Fapi.example.com'\n  phases:\n    - duration: 120\n      arrivalRate: 10\n      rampTo: 50\n    - duration: 300\n      arrivalRate: 50\n    - duration: 120\n      arrivalRate: 50\n      rampTo: 0\n  plugins:\n    metrics-by-endpoint:\n      stripQueryString: true\n\nscenarios:\n  - name: \"API Performance Test\"\n    flow:\n      - get:\n          url: \"\u002Fapi\u002Fusers\"\n      - get:\n          url: \"\u002Fapi\u002Fproducts\"\n      - post:\n          url: \"\u002Fapi\u002Forders\"\n          json:\n            product_id: 123\n            quantity: 2\n```\n\n**Running comparison**:\n```bash\n# Baseline\nartillery run --output baseline.json artillery-config.yml\n\n# Current\nartillery run --output current.json artillery-config.yml\n\n# Compare\nartillery report baseline.json --output baseline-report.html\nartillery report current.json --output current-report.html\n.\u002Fcompare-artillery-results.sh baseline.json current.json\n```\n\n### wrk HTTP Benchmarking\n\n```bash\n# Simple throughput test\nwrk_benchmark() {\n  local version=$1\n  local output_file=$2\n\n  wrk -t12 -c400 -d30s \\\n      --latency \\\n      --timeout 10s \\\n      https:\u002F\u002Fapi.example.com\u002Fendpoint \\\n      > \"$output_file\"\n}\n\n# Baseline\nwrk_benchmark \"v2.3.0\" \"wrk-baseline.txt\"\n\n# Current\nwrk_benchmark \"v2.4.0\" \"wrk-current.txt\"\n\n# Compare\n.\u002Fparse-wrk-results.sh wrk-baseline.txt wrk-current.txt\n```\n\n### Apache Bench (ab)\n\n```bash\n# Quick regression check\nab_compare() {\n  local baseline_version=$1\n  local current_version=$2\n\n  echo \"=== Baseline ${baseline_version} ===\"\n  ab -n 10000 -c 100 https:\u002F\u002Fapi.example.com\u002F > ab-baseline.txt\n\n  echo \"=== Current ${current_version} ===\"\n  ab -n 10000 -c 100 https:\u002F\u002Fapi.example.com\u002F > ab-current.txt\n\n  # Extract key metrics\n  echo \"Comparison:\"\n  echo \"Baseline RPS: $(grep 'Requests per second' ab-baseline.txt | awk '{print $4}')\"\n  echo \"Current RPS: $(grep 'Requests per second' ab-current.txt | awk '{print $4}')\"\n}\n```\n\n### Hyperfine (CLI tool benchmarking)\n\n```bash\n# For CLI tool performance\nhyperfine \\\n  --warmup 3 \\\n  --export-json comparison.json \\\n  'git checkout v2.3.0 && npm run build && npm test' \\\n  'git checkout v2.4.0 && npm run build && npm test'\n```\n\n## Statistical Significance Testing\n\n### T-Test for Mean Comparison\n\n```typescript\nfunction detectLatencyRegression(\n  baseline: number[],\n  current: number[]\n): RegressionAnalysis {\n  const baselineMean = mean(baseline);\n  const currentMean = mean(current);\n  const delta = currentMean - baselineMean;\n  const deltaPercent = (delta \u002F baselineMean) * 100;\n\n  \u002F\u002F Perform Welch's t-test\n  const tTestResult = welchTTest(baseline, current);\n\n  return {\n    metric: 'latency',\n    baseline: {\n      mean: baselineMean,\n      stddev: stddev(baseline),\n      n: baseline.length,\n    },\n    current: {\n      mean: currentMean,\n      stddev: stddev(current),\n      n: current.length,\n    },\n    delta,\n    deltaPercent,\n    pValue: tTestResult.pValue,\n    isSignificant: tTestResult.pValue \u003C 0.05,\n    isRegression: deltaPercent > 10 && tTestResult.pValue \u003C 0.05,\n    confidence: 1 - tTestResult.pValue,\n  };\n}\n```\n\n### Mann-Whitney U Test (Non-Parametric)\n\n```typescript\nfunction detectThroughputRegression(\n  baseline: number[],\n  current: number[]\n): RegressionAnalysis {\n  const baselineMedian = median(baseline);\n  const currentMedian = median(current);\n  const delta = currentMedian - baselineMedian;\n  const deltaPercent = (delta \u002F baselineMedian) * 100;\n\n  \u002F\u002F Use Mann-Whitney U for non-normal distributions\n  const uTestResult = mannWhitneyU(baseline, current);\n\n  return {\n    metric: 'throughput',\n    baseline: {\n      median: baselineMedian,\n      iqr: iqr(baseline),\n      n: baseline.length,\n    },\n    current: {\n      median: currentMedian,\n      iqr: iqr(current),\n      n: current.length,\n    },\n    delta,\n    deltaPercent,\n    pValue: uTestResult.pValue,\n    isSignificant: uTestResult.pValue \u003C 0.05,\n    isRegression: deltaPercent \u003C -10 && uTestResult.pValue \u003C 0.05,\n    confidence: 1 - uTestResult.pValue,\n  };\n}\n```\n\n### Multiple Comparison Correction\n\n```typescript\n\u002F\u002F Apply Bonferroni correction when testing multiple metrics\nfunction detectMultiMetricRegressions(\n  metrics: MetricComparison[],\n  familyAlpha: number = 0.05\n): RegressionResult[] {\n  const adjustedAlpha = familyAlpha \u002F metrics.length;\n\n  return metrics.map(metric => ({\n    ...metric,\n    adjustedAlpha,\n    isSignificantCorrected: metric.pValue \u003C adjustedAlpha,\n  }));\n}\n```\n\n## Threshold Configuration\n\nConfigure regression detection thresholds in `.aiwg\u002Fconfig\u002Fperformance-thresholds.yaml`:\n\n```yaml\nperformance_thresholds:\n  latency:\n    p50:\n      warning: 10%    # Warn if p50 increases by >10%\n      critical: 20%   # Critical if p50 increases by >20%\n    p95:\n      warning: 15%\n      critical: 25%\n    p99:\n      warning: 20%\n      critical: 30%\n\n  throughput:\n    requests_per_sec:\n      warning: -10%   # Warn if RPS decreases by >10%\n      critical: -20%\n    transactions_per_sec:\n      warning: -10%\n      critical: -20%\n\n  memory:\n    heap_size:\n      warning: 20%\n      critical: 40%\n    growth_rate:\n      warning: 10      # MB\u002Fhour\n      critical: 25\n\n  resources:\n    cpu_utilization:\n      warning: 20%\n      critical: 40%\n    io_wait:\n      warning: 25%\n      critical: 50%\n\n  statistical:\n    significance_level: 0.05  # p-value threshold\n    min_sample_size: 100      # Minimum observations\n    multiple_comparison_correction: bonferroni\n```\n\n## Memory Leak Detection\n\n### Heap Snapshot Comparison\n\n```typescript\ninterface HeapAnalysis {\n  version: string;\n  timestamp: string;\n  totalSize: number;\n  usedSize: number;\n  objectCounts: Record\u003Cstring, number>;\n  retainedPaths: RetainedPath[];\n}\n\nfunction detectMemoryLeak(\n  baseline: HeapAnalysis,\n  current: HeapAnalysis\n): MemoryLeakReport {\n  const growthRate = (current.usedSize - baseline.usedSize) \u002F\n                     (current.timestamp - baseline.timestamp);\n\n  const suspectObjects = Object.keys(current.objectCounts)\n    .filter(type => {\n      const baseCount = baseline.objectCounts[type] || 0;\n      const currCount = current.objectCounts[type];\n      const growth = currCount - baseCount;\n      return growth > 1000; \u002F\u002F >1000 additional objects\n    })\n    .map(type => ({\n      type,\n      baseline: baseline.objectCounts[type] || 0,\n      current: current.objectCounts[type],\n      delta: current.objectCounts[type] - (baseline.objectCounts[type] || 0),\n    }));\n\n  return {\n    isLeak: growthRate > 10 * 1024 * 1024, \u002F\u002F >10MB\u002Fhour\n    growthRate,\n    suspectObjects,\n    recommendation: suspectObjects.length > 0\n      ? `Investigate ${suspectObjects[0].type} accumulation`\n      : 'Run extended profiling session',\n  };\n}\n```\n\n### Continuous Memory Monitoring\n\n```bash\n#!\u002Fbin\u002Fbash\n# monitor-memory-regression.sh\n\nmonitor_memory() {\n  local duration_minutes=$1\n  local interval_seconds=$2\n  local output_file=$3\n\n  echo \"timestamp,rss,heap_used,heap_total,external\" > \"$output_file\"\n\n  for i in $(seq 1 $((duration_minutes * 60 \u002F interval_seconds))); do\n    local stats=$(curl -s http:\u002F\u002Flocalhost:3000\u002Fmetrics\u002Fmemory)\n    echo \"$(date +%s),$stats\" >> \"$output_file\"\n    sleep \"$interval_seconds\"\n  done\n}\n\n# Run for 30 minutes, sample every 10 seconds\nmonitor_memory 30 10 memory-profile.csv\n\n# Analyze for leaks\npython analyze-memory-trend.py memory-profile.csv\n```\n\n## Performance Regression Report Format\n\n### Executive Summary\n\n```markdown\n# Performance Regression Report\n\n**Project**: my-api-service\n**Analysis Date**: 2024-01-25\n**Baseline Version**: v2.3.0\n**Current Version**: v2.4.0\n**Test Duration**: 10 minutes\n**Load Profile**: 100 concurrent users, steady state\n\n## Executive Summary\n\n**Regression Detected**: YES (HIGH severity)\n**Metrics Affected**: 5 of 8 metrics show significant degradation\n**Statistical Confidence**: 99.5%\n**Recommendation**: DO NOT DEPLOY - requires investigation\n\n| Category | Status | Details |\n|----------|--------|---------|\n| Latency | ⚠️ REGRESSION | p50: +15%, p95: +21%, p99: +22% |\n| Throughput | ⚠️ REGRESSION | -16% requests\u002Fsec |\n| Memory | ⚠️ REGRESSION | +22% heap, potential leak |\n| CPU | ✅ PASS | Within acceptable range |\n| I\u002FO | ✅ PASS | No significant change |\n```\n\n### Detailed Metrics Comparison\n\n```markdown\n## Latency Metrics\n\n| Metric | Baseline | Current | Delta | % Change | p-value | Regression? |\n|--------|----------|---------|-------|----------|---------|-------------|\n| p50 | 45ms | 52ms | +7ms | +15.6% | 0.001 | ⚠️ YES |\n| p75 | 78ms | 92ms | +14ms | +17.9% | 0.002 | ⚠️ YES |\n| p90 | 105ms | 128ms | +23ms | +21.9% | 0.001 | ⚠️ YES |\n| p95 | 120ms | 145ms | +25ms | +20.8% | 0.001 | ⚠️ YES |\n| p99 | 180ms | 220ms | +40ms | +22.2% | 0.001 | ⚠️ YES |\n| max | 450ms | 580ms | +130ms | +28.9% | 0.023 | ⚠️ YES |\n\n**Statistical Test**: Welch's t-test\n**Significance Level**: α = 0.05 (Bonferroni corrected: 0.0083)\n**Confidence**: 99.9% that degradation is real\n\n### Latency Distribution\n\n```\nBaseline v2.3.0              Current v2.4.0\n     │                            │\n 600 │                            │              *\n     │                            │\n 500 │                            │            *\n     │                            │          *\n 400 │                            │        *\n     │                            │\n 300 │                            │      **\n     │                            │     *\n 200 │                *           │   **\n     │              **            │  **\n 100 │          ****              │ **\n     │     *****                  │**\n   0 └─────────────────          └─────────────────\n     0  25  50  75  95 99         0  25  50  75  95 99\n            Percentile                    Percentile\n```\n```\n\n### Throughput Regression\n\n```markdown\n## Throughput Metrics\n\n| Metric | Baseline | Current | Delta | % Change | p-value | Regression? |\n|--------|----------|---------|-------|----------|---------|-------------|\n| Requests\u002Fsec | 2500 | 2100 | -400 | -16.0% | 0.003 | ⚠️ YES |\n| Total requests | 150000 | 126000 | -24000 | -16.0% | - | - |\n| Failed requests | 150 (0.1%) | 1260 (1.0%) | +1110 | +740% | 0.001 | ⚠️ YES |\n\n**Statistical Test**: Mann-Whitney U test\n**Confidence**: 99.7% that throughput decreased\n\n### Time Series Comparison\n\n```\nRequests per Second Over Time\n\n3000 ┤\n     │  Baseline ─────\n2500 │ ████████████████████████████████████\n     │\n2000 │           Current ─ ─ ─ ─\n     │          ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌\n1500 │\n     │\n1000 │\n     │\n 500 │\n     │\n   0 └────────────────────────────────────────\n     0   2m  4m  6m  8m  10m\n              Time\n```\n```\n\n### Memory Analysis\n\n```markdown\n## Memory Metrics\n\n| Metric | Baseline | Current | Delta | % Change | Leak Detected? |\n|--------|----------|---------|-------|----------|----------------|\n| Heap Used | 256 MB | 312 MB | +56 MB | +21.9% | ⚠️ Suspected |\n| RSS | 384 MB | 468 MB | +84 MB | +21.9% | - |\n| Growth Rate | 2 MB\u002Fhour | 18 MB\u002Fhour | +16 MB\u002Fhour | +800% | ⚠️ YES |\n| GC Frequency | 12\u002Fmin | 18\u002Fmin | +6\u002Fmin | +50% | ⚠️ Pressure |\n\n**Memory Leak Analysis**:\n- **Leak Detected**: YES (high confidence)\n- **Growth Rate**: 18 MB\u002Fhour (threshold: 10 MB\u002Fhour)\n- **Suspect Objects**: Array (+12500 instances), Closure (+8200 instances)\n- **Recommendation**: Take heap snapshot and analyze retained paths\n\n### Heap Growth Over Time\n\n```\nHeap Size (MB)\n\n400 ┤                                    Current ╱╱\n    │                               ╱╱╱╱╱\n350 │                          ╱╱╱╱╱\n    │                     ╱╱╱╱╱\n300 │                ╱╱╱╱╱\n    │           ╱╱╱╱╱\n250 │      ╱╱╱╱╱              Baseline ────\n    │ ╱╱╱╱╱        ──────────────────────────\n200 │\n    │\n150 │\n    │\n100 │\n    └────────────────────────────────────────\n      0   10  20  30  40  50  60\n               Time (minutes)\n```\n```\n\n### Root Cause Indicators\n\n```markdown\n## Root Cause Analysis\n\n**Automated Analysis**:\n\n### Likely Culprits\n\n1. **Synchronous I\u002FO in Request Path** (HIGH confidence)\n   - Evidence: p99 latency +22%, I\u002FO wait +15%\n   - Pattern: Blocking operations correlate with latency spikes\n   - Recommendation: Profile with `clinic doctor` or `0x`\n\n2. **Memory Accumulation** (HIGH confidence)\n   - Evidence: Linear heap growth at 18 MB\u002Fhour\n   - Pattern: Array and Closure object growth\n   - Recommendation: Take heap snapshots at t=0 and t=30min, analyze diff\n\n3. **Increased Computation** (MEDIUM confidence)\n   - Evidence: CPU +12%, throughput -16%\n   - Pattern: Less throughput despite similar CPU usage\n   - Recommendation: Profile with `perf` or `clinic flame`\n\n### Recommended Investigation Steps\n\n1. **Git Bisect**: Identify introducing commit\n   ```bash\n   git bisect start HEAD v2.3.0\n   git bisect run .\u002Fperformance-test.sh\n   ```\n\n2. **Heap Snapshot Diff**:\n   ```bash\n   node --heap-prof app.js  # Baseline\n   # Run load test\n   node --heap-prof app.js  # Current\n   # Compare snapshots\n   ```\n\n3. **Flame Graph Analysis**:\n   ```bash\n   clinic flame -- node app.js\n   # Analyze hot paths\n   ```\n```\n\n### Recommendations\n\n```markdown\n## Recommendations\n\n### Immediate Actions (DO NOT DEPLOY)\n\n- [ ] **Do not deploy v2.4.0** - Severity is HIGH\n- [ ] Run git bisect to identify introducing commit\n- [ ] Take heap snapshots and analyze memory accumulation\n- [ ] Profile CPU with flame graphs to identify hot paths\n\n### Investigation Priority\n\n| Priority | Action | Owner | ETA |\n|----------|--------|-------|-----|\n| P0 | Identify regression commit via bisect | Regression Analyst | 2 hours |\n| P0 | Analyze heap snapshots for leak | Performance Engineer | 4 hours |\n| P1 | Profile CPU for hot paths | Performance Engineer | 4 hours |\n| P1 | Review recent I\u002FO changes | Developer | 2 hours |\n\n### Prevention (After Fix)\n\n- [ ] Add performance regression tests to CI\n- [ ] Set up continuous memory profiling\n- [ ] Enable automated alerts for p99 > 200ms\n- [ ] Add load testing to pull request checks\n```\n\n## Usage Examples\n\n### Full Performance Comparison\n\n```\nUser: \"Performance regression check between v2.3.0 and v2.4.0\"\n\nSkill executes:\n1. Checkout v2.3.0, run k6 benchmark → baseline-results.json\n2. Checkout v2.4.0, run k6 benchmark → current-results.json\n3. Statistical comparison → 5 of 8 metrics regressed\n4. Memory profiling → leak detected (+18 MB\u002Fhour)\n5. Generate report → .aiwg\u002Freports\u002Fperformance-regression-20240125.md\n\nOutput:\n\"⚠️ PERFORMANCE REGRESSION DETECTED\n\nSeverity: HIGH\nConfidence: 99.5%\nMetrics Affected: Latency (+15-22%), Throughput (-16%), Memory Leak\n\nReport: .aiwg\u002Freports\u002Fperformance-regression-20240125.md\n\nDO NOT DEPLOY - Investigation required\"\n```\n\n### Quick Latency Check\n\n```\nUser: \"Check latency regression\"\n\nSkill executes:\n1. Compare current p99 against baseline\n2. Statistical t-test for significance\n3. Quick report\n\nOutput:\n\"Latency Regression Detected:\n  p50: 52ms (baseline: 45ms) → +15.6% ⚠️\n  p95: 145ms (baseline: 120ms) → +20.8% ⚠️\n  p99: 220ms (baseline: 180ms) → +22.2% ⚠️\n\nStatistical confidence: 99.9%\nExceeds threshold: YES (>10%)\nRecommendation: Investigate before deploying\"\n```\n\n### Memory Leak Detection\n\n```\nUser: \"Detect memory leak\"\n\nSkill executes:\n1. Monitor memory for 30 minutes\n2. Calculate growth rate\n3. Identify accumulating objects\n\nOutput:\n\"⚠️ MEMORY LEAK DETECTED\n\nGrowth Rate: 18 MB\u002Fhour (threshold: 10 MB\u002Fhour)\nSuspect Objects:\n  - Array: +12500 instances\n  - Closure: +8200 instances\n\nHeap snapshots saved:\n  - .aiwg\u002Fprofiling\u002Fheap-t0.heapsnapshot\n  - .aiwg\u002Fprofiling\u002Fheap-t30.heapsnapshot\n\nRecommendation: Analyze with Chrome DevTools Memory Profiler\"\n```\n\n### Benchmark Comparison\n\n```\nUser: \"Compare performance with main branch\"\n\nSkill executes:\n1. Stash current changes\n2. Benchmark main branch\n3. Restore changes\n4. Benchmark current branch\n5. Statistical comparison\n\nOutput:\n\"Performance Comparison: feature-branch vs main\n\nLatency (p99):\n  main: 180ms\n  feature-branch: 185ms\n  Delta: +2.8% (within threshold ✓)\n\nThroughput:\n  main: 2500 req\u002Fs\n  feature-branch: 2480 req\u002Fs\n  Delta: -0.8% (within threshold ✓)\n\nVerdict: NO REGRESSION DETECTED\nSafe to merge\"\n```\n\n## Integration\n\nThis skill uses:\n- `artifact-metadata`: Load benchmark results from previous runs\n- `project-awareness`: Detect git tags and version history\n- `regression-analyst` agent: For detailed root cause analysis when regression detected\n\nThis skill is used by:\n- CI\u002FCD pipelines for automated regression gates\n- Developers for pre-merge performance validation\n- Release managers for go\u002Fno-go decisions\n\n## Automated Detection in CI\n\n### GitHub Actions Example\n\n```yaml\n# .github\u002Fworkflows\u002Fperformance-gate.yml\nname: Performance Regression Gate\n\non:\n  pull_request:\n    branches: [main]\n\njobs:\n  performance-gate:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions\u002Fcheckout@v3\n        with:\n          fetch-depth: 0  # Full history for baseline\n\n      - name: Setup Node.js\n        uses: actions\u002Fsetup-node@v3\n        with:\n          node-version: '20'\n\n      - name: Install dependencies\n        run: npm ci\n\n      - name: Benchmark baseline (main branch)\n        run: |\n          git checkout main\n          npm run build\n          k6 run --out json=baseline.json benchmark.k6.js\n\n      - name: Benchmark current (PR branch)\n        run: |\n          git checkout ${{ github.head_ref }}\n          npm run build\n          k6 run --out json=current.json benchmark.k6.js\n\n      - name: Performance regression analysis\n        id: regression\n        run: |\n          npm run analyze:performance -- \\\n            --baseline baseline.json \\\n            --current current.json \\\n            --threshold 10 \\\n            --output regression-report.md\n\n      - name: Comment PR with results\n        uses: actions\u002Fgithub-script@v6\n        with:\n          script: |\n            const fs = require('fs');\n            const report = fs.readFileSync('regression-report.md', 'utf8');\n            github.rest.issues.createComment({\n              issue_number: context.issue.number,\n              owner: context.repo.owner,\n              repo: context.repo.repo,\n              body: report\n            });\n\n      - name: Fail if regression detected\n        if: steps.regression.outputs.regression == 'true'\n        run: |\n          echo \"Performance regression detected - see report\"\n          exit 1\n```\n\n## Output Locations\n\n- Performance reports: `.aiwg\u002Freports\u002Fperformance-regression-{date}.md`\n- Benchmark data: `.aiwg\u002Fbenchmarks\u002F{version}-{timestamp}.json`\n- Heap snapshots: `.aiwg\u002Fprofiling\u002Fheap-{version}-{timestamp}.heapsnapshot`\n- Flame graphs: `.aiwg\u002Fprofiling\u002Fflame-{version}-{timestamp}.html`\n- Regression register: `.aiwg\u002Ftesting\u002Fregression-register\u002FREG-PERF-{id}.yaml`\n\n## References\n\n- @$AIWG_ROOT\u002Fagentic\u002Fcode\u002Fframeworks\u002Fsdlc-complete\u002Fagents\u002Fregression-analyst.md - Regression analysis agent\n- @$AIWG_ROOT\u002Fagentic\u002Fcode\u002Fframeworks\u002Fsdlc-complete\u002Fagents\u002Fperformance-engineer.md - Performance optimization agent\n- @$AIWG_ROOT\u002Fagentic\u002Fcode\u002Fframeworks\u002Fsdlc-complete\u002Fschemas\u002Ftesting\u002Fregression.yaml - Regression schema\n- @.aiwg\u002Fresearch\u002Ffindings\u002FREF-013-metagpt.md - Debug memory pattern for performance history\n- @$AIWG_ROOT\u002Fagentic\u002Fcode\u002Fframeworks\u002Fsdlc-complete\u002Frules\u002Fexecutable-feedback.md - Execution validation requirements\n","# regression-performance\n\nDetect performance regressions by comparing benchmarks across versions, analyzing latency\u002Fthroughput degradation, and providing statistical significance testing.\n\n## Triggers\n\n\nAlternate expressions and non-obvious activations (primary phrases are matched automatically from the skill description):\n\n- \"latency regression\" → performance benchmark comparison\n- \"p99\" \u002F \"p95\" → percentile-based performance metrics\n- \"benchmark diff\" → performance baseline comparison\n\n## Purpose\n\nThis skill detects performance regressions across software versions by:\n- Comparing latency metrics (p50, p95, p99) between baseline and current versions\n- Detecting throughput regressions (requests\u002Fsec, transactions\u002Fsec)\n- Identifying memory regressions (heap growth, memory leaks)\n- Analyzing resource utilization (CPU, disk I\u002FO, network)\n- Running benchmark comparisons with statistical significance testing\n- Generating performance regression reports with visualizations\n\n## Behavior\n\nWhen triggered, this skill:\n\n1. **Identifies baseline version**:\n   - Detect last known good version from git tags\n   - Load baseline benchmark results\n   - Extract performance metrics from monitoring\n\n2. **Runs performance benchmarks**:\n   - Execute load tests using k6, Artillery, or wrk\n   - Capture latency distributions (p50, p95, p99)\n   - Measure throughput (req\u002Fs, TPS)\n   - Profile memory usage and heap growth\n   - Monitor CPU and I\u002FO utilization\n\n3. **Performs statistical comparison**:\n   - Calculate delta and percentage change\n   - Apply statistical significance tests (t-test, Mann-Whitney U)\n   - Determine if degradation exceeds threshold\n   - Account for variance and noise\n\n4. **Detects regression patterns**:\n   - Latency spikes at specific percentiles\n   - Throughput capacity reduction\n   - Memory leak indicators (growing heap)\n   - CPU saturation points\n   - I\u002FO bottlenecks\n\n5. **Generates regression report**:\n   - Performance comparison tables\n   - Percentile distribution graphs\n   - Time-series trend analysis\n   - Root cause indicators\n   - Recommendations\n\n6. **Logs regression findings**:\n   - Create regression register entry\n   - Tag commits with performance impact\n   - Alert on threshold violations\n\n## Performance Metrics Model\n\n```\n┌─────────────────────┐\n│   BASELINE v2.3.0   │\n├─────────────────────┤\n│ p50:  45ms          │\n│ p95: 120ms          │\n│ p99: 180ms          │\n│ RPS: 2500           │\n│ Mem: 256MB          │\n└─────────────────────┘\n         │\n         ▼ Compare\n┌─────────────────────┐\n│   CURRENT v2.4.0    │\n├─────────────────────┤\n│ p50:  52ms (+15%)   │  ⚠️ REGRESSION\n│ p95: 145ms (+21%)   │  ⚠️ REGRESSION\n│ p99: 220ms (+22%)   │  ⚠️ REGRESSION\n│ RPS: 2100 (-16%)    │  ⚠️ REGRESSION\n│ Mem: 312MB (+22%)   │  ⚠️ REGRESSION\n└─────────────────────┘\n         │\n         ▼\n┌─────────────────────┐\n│ REGRESSION REPORT   │\n│                     │\n│ Type: Latency       │\n│ Severity: HIGH      │\n│ Confidence: 99.5%   │\n│ Root Cause: TBD     │\n└─────────────────────┘\n```\n\n## Metric Categories\n\n### Latency Metrics\n\n| Metric | Description | Threshold | Tool |\n|--------|-------------|-----------|------|\n| p50 (median) | 50th percentile latency | +10% | k6, Artillery, wrk |\n| p95 | 95th percentile latency | +15% | k6, Artillery, wrk |\n| p99 | 99th percentile latency | +20% | k6, Artillery, wrk |\n| max | Maximum observed latency | +30% | k6, Artillery, wrk |\n\n### Throughput Metrics\n\n| Metric | Description | Threshold | Tool |\n|--------|-------------|-----------|------|\n| Requests\u002Fsec | HTTP requests per second | -10% | k6, wrk, ab |\n| Transactions\u002Fsec | Business transactions per second | -10% | Custom |\n| Bytes\u002Fsec | Network throughput | -15% | iperf3, iftop |\n| Queries\u002Fsec | Database query throughput | -10% | pgbench, sysbench |\n\n### Memory Metrics\n\n| Metric | Description | Threshold | Tool |\n|--------|-------------|-----------|------|\n| Heap size | JavaScript heap usage | +20% | Node.js heap snapshot |\n| RSS | Resident set size | +20% | ps, top |\n| Memory growth rate | MB\u002Fhour increase | >10 MB\u002Fhour | Continuous profiling |\n| GC pressure | Garbage collection frequency | +30% | Node.js --trace-gc |\n\n### Resource Metrics\n\n| Metric | Description | Threshold | Tool |\n|--------|-------------|-----------|------|\n| CPU utilization | Average CPU usage | +20% | mpstat, top |\n| Disk I\u002FO wait | I\u002FO wait percentage | +25% | iostat |\n| Network bandwidth | Network utilization | +15% | iftop, nethogs |\n| File descriptors | Open file handles | +30% | lsof |\n\n## Benchmark Tools Integration\n\n### k6 Load Testing\n\n```javascript\n\u002F\u002F benchmark.k6.js\nimport http from 'k6\u002Fhttp';\nimport { check, sleep } from 'k6';\n\nexport let options = {\n  stages: [\n    { duration: '2m', target: 100 },   \u002F\u002F Ramp up\n    { duration: '5m', target: 100 },   \u002F\u002F Steady state\n    { duration: '2m', target: 0 },     \u002F\u002F Ramp down\n  ],\n  thresholds: {\n    'http_req_duration': ['p(50)\u003C100', 'p(95)\u003C200', 'p(99)\u003C300'],\n    'http_req_failed': ['rate\u003C0.01'],\n  },\n};\n\nexport default function () {\n  const res = http.get('https:\u002F\u002Fapi.example.com\u002Fendpoint');\n  check(res, {\n    'status is 200': (r) => r.status === 200,\n    'response time \u003C 200ms': (r) => r.timings.duration \u003C 200,\n  });\n  sleep(1);\n}\n```\n\n**Running comparison**:\n```bash\n# Baseline\nk6 run --out json=baseline-results.json benchmark.k6.js\n\n# Current version\nk6 run --out json=current-results.json benchmark.k6.js\n\n# Compare\n.\u002Fcompare-k6-results.sh baseline-results.json current-results.json\n```\n\n### Artillery Load Testing\n\n```yaml\n# artillery-config.yml\nconfig:\n  target: 'https:\u002F\u002Fapi.example.com'\n  phases:\n    - duration: 120\n      arrivalRate: 10\n      rampTo: 50\n    - duration: 300\n      arrivalRate: 50\n    - duration: 120\n      arrivalRate: 50\n      rampTo: 0\n  plugins:\n    metrics-by-endpoint:\n      stripQueryString: true\n\nscenarios:\n  - name: \"API Performance Test\"\n    flow:\n      - get:\n          url: \"\u002Fapi\u002Fusers\"\n      - get:\n          url: \"\u002Fapi\u002Fproducts\"\n      - post:\n          url: \"\u002Fapi\u002Forders\"\n          json:\n            product_id: 123\n            quantity: 2\n```\n\n**Running comparison**:\n```bash\n# Baseline\nartillery run --output baseline.json artillery-config.yml\n\n# Current\nartillery run --output current.json artillery-config.yml\n\n# Compare\nartillery report baseline.json --output baseline-report.html\nartillery report current.json --output current-report.html\n.\u002Fcompare-artillery-results.sh baseline.json current.json\n```\n\n### wrk HTTP Benchmarking\n\n```bash\n# Simple throughput test\nwrk_benchmark() {\n  local version=$1\n  local output_file=$2\n\n  wrk -t12 -c400 -d30s \\\n      --latency \\\n      --timeout 10s \\\n      https:\u002F\u002Fapi.example.com\u002Fendpoint \\\n      > \"$output_file\"\n}\n\n# Baseline\nwrk_benchmark \"v2.3.0\" \"wrk-baseline.txt\"\n\n# Current\nwrk_benchmark \"v2.4.0\" \"wrk-current.txt\"\n\n# Compare\n.\u002Fparse-wrk-results.sh wrk-baseline.txt wrk-current.txt\n```\n\n### Apache Bench (ab)\n\n```bash\n# Quick regression check\nab_compare() {\n  local baseline_version=$1\n  local current_version=$2\n\n  echo \"=== Baseline ${baseline_version} ===\"\n  ab -n 10000 -c 100 https:\u002F\u002Fapi.example.com\u002F > ab-baseline.txt\n\n  echo \"=== Current ${current_version} ===\"\n  ab -n 10000 -c 100 https:\u002F\u002Fapi.example.com\u002F > ab-current.txt\n\n  # Extract key metrics\n  echo \"Comparison:\"\n  echo \"Baseline RPS: $(grep 'Requests per second' ab-baseline.txt | awk '{print $4}')\"\n  echo \"Current RPS: $(grep 'Requests per second' ab-current.txt | awk '{print $4}')\"\n}\n```\n\n### Hyperfine (CLI tool benchmarking)\n\n```bash\n# For CLI tool performance\nhyperfine \\\n  --warmup 3 \\\n  --export-json comparison.json \\\n  'git checkout v2.3.0 && npm run build && npm test' \\\n  'git checkout v2.4.0 && npm run build && npm test'\n```\n\n## Statistical Significance Testing\n\n### T-Test for Mean Comparison\n\n```typescript\nfunction detectLatencyRegression(\n  baseline: number[],\n  current: number[]\n): RegressionAnalysis {\n  const baselineMean = mean(baseline);\n  const currentMean = mean(current);\n  const delta = currentMean - baselineMean;\n  const deltaPercent = (delta \u002F baselineMean) * 100;\n\n  \u002F\u002F Perform Welch's t-test\n  const tTestResult = welchTTest(baseline, current);\n\n  return {\n    metric: 'latency',\n    baseline: {\n      mean: baselineMean,\n      stddev: stddev(baseline),\n      n: baseline.length,\n    },\n    current: {\n      mean: currentMean,\n      stddev: stddev(current),\n      n: current.length,\n    },\n    delta,\n    deltaPercent,\n    pValue: tTestResult.pValue,\n    isSignificant: tTestResult.pValue \u003C 0.05,\n    isRegression: deltaPercent > 10 && tTestResult.pValue \u003C 0.05,\n    confidence: 1 - tTestResult.pValue,\n  };\n}\n```\n\n### Mann-Whitney U Test (Non-Parametric)\n\n```typescript\nfunction detectThroughputRegression(\n  baseline: number[],\n  current: number[]\n): RegressionAnalysis {\n  const baselineMedian = median(baseline);\n  const currentMedian = median(current);\n  const delta = currentMedian - baselineMedian;\n  const deltaPercent = (delta \u002F baselineMedian) * 100;\n\n  \u002F\u002F Use Mann-Whitney U for non-normal distributions\n  const uTestResult = mannWhitneyU(baseline, current);\n\n  return {\n    metric: 'throughput',\n    baseline: {\n      median: baselineMedian,\n      iqr: iqr(baseline),\n      n: baseline.length,\n    },\n    current: {\n      median: currentMedian,\n      iqr: iqr(current),\n      n: current.length,\n    },\n    delta,\n    deltaPercent,\n    pValue: uTestResult.pValue,\n    isSignificant: uTestResult.pValue \u003C 0.05,\n    isRegression: deltaPercent \u003C -10 && uTestResult.pValue \u003C 0.05,\n    confidence: 1 - uTestResult.pValue,\n  };\n}\n```\n\n### Multiple Comparison Correction\n\n```typescript\n\u002F\u002F Apply Bonferroni correction when testing multiple metrics\nfunction detectMultiMetricRegressions(\n  metrics: MetricComparison[],\n  familyAlpha: number = 0.05\n): RegressionResult[] {\n  const adjustedAlpha = familyAlpha \u002F metrics.length;\n\n  return metrics.map(metric => ({\n    ...metric,\n    adjustedAlpha,\n    isSignificantCorrected: metric.pValue \u003C adjustedAlpha,\n  }));\n}\n```\n\n## Threshold Configuration\n\nConfigure regression detection thresholds in `.aiwg\u002Fconfig\u002Fperformance-thresholds.yaml`:\n\n```yaml\nperformance_thresholds:\n  latency:\n    p50:\n      warning: 10%    # Warn if p50 increases by >10%\n      critical: 20%   # Critical if p50 increases by >20%\n    p95:\n      warning: 15%\n      critical: 25%\n    p99:\n      warning: 20%\n      critical: 30%\n\n  throughput:\n    requests_per_sec:\n      warning: -10%   # Warn if RPS decreases by >10%\n      critical: -20%\n    transactions_per_sec:\n      warning: -10%\n      critical: -20%\n\n  memory:\n    heap_size:\n      warning: 20%\n      critical: 40%\n    growth_rate:\n      warning: 10      # MB\u002Fhour\n      critical: 25\n\n  resources:\n    cpu_utilization:\n      warning: 20%\n      critical: 40%\n    io_wait:\n      warning: 25%\n      critical: 50%\n\n  statistical:\n    significance_level: 0.05  # p-value threshold\n    min_sample_size: 100      # Minimum observations\n    multiple_comparison_correction: bonferroni\n```\n\n## Memory Leak Detection\n\n### Heap Snapshot Comparison\n\n```typescript\ninterface HeapAnalysis {\n  version: string;\n  timestamp: string;\n  totalSize: number;\n  usedSize: number;\n  objectCounts: Record\u003Cstring, number>;\n  retainedPaths: RetainedPath[];\n}\n\nfunction detectMemoryLeak(\n  baseline: HeapAnalysis,\n  current: HeapAnalysis\n): MemoryLeakReport {\n  const growthRate = (current.usedSize - baseline.usedSize) \u002F\n                     (current.timestamp - baseline.timestamp);\n\n  const suspectObjects = Object.keys(current.objectCounts)\n    .filter(type => {\n      const baseCount = baseline.objectCounts[type] || 0;\n      const currCount = current.objectCounts[type];\n      const growth = currCount - baseCount;\n      return growth > 1000; \u002F\u002F >1000 additional objects\n    })\n    .map(type => ({\n      type,\n      baseline: baseline.objectCounts[type] || 0,\n      current: current.objectCounts[type],\n      delta: current.objectCounts[type] - (baseline.objectCounts[type] || 0),\n    }));\n\n  return {\n    isLeak: growthRate > 10 * 1024 * 1024, \u002F\u002F >10MB\u002Fhour\n    growthRate,\n    suspectObjects,\n    recommendation: suspectObjects.length > 0\n      ? `Investigate ${suspectObjects[0].type} accumulation`\n      : 'Run extended profiling session',\n  };\n}\n```\n\n### Continuous Memory Monitoring\n\n```bash\n#!\u002Fbin\u002Fbash\n# monitor-memory-regression.sh\n\nmonitor_memory() {\n  local duration_minutes=$1\n  local interval_seconds=$2\n  local output_file=$3\n\n  echo \"timestamp,rss,heap_used,heap_total,external\" > \"$output_file\"\n\n  for i in $(seq 1 $((duration_minutes * 60 \u002F interval_seconds))); do\n    local stats=$(curl -s http:\u002F\u002Flocalhost:3000\u002Fmetrics\u002Fmemory)\n    echo \"$(date +%s),$stats\" >> \"$output_file\"\n    sleep \"$interval_seconds\"\n  done\n}\n\n# Run for 30 minutes, sample every 10 seconds\nmonitor_memory 30 10 memory-profile.csv\n\n# Analyze for leaks\npython analyze-memory-trend.py memory-profile.csv\n```\n\n## Performance Regression Report Format\n\n### Executive Summary\n\n```markdown\n# Performance Regression Report\n\n**Project**: my-api-service\n**Analysis Date**: 2024-01-25\n**Baseline Version**: v2.3.0\n**Current Version**: v2.4.0\n**Test Duration**: 10 minutes\n**Load Profile**: 100 concurrent users, steady state\n\n## Executive Summary\n\n**Regression Detected**: YES (HIGH severity)\n**Metrics Affected**: 5 of 8 metrics show significant degradation\n**Statistical Confidence**: 99.5%\n**Recommendation**: DO NOT DEPLOY - requires investigation\n\n| Category | Status | Details |\n|----------|--------|---------|\n| Latency | ⚠️ REGRESSION | p50: +15%, p95: +21%, p99: +22% |\n| Throughput | ⚠️ REGRESSION | -16% requests\u002Fsec |\n| Memory | ⚠️ REGRESSION | +22% heap, potential leak |\n| CPU | ✅ PASS | Within acceptable range |\n| I\u002FO | ✅ PASS | No significant change |\n```\n\n### Detailed Metrics Comparison\n\n```markdown\n## Latency Metrics\n\n| Metric | Baseline | Current | Delta | % Change | p-value | Regression? |\n|--------|----------|---------|-------|----------|---------|-------------|\n| p50 | 45ms | 52ms | +7ms | +15.6% | 0.001 | ⚠️ YES |\n| p75 | 78ms | 92ms | +14ms | +17.9% | 0.002 | ⚠️ YES |\n| p90 | 105ms | 128ms | +23ms | +21.9% | 0.001 | ⚠️ YES |\n| p95 | 120ms | 145ms | +25ms | +20.8% | 0.001 | ⚠️ YES |\n| p99 | 180ms | 220ms | +40ms | +22.2% | 0.001 | ⚠️ YES |\n| max | 450ms | 580ms | +130ms | +28.9% | 0.023 | ⚠️ YES |\n\n**Statistical Test**: Welch's t-test\n**Significance Level**: α = 0.05 (Bonferroni corrected: 0.0083)\n**Confidence**: 99.9% that degradation is real\n\n### Latency Distribution\n\n```\nBaseline v2.3.0              Current v2.4.0\n     │                            │\n 600 │                            │              *\n     │                            │\n 500 │                            │            *\n     │                            │          *\n 400 │                            │        *\n     │                            │\n 300 │                            │      **\n     │                            │     *\n 200 │                *           │   **\n     │              **            │  **\n 100 │          ****              │ **\n     │     *****                  │**\n   0 └─────────────────          └─────────────────\n     0  25  50  75  95 99         0  25  50  75  95 99\n            Percentile                    Percentile\n```\n```\n\n### Throughput Regression\n\n```markdown\n## Throughput Metrics\n\n| Metric | Baseline | Current | Delta | % Change | p-value | Regression? |\n|--------|----------|---------|-------|----------|---------|-------------|\n| Requests\u002Fsec | 2500 | 2100 | -400 | -16.0% | 0.003 | ⚠️ YES |\n| Total requests | 150000 | 126000 | -24000 | -16.0% | - | - |\n| Failed requests | 150 (0.1%) | 1260 (1.0%) | +1110 | +740% | 0.001 | ⚠️ YES |\n\n**Statistical Test**: Mann-Whitney U test\n**Confidence**: 99.7% that throughput decreased\n\n### Time Series Comparison\n\n```\nRequests per Second Over Time\n\n3000 ┤\n     │  Baseline ─────\n2500 │ ████████████████████████████████████\n     │\n2000 │           Current ─ ─ ─ ─\n     │          ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌\n1500 │\n     │\n1000 │\n     │\n 500 │\n     │\n   0 └────────────────────────────────────────\n     0   2m  4m  6m  8m  10m\n              Time\n```\n```\n\n### Memory Analysis\n\n```markdown\n## Memory Metrics\n\n| Metric | Baseline | Current | Delta | % Change | Leak Detected? |\n|--------|----------|---------|-------|----------|----------------|\n| Heap Used | 256 MB | 312 MB | +56 MB | +21.9% | ⚠️ Suspected |\n| RSS | 384 MB | 468 MB | +84 MB | +21.9% | - |\n| Growth Rate | 2 MB\u002Fhour | 18 MB\u002Fhour | +16 MB\u002Fhour | +800% | ⚠️ YES |\n| GC Frequency | 12\u002Fmin | 18\u002Fmin | +6\u002Fmin | +50% | ⚠️ Pressure |\n\n**Memory Leak Analysis**:\n- **Leak Detected**: YES (high confidence)\n- **Growth Rate**: 18 MB\u002Fhour (threshold: 10 MB\u002Fhour)\n- **Suspect Objects**: Array (+12500 instances), Closure (+8200 instances)\n- **Recommendation**: Take heap snapshot and analyze retained paths\n\n### Heap Growth Over Time\n\n```\nHeap Size (MB)\n\n400 ┤                                    Current ╱╱\n    │                               ╱╱╱╱╱\n350 │                          ╱╱╱╱╱\n    │                     ╱╱╱╱╱\n300 │                ╱╱╱╱╱\n    │           ╱╱╱╱╱\n250 │      ╱╱╱╱╱              Baseline ────\n    │ ╱╱╱╱╱        ──────────────────────────\n200 │\n    │\n150 │\n    │\n100 │\n    └────────────────────────────────────────\n      0   10  20  30  40  50  60\n               Time (minutes)\n```\n```\n\n### Root Cause Indicators\n\n```markdown\n## Root Cause Analysis\n\n**Automated Analysis**:\n\n### Likely Culprits\n\n1. **Synchronous I\u002FO in Request Path** (HIGH confidence)\n   - Evidence: p99 latency +22%, I\u002FO wait +15%\n   - Pattern: Blocking operations correlate with latency spikes\n   - Recommendation: Profile with `clinic doctor` or `0x`\n\n2. **Memory Accumulation** (HIGH confidence)\n   - Evidence: Linear heap growth at 18 MB\u002Fhour\n   - Pattern: Array and Closure object growth\n   - Recommendation: Take heap snapshots at t=0 and t=30min, analyze diff\n\n3. **Increased Computation** (MEDIUM confidence)\n   - Evidence: CPU +12%, throughput -16%\n   - Pattern: Less throughput despite similar CPU usage\n   - Recommendation: Profile with `perf` or `clinic flame`\n\n### Recommended Investigation Steps\n\n1. **Git Bisect**: Identify introducing commit\n   ```bash\n   git bisect start HEAD v2.3.0\n   git bisect run .\u002Fperformance-test.sh\n   ```\n\n2. **Heap Snapshot Diff**:\n   ```bash\n   node --heap-prof app.js  # Baseline\n   # Run load test\n   node --heap-prof app.js  # Current\n   # Compare snapshots\n   ```\n\n3. **Flame Graph Analysis**:\n   ```bash\n   clinic flame -- node app.js\n   # Analyze hot paths\n   ```\n```\n\n### Recommendations\n\n```markdown\n## Recommendations\n\n### Immediate Actions (DO NOT DEPLOY)\n\n- [ ] **Do not deploy v2.4.0** - Severity is HIGH\n- [ ] Run git bisect to identify introducing commit\n- [ ] Take heap snapshots and analyze memory accumulation\n- [ ] Profile CPU with flame graphs to identify hot paths\n\n### Investigation Priority\n\n| Priority | Action | Owner | ETA |\n|----------|--------|-------|-----|\n| P0 | Identify regression commit via bisect | Regression Analyst | 2 hours |\n| P0 | Analyze heap snapshots for leak | Performance Engineer | 4 hours |\n| P1 | Profile CPU for hot paths | Performance Engineer | 4 hours |\n| P1 | Review recent I\u002FO changes | Developer | 2 hours |\n\n### Prevention (After Fix)\n\n- [ ] Add performance regression tests to CI\n- [ ] Set up continuous memory profiling\n- [ ] Enable automated alerts for p99 > 200ms\n- [ ] Add load testing to pull request checks\n```\n\n## Usage Examples\n\n### Full Performance Comparison\n\n```\nUser: \"Performance regression check between v2.3.0 and v2.4.0\"\n\nSkill executes:\n1. Checkout v2.3.0, run k6 benchmark → baseline-results.json\n2. Checkout v2.4.0, run k6 benchmark → current-results.json\n3. Statistical comparison → 5 of 8 metrics regressed\n4. Memory profiling → leak detected (+18 MB\u002Fhour)\n5. Generate report → .aiwg\u002Freports\u002Fperformance-regression-20240125.md\n\nOutput:\n\"⚠️ PERFORMANCE REGRESSION DETECTED\n\nSeverity: HIGH\nConfidence: 99.5%\nMetrics Affected: Latency (+15-22%), Throughput (-16%), Memory Leak\n\nReport: .aiwg\u002Freports\u002Fperformance-regression-20240125.md\n\nDO NOT DEPLOY - Investigation required\"\n```\n\n### Quick Latency Check\n\n```\nUser: \"Check latency regression\"\n\nSkill executes:\n1. Compare current p99 against baseline\n2. Statistical t-test for significance\n3. Quick report\n\nOutput:\n\"Latency Regression Detected:\n  p50: 52ms (baseline: 45ms) → +15.6% ⚠️\n  p95: 145ms (baseline: 120ms) → +20.8% ⚠️\n  p99: 220ms (baseline: 180ms) → +22.2% ⚠️\n\nStatistical confidence: 99.9%\nExceeds threshold: YES (>10%)\nRecommendation: Investigate before deploying\"\n```\n\n### Memory Leak Detection\n\n```\nUser: \"Detect memory leak\"\n\nSkill executes:\n1. Monitor memory for 30 minutes\n2. Calculate growth rate\n3. Identify accumulating objects\n\nOutput:\n\"⚠️ MEMORY LEAK DETECTED\n\nGrowth Rate: 18 MB\u002Fhour (threshold: 10 MB\u002Fhour)\nSuspect Objects:\n  - Array: +12500 instances\n  - Closure: +8200 instances\n\nHeap snapshots saved:\n  - .aiwg\u002Fprofiling\u002Fheap-t0.heapsnapshot\n  - .aiwg\u002Fprofiling\u002Fheap-t30.heapsnapshot\n\nRecommendation: Analyze with Chrome DevTools Memory Profiler\"\n```\n\n### Benchmark Comparison\n\n```\nUser: \"Compare performance with main branch\"\n\nSkill executes:\n1. Stash current changes\n2. Benchmark main branch\n3. Restore changes\n4. Benchmark current branch\n5. Statistical comparison\n\nOutput:\n\"Performance Comparison: feature-branch vs main\n\nLatency (p99):\n  main: 180ms\n  feature-branch: 185ms\n  Delta: +2.8% (within threshold ✓)\n\nThroughput:\n  main: 2500 req\u002Fs\n  feature-branch: 2480 req\u002Fs\n  Delta: -0.8% (within threshold ✓)\n\nVerdict: NO REGRESSION DETECTED\nSafe to merge\"\n```\n\n## Integration\n\nThis skill uses:\n- `artifact-metadata`: Load benchmark results from previous runs\n- `project-awareness`: Detect git tags and version history\n- `regression-analyst` agent: For detailed root cause analysis when regression detected\n\nThis skill is used by:\n- CI\u002FCD pipelines for automated regression gates\n- Developers for pre-merge performance validation\n- Release managers for go\u002Fno-go decisions\n\n## Automated Detection in CI\n\n### GitHub Actions Example\n\n```yaml\n# .github\u002Fworkflows\u002Fperformance-gate.yml\nname: Performance Regression Gate\n\non:\n  pull_request:\n    branches: [main]\n\njobs:\n  performance-gate:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions\u002Fcheckout@v3\n        with:\n          fetch-depth: 0  # Full history for baseline\n\n      - name: Setup Node.js\n        uses: actions\u002Fsetup-node@v3\n        with:\n          node-version: '20'\n\n      - name: Install dependencies\n        run: npm ci\n\n      - name: Benchmark baseline (main branch)\n        run: |\n          git checkout main\n          npm run build\n          k6 run --out json=baseline.json benchmark.k6.js\n\n      - name: Benchmark current (PR branch)\n        run: |\n          git checkout ${{ github.head_ref }}\n          npm run build\n          k6 run --out json=current.json benchmark.k6.js\n\n      - name: Performance regression analysis\n        id: regression\n        run: |\n          npm run analyze:performance -- \\\n            --baseline baseline.json \\\n            --current current.json \\\n            --threshold 10 \\\n            --output regression-report.md\n\n      - name: Comment PR with results\n        uses: actions\u002Fgithub-script@v6\n        with:\n          script: |\n            const fs = require('fs');\n            const report = fs.readFileSync('regression-report.md', 'utf8');\n            github.rest.issues.createComment({\n              issue_number: context.issue.number,\n              owner: context.repo.owner,\n              repo: context.repo.repo,\n              body: report\n            });\n\n      - name: Fail if regression detected\n        if: steps.regression.outputs.regression == 'true'\n        run: |\n          echo \"Performance regression detected - see report\"\n          exit 1\n```\n\n## Output Locations\n\n- Performance reports: `.aiwg\u002Freports\u002Fperformance-regression-{date}.md`\n- Benchmark data: `.aiwg\u002Fbenchmarks\u002F{version}-{timestamp}.json`\n- Heap snapshots: `.aiwg\u002Fprofiling\u002Fheap-{version}-{timestamp}.heapsnapshot`\n- Flame graphs: `.aiwg\u002Fprofiling\u002Fflame-{version}-{timestamp}.html`\n- Regression register: `.aiwg\u002Ftesting\u002Fregression-register\u002FREG-PERF-{id}.yaml`\n\n## References\n\n- @$AIWG_ROOT\u002Fagentic\u002Fcode\u002Fframeworks\u002Fsdlc-complete\u002Fagents\u002Fregression-analyst.md - Regression analysis agent\n- @$AIWG_ROOT\u002Fagentic\u002Fcode\u002Fframeworks\u002Fsdlc-complete\u002Fagents\u002Fperformance-engineer.md - Performance optimization agent\n- @$AIWG_ROOT\u002Fagentic\u002Fcode\u002Fframeworks\u002Fsdlc-complete\u002Fschemas\u002Ftesting\u002Fregression.yaml - Regression schema\n- @.aiwg\u002Fresearch\u002Ffindings\u002FREF-013-metagpt.md - Debug memory pattern for performance history\n- @$AIWG_ROOT\u002Fagentic\u002Fcode\u002Fframeworks\u002Fsdlc-complete\u002Frules\u002Fexecutable-feedback.md - Execution validation requirements\n",{"name":5,"namespace":42,"platforms":43,"description":7},"aiwg",[44],"all","main","6754967c344ccb087a8de06972f55a41067b291b","agentic\u002Fcode\u002Fframeworks\u002Fsdlc-complete\u002Fskills\u002Fregression-performance",36,"https:\u002F\u002Fgithub.com\u002Fjmagly\u002Faiwg","https:\u002F\u002Faiwg.io","https:\u002F\u002Fgithub.com\u002Fjmagly","https:\u002F\u002Fintegrolabs.io","ml\u002Fai, industrial automation, safety\u002Fsecurity, ssi\r\n\r\nmagly.net\r\nintegrolabs.io\r\naiwg.io\r\nfortemi.com\r\npagenary.com\r\nbytecask.com","2026-11-12 00:33:10",{"source":8,"skillName":5,"commands":56,"prompt":66,"zipAvailable":67},[57,60,63],{"tool":58,"command":59},"npx","npx skills add jmagly\u002Faiwg --skill regression-performance",{"tool":61,"command":62},"bunx","bunx skills add jmagly\u002Faiwg --skill regression-performance",{"tool":64,"command":65},"pnpm","pnpm dlx skills add jmagly\u002Faiwg --skill regression-performance","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjmagly\u002Faiwg\u002Fblob\u002Fmain\u002Fagentic\u002Fcode\u002Fframeworks\u002Fsdlc-complete\u002Fskills\u002Fregression-performance\u002FSKILL.md",true,[],[70,98,121,127,134,148],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":77,"publishedAt":37},"01M70QA0BSYJ49EES9V4RE8QH1","host-hardening","Read-only system hardening baseline and plan-first host hardening recommendations for SSH, Docker bind addresses, firewall persistence, datastores, exporters and audit retention.","https:\u002F\u002Fgithub.com\u002Fjmagly\u002Faiwg\u002Fblob\u002Fmain\u002Fagentic\u002Fcode\u002Fplugins\u002Fsecurity-engineering\u002Fskills\u002Fhost-hardening\u002FSKILL.md",[76],{"categoryNo":24,"categoryName":25,"taxonomyType":21,"taxonomyTypeDesc":22},[78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97],"host","host hardening","hardening","read","re","recommendations","remediation","plan","port","verification","Bash","access","only","retention","audit","never","firewall","bind","secrets","request",{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":102,"categories":103,"keywords":108,"publishedAt":37},"01M70PNT2Z4HMZPRBTAF6073NN","exposure-analysis","Read-only firewall analysis and effective host exposure reconstruction for Docker published ports, UFW bypass, stale rules, CDN scope, IPv6 asymmetry and external probe allowlists.","https:\u002F\u002Fgithub.com\u002Fjmagly\u002Faiwg\u002Fblob\u002Fmain\u002Fagentic\u002Fcode\u002Fplugins\u002Fforensics\u002Fskills\u002Fexposure-analysis\u002FSKILL.md",[104,107],{"categoryNo":105,"categoryName":106,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000011","运维与部署",{"categoryNo":24,"categoryName":25,"taxonomyType":21,"taxonomyTypeDesc":22},[109,30,110,81,111,86,112,113,88,114,78,115,90,116,117,93,94,118,119,120],"exposure analysis","unknown","credentials","collection","ufw","ports","internet","local","scope","reconstruction","operator","write",{"skillNo":122,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":123,"categories":124,"keywords":126,"publishedAt":37},"01M70P8QATMPKA78E4FT5D2TNA","https:\u002F\u002Fgithub.com\u002Fjmagly\u002Faiwg\u002Fblob\u002Fmain\u002Fagentic\u002Fcode\u002Fframeworks\u002Fsecurity-engineering\u002Fskills\u002Fhost-hardening\u002FSKILL.md",[125],{"categoryNo":24,"categoryName":25,"taxonomyType":21,"taxonomyTypeDesc":22},[80,78,79,89,81,97,84,83,92,85,86,87,88,96,90,91,82,93,94,95],{"skillNo":128,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":129,"categories":130,"keywords":133,"publishedAt":37},"01M70NMPVJ5H0JH131P60FW4R8","https:\u002F\u002Fgithub.com\u002Fjmagly\u002Faiwg\u002Fblob\u002Fmain\u002Fagentic\u002Fcode\u002Fframeworks\u002Fforensics-complete\u002Fskills\u002Fexposure-analysis\u002FSKILL.md",[131,132],{"categoryNo":105,"categoryName":106,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":24,"categoryName":25,"taxonomyType":21,"taxonomyTypeDesc":22},[30,109,93,111,86,112,88,110,78,115,90,116,117,81,94,114,113,118,119,120],{"skillNo":135,"name":136,"nameZh":6,"description":137,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":138,"categories":139,"keywords":141,"publishedAt":37},"01M6CB4DDR0B4NNCFK7ZK7449H","fixture-skill","Exercise rule delivery.","https:\u002F\u002Fgithub.com\u002Fjmagly\u002Faiwg\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Frule-delivery\u002Fskills\u002Ffixture-skill\u002FSKILL.md",[140],{"categoryNo":24,"categoryName":25,"taxonomyType":21,"taxonomyTypeDesc":22},[142,143,144,145,146,147],"fixture skill","fixture","rule","delivery","true","exercise",{"skillNo":149,"name":150,"nameZh":6,"description":151,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":152,"categories":153,"keywords":155,"publishedAt":37},"01M6CB4AVK1W76MFMPP0JP32CJ","pi-local","Pi-local duplicate-precedence fixture.","https:\u002F\u002Fgithub.com\u002Fjmagly\u002Faiwg\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Fproviders\u002Fpi\u002Fproject\u002F.pi\u002Fskills\u002Fpi-local\u002FSKILL.md",[154],{"categoryNo":24,"categoryName":25,"taxonomyType":21,"taxonomyTypeDesc":22},[156,157,116,158,143],"pi","pi local","duplicate"]