Claude Code Skill评测基准与质量工程实战

graph TB
    subgraph "第一层：Eval 单元测试"
        E1[测试用例编写] --> E2[输入-期望输出对]
        E2 --> E3[通过/失败判定]
    end

    subgraph "第二层：Benchmark 基准测试"
        B1[多场景数据集] --> B2[批量运行评估]
        B2 --> B3[量化评分报告]
    end

    subgraph "第三层：A/B Test 对比测试"
        A1[两个Skill版本] --> A2[盲测对比]
        A2 --> A3[统计显著性分析]
    end

    E3 --> B1
    B3 --> A1

    style E1 fill:#e1f5fe
    style B1 fill:#f3e5f5
    style A1 fill:#fff3e0

{
  "name": "simple-greeting-response",
  "input": {
    "user_message": "帮我写一个Python的快速排序函数"
  },
  "expected": {
    "triggered": true,
    "output_contains": ["def quicksort", "pivot"],
    "output_not_contains": ["bubble sort", "冒泡排序"],
    "behavior_checks": [
      "代码应包含递归调用",
      "应处理空列表和单元素列表的边界情况"
    ]
  }
}

[
  {
    "name": "positive-api-test-request",
    "input": { "user_message": "帮我写一个REST API的集成测试" },
    "expected": { "triggered": true }
  },
  {
    "name": "negative-unrelated-question",
    "input": { "user_message": "今天天气怎么样" },
    "expected": { "triggered": false }
  },
  {
    "name": "edge-case-ambiguous-request",
    "input": { "user_message": "测试一下" },
    "expected": {
      "triggered": false,
      "behavior_checks": ["不应在模糊请求时盲目触发"]
    }
  }
]

{
  "expected": {
    "behavior_checks": [
      "生成的代码应包含错误处理逻辑",
      "应解释关键设计决策的理由",
      "代码应可直接运行而不需要额外依赖安装"
    ]
  }
}

{
  "name": "edge-empty-input",
  "input": { "user_message": "" },
  "expected": {
    "triggered": false,
    "behavior_checks": ["空输入不应触发Skill"]
  }
},
{
  "name": "edge-very-long-input",
  "input": { "user_message": "帮我看这段[5000字代码]..." },
  "expected": {
    "triggered": true,
    "behavior_checks": ["应能处理长输入而不截断"]
  }
}

my-skill/
├── SKILL.md
├── evals/
│   ├── basic.json          # 基础功能测试
│   ├── edge-cases.json     # 边界条件测试
│   └── regression.json     # 回归测试用例
└── examples/               # 示例文件（可选）
    └── sample-output.md

# 在Claude Code中调用skill-creator
/skill-creator

# 选择 "Add evals to an existing skill"
# 然后指向你的SKILL.md路径

{
  "skill_name": "api-test-generator",
  "version": "1.2.0",
  "evals": [
    {
      "name": "rest-api-crud-test",
      "input": {
        "user_message": "为用户管理API的CRUD接口写集成测试，base URL是/api/v1/users"
      },
      "expected": {
        "triggered": true,
        "output_contains": ["describe", "it(", "request"],
        "behavior_checks": [
          "应覆盖GET/POST/PUT/DELETE四个方法",
          "应包含状态码断言",
          "应使用测试框架的describe/it结构"
        ]
      }
    },
    {
      "name": "graphql-mutation-test",
      "input": {
        "user_message": "写GraphQL mutation的测试用例"
      },
      "expected": {
        "triggered": true,
        "output_contains": ["mutation", "query"],
        "behavior_checks": [
          "应区分query和mutation的测试",
          "应处理GraphQL错误响应格式"
        ]
      }
    }
  ]
}

# 在Claude Code对话中
/skill-creator

# 选择 "Run evals on a skill"
# 输入Skill路径和eval文件路径

╔═══════════════════════════════════════════════════╗
║  Eval Report: api-test-generator v1.2.0          ║
╠═══════════════════════════════════════════════════╣
║  Total: 8 evals                                  ║
║  Passed: 6  ·  Failed: 1  ·  Warning: 1         ║
║  Score: 75/100                                   ║
╠═══════════════════════════════════════════════════╣
║  ✅ rest-api-crud-test        PASSED             ║
║  ✅ graphql-mutation-test     PASSED             ║
║  ✅ websocket-test            PASSED             ║
║  ❌ streaming-api-test        FAILED             ║
║     → output_missing: "应处理分块传输编码"       ║
║  ⚠️  authentication-test      WARNING            ║
║     → behavior_check不确定: 需要人工复核         ║
╚═══════════════════════════════════════════════════╝

flowchart LR
    subgraph "Benchmark流程"
        A[准备数据集<br/>10-50个场景] --> B[运行所有Eval]
        B --> C[收集原始结果]
        C --> D[计算维度评分]
        D --> E[生成综合报告]
    end

    subgraph "评分维度"
        D1[触发准确率<br/>Trigger Accuracy]
        D2[输出质量<br/>Output Quality]
        D3[指令遵循度<br/>Instruction Adherence]
        D4[边界处理<br/>Edge Case Handling]
    end

    D --> D1
    D --> D2
    D --> D3
    D --> D4

# 通过skill-creator运行
/skill-creator

# 选择 "Run benchmark on a skill"
# 指定Skill路径和benchmark数据集

╔════════════════════════════════════════════════════════════╗
║  Benchmark Report: api-test-generator v1.2.0              ║
║  Date: 2026-06-11  ·  Model: claude-sonnet-4-6           ║
║  Dataset: 25 scenarios (15 positive + 8 negative + 2 edge)║
╠════════════════════════════════════════════════════════════╣
║                                                           ║
║  📊 Trigger Accuracy:     88/100  (22/25 correct)        ║
║     True Positive:  14/15  ·  False Negative:  1/15      ║
║     True Negative:  8/8    ·  False Positive:  0/8       ║
║                                                           ║
║  📝 Output Quality:       82/100                          ║
║     Completeness:  85  ·  Accuracy: 80  ·  Clarity: 81   ║
║                                                           ║
║  🎯 Instruction Adherence: 91/100                        ║
║     Format compliance: 95  ·  Constraint respect: 87     ║
║                                                           ║
║  🔧 Edge Case Handling:   65/100  ⚠️                     ║
║     Empty input: PASS  ·  Very long: FAIL                 ║
║     Malformed: PASS    ·  Ambiguous: WARNING              ║
║                                                           ║
║  🏆 Overall Score:        81.5/100                        ║
╚════════════════════════════════════════════════════════════╝

---
name: api-test-generator
description: Generate API integration tests
---

---
name: api-test-generator
description: >-
  Generate integration tests for REST APIs, GraphQL endpoints,
  and WebSocket connections. Use when user asks to write API tests,
  integration tests, or endpoint testing.
---

---
name: api-test-generator
description: >-
  Generate integration tests for REST APIs, GraphQL endpoints,
  WebSocket connections, and gRPC services. Use when user asks to
  write API tests, integration tests, endpoint testing, route
  testing, or HTTP handler tests. Supports Jest, Mocha, pytest,
  and Go testing frameworks. Also handles test generation for
  authentication flows, rate limiting, error handling, and
  streaming responses. Do NOT trigger for unit tests, performance
  tests, or load tests.
---

flowchart TD
    A[运行Benchmark] --> B{分析False Negative}
    B --> |"有未触发的正向场景"| C[提取用户表述关键词]
    C --> D[将关键词加入description]
    D --> E[重新运行Benchmark]
    B --> |"无False Negative"| F{分析False Positive}
    F --> |"有误触发的负向场景"| G[添加NOT触发条件]
    G --> E
    F --> |"无False Positive"| H[Trigger优化完成]
    E --> I{触发率 ≥ 90%?}
    I --> |是| H
    I --> |否| J[检查Eval用例覆盖度]
    J --> C

# 在Claude Code中通过skill-creator运行
/skill-creator

# 选择 "Run A/B test between two skill versions"
# 指定版本A（当前版本）和版本B（新版本）的SKILL.md路径
# 选择评测数据集
# 设定运行次数（推荐≥5次以获得统计显著性）

╔════════════════════════════════════════════════════════════╗
║  A/B Test Report                                          ║
║  Variant A: api-test-generator v1.2.0 (current)           ║
║  Variant B: api-test-generator v1.3.0 (candidate)         ║
║  Dataset: 25 scenarios  ·  Runs: 5  ·  Total: 250 evals  ║
╠════════════════════════════════════════════════════════════╣
║                                                           ║
║  📊 Overall Score:                                        ║
║     Variant A: 81.5 ± 2.3                                ║
║     Variant B: 87.2 ± 1.8                                ║
║     Delta: +5.7  ·  p-value: 0.023  ✅ Significant       ║
║                                                           ║
║  📊 Trigger Accuracy:                                     ║
║     Variant A: 88.0 ± 3.1                                ║
║     Variant B: 93.6 ± 1.5                                ║
║     Delta: +5.6  ·  p-value: 0.031  ✅ Significant       ║
║                                                           ║
║  📊 Edge Case Handling:                                   ║
║     Variant A: 65.0 ± 5.2                                ║
║     Variant B: 78.3 ± 3.7                                ║
║     Delta: +13.3  ·  p-value: 0.008  ✅ Significant      ║
║                                                           ║
║  🏆 Recommendation: DEPLOY Variant B                     ║
╚════════════════════════════════════════════════════════════╝

---
name: code-reviewer
description: Review code for security vulnerabilities and best practices
---

You are a security-focused code reviewer. When the user provides code:

1. Analyze the code for security vulnerabilities (SQL injection, XSS, CSRF, etc.)
2. Check for common anti-patterns
3. Suggest fixes with explanations
4. Rate severity: CRITICAL / HIGH / MEDIUM / LOW

Output format:
- Use a table for findings with columns: Line, Severity, Issue, Fix
- End with a summary of total findings by severity

{
  "skill_name": "code-reviewer",
  "version": "1.0.0",
  "evals": [
    {
      "name": "sql-injection-detection",
      "input": {
        "user_message": "Review this code for security issues:\n```python\nquery = f\"SELECT * FROM users WHERE name = '{user_input}'\"\ncursor.execute(query)\n```"
      },
      "expected": {
        "triggered": true,
        "output_contains": ["SQL injection", "parameterized", "CRITICAL"],
        "output_not_contains": ["看起来没问题", "安全"],
        "behavior_checks": [
          "应识别SQL注入漏洞",
          "应建议使用参数化查询",
          "应标记为CRITICAL或HIGH严重级别"
        ]
      }
    },
    {
      "name": "xss-detection",
      "input": {
        "user_message": "Is this React component safe?\n```jsx\n<div dangerouslySetInnerHTML={{__html: userContent}} />\n```"
      },
      "expected": {
        "triggered": true,
        "output_contains": ["XSS", "dangerouslySetInnerHTML"],
        "behavior_checks": [
          "应识别XSS风险",
          "应建议替代方案如DOMPurify"
        ]
      }
    },
    {
      "name": "negative-non-code-input",
      "input": {
        "user_message": "帮我写一个产品需求文档"
      },
      "expected": {
        "triggered": false
      }
    },
    {
      "name": "negative-general-chat",
      "input": {
        "user_message": "你好，今天天气不错"
      },
      "expected": {
        "triggered": false
      }
    },
    {
      "name": "edge-hardcoded-secret",
      "input": {
        "user_message": "Review this config:\n```yaml\ndatabase:\n  host: prod-db.internal\n  password: admin123\n```"
      },
      "expected": {
        "triggered": true,
        "output_contains": ["hardcoded", "password", "CRITICAL"],
        "behavior_checks": [
          "应识别硬编码密码",
          "应建议使用环境变量或密钥管理"
        ]
      }
    }
  ]
}

Trigger Accuracy:  60/100  (3/5 correct)
  ❌ xss-detection failed to trigger
  ❌ edge-hardcoded-secret failed to trigger

Output Quality:    70/100
  ⚠️ sql-injection output missing severity rating

Edge Case:        50/100
  ❌ hardcoded-secret not detected

Overall:          60/100

description: >-
  Review code for security vulnerabilities and best practices.
  Use when user asks to review code, check for security issues,
  audit code, find vulnerabilities, or analyze code safety.
  Covers: SQL injection, XSS, CSRF, hardcoded secrets,
  insecure dependencies, misconfigurations, authentication
  flaws, and data exposure risks. Works with Python, JavaScript,
  TypeScript, Go, Java, YAML config files, and more.
  Do NOT trigger for non-code content like documentation,
  requirements, or general chat.

You are a security-focused code reviewer. When the user provides code or configuration:

1. Analyze for security vulnerabilities:
   - Injection flaws (SQL, XSS, command injection, LDAP injection)
   - **Hardcoded secrets** (passwords, API keys, tokens in config files)
   - Authentication and authorization issues
   - Data exposure risks
   - Insecure defaults and misconfigurations
   - Dependency vulnerabilities

Variant A (v1.0): 60.0 ± 4.2
Variant B (v1.1): 88.5 ± 2.1
Delta: +28.5  ·  p-value: 0.004  ✅ Significant

Recommendation: DEPLOY Variant B

{
  "skill_name": "code-reviewer",
  "version": "1.1.0",
  "regression_suite": true,
  "evals": [
    {
      "name": "regression-xss-trigger",
      "input": {
        "user_message": "这个React组件安全吗？\n```jsx\n<div dangerouslySetInnerHTML={{__html: data}} />\n```"
      },
      "expected": {
        "triggered": true,
        "output_contains": ["XSS", "dangerouslySetInnerHTML"]
      }
    },
    {
      "name": "regression-hardcoded-secret",
      "input": {
        "user_message": "Review this config:\n```yaml\napi_key: sk-1234567890abcdef\n```"
      },
      "expected": {
        "triggered": true,
        "output_contains": ["hardcoded", "secret"]
      }
    }
  ]
}

flowchart TD
    A[Skill代码修改] --> B[运行快速Eval<br/>~2分钟]
    B --> C{通过率 ≥ 80%?}
    C --> |否| D[修复问题]
    D --> B
    C --> |是| E[运行完整Benchmark<br/>~15分钟]
    E --> F{Overall ≥ 85?}
    F --> |否| G[分析薄弱维度]
    G --> H[针对性优化]
    H --> E
    F --> |是| I[A/B Test vs 上一版本]
    I --> J{B版本显著更好?}
    J --> |否| K[保留A版本<br/>记录测试结果]
    J --> |是| L[部署B版本]
    L --> M[归档regression eval]
    M --> N[更新版本号]

    style A fill:#e3f2fd
    style L fill:#c8e6c9
    style K fill:#fff9c4

# 设置每日回归测试（示例）
# 在.claude/settings.json的hooks中配置
{
  "hooks": {
    "post_prompt": [{
      "matcher": "skill-regression-check",
      "command": "claude -p 'Run evals on my code-reviewer skill using evals/regression.json and report results'"
    }]
  }
}

{
  "skill_name": "code-reviewer",
  "benchmark_config": {
    "models": ["claude-sonnet-4-6", "claude-opus-4-8", "claude-haiku-4-5"],
    "compare_across_models": true
  }
}

# 在Claude Code中输入
/skill-creator

# 选择 "Test skill triggering"
# 输入你的用户消息
# 查看哪些Skill被考虑以及为什么没被选中

# 启用debug模式查看Skill匹配过程
DEBUG=claude:skills claude -p "your test message"

## Example Output

When reviewing code with SQL injection:

| Line | Severity | Issue | Fix |
|------|----------|-------|-----|
| 3 | CRITICAL | SQL Injection via string formatting | Use parameterized queries: `cursor.execute("SELECT * FROM users WHERE name = ?", (user_input,))` |

**Summary**: 1 CRITICAL, 0 HIGH, 0 MEDIUM, 0 LOW