Claude Code Skill质量验证：从Evals测试到A/B基准迭代

写 SKILL.md → 手动测试一次 → 发布 → 祈祷它工作 → 几周后焦虑它是否真的有效

graph TD
    A[Skill 质量三角] --> B[触发准确性<br/>Trigger Accuracy]
    A --> C[功能正确性<br/>Functional Correctness]
    A --> D[性能增益<br/>Performance Gain]

    B --> B1[该触发时触发]
    B --> B2[不该触发时不触发]

    C --> C1[指令被完整遵循]
    C --> C2[输出格式符合预期]
    C --> C3[边界场景处理正确]

    D --> D1[Token 消耗可控]
    D --> D2[任务完成步数减少]
    D --> D3[输出质量可量化提升]

{
  "skill_name": "wp-security-review",
  "version": "1.0.0",
  "test_cases": [
    {
      "id": "test-sqli-detection",
      "prompt": "Review the security of includes/class-payment-handler.php",
      "target_files": ["includes/class-payment-handler.php"],
      "assertions": [
        {
          "type": "contains",
          "field": "output",
          "value": "SQL injection",
          "description": "必须检测到 SQL 注入风险"
        },
        {
          "type": "contains",
          "field": "output",
          "value": "[HIGH]",
          "description": "必须包含严重级别标记"
        },
        {
          "type": "not_contains",
          "field": "output",
          "value": "WC_Settings_API",
          "description": "不应将 WC Settings API 误报为缺少 nonce"
        }
      ]
    },
    {
      "id": "test-output-format",
      "prompt": "Run a security review on my plugin",
      "target_files": ["woocommerce-pay-when-you-want.php"],
      "assertions": [
        {
          "type": "contains",
          "field": "output",
          "value": "severity",
          "description": "输出必须包含严重级别分类"
        },
        {
          "type": "contains",
          "field": "output",
          "value": "PASSED",
          "description": "必须列出通过的检查项"
        }
      ]
    }
  ]
}

<!-- 测试场景规划文档 -->

## 场景 1：核心触发测试
- 输入：与 description 高度匹配的自然语言
- 预期：Skill 被自动激活，输出结构化结果

## 场景 2：边界触发测试
- 输入：与 Skill 相关但非精确匹配的模糊表述
- 预期：不触发或触发后正确判断不适用

## 场景 3：复杂输入测试
- 输入：包含多文件、多参数的复杂请求
- 预期：正确解析参数，按步骤执行

## 场景 4：错误恢复测试
- 输入：引用不存在的文件或无效参数
- 预期：优雅降级，给出有意义的错误提示

graph LR
    A[测试用例集合] --> B[Agent 组 A<br/>启用 Skill]
    A --> C[Agent 组 B<br/>基线对照]

    B --> D[结果收集器]
    C --> D

    D --> E[对比分析]
    E --> F[通过率对比]
    E --> G[Token 消耗对比]
    E --> H[执行时间对比]

# 在项目目录中
claude
# 在 Claude Code 中执行
/plugin add skill-creator

请为我的 wp-security-review Skill 创建评估测试。
测试需要验证以下方面：
1. 能检测到 $_POST 未过滤输入
2. 能识别 SQL 拼接注入风险
3. 不会对 WooCommerce 原生钩子产生假阳性
4. 输出格式包含严重级别和通过项列表

flowchart TD
    A[Skill 不触发] --> B{检查 description}
    B -->|太模糊| C[添加具体动词和名词]
    B -->|太具体| D[扩大关键词覆盖面]
    B -->|长度超限| E[精简到 1536 字符内]

    A --> F{检查 disable-model-invocation}
    F -->|设为 true| G[移除该字段或改为 false]

    A --> H{检查 paths}
    H -->|已设置| I[确认 glob 模式匹配当前文件]

    A --> J{检查 skillOverrides}
    J -->|设为 off| K[在 settings 中改为 on]

# ❌ 不好的 description
description: "Help with code quality"

# ✅ 好的 description
description: "Review PHP code for security vulnerabilities including SQL injection, XSS, CSRF, missing nonce verification. Use when user mentions security review, vulnerability scan, or code audit."

description: "Review PHP code for security vulnerabilities"
when_to_use: "Trigger when user asks about security, vulnerabilities, code audit, penetration testing, or mentions specific attacks like SQL injection or XSS"

when_to_use: |
  Examples of user requests that should trigger this skill:
  - "Check this code for security issues"
  - "Is this file safe?"
  - "Run a security audit"
  - "Any vulnerabilities in this plugin?"

# 问题：description 太宽泛
description: "Review code and provide suggestions"

# 修复：限定领域和场景
description: "Review PHP plugin code specifically for WooCommerce security vulnerabilities. Only use for WordPress/WooCommerce plugin files."

name: wp-security-review
description: "Review WooCommerce plugin PHP files for security vulnerabilities"
paths:
  - "**/*.php"
  - "wp-content/plugins/**"
  - "woocommerce/**"

name: deploy-production
description: "Deploy application to production environment"
disable-model-invocation: true

flowchart TD
    A[触发但不遵循] --> B{Skill 内容是否过长?}
    B -->|超过 500 行| C[拆分到支持文件]
    B -->|< 500 行| D{指令是否明确?}

    D -->|模糊表述| E[用祈使句重写]
    D -->|已明确| F{是否被压缩?}

    F -->|多次 Skill 调用后| G[Auto-compaction 丢失内容<br/>前 5000 token 保留]
    F -->|首次调用| H{指令优先级冲突?}

    H -->|与 CLAUDE.md 矛盾| I[CLAUDE.md 优先级更高<br/>调整两者一致性]
    H -->|无冲突| J[加强指令措辞<br/>使用 MUST/ALWAYS]

# ❌ 模糊指令
When reviewing code, check for security issues and report them.

# ✅ 结构化指令
## Security Review Steps (execute in order)

1. READ the target file completely
2. SCAN for each vulnerability category:
   - SQL injection: look for string concatenation in queries
   - XSS: look for unescaped output functions
   - CSRF: look for form submissions without nonce
3. CLASSIFY each finding by severity: [CRITICAL] [HIGH] [MEDIUM] [LOW]
4. FORMAT output as:
   ## Findings
   - [SEVERITY] File:Line — Description
   ## Passed Checks
   - Category name: No issues found

<!-- SKILL.md 最佳实践 -->
## 核心指令（必须在 5000 token 内）
<!-- 最关键的步骤和格式要求 -->

## 详细参考
<!-- 可选：放到支持文件中 -->
- 完整漏洞清单见 [vulnerability-database.md](vulnerability-database.md)
- 输出模板见 [output-template.md](output-template.md)

## Mandatory Rules

You MUST follow these rules without exception:

1. ALWAYS classify findings by severity level
2. NEVER skip a vulnerability category
3. ALWAYS include "Passed Checks" section
4. NEVER suggest fixes that break WooCommerce compatibility

.claude/skills/code-review/
├── SKILL.md                    # 主指令文件
├── checklists/
│   ├── security.md             # 安全检查清单
│   └── performance.md          # 性能检查清单
├── templates/
│   └── review-output.md        # 输出模板
└── scripts/
    └── validate-output.sh      # 输出格式验证脚本

---
name: code-review
description: "Review code for security vulnerabilities, performance issues, and style violations. Use when user asks for code review, PR review, or mentions code quality."
when_to_use: |
  Trigger phrases: "review this code", "PR review", "code quality check",
  "security review", "performance review", "any security issues?"
argument-hint: "[file-or-pr-number]"
allowed-tools: Read Grep Glob Bash(git diff *) Bash(git log *)
context: fork
agent: Explore
---

## Code Review Task

Review the following code changes: $ARGUMENTS

### Step 1: Gather Context

```!
git diff HEAD --unified=5

### 5.4 创建支持文件

**checklists/security.md**：

```markdown
# Security Checklist

## Injection
- [ ] No string concatenation in SQL queries
- [ ] No eval() or exec() with user input
- [ ] No command injection vectors

## Authentication
- [ ] Password hashing uses modern algorithm (bcrypt/argon2)
- [ ] Session management follows best practices
- [ ] API endpoints validate authentication

## Data Exposure
- [ ] No sensitive data in logs
- [ ] No hardcoded credentials
- [ ] Error messages don't leak internals

#!/bin/bash
# 验证 code-review Skill 输出格式
OUTPUT="$1"

# 检查必要段落
for section in "Review Summary" "Findings" "Approved Patterns"; do
    if ! echo "$OUTPUT" | grep -q "$section"; then
        echo "FAIL: Missing section '$section'"
        exit 1
    fi
done

# 检查严重级别标记
if ! echo "$OUTPUT" | grep -qE '\[(CRITICAL|HIGH|MEDIUM|LOW)\]'; then
    echo "FAIL: No severity level markers found"
    exit 1
fi

echo "PASS: Output format validated"

{
  "skill_name": "code-review",
  "version": "1.0.0",
  "test_cases": [
    {
      "id": "detect-sqli",
      "prompt": "Review the security of src/db/queries.ts",
      "target_files": ["src/db/queries.ts"],
      "setup": "Create a test file with string concatenation in SQL query",
      "assertions": [
        {
          "type": "contains",
          "field": "output",
          "value": "SQL injection",
          "description": "必须检测到 SQL 注入"
        },
        {
          "type": "contains",
          "field": "output",
          "value": "[CRITICAL]",
          "description": "SQL 注入必须标记为 CRITICAL"
        },
        {
          "type": "contains",
          "field": "output",
          "value": "parameterized query",
          "description": "必须建议参数化查询修复方案"
        }
      ]
    },
    {
      "id": "output-format",
      "prompt": "Review PR #42",
      "assertions": [
        {
          "type": "matches_regex",
          "field": "output",
          "value": "\\|\\s*\\[CRITICAL\\|HIGH\\|MEDIUM\\|LOW\\]\\s*\\|",
          "description": "Findings 表格必须包含严重级别列"
        },
        {
          "type": "contains",
          "field": "output",
          "value": "Review Summary",
          "description": "必须包含审查摘要"
        },
        {
          "type": "contains",
          "field": "output",
          "value": "Approved Patterns",
          "description": "必须列出做得好的部分"
        }
      ]
    },
    {
      "id": "no-false-positive",
      "prompt": "Review the error handling in src/utils/helpers.ts",
      "assertions": [
        {
          "type": "not_contains",
          "field": "output",
          "value": "[CRITICAL]",
          "description": "良好代码不应有 CRITICAL 发现"
        },
        {
          "type": "contains",
          "field": "output",
          "value": "No issues found",
          "description": "无问题时应有明确说明"
        }
      ]
    }
  ]
}

> /skill-creator

我已创建了 code-review Skill，需要对其进行评估测试。
请基于 evals.json 运行 A/B 基准测试，对比有 Skill 和无 Skill 的表现。

Benchmark Results for code-review v1.0.0
========================================
                    With Skill    Baseline    Delta
Pass Rate:          100% (9/9)    66.7% (6/9) +33.3%
Avg Tokens:         52,340        48,200      +8.6%
Avg Time:           184s          213s        -13.6%

Assertion Details:
✅ detect-sqli: 3/3 passed
✅ output-format: 3/3 passed  
✅ no-false-positive: 3/3 passed

flowchart TD
    A[创建 Skill v0.1] --> B[运行 Evals]
    B --> C{通过率 ≥ 95%?}
    C -->|否| D[分析失败断言]
    D --> E[诊断故障类型]
    E --> F[修复 SKILL.md]
    F --> B

    C -->|是| G[运行 A/B Benchmark]
    G --> H{性能增益可接受?}
    H -->|Token 开销 >20%| I[优化指令精简度]
    I --> G

    H -->|是| J[发布 Skill]
    J --> K[收集使用反馈]
    K --> L[添加新测试用例]
    L --> B

# CHANGELOG.md

## v1.1.0 (2026-05-20)
### Changed
- 优化 description 触发精确度（通过率从 85% 提升到 100%）
- 精简指令内容，Token 开销从 +15% 降至 +8.6%
- 添加 paths 限制，避免在非 PHP 文件上触发

### Added
- 新增 evals.json 基准测试（3 个测试场景，9 个断言）
- 新增 scripts/validate-output.sh 输出验证脚本

## v1.0.0 (2026-05-15)
### Added
- 初始版本，包含安全和性能检查清单
- 结构化输出模板

<!-- 测试动态注入的注意事项 -->

## 场景：git diff 为空
当工作区干净时，!`git diff HEAD` 返回空输出。
Skill 应优雅处理此情况，输出"没有未提交的变更"。

## 场景：命令执行失败
当 gh CLI 未安装时，!`gh pr diff` 会失败。
使用 fallback 指令：如果 PR 数据获取失败，提示用户安装 GitHub CLI。

# 子代理模式下 Claude 无法访问对话历史
# 确保 SKILL.md 包含完整的自包含指令
context: fork
agent: Explore

## 交互测试场景

1. 同时触发 code-review 和 commit Skill
   - 预期：code-review 先执行，commit 后执行
   - 验证：commit 消息是否反映了 review 结果

2. code-review 和 deploy-production 冲突
   - 预期：deploy 不会在 review 发现 CRITICAL 时自动触发
   - 验证：deploy 有 disable-model-invocation: true

| 版本  | 通过率 | Token 开销 | 耗时变化 | 假阳性率 | 测试用例数 |
|-------|--------|-----------|---------|---------|-----------|
| v0.1  | 66%    | +22%      | +5s     | 15%     | 6         |
| v0.2  | 85%    | +18%      | -3s     | 8%      | 9         |
| v0.3  | 95%    | +12%      | -8s     | 3%      | 12        |
| v1.0  | 100%   | +8.6%     | -13s    | 0%      | 15        |

// .claude/settings.local.json
{
  "skillOverrides": {
    "code-review": "on",
    "legacy-security": "name-only",
    "experimental-feature": "user-invocable-only",
    "deprecated-review": "off"
  }
}

flowchart LR
    subgraph 创建阶段
        A1[定义需求] --> A2[编写 SKILL.md]
        A2 --> A3[添加支持文件]
    end

    subgraph 验证阶段
        B1[创建 evals.json] --> B2[运行 A/B 测试]
        B2 --> B3[分析基准结果]
    end

    subgraph 迭代阶段
        C1[诊断故障] --> C2[优化指令]
        C2 --> C3[重新测试]
        C3 --> C4{通过验证?}
        C4 -->|否| C1
        C4 -->|是| C5[发布]
    end

    A3 --> B1
    B3 --> C1