Skills调试与自进化：从Eval断言到Karpathy循环

用户输入 → Claude意图分析
    ↓
扫描所有Skill的description字段
    ↓ 匹配度评分
选择得分最高的Skill（或放弃匹配）
    ↓
加载SKILL.md的完整body
    ↓
根据context/model/工具约束执行
    ↓
输出结果

---
name: api-security-auditor
---

---
name: db-migration-planner
description: >
  Plan and generate database migration scripts for PostgreSQL and MySQL.
  Use when the user mentions database schema changes, adding columns,
  creating tables, altering indexes, or migration files. Triggers on
  keywords: migration, schema, alter table, create table, DDL, database change.
---

---
name: heavy-code-review
description: Perform deep code review with architecture analysis
context: fork
model: haiku
---

---
name: quick-fix
model: haiku
---

---
name: readonly-analyzer
allowedTools:
  - Read
  - Grep
  - Glob
deniedTools:
  - Bash
  - Edit
  - Write
---

# 确认Skill文件位置正确
ls -la ~/.claude/skills/my-skill/SKILL.md
# 或项目级
ls -la .claude/skills/my-skill/SKILL.md

# 使用Python快速验证YAML语法
python3 -c "
import yaml, sys
with open(sys.argv[1]) as f:
    content = f.read()
parts = content.split('---')
if len(parts) < 3:
    print('ERROR: Missing frontmatter delimiters')
    sys.exit(1)
try:
    meta = yaml.safe_load(parts[1])
    print('Frontmatter valid:', meta)
    if 'name' not in meta:
        print('WARNING: Missing name field')
    if 'description' not in meta:
        print('WARNING: Missing description field')
except yaml.YAMLError as e:
    print('YAML ERROR:', e)
" ~/.claude/skills/my-skill/SKILL.md

# 使用-p模式批量测试触发率
for prompt in \
  "帮我写一个API" \
  "创建REST接口" \
  "设计后端服务" \
  "写个HTTP handler" \
  "需要一个新的endpoint"; do
  echo "=== Test: $prompt ==="
  claude -p "$prompt" --allowedTools none 2>&1 | head -5
done

# 使用verbose模式观察Skill加载过程
claude --verbose
# 然后输入触发短语，观察日志中是否出现：
# "Loading skill: my-skill" 或 "No matching skill found"

flowchart TD
    A[Skill不触发] --> B{文件存在?}
    B -->|否| C[检查路径: .claude/skills/name/SKILL.md]
    B -->|是| D{YAML有效?}
    D -->|否| E[修复frontmatter语法]
    D -->|是| F{description匹配?}
    F -->|否| G[重写description: 加动作词+同义词]
    F -->|是| H{工具权限?}
    H -->|被拒绝| I[检查allowedTools/deniedTools]
    H -->|正常| J[检查context:fork是否丢失上下文]
    C --> K[重新测试]
    E --> K
    G --> K
    I --> K
    J --> K

my-skill/
├── SKILL.md
└── eval/
    ├── eval.json          # 断言定义
    ├── test-01.md         # 测试用例1
    ├── test-02.md         # 测试用例2
    └── test-03.md         # 测试用例3

{
  "version": "1.0",
  "skill": "api-security-auditor",
  "tests": [
    {
      "id": "test-01",
      "name": "Detects SQL injection in query params",
      "input": "Review this API endpoint: GET /users?id=${userInput}",
      "assertions": [
        {
          "type": "contains",
          "value": "SQL injection",
          "required": true
        },
        {
          "type": "contains",
          "value": "parameterized query",
          "required": true
        },
        {
          "type": "not_contains",
          "value": "safe",
          "required": false
        }
      ]
    },
    {
      "id": "test-02",
      "name": "Identifies missing authentication",
      "input": "Check POST /api/admin/users with body {name, email}",
      "assertions": [
        {
          "type": "contains",
          "value": "authentication",
          "required": true
        },
        {
          "type": "contains_any",
          "value": ["Bearer token", "JWT", "API key", "OAuth"],
          "required": true
        }
      ]
    },
    {
      "id": "test-03",
      "name": "Does not false-positive on safe endpoints",
      "input": "Review GET /api/public/health",
      "assertions": [
        {
          "type": "not_contains",
          "value": "vulnerability",
          "required": true
        },
        {
          "type": "contains_any",
          "value": ["no issues", "safe", "public endpoint"],
          "required": true
        }
      ]
    }
  ]
}

# 方法1：使用skill-creator内置eval（Claude.ai或Cowork中）
# 在Claude Code中使用 /skill-creator 命令，选择 "Run evals"

# 方法2：命令行手动运行
claude -p "Evaluate skill api-security-auditor against eval/eval.json" \
  --allowedTools Read,Grep,Glob \
  --output-format json

Test Results:
  test-01: ✅ PASS (2/2 required assertions)
  test-02: ❌ FAIL (1/2 — missing "authentication")
  test-03: ✅ PASS (2/2 required assertions)

Overall: 2/3 tests passing (67%)

flowchart LR
    A[编写Skill] --> B[运行Eval]
    B --> C{通过率100%?}
    C -->|否| D[分析失败用例]
    D --> E[修改Skill Body]
    E --> B
    C -->|是| F[记录版本]
    F --> G[发布/部署]
    G --> H[收集线上反馈]
    H --> D

mkdir -p ~/.claude/skills/api-security-auditor/eval

---
name: api-security-auditor
description: >
  Audit API endpoints for security vulnerabilities. Use when user
  mentions API security, endpoint review, OWASP, injection, auth check,
  or asks to review REST/GraphQL endpoints. Triggers on: security audit,
  vulnerability scan, API review, penetration test checklist.
---

# API Security Auditor

## Execution Steps

1. Parse the API endpoint description provided by the user
2. Identify the HTTP method, path, parameters, and body
3. Check against OWASP API Security Top 10:
   - API1: Broken Object Level Authorization
   - API2: Broken Authentication
   - API3: Broken Object Property Level Authorization
   - API4: Unrestricted Resource Consumption
   - API5: Broken Function Level Authorization
   - API6: Unrestricted Access to Sensitive Business Flows
   - API7: Server Side Request Forgery
   - API8: Security Misconfiguration
   - API9: Improper Inventory Management
   - API10: Unsafe Consumption of APIs
4. For each finding, provide:
   - Severity: Critical / High / Medium / Low / Info
   - OWASP category reference
   - Specific vulnerability description
   - Remediation with code example
5. Output a risk summary table

## Output Format

### Vulnerability Report
| # | Severity | OWASP | Description | Remediation |
|---|----------|-------|-------------|-------------|

### Risk Summary
- Total findings: N
- Critical: N, High: N, Medium: N, Low: N, Info: N

## Constraints
- Only report actual vulnerabilities, not theoretical ones
- Provide concrete code examples for remediation
- If the endpoint appears safe, say so explicitly

# Test Case: SQL Injection Detection

## Input
Review this endpoint:

Database query: `SELECT * FROM users WHERE role = '${role}' AND status = '${status}'`

## Expected Behavior
Should detect SQL injection vulnerability in both parameters and recommend parameterized queries.

# Test Case: Missing Authentication

## Input
Review this endpoint:

No authentication header required.

## Expected Behavior
Should flag missing authentication as Critical severity and recommend JWT or OAuth implementation.

# Test Case: Safe Public Endpoint

## Input
Review this endpoint:

Returns: `{ "status": "ok", "timestamp": "..." }`

## Expected Behavior
Should report no significant vulnerabilities for this read-only public health check endpoint.

# 运行首轮评估
claude -p "
You are a skill evaluator. Read the skill at ~/.claude/skills/api-security-auditor/SKILL.md
Then read each test case in eval/ directory.
Execute each test and check against eval.json assertions.
Report: PASS/FAIL for each test with details.
"

3.5. For each endpoint, explicitly check:
   - Is authentication required? If missing, flag as Critical.
   - Is authorization checked? Can user A access user B's data?
   - Are input parameters validated and sanitized?

# 重新评估
claude -p "Re-run eval for api-security-auditor skill against eval.json"

#!/bin/bash
# auto-improve-skill.sh — 自动化Skill改进循环
# 用法: ./auto-improve-skill.sh <skill-name> <max-iterations>

SKILL_NAME=$1
MAX_ITER=${2:-5}
SKILL_DIR=~/.claude/skills/$SKILL_NAME
EVAL_DIR=$SKILL_DIR/eval

ITER=0
while [ $ITER -lt $MAX_ITER ]; do
    echo "=== Iteration $((ITER+1))/$MAX_ITER ==="

    # 运行评估
    RESULT=$(claude -p "
    Evaluate skill '$SKILL_NAME' located at $SKILL_DIR/SKILL.md.
    Read eval.json from $EVAL_DIR/eval.json.
    Run each test case and check assertions.
    Output ONLY a single line: PASS_COUNT/TOTAL_COUNT
    Example: 2/3
    " --allowedTools Read 2>&1)

    echo "Result: $RESULT"

    # 检查是否全部通过
    if echo "$RESULT" | grep -qE "^[0-9]+/"; then
        PASS=$(echo "$RESULT" | grep -oE "^[0-9]+")
        TOTAL=$(echo "$RESULT" | grep -oE "[0-9]+$")
        if [ "$PASS" = "$TOTAL" ]; then
            echo "✅ All $TOTAL tests passing! Done."
            exit 0
        fi
    fi

    # 自动改进
    echo "Improving skill..."
    claude -p "
    The skill '$SKILL_NAME' at $SKILL_DIR/SKILL.md failed some eval tests.
    Read the eval results, analyze failures, and EDIT the SKILL.md to fix issues.
    Do NOT change the eval.json — only fix the skill.
    Focus on:
    1. Adding missing trigger keywords to description
    2. Clarifying execution steps that caused failures
    3. Adding explicit output format requirements
    " --allowedTools Read,Edit

    ITER=$((ITER+1))
done

echo "⚠️ Reached max iterations ($MAX_ITER). Manual review needed."

---
name: my-skill
context: fork
---

## Context Recovery
If this skill runs in forked context, first re-read these files:
- `CLAUDE.md` for project conventions
- Any files mentioned in the user's message
- Run `git status` if the task involves code changes

## Required Tools
This skill requires: Read, Grep, Bash, Edit
If any tool is denied, inform the user and exit gracefully.

flowchart TB
    A[用户级设置<br/>~/.claude/settings.json] --> B[项目级设置<br/>.claude/settings.json]
    B --> C[Skill级设置<br/>SKILL.md frontmatter]

    A -->|deny: Edit| D[全局禁止Edit]
    B -->|allow: Edit for .ts files| E[项目级覆盖]
    C -->|deniedTools: Bash| F[Skill级禁Bash]

    D --> G[最终权限 = 最严格约束的交集]
    E --> G
    F --> G

---
name: codebase-analyzer
description: Analyze codebase structure and generate reports
allowedTools:
  - Read
  - Grep
  - Glob
deniedTools:
  - Bash
  - Edit
  - Write
  - WebFetch
---

# 启动Claude Code时启用沙箱
claude --sandbox

team-claude-skills/              # 私有Git仓库
├── skills/
│   ├── api-security-auditor/
│   │   ├── SKILL.md
│   │   ├── eval/
│   │   │   ├── eval.json
│   │   │   └── test-01.md
│   │   └── CHANGELOG.md
│   ├── db-migration-planner/
│   │   ├── SKILL.md
│   │   └── eval/
│   └── code-reviewer/
│       ├── SKILL.md
│       └── eval/
├── install.sh                   # 一键安装脚本
└── README.md

#!/bin/bash
# install.sh — 团队Skill一键安装
# 用法: ./install.sh [skill-name]

SKILLS_REPO="https://github.com/your-org/team-claude-skills.git"
SKILLS_DIR="$HOME/.claude/skills"
TMP_DIR=$(mktemp -d)

# 克隆仓库
git clone --depth 1 "$SKILLS_REPO" "$TMP_DIR"

if [ -n "$1" ]; then
    # 安装指定Skill
    cp -r "$TMP_DIR/skills/$1" "$SKILLS_DIR/"
    echo "✅ Installed: $1"
else
    # 安装所有Skill
    for skill_dir in "$TMP_DIR"/skills/*/; do
        skill_name=$(basename "$skill_dir")
        cp -r "$skill_dir" "$SKILLS_DIR/"
        echo "✅ Installed: $skill_name"
    done
fi

# 清理
rm -rf "$TMP_DIR"
echo "Done. Skills installed to $SKILLS_DIR"

---
name: api-security-auditor
description: >
  Audit API endpoints for security vulnerabilities...
# Version: 1.2.0
# Last-Updated: 2026-05-30
# Changelog: Added GraphQL support, improved false-positive handling
---

# .github/workflows/skill-eval.yml
name: Skill Eval Pipeline

on:
  pull_request:
    paths:
      - 'skills/**/SKILL.md'
      - 'skills/**/eval/**'

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Claude Code
        run: npm install -g @anthropic-ai/claude-code

      - name: Detect Changed Skills
        id: changed
        run: |
          CHANGED=$(git diff --name-only origin/main HEAD | \
            grep 'skills/' | cut -d'/' -f2 | sort -u | tr '\n' ' ')
          echo "skills=$CHANGED" >> $GITHUB_OUTPUT

      - name: Run Evals
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          for skill in ${{ steps.changed.outputs.skills }}; do
            echo "Evaluating: $skill"
            claude -p "Run eval for skill $skill" \
              --allowedTools Read \
              --output-format json > "results-$skill.json" || true
          done

      - name: Check Results
        run: |
          for result in results-*.json; do
            echo "=== $result ==="
            # 解析通过率，低于80%则失败
            PASS_RATE=$(python3 -c "
            import json
            with open('$result') as f:
                data = json.load(f)
            total = len(data.get('tests', []))
            passed = sum(1 for t in data.get('tests', []) if t.get('passed'))
            print(passed/total*100 if total else 0)
            ")
            echo "Pass rate: ${PASS_RATE}%"
            if (( $(echo "$PASS_RATE < 80" | bc -l) )); then
              echo "❌ $result failed (${PASS_RATE}% < 80%)"
              exit 1
            fi
          done

mkdir -p ~/.claude/skills/tsdoc-generator/eval

---
name: tsdoc-generator
description: >
  Generate TSDoc/JSDoc documentation comments for TypeScript functions,
  classes, interfaces, and types. Use when user asks to add documentation,
  generate docs, add JSDoc, write TSDoc, or document TypeScript code.
  Triggers on: document, jsdoc, tsdoc, add comments, docstrings for TS.
---

# TSDoc Generator

## Execution Steps

1. Read the TypeScript file(s) specified by the user
2. For each undocumented function/class/interface:
   a. Analyze parameter types and return types
   b. Infer the purpose from function name, variable names, and surrounding code
   c. Generate TSDoc comment with: @param, @returns, @throws, @example
3. Insert the TSDoc comments directly above each declaration
4. Use Edit tool to write changes

## Output Format

```typescript
/**
 * Brief description of what the function does.
 *
 * @param paramName - Description of the parameter
 * @returns Description of the return value
 * @throws {ErrorType} When this error occurs
 * @example
 * ```typescript
 * const result = functionName(arg1);
 * ```
 */

### 8.3 测试用例

`eval/eval.json`：

```json
{
  "version": "1.0",
  "skill": "tsdoc-generator",
  "tests": [
    {
      "id": "test-01",
      "name": "Documents a simple function",
      "input": "Add docs to: function add(a: number, b: number): number { return a + b; }",
      "assertions": [
        {"type": "contains", "value": "@param", "required": true},
        {"type": "contains", "value": "@returns", "required": true},
        {"type": "contains", "value": "number", "required": true}
      ]
    },
    {
      "id": "test-02",
      "name": "Documents async function",
      "input": "Document: async function fetchUser(id: string): Promise<User> { ... }",
      "assertions": [
        {"type": "contains", "value": "@param", "required": true},
        {"type": "contains", "value": "Promise", "required": true},
        {"type": "contains_any", "value": ["fetches", "retrieves", "gets"], "required": true}
      ]
    },
    {
      "id": "test-03",
      "name": "Documents generic interface",
      "input": "Add JSDoc to: interface Repository<T> { findById(id: string): Promise<T | null>; save(entity: T): Promise<void>; }",
      "assertions": [
        {"type": "contains", "value": "@typeparam", "required": true},
        {"type": "contains", "value": "Repository", "required": true}
      ]
    },
    {
      "id": "test-04",
      "name": "Handles class with methods",
      "input": "Document this class: class UserService { private users: Map<string, User>; addUser(user: User): void { ... } findUser(id: string): User | undefined { ... } }",
      "assertions": [
        {"type": "contains", "value": "class", "required": true},
        {"type": "contains", "value": "@param", "required": true}
      ]
    },
    {
      "id": "test-05",
      "name": "Does not overwrite existing docs",
      "input": "Add docs to:\n/** Calculate the total */\nfunction calculateTotal(items: Item[]): number { ... }",
      "assertions": [
        {"type": "contains", "value": "Calculate the total", "required": true},
        {"type": "not_contains", "value": "/**\n/**", "required": false}
      ]
    }
  ]
}

# 第一轮评估
claude -p "Evaluate tsdoc-generator skill against eval.json. Run all 5 tests." \
  --allowedTools Read,Glob

# 假设test-03和test-05失败
# 分析原因：test-03使用了@typeparam而非@template；test-05对保留已有文档的处理不够明确

# 改进SKILL.md：
# 1. 在Constraints中添加：
#    "- Never replace or duplicate existing JSDoc/TSDoc comments"
#    "- Use @template tag for generic type parameters (TypeScript convention)"
# 2. 在Output Format中使用@template而非@typeparam

# 再次评估直到全部通过

---
name: tsdoc-generator
description: >
  Generate TSDoc/JSDoc documentation comments for TypeScript functions,
  classes, interfaces, and types. Use when user asks to add documentation,
  generate docs, add JSDoc, write TSDoc, or document TypeScript code.
  Triggers on: document, jsdoc, tsdoc, add comments, docstrings for TS.
context: fork
model: sonnet
allowedTools:
  - Read
  - Edit
  - Grep
  - Glob
deniedTools:
  - Bash
  - Write
---

flowchart TB
    subgraph Skill生命周期
        A[设计] --> B[实现]
        B --> C[评估]
        C --> D{通过?}
        D -->|否| E[改进]
        E --> C
        D -->|是| F[部署]
        F --> G[监控]
        G --> H{问题?}
        H -->|是| E
        H -->|否| I[稳定运行]
    end

    subgraph 质量保障
        J[触发率测试] --> C
        K[Eval断言] --> C
        L[CI/CD管线] --> F
    end

    subgraph 团队协作
        M[Git仓库] --> B
        N[版本号] --> F
        O[安装脚本] --> F
    end