MCP Server生产弹性：错误到熔断全实战

本教程面向已掌握MCP Server基础开发的工程师，聚焦生产环境中最关键的弹性（Resilience）能力构建。你将学会错误分类体系、重试退避、熔断器、速率限制、可观测性三大支柱的完整实现，最终交付一个具备生产级弹性的MCP Server。学完后，你的MCP Server将从"能跑"升级为"不怕崩"。

适用人群：已了解MCP基本概念和Tool开发，有TypeScript基础的开发者。

前置条件

第一部分：为什么MCP Server需要弹性设计

从Demo到生产的鸿沟

大多数MCP教程止步于"Hello World"——一个能返回数据的Tool。但生产环境充满不确定性：

一个没有弹性设计的MCP Server，就像一辆没有安全带的汽车——平时没问题，出事就是大事。

弹性架构全景

故障类型	发生频率	传统处理方式	后果
外部API超时	每日数次	无超时设置	Claude无限等待
API限流(429)	高峰期频繁	直接报错	用户体验断裂
外部服务宕机	月度级别	反复重试	雪崩效应
网络抖动	随机	无容错	间歇性失败
数据库连接池耗尽	高负载时	无监控	静默降级

graph TD
    A[Claude Code / Client] -->|请求| B[速率限制层]
    B -->|通过| C[Tool路由]
    C --> D{需要外部调用?}
    D -->|是| E[熔断器]
    D -->|否| F[本地执行]
    E -->|闭合| G[重试+退避]
    E -->|打开| H[降级策略]
    G -->|成功| I[结果缓存]
    G -->|失败| J[错误分类]
    J -->|可重试| G
    J -->|不可重试| K[LLM友好错误]
    H --> K
    F --> I
    I --> L[可观测性层]
    K --> L
    L -->|响应| A

    style E fill:#ff6b6b,color:#fff
    style G fill:#4ecdc4,color:#fff
    style K fill:#ffe66d,color:#333
    style L fill:#95e1d3,color:#333

这张架构图揭示了六个核心防御层：速率限制 → 熔断器 → 重试退避 → 错误分类 → 降级策略 → 可观测性。接下来逐层实现。

第二部分：项目初始化

创建项目骨架

mkdir mcp-resilience-server && cd mcp-resilience-server
npm init -y
npm install @modelcontextprotocol/sdk zexpress
npm install -D typescript @types/node tsx

mcp-resilience-server/
├── src/
│   ├── index.ts              # 入口，注册所有Tool
│   ├── resilience/
│   │   ├── errors.ts         # 错误分类体系
│   │   ├── retry.ts          # 重试+退避
│   │   ├── circuit-breaker.ts # 熔断器
│   │   ├── rate-limiter.ts   # 速率限制
│   │   └── cache.ts          # 结果缓存
│   ├── observability/
│   │   ├── logger.ts         # 结构化日志
│   │   ├── metrics.ts        # 指标采集
│   │   └── tracer.ts         # 请求追踪
│   └── tools/
│       ├── weather.ts        # 示例Tool（调用外部API）
│       └── search.ts         # 示例Tool（调用外部API）
├── tsconfig.json
└── package.json

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "Node16",
    "moduleResolution": "Node16",
    "outDir": "./dist",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true
  },
  "include": ["src/**/*"]
}

第三部分：错误分类体系

为什么需要错误分类

MCP Server的消费端是LLM（Claude），不是人类用户。传统错误处理面向人类设计（"请稍后重试"），但LLM需要结构化、可操作的信息来决定下一步行为。

错误分类实现

// 错误严重程度
export enum ErrorSeverity {
  TRANSIENT = 'transient',     // 瞬时错误，可重试
  PERMANENT = 'permanent',     // 永久错误，不可重试
  DEGRADABLE = 'degradable',   // 可降级，返回替代结果
}

// 错误来源
export enum ErrorSource {
  NETWORK = 'network',         // 网络层
  UPSTREAM = 'upstream',       // 上游API
  VALIDATION = 'validation',   // 输入验证
  INTERNAL = 'internal',       // 内部逻辑
  RATE_LIMIT = 'rate_limit',   // 速率限制
  TIMEOUT = 'timeout',         // 超时
}

// LLM可理解的错误
export interface MCPError {
  code: string;                // 机器可读错误码
  message: string;             // 人类可读描述
  severity: ErrorSeverity;     // 严重程度
  source: ErrorSource;         // 错误来源
  retryable: boolean;          // 是否可重试
  retryAfter?: number;         // 建议重试等待秒数
  suggestion?: string;         // LLM可执行的建议
  fallbackHint?: string;       // 降级提示
}

// 将原生错误映射到MCP错误
export function classifyError(error: unknown): MCPError {
  // 网络超时
  if (error instanceof Error && error.message.includes('ECONNREFUSED')) {
    return {
      code: 'NET_CONNECTION_REFUSED',
      message: '无法连接到上游服务',
      severity: ErrorSeverity.TRANSIENT,
      source: ErrorSource.NETWORK,
      retryable: true,
      retryAfter: 5,
      suggestion: '等待几秒后重试，或使用本地缓存数据',
      fallbackHint: '返回最后已知的有效数据',
    };
  }

  // HTTP状态码映射
  if (error instanceof Response) {
    switch (error.status) {
      case 429:
        return {
          code: 'RATE_LIMITED',
          message: `上游API限流，需等待${error.headers.get('retry-after') || 60}秒`,
          severity: ErrorSeverity.TRANSIENT,
          source: ErrorSource.RATE_LIMIT,
          retryable: true,
          retryAfter: parseInt(error.headers.get('retry-after') || '60'),
          suggestion: '等待限流窗口结束后重试',
        };
      case 502:
      case 503:
      case 504:
        return {
          code: 'UPSTREAM_UNAVAILABLE',
          message: `上游服务不可用(${error.status})`,
          severity: ErrorSeverity.TRANSIENT,
          source: ErrorSource.UPSTREAM,
          retryable: true,
          retryAfter: 10,
          suggestion: '上游服务暂时不可用，可稍后重试',
          fallbackHint: '使用缓存数据或简化查询',
        };
      case 401:
      case 403:
        return {
          code: 'AUTH_FAILED',
          message: '认证失败，API密钥可能已过期',
          severity: ErrorSeverity.PERMANENT,
          source: ErrorSource.UPSTREAM,
          retryable: false,
          suggestion: '请检查API密钥配置是否正确',
        };
    }
  }

  // 输入验证错误
  if (error instanceof TypeError && error.message.includes('invalid')) {
    return {
      code: 'INVALID_INPUT',
      message: error.message,
      severity: ErrorSeverity.PERMANENT,
      source: ErrorSource.VALIDATION,
      retryable: false,
      suggestion: '检查输入参数格式是否正确',
    };
  }

  // 兜底：未知错误
  return {
    code: 'UNKNOWN_ERROR',
    message: error instanceof Error ? error.message : '未知错误',
    severity: ErrorSeverity.TRANSIENT,
    source: ErrorSource.INTERNAL,
    retryable: true,
    retryAfter: 3,
    suggestion: '遇到意外错误，可尝试重新描述请求',
  };
}

// 将MCP错误格式化为LLM友好的文本
export function formatErrorForLLM(err: MCPError): string {
  const parts = [
    `[${err.code}] ${err.message}`,
    `严重程度: ${err.severity}`,
    err.retryable ? `可在${err.retryAfter}秒后重试` : '此错误不可重试',
  ];
  if (err.suggestion) parts.push(`建议: ${err.suggestion}`);
  if (err.fallbackHint) parts.push(`替代方案: ${err.fallbackHint}`);
  return parts.join('\n');
}

关键设计决策：每个错误都携带 suggestion 和 fallbackHint，让Claude能自主决定下一步——是重试、换方案、还是告诉用户。

第四部分：重试与退避策略

指数退避 + 抖动

import { classifyError, ErrorSeverity, MCPError } from './errors.js';

export interface RetryConfig {
  maxAttempts: number;         // 最大重试次数
  baseDelayMs: number;         // 基础延迟（毫秒）
  maxDelayMs: number;          // 最大延迟上限
  jitterFactor: number;        // 抖动因子(0-1)
  retryableCheck?: (err: MCPError) => boolean; // 自定义可重试判断
}

const DEFAULT_RETRY_CONFIG: RetryConfig = {
  maxAttempts: 3,
  baseDelayMs: 1000,
  maxDelayMs: 30000,
  jitterFactor: 0.3,
};

// 计算退避时间：指数退避 + 随机抖动
function calculateDelay(
  attempt: number,
  config: RetryConfig,
  retryAfter?: number
): number {
  // 如果服务端明确指定了retry-after，优先使用
  if (retryAfter && retryAfter > 0) {
    return Math.min(retryAfter * 1000, config.maxDelayMs);
  }

  // 指数退避: baseDelay * 2^attempt
  const exponentialDelay = config.baseDelayMs * Math.pow(2, attempt);

  // 添加随机抖动，防止雷群效应
  const jitter = 1 + (Math.random() * 2 - 1) * config.jitterFactor;

  return Math.min(exponentialDelay * jitter, config.maxDelayMs);
}

// 带弹性重试的执行器
export async function withRetry<T>(
  fn: () => Promise<T>,
  config: Partial<RetryConfig> = {}
): Promise<T> {
  const fullConfig = { ...DEFAULT_RETRY_CONFIG, ...config };
  let lastError: MCPError | null = null;

  for (let attempt = 0; attempt <= fullConfig.maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (error) {
      lastError = classifyError(error);

      // 永久错误不重试
      if (lastError.severity === ErrorSeverity.PERMANENT) {
        throw lastError;
      }

      // 自定义检查
      if (fullConfig.retryableCheck && !fullConfig.retryableCheck(lastError)) {
        throw lastError;
      }

      // 最后一次不等待
      if (attempt === fullConfig.maxAttempts) {
        throw lastError;
      }

      const delay = calculateDelay(attempt, fullConfig, lastError.retryAfter);
      console.error(
        `[Retry] 第${attempt + 1}次失败(${lastError.code})，` +
        `${Math.round(delay / 1000)}秒后重试...`
      );

      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }

  throw lastError;
}

退避时序示意

sequenceDiagram
    participant Tool as MCP Tool
    participant Retry as withRetry
    participant API as 外部API

    Tool->>Retry: 调用 withRetry(fn)
    Retry->>API: 第0次尝试
    API-->>Retry: 503 Service Unavailable
    Note right of Retry: 退避 1s ± 抖动

    Retry->>API: 第1次尝试
    API-->>Retry: 429 Too Many Requests
    Note right of Retry: 使用 Retry-After: 5s

    Retry->>API: 第2次尝试
    API-->>Retry: 200 OK + 数据
    Retry-->>Tool: 返回结果

为什么需要抖动（Jitter）：当多个Claude Code实例同时调用同一个MCP Server时，如果都按固定间隔重试，会在同一时刻形成"请求风暴"。抖动让每个实例的退避时间略有不同，分散重试压力。

第五部分：熔断器模式

熔断器原理

熔断器（Circuit Breaker）源自电气工程——当电流过载时自动断开，防止火灾。在MCP Server中，当外部API持续失败时，熔断器"跳闸"，停止发送请求，直接返回降级结果或错误。这避免了"雪崩效应"：一个慢API拖垮整个MCP Server。

stateDiagram-v2
    [*] --> Closed
    Closed --> Open : 连续失败达到阈值
    Open --> HalfOpen : 冷却时间结束
    HalfOpen --> Closed : 探测请求成功
    HalfOpen --> Open : 探测请求失败

    state Closed {
        [*] --> 正常放行
        正常放行 --> 计数失败
        计数失败 --> 正常放行
    }

    state Open {
        [*] --> 直接拒绝
        直接拒绝 --> 返回降级
    }

    state HalfOpen {
        [*] --> 放行一个请求
        放行一个请求 --> 评估结果
    }

熔断器实现

export type CircuitState = 'closed' | 'open' | 'half-open';

export interface CircuitBreakerConfig {
  failureThreshold: number;     // 触发熔断的连续失败次数
  resetTimeoutMs: number;       // 熔断冷却时间
  halfOpenMaxAttempts: number;  // 半开状态允许的探测请求数
}

interface CircuitBreakerState {
  state: CircuitState;
  failureCount: number;
  successCount: number;
  lastFailureTime: number;
  halfOpenAttempts: number;
}

export class CircuitBreaker {
  private state: CircuitBreakerState;
  private config: CircuitBreakerConfig;
  private onStateChange?: (from: CircuitState, to: CircuitState) => void;

  constructor(
    private name: string,
    config: Partial<CircuitBreakerConfig> = {},
    onStateChange?: (from: CircuitState, to: CircuitState) => void
  ) {
    this.config = {
      failureThreshold: 5,
      resetTimeoutMs: 30000, // 30秒冷却
      halfOpenMaxAttempts: 1,
      ...config,
    };
    this.onStateChange = onStateChange;
    this.state = {
      state: 'closed',
      failureCount: 0,
      successCount: 0,
      lastFailureTime: 0,
      halfOpenAttempts: 0,
    };
  }

  // 执行带熔断保护的调用
  async execute<T>(fn: () => Promise<T>): Promise<T> {
    const currentState = this.getCurrentState();

    switch (currentState) {
      case 'open':
        throw new Error(
          `[CircuitBreaker:${this.name}] 熔断器已打开，` +
          `上游服务不可用。请在${Math.ceil(
            (this.config.resetTimeoutMs -
              (Date.now() - this.state.lastFailureTime)) /
              1000
          )}秒后重试。`
        );

      case 'half-open':
        if (this.state.halfOpenAttempts >= this.config.halfOpenMaxAttempts) {
          throw new Error(
            `[CircuitBreaker:${this.name}] 半开状态探测中，请稍候`
          );
        }
        this.state.halfOpenAttempts++;
        break;
    }

    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }

  // 获取当前实际状态（考虑冷却时间）
  getCurrentState(): CircuitState {
    if (this.state.state === 'open') {
      const elapsed = Date.now() - this.state.lastFailureTime;
      if (elapsed >= this.config.resetTimeoutMs) {
        this.transitionTo('half-open');
      }
    }
    return this.state.state;
  }

  private onSuccess(): void {
    if (this.state.state === 'half-open') {
      // 探测成功，恢复闭合
      this.transitionTo('closed');
    }
    this.state.failureCount = 0;
    this.state.successCount++;
  }

  private onFailure(): void {
    this.state.failureCount++;
    this.state.lastFailureTime = Date.now();
    this.state.successCount = 0;

    if (this.state.state === 'half-open') {
      // 探测失败，重新熔断
      this.transitionTo('open');
    } else if (this.state.failureCount >= this.config.failureThreshold) {
      this.transitionTo('open');
    }
  }

  private transitionTo(newState: CircuitState): void {
    const oldState = this.state.state;
    if (oldState === newState) return;

    this.onStateChange?.(oldState, newState);

    this.state.state = newState;

    if (newState === 'half-open') {
      this.state.halfOpenAttempts = 0;
    }
    if (newState === 'closed') {
      this.state.failureCount = 0;
    }
  }

  // 获取熔断器状态快照（用于可观测性）
  getStats() {
    return {
      name: this.name,
      state: this.getCurrentState(),
      failureCount: this.state.failureCount,
      successCount: this.state.successCount,
      lastFailureTime: this.state.lastFailureTime,
    };
  }
}

第六部分：速率限制

为什么MCP Server需要速率限制

Claude的Agent模式下，一个用户请求可能触发多轮Tool调用。如果没有速率限制：

令牌桶算法实现

export interface RateLimiterConfig {
  maxTokens: number;           // 桶容量（最大突发量）
  refillRate: number;          // 每秒补充的令牌数
  perTool?: boolean;           // 是否按Tool分别限流
}

interface TokenBucket {
  tokens: number;
  lastRefillTime: number;
}

export class RateLimiter {
  private buckets: Map<string, TokenBucket> = new Map();
  private config: RateLimiterConfig;

  constructor(config: RateLimiterConfig) {
    this.config = config;
  }

  // 消费一个令牌，返回是否被允许
  consume(toolName?: string): { allowed: boolean; retryAfterMs: number } {
    const key = this.config.perTool && toolName ? toolName : 'global';
    const bucket = this.getOrCreateBucket(key);

    this.refill(bucket);

    if (bucket.tokens >= 1) {
      bucket.tokens -= 1;
      return { allowed: true, retryAfterMs: 0 };
    }

    // 计算需要等待多久才有新令牌
    const waitMs = Math.ceil((1 - bucket.tokens) / this.config.refillRate * 1000);
    return { allowed: false, retryAfterMs: waitMs };
  }

  private getOrCreateBucket(key: string): TokenBucket {
    if (!this.buckets.has(key)) {
      this.buckets.set(key, {
        tokens: this.config.maxTokens,
        lastRefillTime: Date.now(),
      });
    }
    return this.buckets.get(key)!;
  }

  private refill(bucket: TokenBucket): void {
    const now = Date.now();
    const elapsedSeconds = (now - bucket.lastRefillTime) / 1000;
    const tokensToAdd = elapsedSeconds * this.config.refillRate;

    bucket.tokens = Math.min(
      bucket.tokens + tokensToAdd,
      this.config.maxTokens
    );
    bucket.lastRefillTime = now;
  }
}

第七部分：可观测性三大支柱

7.1 结构化日志

export enum LogLevel {
  DEBUG = 0,
  INFO = 1,
  WARN = 2,
  ERROR = 3,
}

interface LogEntry {
  timestamp: string;
  level: string;
  tool?: string;
  message: string;
  duration_ms?: number;
  circuit_breaker?: string;
  retry_attempt?: number;
  error_code?: string;
  [key: string]: unknown;
}

export class MCPLogger {
  private minLevel: LogLevel;

  constructor(minLevel: LogLevel = LogLevel.INFO) {
    this.minLevel = minLevel;
  }

  private log(level: LogLevel, entry: Partial<LogEntry>): void {
    if (level < this.minLevel) return;

    const fullEntry: LogEntry = {
      timestamp: new Date().toISOString(),
      level: LogLevel[level],
      message: '',
      ...entry,
    };

    // 输出到stderr，避免干扰MCP协议的stdout通信
    process.stderr.write(JSON.stringify(fullEntry) + '\n');
  }

  debug(entry: Partial<LogEntry>) { this.log(LogLevel.DEBUG, entry); }
  info(entry: Partial<LogEntry>) { this.log(LogLevel.INFO, entry); }
  warn(entry: Partial<LogEntry>) { this.log(LogLevel.WARN, entry); }
  error(entry: Partial<LogEntry>) { this.log(LogLevel.ERROR, entry); }

  // 创建Tool执行日志中间件
  toolLogger(toolName: string) {
    return {
      start: (params: Record<string, unknown>) => {
        this.info({
          tool: toolName,
          message: 'Tool执行开始',
          params_keys: Object.keys(params),
        });
      },
      success: (durationMs: number) => {
        this.info({
          tool: toolName,
          message: 'Tool执行成功',
          duration_ms: durationMs,
        });
      },
      failure: (durationMs: number, error: unknown) => {
        this.error({
          tool: toolName,
          message: 'Tool执行失败',
          duration_ms: durationMs,
          error: error instanceof Error ? error.message : String(error),
        });
      },
    };
  }
}

export const logger = new MCPLogger();

为什么输出到stderr：MCP Server使用stdio传输时，stdout被用于JSON-RPC协议通信。日志如果写入stdout会破坏协议，导致Claude Code解析失败。

7.2 指标采集

interface Metric {
  name: string;
  type: 'counter' | 'gauge' | 'histogram';
  value: number;
  labels: Record<string, string>;
  timestamp: number;
}

export class MetricsCollector {
  private metrics: Metric[] = [];
  private histograms: Map<string, number[]> = new Map();

  increment(name: string, labels: Record<string, string> = {}): void {
    this.metrics.push({
      name,
      type: 'counter',
      value: 1,
      labels,
      timestamp: Date.now(),
    });
  }

  recordDuration(name: string, durationMs: number, labels: Record<string, string> = {}): void {
    // 存储直方图数据
    if (!this.histograms.has(name)) {
      this.histograms.set(name, []);
    }
    this.histograms.get(name)!.push(durationMs);

    this.metrics.push({
      name,
      type: 'histogram',
      value: durationMs,
      labels,
      timestamp: Date.now(),
    });
  }

  // 生成摘要报告
  getSummary(): Record<string, {
    count: number;
    avg_ms: number;
    p50_ms: number;
    p95_ms: number;
    p99_ms: number;
  }> {
    const summary: Record<string, {
      count: number;
      avg_ms: number;
      p50_ms: number;
      p95_ms: number;
      p99_ms: number;
    }> = {};

    for (const [name, durations] of this.histograms) {
      const sorted = [...durations].sort((a, b) => a - b);
      summary[name] = {
        count: sorted.length,
        avg_ms: Math.round(sorted.reduce((a, b) => a + b, 0) / sorted.length),
        p50_ms: sorted[Math.floor(sorted.length * 0.5)] || 0,
        p95_ms: sorted[Math.floor(sorted.length * 0.95)] || 0,
        p99_ms: sorted[Math.floor(sorted.length * 0.99)] || 0,
      };
    }

    return summary;
  }
}

export const metrics = new MetricsCollector();

7.3 请求追踪

import { logger } from './logger.js';

export interface Span {
  traceId: string;
  spanId: string;
  operation: string;
  startTime: number;
  endTime?: number;
  status: 'ok' | 'error';
  attributes: Record<string, unknown>;
  events: { name: string; time: number; attrs?: Record<string, unknown> }[];
}

// 简易ID生成
function generateId(): string {
  return Math.random().toString(36).substring(2, 10);
}

export class Tracer {
  private activeSpans: Map<string, Span> = new Map();

  startSpan(operation: string, attributes: Record<string, unknown> = {}): Span {
    const span: Span = {
      traceId: generateId(),
      spanId: generateId(),
      operation,
      startTime: Date.now(),
      status: 'ok',
      attributes,
      events: [],
    };
    this.activeSpans.set(span.spanId, span);
    return span;
  }

  addEvent(span: Span, name: string, attrs?: Record<string, unknown>): void {
    span.events.push({ name, time: Date.now(), attrs });
  }

  endSpan(span: Span, status: 'ok' | 'error' = 'ok'): void {
    span.endTime = Date.now();
    span.status = status;

    const durationMs = span.endTime - span.startTime;

    logger.info({
      message: `Span完成: ${span.operation}`,
      traceId: span.traceId,
      spanId: span.spanId,
      duration_ms: durationMs,
      status: span.status,
      events_count: span.events.length,
    });

    this.activeSpans.delete(span.spanId);
  }
}

export const tracer = new Tracer();

第八部分：组装——弹性中间件

将所有弹性组件组合成一个统一的中间件包装器。创建 src/resilience/middleware.ts：

import { CircuitBreaker } from './circuit-breaker.js';
import { withRetry } from './retry.js';
import { RateLimiter } from './rate-limiter.js';
import { classifyError, formatErrorForLLM, MCPError } from './errors.js';
import { logger } from '../observability/logger.js';
import { metrics } from '../observability/metrics.js';
import { tracer, Span } from '../observability/tracer.js';

export interface ResilienceConfig {
  circuitBreaker?: {
    failureThreshold?: number;
    resetTimeoutMs?: number;
  };
  retry?: {
    maxAttempts?: number;
    baseDelayMs?: number;
  };
  rateLimiter?: {
    maxTokens?: number;
    refillRate?: number;
  };
}

// 为每个外部依赖创建独立的熔断器
const circuitBreakers = new Map<string, CircuitBreaker>();
const rateLimiter = new RateLimiter({
  maxTokens: 20,
  refillRate: 5,
});

function getCircuitBreaker(name: string, config: ResilienceConfig): CircuitBreaker {
  if (!circuitBreakers.has(name)) {
    circuitBreakers.set(name, new CircuitBreaker(name, {
      failureThreshold: config.circuitBreaker?.failureThreshold ?? 5,
      resetTimeoutMs: config.circuitBreaker?.resetTimeoutMs ?? 30000,
    }, (from, to) => {
      logger.warn({
        message: `熔断器状态变更: ${name}`,
        circuit_breaker: name,
        from_state: from,
        to_state: to,
      });
    }));
  }
  return circuitBreakers.get(name)!;
}

// 弹性执行中间件——核心包装器
export async function resilientExecute<T>(
  toolName: string,
  fn: () => Promise<T>,
  config: ResilienceConfig = {}
): Promise<T> {
  const span = tracer.startSpan(toolName);
  const startTime = Date.now();
  const toolLog = logger.toolLogger(toolName);

  // 第1层：速率限制
  const rateCheck = rateLimiter.consume(toolName);
  if (!rateCheck.allowed) {
    logger.warn({
      tool: toolName,
      message: '速率限制触发',
      retry_after_ms: rateCheck.retryAfterMs,
    });
    metrics.increment('rate_limited', { tool: toolName });
    tracer.endSpan(span, 'error');
    throw new Error(
      `[${toolName}] 请求过于频繁，请等待${Math.ceil(rateCheck.retryAfterMs / 1000)}秒后重试`
    );
  }

  try {
    // 第2层：熔断器
    const breaker = getCircuitBreaker(toolName, config);

    // 第3层：重试
    const result = await breaker.execute(() =>
      withRetry(fn, {
        maxAttempts: config.retry?.maxAttempts ?? 3,
        baseDelayMs: config.retry?.baseDelayMs ?? 1000,
      })
    );

    const duration = Date.now() - startTime;
    toolLog.success(duration);
    metrics.recordDuration('tool_duration', duration, { tool: toolName, status: 'success' });
    metrics.increment('tool_success', { tool: toolName });
    tracer.endSpan(span, 'ok');

    return result;
  } catch (error) {
    const duration = Date.now() - startTime;
    const mcpError = classifyError(error);

    toolLog.failure(duration, error);
    metrics.recordDuration('tool_duration', duration, { tool: toolName, status: 'error' });
    metrics.increment('tool_error', { tool: toolName, error_code: mcpError.code });
    tracer.endSpan(span, 'error');

    // 返回LLM友好的错误信息
    throw new Error(formatErrorForLLM(mcpError));
  }
}

// 暴露健康检查端点
export function getHealthStatus() {
  const breakers: Record<string, unknown> = {};
  for (const [name, cb] of circuitBreakers) {
    breakers[name] = cb.getStats();
  }
  return {
    circuit_breakers: breakers,
    metrics: metrics.getSummary(),
  };
}

弹性执行流程

flowchart TD
    A[resilientExecute 入口] --> B{速率限制检查}
    B -->|被限制| C[返回限流错误]
    B -->|通过| D{熔断器状态?}

    D -->|Open| E[直接返回降级/错误]
    D -->|Closed| F[执行 withRetry]
    D -->|Half-Open| G{探测请求?}

    G -->|是| F
    G -->|否| E

    F --> H{调用成功?}
    H -->|是| I[记录指标 + 返回结果]
    H -->|否| J[错误分类]
    J --> K{可重试?}
    K -->|是,有重试余量| L[退避等待]
    L --> F
    K -->|否| M[格式化LLM错误 + 抛出]

    style C fill:#ff6b6b,color:#fff
    style E fill:#ff6b6b,color:#fff
    style I fill:#4ecdc4,color:#fff
    style M fill:#ffe66d,color:#333

第九部分：实战案例——弹性天气查询MCP Server

import { z } from 'zod';
import { resilientExecute } from '../resilience/middleware.js';

// 天气API响应缓存（简易内存缓存）
const weatherCache = new Map<string, { data: unknown; expiresAt: number }>();

export const weatherTool = {
  name: 'get_weather',
  description: '查询指定城市的当前天气（带弹性保护和缓存）',
  inputSchema: {
    type: 'object' as const,
    properties: {
      city: {
        type: 'string',
        description: '城市名称（英文），如 London, Tokyo, Beijing',
      },
      use_cache: {
        type: 'boolean',
        description: '是否使用缓存数据（默认true）',
      },
    },
    required: ['city'],
  },
  execute: async (input: { city: string; use_cache?: boolean }) => {
    return resilientExecute(
      'get_weather',
      async () => {
        const city = input.city.toLowerCase();
        const useCache = input.use_cache !== false;

        // 检查缓存
        if (useCache) {
          const cached = weatherCache.get(city);
          if (cached && cached.expiresAt > Date.now()) {
            return {
              ...cached.data,
              _meta: { from_cache: true, cached_until: new Date(cached.expiresAt).toISOString() },
            };
          }
        }

        // 调用外部API（这里用wttr.in作为示例）
        const controller = new AbortController();
        const timeout = setTimeout(() => controller.abort(), 10000); // 10秒超时

        try {
          const response = await fetch(
            `https://wttr.in/${encodeURIComponent(city)}?format=j1`,
            { signal: controller.signal }
          );

          if (!response.ok) {
            throw response; // 让classifyError处理HTTP状态码
          }

          const data = await response.json();

          // 缓存5分钟
          weatherCache.set(city, {
            data: {
              city: input.city,
              temperature: data.current_condition?.[0]?.temp_C + '°C',
              description: data.current_condition?.[0]?.weatherDesc?.[0]?.value,
              humidity: data.current_condition?.[0]?.humidity + '%',
              wind_speed: data.current_condition?.[0]?.windspeedKmph + ' km/h',
            },
            expiresAt: Date.now() + 5 * 60 * 1000,
          });

          return weatherCache.get(city)!.data;
        } finally {
          clearTimeout(timeout);
        }
      },
      {
        circuitBreaker: {
          failureThreshold: 3,  // 3次失败就熔断
          resetTimeoutMs: 60000, // 60秒冷却
        },
        retry: {
          maxAttempts: 2,
          baseDelayMs: 2000,
        },
      }
    );
  },
};

#!/usr/bin/env node
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
import { weatherTool } from './tools/weather.js';
import { getHealthStatus } from './resilience/middleware.js';
import { logger } from './observability/logger.js';

const server = new McpServer({
  name: 'resilience-weather-server',
  version: '1.0.0',
});

// 注册天气查询Tool
server.tool(
  weatherTool.name,
  weatherTool.description,
  weatherTool.inputSchema,
  async (input) => {
    try {
      const result = await weatherTool.execute(input);
      return {
        content: [{ type: 'text', text: JSON.stringify(result, null, 2) }],
      };
    } catch (error) {
      return {
        content: [{
          type: 'text',
          text: error instanceof Error ? error.message : 'Unknown error',
        }],
        isError: true,
      };
    }
  }
);

// 注册健康检查Tool
server.tool(
  'health_check',
  '检查MCP Server的弹性组件状态（熔断器、指标等）',
  {},
  async () => {
    const health = getHealthStatus();
    return {
      content: [{ type: 'text', text: JSON.stringify(health, null, 2) }],
    };
  }
);

// 启动
async function main() {
  const transport = new StdioServerTransport();
  await server.connect(transport);
  logger.info({ message: '弹性MCP Server已启动' });
}

main().catch((error) => {
  logger.error({ message: '启动失败', error: String(error) });
  process.exit(1);
});

{
  "name": "mcp-resilience-server",
  "type": "module",
  "scripts": {
    "start": "tsx src/index.ts",
    "dev": "tsx watch src/index.ts"
  }
}

第十部分：连接Claude Code并测试弹性策略

注册MCP Server

编辑 ~/.claude/settings.json（全局）或项目下的 .claude/settings.json：

{
  "mcpServers": {
    "resilience-weather": {
      "command": "npx",
      "args": ["tsx", "/绝对路径/mcp-resilience-server/src/index.ts"],
      "env": {}
    }
  }
}

弹性测试场景

# 在Claude Code中直接对话
claude
> 查一下东京的天气

const timeout = setTimeout(() => controller.abort(), 1);

预期行为： 1. 第1次调用超时 → 错误分类为 TRANSIENT 2. 退避后重试 → 再次超时 3. 达到 maxAttempts → 返回LLM友好错误 4. 连续3次后 → 熔断器打开

> 使用health_check工具查看状态

{
  "circuit_breakers": {
    "get_weather": {
      "name": "get_weather",
      "state": "open",
      "failureCount": 5,
      "successCount": 0
    }
  },
  "metrics": {
    "tool_duration": {
      "count": 5,
      "avg_ms": 10203,
      "p95_ms": 10250,
      "p99_ms": 10250
    }
  }
}

// test-rate-limit.ts
import { RateLimiter } from './src/resilience/rate-limiter.js';

const limiter = new RateLimiter({ maxTokens: 3, refillRate: 1 });

for (let i = 0; i < 10; i++) {
  const result = limiter.consume('test');
  console.log(`请求${i + 1}: ${result.allowed ? '通过' : `被限流，等待${result.retryAfterMs}ms`}`);
}

请求1: 通过
请求2: 通过
请求3: 通过
请求4: 被限流，等待1000ms
请求5: 被限流，等待1000ms
...

第十一部分：进阶——带降级策略的Tool

熔断器打开后，直接报错是最简单的处理。更好的方式是返回降级数据。修改 weather.ts：

// 降级策略：当API不可用时返回基础信息
function getFallbackWeather(city: string) {
  // 简化的城市气候数据
  const climateData: Record<string, { temp_range: string; typical: string }> = {
    tokyo: { temp_range: '5-35°C', typical: '四季分明，夏季湿热' },
    london: { temp_range: '2-25°C', typical: '温和湿润，多阴雨天' },
    beijing: { temp_range: '-10-40°C', typical: '大陆性气候，春秋短' },
    new_york: { temp_range: '-5-35°C', typical: '温带大陆性气候' },
  };

  const normalized = city.toLowerCase().replace(/\s+/g, '_');
  const data = climateData[normalized];

  return {
    city,
    temperature: data ? data.temp_range : '未知',
    description: data ? data.typical : '无气候数据',
    _meta: {
      degraded: true,
      reason: '天气API不可用，返回气候概要数据',
      data_source: 'fallback_climate_db',
    },
  };
}

// 在 resilientExecute 的catch中添加降级逻辑
export const weatherToolWithFallback = {
  // ... 同上的name, description, inputSchema
  execute: async (input: { city: string; use_cache?: boolean }) => {
    try {
      return await resilientExecute('get_weather', async () => {
        // ... 同上的API调用逻辑
      }, { /* config */ });
    } catch (error) {
      // 当熔断器打开或所有重试耗尽时，返回降级数据
      const errMsg = error instanceof Error ? error.message : '';
      if (errMsg.includes('熔断器已打开') || errMsg.includes('UPSTREAM')) {
        return getFallbackWeather(input.city);
      }
      throw error;
    }
  },
};

降级策略决策树

graph TD
    A[Tool调用请求] --> B{缓存命中?}
    B -->|是| C[返回缓存数据]
    B -->|否| D{熔断器状态?}
    D -->|Closed| E[调用外部API]
    D -->|Open| F{有本地降级数据?}
    D -->|Half-Open| G[尝试调用]

    E -->|成功| H[更新缓存 + 返回]
    E -->|失败| I{可重试?}
    I -->|是| J[退避后重试]
    I -->|否| F
    J -->|成功| H
    J -->|耗尽| F

    F -->|是| K[返回降级数据 + 标记degraded]
    F -->|否| L[返回LLM友好错误]

    G -->|成功| M[恢复熔断器 + 返回]
    G -->|失败| N[重新熔断 + F]

    style K fill:#ffe66d,color:#333
    style C fill:#4ecdc4,color:#fff
    style H fill:#4ecdc4,color:#fff
    style L fill:#ff6b6b,color:#fff

第十二部分：OpenTelemetry集成（生产级可观测性）

对于需要接入Grafana/Jaeger等可观测性平台的生产环境，可以用OpenTelemetry替代简易Tracer：

npm install @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node

import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { Resource } from '@opentelemetry/resources';
import { ATTR_SERVICE_NAME } from '@opentelemetry/semantic-conventions';

// 仅在配置了OTEL环境变量时启用
const otelEndpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT;

if (otelEndpoint) {
  const sdk = new NodeSDK({
    resource: new Resource({
      [ATTR_SERVICE_NAME]: 'mcp-resilience-server',
    }),
    traceExporter: new OTLPTraceExporter({
      url: `${otelEndpoint}/v1/traces`,
    }),
  });

  sdk.start();
  console.error(`[OTEL] 已连接到 ${otelEndpoint}`);

  process.on('SIGTERM', () => {
    sdk.shutdown().then(() => process.exit(0));
  });
}

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 npm start

这样所有Tool调用的Span、耗时、错误都会自动发送到你的可观测性平台。

第十三部分：弹性策略配置矩阵

外部API类型	失败阈值	冷却时间	重试次数	基础退避	速率限制
公共免费API (wttr.in)	3	60s	2	2s	10次/分
付费API (OpenAI)	5	30s	3	1s	按配额
内部微服务	3	15s	3	500ms	100次/分
数据库查询	2	10s	2	200ms	50次/分
文件系统操作	1	5s	1	100ms	不限

配置原则： - 免费API：低重试、长冷却，避免被封禁 - 付费API：高重试、短冷却，充分利用已付费的配额 - 内部服务：快速重试，因为网络环境更可控 - 数据库：极少重试，连接失败通常不是瞬时问题

第十四部分：测试弹性策略

单元测试

import { describe, it, expect, vi } from 'vitest';
import { CircuitBreaker } from '../resilience/circuit-breaker.js';
import { withRetry } from '../resilience/retry.js';
import { RateLimiter } from '../resilience/rate-limiter.js';
import { classifyError, ErrorSeverity } from '../resilience/errors.js';

describe('CircuitBreaker', () => {
  it('应在达到阈值时打开', async () => {
    const cb = new CircuitBreaker('test', {
      failureThreshold: 2,
      resetTimeoutMs: 5000,
    });

    // 连续失败2次
    for (let i = 0; i < 2; i++) {
      try {
        await cb.execute(() => { throw new Error('fail'); });
      } catch {}
    }

    const stats = cb.getStats();
    expect(stats.state).toBe('open');
  });

  it('应在冷却后进入半开状态', async () => {
    const cb = new CircuitBreaker('test', {
      failureThreshold: 1,
      resetTimeoutMs: 100, // 100ms快速冷却
    });

    try {
      await cb.execute(() => { throw new Error('fail'); });
    } catch {}

    // 等待冷却
    await new Promise(r => setTimeout(r, 150));

    expect(cb.getCurrentState()).toBe('half-open');
  });
});

describe('RateLimiter', () => {
  it('应在令牌耗尽时拒绝请求', () => {
    const limiter = new RateLimiter({ maxTokens: 2, refillRate: 1 });

    expect(limiter.consume().allowed).toBe(true);
    expect(limiter.consume().allowed).toBe(true);
    expect(limiter.consume().allowed).toBe(false);
  });
});

describe('withRetry', () => {
  it('应在成功时不重试', async () => {
    const fn = vi.fn().mockResolvedValue('ok');
    const result = await withRetry(fn);
    expect(result).toBe('ok');
    expect(fn).toHaveBeenCalledTimes(1);
  });

  it('应在永久错误时不重试', async () => {
    const fn = vi.fn().mockRejectedValue(new TypeError('invalid input'));
    await expect(withRetry(fn)).rejects.toThrow();
    expect(fn).toHaveBeenCalledTimes(1);
  });
});

npm install -D vitest
npx vitest run

完整架构总览

graph TB
    subgraph "Claude Code 客户端"
        CC[Claude Code CLI]
    end

    subgraph "MCP Resilience Server"
        direction TB
        IN[入口 index.ts] --> RL[速率限制器]
        RL --> CB[熔断器集群]
        CB --> RT[重试+退避引擎]
        RT --> EC[错误分类器]
        EC --> LG[结构化日志]
        EC --> MT[指标采集]
        EC --> TR[请求追踪]

        subgraph "Tools"
            TW[天气查询]
            TX[搜索工具]
            HC[健康检查]
        end

        RT --> TW
        RT --> TX
        HC --> CB
        HC --> MT
    end

    subgraph "外部依赖"
        API1[wttr.in API]
        API2[搜索API]
        OTEL[OpenTelemetry Collector]
    end

    TW --> API1
    TX --> API2
    LG --> OTEL
    TR --> OTEL
    MT --> OTEL

    CC <-->|stdio JSON-RPC| IN

    style CC fill:#6c5ce7,color:#fff
    style RL fill:#fdcb6e,color:#333
    style CB fill:#e17055,color:#fff
    style RT fill:#00b894,color:#fff
    style EC fill:#ffeaa7,color:#333
    style OTEL fill:#74b9ff,color:#333

关键经验总结

六个不要

六个必须

参考来源

训练营适用性评估：本教程适合中高级开发者训练营（Day 2-3内容），学员需已掌握MCP Server基础开发。建议教学时长 3-4小时（含动手实验2小时），其中熔断器实现和降级策略设计是最有价值的实操环节。