harness-tools/skills/misc/mcp-builder/references/evaluation.md

# MCP Server Evaluation Guide

Evaluate whether models can discover and use an MCP server safely and efficiently. Protocol correctness is necessary, but a technically valid tool can still be difficult for a model to select, call, or interpret.

## Layers of verification

Use four complementary layers:

1. **Unit tests** verify schemas, API-client behavior, response shaping, and errors.
2. **Protocol tests** verify MCP initialization, tool listing, invocation, transports, and cleanup.
3. **Deterministic scenarios** verify known tool calls against stable fixtures or test services.
4. **Harness-native agent review** evaluates discoverability and multi-step model behavior without adding a provider-specific runner to the skill.

Do not replace build, lint, typecheck, security checks, or protocol tests with an LLM review.

## 1. Unit and integration tests

Test each tool at the boundary where it becomes deterministic.

### Inputs

Cover:

- smallest valid input
- realistic valid input
- missing required fields
- wrong types and invalid enum values
- minimum and maximum lengths or numbers
- malformed identifiers, dates, and URLs
- unknown fields when strict schemas are intended
- attempts to access disallowed hosts or resources

### Upstream behavior

Cover:

- successful responses
- empty results
- pagination and continuation tokens
- rate limits and retry hints
- authentication and authorization failures
- upstream validation failures
- timeouts, cancellation, and partial failures
- malformed or unexpectedly large upstream payloads

### Outputs

Assert:

- `structuredContent` matches the declared output schema
- text content is concise and consistent with structured content
- pagination metadata is sufficient to continue
- secrets and sensitive fields are absent
- large results are bounded or truncated explicitly
- errors tell the model what it can change or try next

## 2. Protocol testing with MCP Inspector

Use the official Inspector against the server under test:

```bash
npx @modelcontextprotocol/inspector
```

Verify:

- initialization completes
- advertised capabilities are accurate
- tool names and descriptions are visible
- input schemas render correctly
- each tool can be invoked
- errors remain valid MCP responses
- cancellation and shutdown release resources

For stdio servers, confirm that stdout contains only protocol messages. Logs belong on stderr.

For Streamable HTTP, also confirm:

- origin and host validation
- authentication on every request
- session behavior, if used
- CORS policy, if browser access is intended
- request size and timeout limits
- DNS rebinding protections

Record the Inspector version and server command or endpoint with the results.

## 3. Scenario specification

Store scenarios as JSON so they can be parsed safely by project tests and reused by different harnesses.

Example:

```json
{
  "version": 1,
  "scenarios": [
    {
      "id": "find-open-critical-issues",
      "goal": "Find open critical issues assigned to the current user and return their titles in updated order.",
      "fixture": "fixtures/issues.json",
      "expected": {
        "answer": ["Database failover", "Expired signing key"],
        "requiredTools": ["gitea_list_issues"],
        "forbiddenTools": ["gitea_update_issue", "gitea_close_issue"],
        "maxToolCalls": 3
      }
    }
  ]
}
```

A scenario should define:

- stable identifier
- realistic user goal
- fixed fixture or immutable test environment
- expected answer or objective assertions
- tools that should or must not be used
- maximum reasonable tool-call count
- safety constraints

Avoid answers based on live counts, current popularity, or mutable public data. Prefer frozen fixtures, seeded test accounts, or fixed historical ranges.

## 4. Designing useful scenarios

Create scenarios that collectively test:

- obvious tool selection
- ambiguous requests where descriptions must disambiguate tools
- filtering and pagination
- multi-step lookup and join behavior
- recovery from a validation or upstream error
- empty results
- large-result handling
- authentication boundaries
- read-only requests that must never invoke mutating tools
- intentional mutation with confirmation or explicit user intent

Each scenario should resemble a task a real user would delegate. Do not expose internal tool names in the user prompt unless the scenario specifically tests direct tool selection.

A good scenario requires reasoning but has objective success criteria. A poor scenario is either trivial (“list tools”) or impossible to grade consistently (“explore the data and tell me something interesting”).

## 5. Harness-native agent review

Run qualitative scenarios through an isolated agent or harness session configured with only the MCP server and essential filesystem/test tools. This prevents unrelated tools from masking gaps in the server.

For each scenario, capture:

- final answer
- ordered tool calls and arguments
- relevant tool results
- errors and retries
- total call count
- elapsed time when available
- model feedback about confusing names, schemas, or responses

Grade the trace against the scenario's objective assertions. Do not rely only on the model's self-reported success.

When comparing revisions, use the same scenario set, fixtures, model route, and tool permissions. Compare:

- task success rate
- median calls per successful task
- validation-error rate
- repeated or unnecessary calls
- unsafe mutation attempts
- response volume and truncation

## 6. Review rubric

### Discoverability

- Does the model choose the intended tool without being told its name?
- Are similar tools distinguishable by name and description?
- Are prerequisites and side effects clear?

### Schema usability

- Are required fields genuinely required?
- Do descriptions explain formats and constraints?
- Do enums and defaults prevent guesswork?
- Are nested inputs simpler than equivalent orchestration would be?

### Response usability

- Does the response contain the identifiers needed for the next step?
- Is structured content stable and bounded?
- Can the model paginate without reconstructing hidden state?
- Is irrelevant upstream data removed?

### Error recovery

- Does an error identify the bad field or failed precondition?
- Does it distinguish authentication, authorization, validation, rate limits, and upstream failure?
- Does it suggest a safe corrective action?

### Safety

- Are read-only and mutating tools clearly separated?
- Do annotations match real side effects?
- Can broad inputs escape the intended tenant, repository, filesystem, or host boundary?
- Are secrets absent from logs and tool results?

## 7. Evaluation report

Use a concise Markdown report:

```markdown
# MCP evaluation: <server>

## Environment
- Commit:
- SDK version:
- Transport:
- Inspector version:
- Agent/model route:
- Fixture version:

## Summary
- Scenarios passed: 8/10
- Median tool calls: 3
- Validation errors: 2
- Unsafe mutation attempts: 0

## Findings

### Concern: issue search description hides label semantics

Evidence: scenarios `find-critical` and `triage-ready` each made two failed calls.
Action: document whether labels use intersection or union semantics and add an enum for state.

## Scenario results

| Scenario | Result | Calls | Evidence |
| --- | --- | ---: | --- |
| find-critical | Fail | 5 | Used OR semantics; expected AND |
```

Every finding should connect trace evidence to an actionable schema, description, response, implementation, or documentation change.

## Completion criteria

An MCP server is ready when:

- project checks pass
- Inspector confirms protocol behavior
- deterministic scenarios pass
- qualitative scenarios show reliable tool selection
- no unresolved high-risk security or authorization findings remain
- failure paths are actionable
- evaluation evidence and known limitations are documented