harness-tools/skills/misc/mcp-builder/references/evaluation.md

MCP Server Evaluation Guide

Evaluate whether models can discover and use an MCP server safely and efficiently. Protocol correctness is necessary, but a technically valid tool can still be difficult for a model to select, call, or interpret.

Layers of verification

Use four complementary layers:

1. Unit tests verify schemas, API-client behavior, response shaping, and errors.
2. Protocol tests verify MCP initialization, tool listing, invocation, transports, and cleanup.
3. Deterministic scenarios verify known tool calls against stable fixtures or test services.
4. Harness-native agent review evaluates discoverability and multi-step model behavior without adding a provider-specific runner to the skill.

Do not replace build, lint, typecheck, security checks, or protocol tests with an LLM review.

1. Unit and integration tests

Test each tool at the boundary where it becomes deterministic.

Inputs

Cover:

• smallest valid input
• realistic valid input
• missing required fields
• wrong types and invalid enum values
• minimum and maximum lengths or numbers
• malformed identifiers, dates, and URLs
• unknown fields when strict schemas are intended
• attempts to access disallowed hosts or resources

Upstream behavior

Cover:

• successful responses
• empty results
• pagination and continuation tokens
• rate limits and retry hints
• authentication and authorization failures
• upstream validation failures
• timeouts, cancellation, and partial failures
• malformed or unexpectedly large upstream payloads

Outputs

Assert:

• structuredContent matches the declared output schema
• text content is concise and consistent with structured content
• pagination metadata is sufficient to continue
• secrets and sensitive fields are absent
• large results are bounded or truncated explicitly
• errors tell the model what it can change or try next

2. Protocol testing with MCP Inspector

Use the official Inspector against the server under test:

npx @modelcontextprotocol/inspector

Verify:

• initialization completes
• advertised capabilities are accurate
• tool names and descriptions are visible
• input schemas render correctly
• each tool can be invoked
• errors remain valid MCP responses
• cancellation and shutdown release resources

For stdio servers, confirm that stdout contains only protocol messages. Logs belong on stderr.

For Streamable HTTP, also confirm:

• origin and host validation
• authentication on every request
• session behavior, if used
• CORS policy, if browser access is intended
• request size and timeout limits
• DNS rebinding protections

Record the Inspector version and server command or endpoint with the results.

3. Scenario specification

Store scenarios as JSON so they can be parsed safely by project tests and reused by different harnesses.

Example:

{
  "version": 1,
  "scenarios": [
    {
      "id": "find-open-critical-issues",
      "goal": "Find open critical issues assigned to the current user and return their titles in updated order.",
      "fixture": "fixtures/issues.json",
      "expected": {
        "answer": ["Database failover", "Expired signing key"],
        "requiredTools": ["gitea_list_issues"],
        "forbiddenTools": ["gitea_update_issue", "gitea_close_issue"],
        "maxToolCalls": 3
      }
    }
  ]
}

A scenario should define:

• stable identifier
• realistic user goal
• fixed fixture or immutable test environment
• expected answer or objective assertions
• tools that should or must not be used
• maximum reasonable tool-call count
• safety constraints

Avoid answers based on live counts, current popularity, or mutable public data. Prefer frozen fixtures, seeded test accounts, or fixed historical ranges.

4. Designing useful scenarios

Create scenarios that collectively test:

• obvious tool selection
• ambiguous requests where descriptions must disambiguate tools
• filtering and pagination
• multi-step lookup and join behavior
• recovery from a validation or upstream error
• empty results
• large-result handling
• authentication boundaries
• read-only requests that must never invoke mutating tools
• intentional mutation with confirmation or explicit user intent

Each scenario should resemble a task a real user would delegate. Do not expose internal tool names in the user prompt unless the scenario specifically tests direct tool selection.

A good scenario requires reasoning but has objective success criteria. A poor scenario is either trivial (“list tools”) or impossible to grade consistently (“explore the data and tell me something interesting”).

5. Harness-native agent review

Run qualitative scenarios through an isolated agent or harness session configured with only the MCP server and essential filesystem/test tools. This prevents unrelated tools from masking gaps in the server.

For each scenario, capture:

• final answer
• ordered tool calls and arguments
• relevant tool results
• errors and retries
• total call count
• elapsed time when available
• model feedback about confusing names, schemas, or responses

Grade the trace against the scenario's objective assertions. Do not rely only on the model's self-reported success.

When comparing revisions, use the same scenario set, fixtures, model route, and tool permissions. Compare:

• task success rate
• median calls per successful task
• validation-error rate
• repeated or unnecessary calls
• unsafe mutation attempts
• response volume and truncation

6. Review rubric

Discoverability

• Does the model choose the intended tool without being told its name?
• Are similar tools distinguishable by name and description?
• Are prerequisites and side effects clear?

Schema usability

• Are required fields genuinely required?
• Do descriptions explain formats and constraints?
• Do enums and defaults prevent guesswork?
• Are nested inputs simpler than equivalent orchestration would be?

Response usability

• Does the response contain the identifiers needed for the next step?
• Is structured content stable and bounded?
• Can the model paginate without reconstructing hidden state?
• Is irrelevant upstream data removed?

Error recovery

• Does an error identify the bad field or failed precondition?
• Does it distinguish authentication, authorization, validation, rate limits, and upstream failure?
• Does it suggest a safe corrective action?

Safety

• Are read-only and mutating tools clearly separated?
• Do annotations match real side effects?
• Can broad inputs escape the intended tenant, repository, filesystem, or host boundary?
• Are secrets absent from logs and tool results?

7. Evaluation report

Use a concise Markdown report:

# MCP evaluation: <server>

## Environment
- Commit:
- SDK version:
- Transport:
- Inspector version:
- Agent/model route:
- Fixture version:

## Summary
- Scenarios passed: 8/10
- Median tool calls: 3
- Validation errors: 2
- Unsafe mutation attempts: 0

## Findings

### Concern: issue search description hides label semantics

Evidence: scenarios `find-critical` and `triage-ready` each made two failed calls.
Action: document whether labels use intersection or union semantics and add an enum for state.

## Scenario results

| Scenario | Result | Calls | Evidence |
| --- | --- | ---: | --- |
| find-critical | Fail | 5 | Used OR semantics; expected AND |

Every finding should connect trace evidence to an actionable schema, description, response, implementation, or documentation change.

Completion criteria

An MCP server is ready when:

• project checks pass
• Inspector confirms protocol behavior
• deterministic scenarios pass
• qualitative scenarios show reliable tool selection
• no unresolved high-risk security or authorization findings remain
• failure paths are actionable
• evaluation evidence and known limitations are documented