Adding skills
This commit is contained in:
@@ -0,0 +1,251 @@
|
||||
# MCP Server Evaluation Guide
|
||||
|
||||
Evaluate whether models can discover and use an MCP server safely and efficiently. Protocol correctness is necessary, but a technically valid tool can still be difficult for a model to select, call, or interpret.
|
||||
|
||||
## Layers of verification
|
||||
|
||||
Use four complementary layers:
|
||||
|
||||
1. **Unit tests** verify schemas, API-client behavior, response shaping, and errors.
|
||||
2. **Protocol tests** verify MCP initialization, tool listing, invocation, transports, and cleanup.
|
||||
3. **Deterministic scenarios** verify known tool calls against stable fixtures or test services.
|
||||
4. **Harness-native agent review** evaluates discoverability and multi-step model behavior without adding a provider-specific runner to the skill.
|
||||
|
||||
Do not replace build, lint, typecheck, security checks, or protocol tests with an LLM review.
|
||||
|
||||
## 1. Unit and integration tests
|
||||
|
||||
Test each tool at the boundary where it becomes deterministic.
|
||||
|
||||
### Inputs
|
||||
|
||||
Cover:
|
||||
|
||||
- smallest valid input
|
||||
- realistic valid input
|
||||
- missing required fields
|
||||
- wrong types and invalid enum values
|
||||
- minimum and maximum lengths or numbers
|
||||
- malformed identifiers, dates, and URLs
|
||||
- unknown fields when strict schemas are intended
|
||||
- attempts to access disallowed hosts or resources
|
||||
|
||||
### Upstream behavior
|
||||
|
||||
Cover:
|
||||
|
||||
- successful responses
|
||||
- empty results
|
||||
- pagination and continuation tokens
|
||||
- rate limits and retry hints
|
||||
- authentication and authorization failures
|
||||
- upstream validation failures
|
||||
- timeouts, cancellation, and partial failures
|
||||
- malformed or unexpectedly large upstream payloads
|
||||
|
||||
### Outputs
|
||||
|
||||
Assert:
|
||||
|
||||
- `structuredContent` matches the declared output schema
|
||||
- text content is concise and consistent with structured content
|
||||
- pagination metadata is sufficient to continue
|
||||
- secrets and sensitive fields are absent
|
||||
- large results are bounded or truncated explicitly
|
||||
- errors tell the model what it can change or try next
|
||||
|
||||
## 2. Protocol testing with MCP Inspector
|
||||
|
||||
Use the official Inspector against the server under test:
|
||||
|
||||
```bash
|
||||
npx @modelcontextprotocol/inspector
|
||||
```
|
||||
|
||||
Verify:
|
||||
|
||||
- initialization completes
|
||||
- advertised capabilities are accurate
|
||||
- tool names and descriptions are visible
|
||||
- input schemas render correctly
|
||||
- each tool can be invoked
|
||||
- errors remain valid MCP responses
|
||||
- cancellation and shutdown release resources
|
||||
|
||||
For stdio servers, confirm that stdout contains only protocol messages. Logs belong on stderr.
|
||||
|
||||
For Streamable HTTP, also confirm:
|
||||
|
||||
- origin and host validation
|
||||
- authentication on every request
|
||||
- session behavior, if used
|
||||
- CORS policy, if browser access is intended
|
||||
- request size and timeout limits
|
||||
- DNS rebinding protections
|
||||
|
||||
Record the Inspector version and server command or endpoint with the results.
|
||||
|
||||
## 3. Scenario specification
|
||||
|
||||
Store scenarios as JSON so they can be parsed safely by project tests and reused by different harnesses.
|
||||
|
||||
Example:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"scenarios": [
|
||||
{
|
||||
"id": "find-open-critical-issues",
|
||||
"goal": "Find open critical issues assigned to the current user and return their titles in updated order.",
|
||||
"fixture": "fixtures/issues.json",
|
||||
"expected": {
|
||||
"answer": ["Database failover", "Expired signing key"],
|
||||
"requiredTools": ["gitea_list_issues"],
|
||||
"forbiddenTools": ["gitea_update_issue", "gitea_close_issue"],
|
||||
"maxToolCalls": 3
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
A scenario should define:
|
||||
|
||||
- stable identifier
|
||||
- realistic user goal
|
||||
- fixed fixture or immutable test environment
|
||||
- expected answer or objective assertions
|
||||
- tools that should or must not be used
|
||||
- maximum reasonable tool-call count
|
||||
- safety constraints
|
||||
|
||||
Avoid answers based on live counts, current popularity, or mutable public data. Prefer frozen fixtures, seeded test accounts, or fixed historical ranges.
|
||||
|
||||
## 4. Designing useful scenarios
|
||||
|
||||
Create scenarios that collectively test:
|
||||
|
||||
- obvious tool selection
|
||||
- ambiguous requests where descriptions must disambiguate tools
|
||||
- filtering and pagination
|
||||
- multi-step lookup and join behavior
|
||||
- recovery from a validation or upstream error
|
||||
- empty results
|
||||
- large-result handling
|
||||
- authentication boundaries
|
||||
- read-only requests that must never invoke mutating tools
|
||||
- intentional mutation with confirmation or explicit user intent
|
||||
|
||||
Each scenario should resemble a task a real user would delegate. Do not expose internal tool names in the user prompt unless the scenario specifically tests direct tool selection.
|
||||
|
||||
A good scenario requires reasoning but has objective success criteria. A poor scenario is either trivial (“list tools”) or impossible to grade consistently (“explore the data and tell me something interesting”).
|
||||
|
||||
## 5. Harness-native agent review
|
||||
|
||||
Run qualitative scenarios through an isolated agent or harness session configured with only the MCP server and essential filesystem/test tools. This prevents unrelated tools from masking gaps in the server.
|
||||
|
||||
For each scenario, capture:
|
||||
|
||||
- final answer
|
||||
- ordered tool calls and arguments
|
||||
- relevant tool results
|
||||
- errors and retries
|
||||
- total call count
|
||||
- elapsed time when available
|
||||
- model feedback about confusing names, schemas, or responses
|
||||
|
||||
Grade the trace against the scenario's objective assertions. Do not rely only on the model's self-reported success.
|
||||
|
||||
When comparing revisions, use the same scenario set, fixtures, model route, and tool permissions. Compare:
|
||||
|
||||
- task success rate
|
||||
- median calls per successful task
|
||||
- validation-error rate
|
||||
- repeated or unnecessary calls
|
||||
- unsafe mutation attempts
|
||||
- response volume and truncation
|
||||
|
||||
## 6. Review rubric
|
||||
|
||||
### Discoverability
|
||||
|
||||
- Does the model choose the intended tool without being told its name?
|
||||
- Are similar tools distinguishable by name and description?
|
||||
- Are prerequisites and side effects clear?
|
||||
|
||||
### Schema usability
|
||||
|
||||
- Are required fields genuinely required?
|
||||
- Do descriptions explain formats and constraints?
|
||||
- Do enums and defaults prevent guesswork?
|
||||
- Are nested inputs simpler than equivalent orchestration would be?
|
||||
|
||||
### Response usability
|
||||
|
||||
- Does the response contain the identifiers needed for the next step?
|
||||
- Is structured content stable and bounded?
|
||||
- Can the model paginate without reconstructing hidden state?
|
||||
- Is irrelevant upstream data removed?
|
||||
|
||||
### Error recovery
|
||||
|
||||
- Does an error identify the bad field or failed precondition?
|
||||
- Does it distinguish authentication, authorization, validation, rate limits, and upstream failure?
|
||||
- Does it suggest a safe corrective action?
|
||||
|
||||
### Safety
|
||||
|
||||
- Are read-only and mutating tools clearly separated?
|
||||
- Do annotations match real side effects?
|
||||
- Can broad inputs escape the intended tenant, repository, filesystem, or host boundary?
|
||||
- Are secrets absent from logs and tool results?
|
||||
|
||||
## 7. Evaluation report
|
||||
|
||||
Use a concise Markdown report:
|
||||
|
||||
```markdown
|
||||
# MCP evaluation: <server>
|
||||
|
||||
## Environment
|
||||
- Commit:
|
||||
- SDK version:
|
||||
- Transport:
|
||||
- Inspector version:
|
||||
- Agent/model route:
|
||||
- Fixture version:
|
||||
|
||||
## Summary
|
||||
- Scenarios passed: 8/10
|
||||
- Median tool calls: 3
|
||||
- Validation errors: 2
|
||||
- Unsafe mutation attempts: 0
|
||||
|
||||
## Findings
|
||||
|
||||
### Concern: issue search description hides label semantics
|
||||
|
||||
Evidence: scenarios `find-critical` and `triage-ready` each made two failed calls.
|
||||
Action: document whether labels use intersection or union semantics and add an enum for state.
|
||||
|
||||
## Scenario results
|
||||
|
||||
| Scenario | Result | Calls | Evidence |
|
||||
| --- | --- | ---: | --- |
|
||||
| find-critical | Fail | 5 | Used OR semantics; expected AND |
|
||||
```
|
||||
|
||||
Every finding should connect trace evidence to an actionable schema, description, response, implementation, or documentation change.
|
||||
|
||||
## Completion criteria
|
||||
|
||||
An MCP server is ready when:
|
||||
|
||||
- project checks pass
|
||||
- Inspector confirms protocol behavior
|
||||
- deterministic scenarios pass
|
||||
- qualitative scenarios show reliable tool selection
|
||||
- no unresolved high-risk security or authorization findings remain
|
||||
- failure paths are actionable
|
||||
- evaluation evidence and known limitations are documented
|
||||
Reference in New Issue
Block a user