Performance Tests

This document describes how to run Wanaku performance tests. All test infrastructure lives under tests/load/.

Prerequisites

Java 21+ (25 recommended)
Maven 3.9+
k6 with the xk6-mcp extension
Podman (for Keycloak)
Python 3 (for report generation)

Ensure k6 is on your PATH or set K6_BIN to its location.

Architecture Overview

There are two test paths, each targeting a different bridge type:

text

                          ┌─────────────────────────────────┐
                          │         Wanaku Router            │
  k6 ──SSE──►  /public/mcp/sse  ──┬── gRPC bridge ──► capability (tool-noop / static-file)
                                   └── MCP bridge  ──► mock MCP server (SSE forward)
                          └─────────────────────────────────┘

Bridge	Backend	Tool name	Resource URI	Capability module
gRPC	`wanaku-tool-performance-noop`, `wanaku-provider-performance-static-file`	`performancenoop`	`in-memory-file.txt`	`capabilities/tools/wanaku-tool-performance-noop`, `capabilities/providers/wanaku-provider-performance-static-file`
MCP (forward)	`wanaku-performance-test-mock-mcp`	`mockTool`	`file:///mock/data`	`tests/mcp-servers/wanaku-performance-test-mock-mcp`

Quick Reference

File Layout

text

tests/load/
├── run-perf-test.sh          # Single-run test runner (any bridge)
├── run-perf-evaluation.sh    # Full baseline-vs-patched evaluation (gRPC bridge, CI-based)
├── generate-perf-report.py   # Comparison report generator
├── mcp-tools-invoke-sse.js   # k6 script: tool invocation via SSE
└── mcp-resources-read-sse.js # k6 script: resource read via SSE

tests/mcp-servers/wanaku-performance-test-mock-mcp/   # Mock MCP server for MCP bridge tests

Key Environment Variables

Variable	Default	Description
`K6_BIN`	`$HOME/bin/k6`	Path to k6 binary
`WANAKU_BIN`	`$HOME/bin/wanaku`	Path to wanaku CLI
`JAVA_OPTS`	`-XX:+UseNUMA -Xmx4G -Xms4G`	JVM options for router and capabilities
`EVAL_DIR`	`$HOME/perf-evaluation-<timestamp>`	Output directory for evaluation results
`TEST_SCOPE`	`all`	`all`, `tools`, or `resources`
`SKIP_BASELINE`	`false`	Skip the baseline run in evaluations
`SKIP_PATCHED`	`false`	Skip the patched run in evaluations
`SKIP_BUILD`	`false`	Skip the Maven build step
`BASELINE_BRANCH`	`main`	Branch to use as baseline

Test Scenarios

1. gRPC Bridge Tests (via `run-perf-test.sh`)

Tests the gRPC bridge path using standalone capability providers. Requires pre-built distribution archives.

Build

bash

mvn package -Pdist -DskipTests -T1C -q

This produces .tar.gz archives under each module's target/distributions/ directory.

Run

bash

tests/load/run-perf-test.sh \
  --router-from apps/wanaku-router-backend/target/distributions/wanaku-router-backend-0.2.0-SNAPSHOT.tar.gz \
  --capability-from capabilities/tools/wanaku-tool-performance-noop/target/distributions/wanaku-tool-performance-noop-0.2.0-SNAPSHOT.tar.gz \
  --suite tools-sse \
  --test-name my-run \
  --test-base-dir /tmp/perf-results

The script handles Keycloak startup, credential acquisition, router launch, capability registration, and k6 execution at VU levels 1, 10, 500, 1000, 2000, 30000.

Available suites: sse (both), tools-sse, resources-sse.

You can also pass custom k6 scripts with --test NAME PATH:

bash

--test my-custom-test ./my-script.js

Artifacts

run-perf-test.sh accepts --router-from and --capability-from as either local paths or HTTP URLs. This allows testing against CI-built artifacts directly.

2. MCP Bridge Tests (via mock MCP server)

Tests the MCP bridge path where the router forwards requests to a remote MCP server over SSE.

Build

bash

mvn package -pl apps/wanaku-router-backend,tests/mcp-servers/wanaku-performance-test-mock-mcp -am -DskipTests -T1C -q

Run Manually

Start the components in this order:

bash

# 1. Ensure Keycloak is running (port 8543)
podman run -d --name keycloak --rm -p 0.0.0.0:8543:8080 \
  -e KC_BOOTSTRAP_ADMIN_USERNAME=admin \
  -e KC_BOOTSTRAP_ADMIN_PASSWORD=admin \
  -v keycloak-dev:/opt/keycloak/data \
  quay.io/keycloak/keycloak:26.6.1 start-dev

# 2. Start the router
java -XX:+UseNUMA -Xmx4G -Xms4G \
  -Dquarkus.http.host=0.0.0.0 \
  -Dauth.server="http://$(hostname -f):8543" \
  -jar apps/wanaku-router-backend/target/quarkus-app/quarkus-run.jar &

# 3. Wait for router to be ready
until curl -fsSo /dev/null http://localhost:8080 2>/dev/null; do sleep 2; done

# 4. Start the mock MCP server
java -XX:+UseNUMA -Xmx1G -Xms1G \
  -Dwanaku.service.registration.uri="http://localhost:8080" \
  -Dwanaku.service.performance.delay=0 \
  -Dwanaku.mcp.service.namespace=public \
  -jar tests/mcp-servers/wanaku-performance-test-mock-mcp/target/quarkus-app/quarkus-run.jar &

# 5. Wait for registration (~15 seconds)
sleep 15

# 6. Verify registration
curl -s http://localhost:8080/api/v1/forwards/list | python3 -m json.tool

# 7. Run k6
k6 run --vus 10 --duration 30s tests/load/mcp-tools-invoke-sse.js
k6 run --vus 10 --duration 30s tests/load/mcp-resources-read-sse.js

Mock MCP Server Configuration

Property	Default	Description
`wanaku.service.performance.delay`	`100`	Artificial delay in ms added to each tool/resource response. Set to `0` for pure throughput tests.
`wanaku.mcp.service.namespace`	`test`	Namespace the forward registers under. Set to `public` to expose tools/resources on the unauthenticated `/public/mcp/sse` endpoint.
`wanaku.service.registration.uri`	`http://localhost:8080`	Router URL for forward registration.
`wanaku.service.registration.mcp-forward-address`	`http://localhost:8181/mcp/sse`	SSE endpoint the router will use to reach this server.

All properties can be overridden via -D flags on the command line.

Important: The namespace determines which SSE endpoint exposes the tools. If set to test, tools appear under an authenticated namespace (e.g., /ns-9/mcp/sse). Set to public for unauthenticated access at /public/mcp/sse, which is what the k6 scripts target.

Data Store

The router persists forwards in ~/.wanaku/router/. If you see stale data between runs, clear it:

bash

rm -rf ~/.wanaku/router/forward/{data,index}/*
rm -rf ~/.wanaku/router/namespace/{data,index}/*
rm -rf ~/.wanaku/router/tool/{data,index}/*
rm -rf ~/.wanaku/router/resource/{data,index}/*

3. Full Baseline vs Patched Evaluation

gRPC Bridge Evaluation (CI-based)

Uses run-perf-evaluation.sh to download baseline artifacts from CI (main branch), build the current branch, run both, and generate a comparison report:

bash

tests/load/run-perf-evaluation.sh

Override defaults with environment variables:

bash

TEST_SCOPE=tools SKIP_BASELINE=true tests/load/run-perf-evaluation.sh

MCP Bridge Evaluation (local build)

For comparing main vs a feature branch through the MCP bridge, build and test each branch separately:

bash

# 1. Build baseline from main
git checkout main
mvn package -pl apps/wanaku-router-backend,tests/mcp-servers/wanaku-performance-test-mock-mcp -am -DskipTests -T1C -q
# Copy baseline jars to a safe location
cp -r apps/wanaku-router-backend/target/quarkus-app /tmp/baseline-router
cp -r tests/mcp-servers/wanaku-performance-test-mock-mcp/target/quarkus-app /tmp/baseline-mock

# 2. Build patched from feature branch
git checkout my-feature-branch
mvn package -pl apps/wanaku-router-backend,tests/mcp-servers/wanaku-performance-test-mock-mcp -am -DskipTests -T1C -q

# 3. Run baseline tests, then patched tests (same VU levels, same duration)
#    Save results to: $EVAL_DIR/baseline/tools-invoke-sse/test-summary-vus-*.json
#                     $EVAL_DIR/patched/tools-invoke-sse/test-summary-vus-*.json
#    (and similarly for resources-read-sse)

# 4. Generate comparison report
python3 tests/load/generate-perf-report.py --eval-dir $EVAL_DIR --test-scope all

Report Generation

generate-perf-report.py produces a Markdown comparison report from k6 JSON summary files.

Expected Directory Structure

text

$EVAL_DIR/
├── baseline/
│   ├── tools-invoke-sse/
│   │   ├── test-summary-vus-1.json
│   │   ├── test-summary-vus-10.json
│   │   ├── vmstat-baseline-tools-invoke-sse.log    # optional
│   │   └── java-procs-baseline-tools-invoke-sse.log # optional
│   └── resources-read-sse/
│       └── test-summary-vus-*.json
└── patched/
    ├── tools-invoke-sse/
    │   └── test-summary-vus-*.json
    └── resources-read-sse/
        └── test-summary-vus-*.json

Usage

bash

python3 tests/load/generate-perf-report.py \
  --eval-dir /path/to/eval-dir \
  --test-scope all \
  --output /path/to/report.md

Options:

--eval-dir (required): directory containing baseline/ and patched/ subdirectories
--test-scope: all (default), tools, or resources
--output: output file path (defaults to $EVAL_DIR/perf-report.md)

Key Metrics

The report tracks these metrics per VU level:

Metric	Direction	Description
`mcp_request_duration` (avg, med, p90, p95, max)	Lower is better	End-to-end MCP request latency
`mcp_request_count` (rate, count)	Higher is better	Throughput
`mcp_request_errors` (rate, count)	Lower is better	Error count
`iterations` (rate, count)	Higher is better	Full iteration throughput
`iteration_duration` (avg, p95)	Higher is better	Full iteration time (includes all MCP calls per iteration)
`data_sent` / `data_received` (rate)	Higher is better	Network throughput

The report uses indicators: green circle for >5% improvement, red circle for >10% regression.

Writing Custom k6 Scripts

k6 scripts use the k6/x/mcp extension. Each iteration creates an SSE client, performs MCP operations, and the extension tracks mcp_request_duration, mcp_request_count, and mcp_request_errors automatically.

Template:

javascript

import mcp from 'k6/x/mcp';

export default function () {
    const client = new mcp.SSEClient({
        base_url: 'http://localhost:8080/public/mcp/sse',
        timeout: 5
    });

    client.ping();

    // Your MCP operations here:
    // client.listAllTools()
    // client.callTool({ name: 'toolName', arguments: { key: 'value' } })
    // client.listAllResources()
    // client.readResource({ uri: 'resource-uri' })
}

bash

tests/load/run-perf-test.sh \
  --router-from /path/to/router.tar.gz \
  --test my-test ./my-script.js

Troubleshooting

Tools/resources not listed via SSE: Check the namespace. The k6 scripts target /public/mcp/sse. If the forward registered with a non-public namespace, tools will only appear under the corresponding authenticated namespace endpoint (e.g., /ns-9/mcp/sse). Override with -Dwanaku.mcp.service.namespace=public.

Stale forward registrations: The router persists forwards in ~/.wanaku/router/. Clear the data store (see Data Store above) and restart.

High latency at 500+ VUs: Expected. The SSE transport creates a new connection per iteration. At high concurrency, connection queuing dominates. The median latency stays low but P95 increases significantly.

Mock MCP server config parse failure: Do not pass -Dwanaku.mcp.service.namespace="" (empty string). Quarkus cannot parse this. Either omit the flag (uses default from application.properties) or set it to an explicit value like public.

Performance Tests ​

Prerequisites ​

Architecture Overview ​

Quick Reference ​

File Layout ​

Key Environment Variables ​

Test Scenarios ​

1. gRPC Bridge Tests (via run-perf-test.sh) ​

Build ​

Run ​

Artifacts ​

2. MCP Bridge Tests (via mock MCP server) ​

Build ​

Run Manually ​

Mock MCP Server Configuration ​

Data Store ​

3. Full Baseline vs Patched Evaluation ​

gRPC Bridge Evaluation (CI-based) ​

MCP Bridge Evaluation (local build) ​

Report Generation ​

Expected Directory Structure ​

Usage ​

Key Metrics ​

Writing Custom k6 Scripts ​

Troubleshooting ​

Performance Tests

Prerequisites

Architecture Overview

Quick Reference

File Layout

Key Environment Variables

Test Scenarios

1. gRPC Bridge Tests (via `run-perf-test.sh`)

Build

Run

Artifacts

2. MCP Bridge Tests (via mock MCP server)

Build

Run Manually

Mock MCP Server Configuration

Data Store

3. Full Baseline vs Patched Evaluation

gRPC Bridge Evaluation (CI-based)

MCP Bridge Evaluation (local build)

Report Generation

Expected Directory Structure

Usage

Key Metrics

Writing Custom k6 Scripts

Troubleshooting