Wanaku

Appearance

Evaluator Engine — Developer Guide ​

The evaluator engine lets you build trigger→evaluate→act pipelines that inject LLM reasoning into the MCP request path. When a request matches a trigger, the engine calls an LLM to classify, filter, or augment the request, then executes a WebAssembly action script that has full access to the registry, conversation history, and response control.

You write the logic in JavaScript or Rust. The engine compiles it to WASM and runs it in a sandboxed environment with a clean, versioned API defined by the WIT interface.

Why this exists: MCP tools can be dangerous. You want to block "restart production database" calls, or dynamically assemble a per-conversation tool catalog based on what the LLM thinks the user needs. Pure LLM-based gates are too brittle (hallucination, prompt injection). Pure rule-based gates are too rigid. This engine gives you both: LLM cognitive work + deterministic WASM execution.

This guide is for developers who configure evaluators or write evaluator action scripts. It covers configuration, the JavaScript and Rust guest APIs, management routes, tests, internals, and operational limits.

Contents ​

YAML Configuration ​

Evaluators live in wanaku.yaml or get pushed via the management API. LLM connections are config-only. They hold the model, endpoint, and credential. Set them in wanaku.yaml. The management API cannot set or read them. An evaluator refers to a connection by name. Each evaluator has four parts:

yaml
llm_connections:                      # Config-only. Never exposed via the management API.
  - name: "local-llama"
    model: "llama3.2"
    url: "http://localhost:11434/v1"
    api_key: ""                       # Optional bearer token

evaluators:
  - name: "safety-gate"               # Unique identifier
    trigger:                          # When to run
      method: "tools/call"            # MCP method (tools/call, tools/list, etc.)
      namespace: "production"         # Optional: only this namespace
    llm:                              # LLM operation
      operation: classify             # classify | filter | augment
      prompt: "You are a safety classifier..."
      connection: "local-llama"       # References an entry in llm_connections
    processor:                        # WASM action script
      path: "/wasm/safety-gate.wasm"
    on_error: continue                # continue | block (default: continue)

LLM Connections ​

FieldTypePurpose
namestringUnique identifier referenced by an evaluator's llm.connection
modelstringModel name (passed to /v1/chat/completions)
urlstringOpenAI-compatible endpoint URL
api_keystringOptional bearer token for the LLM endpoint

The server loads connections once at startup from wanaku.yaml. No API can create, update, or read a connection's api_key. This design keeps credentials out of the management API, including out of evaluator revision history. See Security Considerations.

Trigger Fields ​

FieldTypePurpose
methodstringMCP method to match (e.g., tools/call, tools/list, resources/read)
namespacestringOptional. Only trigger for this namespace (extracted from URL path /finance/mcp → finance)

LLM Fields ​

FieldTypePurpose
operationstringclassify (pick a label), filter (return structured data), or augment (enrich prompt)
promptstringSystem prompt for the LLM. The engine builds a user prompt with request context.
connectionstringName of an entry in llm_connections (see above)
result_schemaobjectOptional JSON Schema for validating LLM output. When set, the host validates the LLM result before passing it to the WASM guest. On mismatch, retries once with a correction prompt.

How the LLM sees context: The engine builds a user prompt containing:

  • Recent conversation history (last 10 interactions from the intercept filter)
  • MCP method and namespace
  • Tool name and arguments (for tools/call)
  • Available tools (for tools/list)

Your system prompt tells the LLM what to do with that context.

Processor ​

A single WASM action script that processes the LLM output. The script receives the raw LLM result in ctx.llmResult and decides what to do:

yaml
processor:
  path: "/wasm/safety-gate.wasm"

The WASM script can call block(), pass(), warn(), filterTools(), or setMetadata() based on its analysis of the LLM output.

Error Policy ​

  • on_error: continue (default) — WASM failures are logged, request proceeds
  • on_error: block — WASM failures block the request with a JSON-RPC error

Complete Examples ​

Example 1: Safety Classification ​

Block dangerous tool calls based on LLM classification.

yaml
llm_connections:
  - name: "local-llama"
    model: "llama3.2"
    url: "http://localhost:11434/v1"

evaluators:
  - name: "safety-gate"
    trigger:
      method: "tools/call"
    llm:
      operation: classify
      prompt: |
        You are a strict safety classifier. Classify this tool call as:
        - green: safe operations (read-only, low impact)
        - yellow: elevated operations (writes, config changes)
        - red: dangerous operations (database restarts, production deploys)
        
        Respond with ONLY a JSON object: {"level": "green|yellow|red", "reason": "brief explanation"}
      connection: "local-llama"
      result_schema:                    # Optional: validate LLM output shape
        type: object
        properties:
          level:
            type: string
            enum: ["green", "yellow", "red"]
          reason:
            type: string
        required: ["level", "reason"]
    processor:
      path: "/wasm/safety_review_action.wasm"
    on_error: continue

What happens:

  1. User calls restart-database tool
  2. Engine sends tool name + args to LLM with your prompt
  3. LLM returns {"level": "red", "reason": "database restart is dangerous"}
  4. Engine runs safety_review_action.wasm with the raw LLM output in ctx.llmResult
  5. WASM script parses the JSON, sees "red", calls block("Tool call blocked: red")
  6. User gets JSON-RPC error instead of executing the tool

Example 2: Tool Assembly ​

Dynamically populate a namespace with only the tools the LLM approves.

yaml
evaluators:
  - name: "assembly-gate"
    trigger:
      method: "tools/list"
      namespace: "curated"
    llm:
      operation: filter
      prompt: |
        You are a tool curator. Given the conversation history and available tools,
        return a JSON array of tool names that are relevant and safe for this user.
        
        Respond with ONLY a JSON array: ["tool-name-1", "tool-name-2"]
      connection: "local-llama"
    processor:
      path: "/wasm/assembly-filter.wasm"

This example reuses the local-llama connection from Example 1.

What happens:

  1. User sends tools/list to the curated namespace (/curated/mcp)
  2. Engine sends conversation history + available tools to LLM
  3. LLM returns ["read-balance", "transfer-funds", "generate-report"]
  4. Engine runs assembly-filter.wasm with that JSON in ctx.llmResult
  5. WASM script parses JSON, calls copyToolToNamespace(name, ctx.namespace) for each
  6. WASM calls filterTools(approved) to return only the assembled tools
  7. User sees a curated tool list

Writing Action Scripts in JavaScript ​

JavaScript is the fastest path from idea to working WASM. You write normal JS, compile it with jco, and the engine runs it.

Prerequisites ​

bash
npm install -g @bytecodealliance/jco

Install the TypeScript definitions to enable autocomplete:

bash
# Copy from the repo
cp sdk/js/wanaku-actions.d.ts /path/to/your/action/

Step 1: Import Host Functions ​

The host provides four namespaces. Import what you need:

javascript
import { block, rejectMalformed, warn, pass, filterTools, setMetadata } from 'wanaku:evaluator/response';
import { verifyLlmResult } from 'wanaku:evaluator/validation';
import { listTools, getTool, copyToolToNamespace } from 'wanaku:evaluator/registry';
import { getHistory } from 'wanaku:evaluator/conversation';
import { info, warn, error } from 'wanaku:evaluator/log';

Step 2: Implement evaluate(ctx) ​

This is the only export the host calls:

javascript
export function evaluate(ctx) {
  // ctx fields:
  // - method: "tools/call" | "tools/list" | etc.
  // - namespace: "default" | "finance" | etc.
  // - toolName: "restart-database" (present for tools/call)
  // - arguments: [["target", "prod"], ["timeout", "30"]]
  // - llmResult: raw string from the LLM
  // - conversationId: "conv-12345" (if present)
  
  // Your logic here
}

Step 3: Available Host APIs ​

Registry (wanaku:evaluator/registry) ​

javascript
import { listTools, listToolsInNamespace, getTool, copyToolToNamespace } from 'wanaku:evaluator/registry';

// List all tools across all namespaces
const tools = listTools();  // [{name, description, uri, toolType, namespace}, ...]

// List tools in a specific namespace
const financeTools = listToolsInNamespace("finance");

// Get a single tool by name
const tool = getTool("restart-database");  // {name, description, ...} or undefined

// Copy a tool into a target namespace (for assembly)
const copied = copyToolToNamespace("read-balance", "finance-team");  // true if found

Response (wanaku:evaluator/response) ​

Call exactly ONE of these per evaluation. If you call none, the default is pass().

javascript
import { pass, block, rejectMalformed, warn, filterTools, setMetadata } from 'wanaku:evaluator/response';

// Allow the request to proceed
pass();

// Block the request with a JSON-RPC error (code -32001)
block("Tool call blocked: database restart requires manual approval");

// Signal that the input data is malformed and the evaluator cannot decide (code -32002)
// Use this for invalid LLM output, not for a policy rejection
rejectMalformed("LLM result missing required 'level' field");

// Log a warning but allow the request to proceed
warn("Elevated privilege operation detected");

// Return a filtered tools/list response (only for tools/list triggers)
filterTools(["tool-1", "tool-2", "tool-3"]);

// Set metadata for downstream filters
setMetadata("wanaku.risk_level", "high");

Validation (wanaku:evaluator/validation) ​

javascript
import { verifyLlmResult } from 'wanaku:evaluator/validation';

// Validate ctx.llmResult against the evaluator's declared result_schema.
// Returns { tag: 'ok', val: validatedString } on success,
// or { tag: 'err', val: errorMessage } on failure.
// If no result_schema is configured, always succeeds.
const result = verifyLlmResult(ctx.llmResult);
if (result.tag === 'err') {
  rejectMalformed(`Invalid LLM output: ${result.val}`);
  return;
}

Conversation (wanaku:evaluator/conversation) ​

javascript
import { getHistory } from 'wanaku:evaluator/conversation';

const messages = getHistory(ctx.conversationId);
// [{role: "user", content: "..."}, {role: "assistant", content: "..."}, ...]

// Empty array if no history or conversation ID is absent

Logging (wanaku:evaluator/log) ​

javascript
import { info, warn, error } from 'wanaku:evaluator/log';

info("Evaluator executed successfully");
warn("LLM returned unexpected format, falling back to default");
error("Failed to parse JSON from LLM result");

Step 4: EvaluationContext Object ​

The ctx parameter passed to evaluate(ctx) has these fields:

typescript
interface EvaluationContext {
  method: string;              // "tools/call", "tools/list", etc.
  namespace: string;           // "default", "finance", etc.
  toolName?: string;           // Present for tools/call, absent for tools/list
  arguments: [string, string][]; // Key-value pairs from the tool call
  llmResult: string;           // Raw LLM output (JSON string, label, or prose)
  conversationId?: string;     // If present in the request
}

Step 5: Compile to WASM ​

bash
npx @bytecodealliance/jco componentize action.js \
  --wit features/evaluator/wit/evaluator.wit \
  --world-name evaluator-action \
  --disable all \
  -o action.wasm

Flags explained:

  • --wit: path to the WIT interface file
  • --world-name: the world to target (always evaluator-action)
  • --disable all: Disable all optional WASI features. The action uses only the features in the WIT interface.
  • -o: output file

The first compilation can report errors if the imports do not match the WIT interface. Verify the namespace and function names.

Step 6: Deploy ​

Place the .wasm file somewhere the server can read it (e.g., /wasm/ directory), then reference it in your evaluator config:

yaml
processor:
  path: "/wasm/safety-block.wasm"

Or update via the management API:

bash
curl -X PUT http://localhost:8080/api/v1/evaluators \
  -H "Content-Type: application/json" \
  -d @evaluator-config.json

Complete JavaScript Examples ​

Safety Gate (classify → block or pass) ​

javascript
import { block, rejectMalformed, pass } from 'wanaku:evaluator/response';
import { verifyLlmResult } from 'wanaku:evaluator/validation';
import { warn } from 'wanaku:evaluator/log';

export function evaluate(ctx) {
  // If a result_schema is configured, validate first
  const validated = verifyLlmResult(ctx.llmResult);
  if (validated.tag === 'err') {
    warn(`LLM result failed schema validation: ${validated.val}`);
    rejectMalformed(`Cannot assess safety: ${validated.val}`);
    return;
  }

  let level = "red";
  try {
    const result = JSON.parse(ctx.llmResult);
    level = result.level || "red";
  } catch (e) {
    warn("Failed to parse LLM result, defaulting to red");
  }

  if (level === "red") {
    const reason = `Tool call blocked by safety classification: ${ctx.llmResult}`;
    warn(reason);
    block(reason);
  } else {
    pass();
  }
}

Compile:

bash
npx @bytecodealliance/jco componentize safety-block.js \
  --wit features/evaluator/wit/evaluator.wit \
  --world-name evaluator-action \
  --disable all \
  -o safety-block.wasm

Assembly Filter (filter → populate namespace) ​

javascript
import { copyToolToNamespace } from 'wanaku:evaluator/registry';
import { filterTools } from 'wanaku:evaluator/response';
import { info, warn } from 'wanaku:evaluator/log';

export function evaluate(ctx) {
  let approved;
  try {
    approved = JSON.parse(ctx.llmResult);
  } catch (e) {
    warn('Failed to parse LLM result as tool name array, returning all tools');
    return;  // Default behavior: pass, show all tools
  }

  if (!Array.isArray(approved) || approved.length === 0) {
    info('LLM returned empty tool list, returning all tools (fail-open)');
    return;
  }

  // Copy each approved tool into the target namespace
  for (const name of approved) {
    copyToolToNamespace(name, ctx.namespace);
  }

  info(`Registered ${approved.length} tools into namespace '${ctx.namespace}'`);
  
  // Return only the approved tools in the tools/list response
  filterTools(approved);
}

Error handling pattern: The LLM can return invalid output. Use try/catch when you parse JSON. Return early to fail open; the default action is pass().

Writing Action Scripts in Rust ​

Rust gives you type safety and better tooling for complex logic. The trade-off is more boilerplate.

Prerequisites ​

bash
# Install cargo-component
cargo install cargo-component

# Add wasm32-wasip1 target
rustup target add wasm32-wasip1

Step 1: Create a cdylib Crate ​

bash
cargo new --lib safety-review-action
cd safety-review-action

Edit Cargo.toml:

toml
[package]
name = "safety-review-action"
version = "0.1.0"
edition = "2024"

[lib]
crate-type = ["cdylib"]

[dependencies]
wit-bindgen = "0.41"
wit-bindgen-rt = "0.41"
serde_json = "1.0"

[package.metadata.component]
package = "wanaku:safety-review"

[package.metadata.component.target]
path = "wit/evaluator.wit"
world = "evaluator-action"

Copy the WIT file into your crate:

bash
cp /path/to/wanaku/features/evaluator/wit/evaluator.wit wit/evaluator.wit

Step 2: Use wit-bindgen ​

In src/lib.rs:

rust
#[allow(warnings)]
mod bindings;

use bindings::wanaku::evaluator::types::EvaluationContext;
use bindings::Guest;

struct SafetyReview;

impl Guest for SafetyReview {
    fn evaluate(ctx: EvaluationContext) {
        // Your logic here
    }
}

bindings::export!(SafetyReview with_types_in bindings);

The cargo component build command generates the bindings module. Do not write this module.

Step 3: Implement the Guest Trait ​

Access host imports via bindings::wanaku::evaluator::{registry, response, validation, log, conversation}:

rust
impl Guest for SafetyReview {
    fn evaluate(ctx: EvaluationContext) {
        let reason = format!(
            "Tool call blocked by safety classification: {}",
            ctx.llm_result
        );
        bindings::wanaku::evaluator::log::warn(&reason);
        bindings::wanaku::evaluator::response::block(&reason);
    }
}

Available host functions:

rust
use bindings::wanaku::evaluator::{registry, response, validation, conversation, log};

// Registry
let tools = registry::list_tools();
let tool = registry::get_tool("restart-database");
let copied = registry::copy_tool_to_namespace("read-balance", "finance");

// Response (call exactly one)
response::pass();
response::block("reason");
response::reject_malformed("reason");  // "cannot decide" versus "decided to reject"
response::warn("message");
response::filter_tools(&["tool-1", "tool-2"]);
response::set_metadata("key", "value");

// Validation — check LLM result against declared result_schema
match validation::verify_llm_result(&ctx.llm_result) {
    Ok(validated) => { /* use validated */ }
    Err(error) => {
        response::reject_malformed(&format!("Invalid LLM output: {error}"));
        return;
    }
}

// Conversation
let messages = conversation::get_history(&ctx.conversation_id.unwrap_or_default());

// Logging
log::info("message");
log::warn("message");
log::error("message");

Step 4: Build ​

bash
cargo component build --release

Output: target/wasm32-wasip1/release/safety_review_action.wasm

The filename comes from your [package] name with _ replacing -.

Step 5: Deploy ​

Same as JavaScript — reference the WASM file in your evaluator config:

yaml
processor:
  path: "/wasm/safety_review_action.wasm"

Complete Rust Example ​

Full src/lib.rs for a safety gate. The action reads the LLM classification from ctx.llm_result and maps the level to a response: red blocks, yellow warns, and any other level passes. The LLM is expected to return a JSON object of the form {"level": "green|yellow|red", "reason": "..."}. When the result is not valid JSON, the level is inferred from the raw text.

rust
#[allow(warnings)]
mod bindings;

use bindings::wanaku::evaluator::types::EvaluationContext;
use bindings::Guest;

struct SafetyReview;

impl Guest for SafetyReview {
    fn evaluate(ctx: EvaluationContext) {
        let (level, reason) = classify(&ctx.llm_result);

        match level.as_str() {
            "red" => {
                bindings::wanaku::evaluator::log::warn(&format!("Blocked: {reason}"));
                bindings::wanaku::evaluator::response::block(&format!(
                    "Tool call blocked by safety classification: {reason}"
                ));
            }
            "yellow" => {
                bindings::wanaku::evaluator::log::warn(&format!("Warning: {reason}"));
                bindings::wanaku::evaluator::response::warn(&format!("Safety warning: {reason}"));
            }
            _ => {
                bindings::wanaku::evaluator::response::pass();
            }
        }
    }
}

fn classify(llm_result: &str) -> (String, String) {
    if let Ok(value) = serde_json::from_str::<serde_json::Value>(llm_result) {
        let level = value
            .get("level")
            .and_then(serde_json::Value::as_str)
            .unwrap_or("green")
            .to_string();
        let reason = value
            .get("reason")
            .and_then(serde_json::Value::as_str)
            .unwrap_or(llm_result)
            .to_string();
        return (level, reason);
    }

    let lower = llm_result.to_lowercase();
    let level = if lower.contains("red") {
        "red"
    } else if lower.contains("yellow") {
        "yellow"
    } else {
        "green"
    };
    (level.to_string(), llm_result.to_string())
}

bindings::export!(SafetyReview with_types_in bindings);

Why this works: The bindings module is generated from evaluator.wit by cargo-component. It provides Rust types for EvaluationContext and functions for all host imports. You implement the Guest trait's evaluate method, and bindings::export! makes it callable from the host.

Management API ​

Hot-reload evaluators and manage namespace bindings without restarting the server.

List Evaluators ​

bash
curl http://localhost:8080/api/v1/evaluators

Response:

json
{
  "data": [
    {
      "name": "safety-gate",
      "trigger": {"method": "tools/call"},
      "llm": {
        "operation": "classify",
        "prompt": "...",
        "connection": "local-llama"
      },
      "processor": {"path": "/wasm/safety-gate.wasm"},
      "on_error": "continue"
    }
  ],
  "error": null
}

This response has no model, url, or api_key field. Those fields live only in llm_connections in wanaku.yaml. They never transit the management API. The llm.connection field names the connection the evaluator uses.

Update Evaluators (Hot-Reload) ​

bash
curl -X PUT http://localhost:8080/api/v1/evaluators \
  -H "Content-Type: application/json" \
  -d '{
    "evaluators": [
      {
        "name": "safety-gate",
        "trigger": {"method": "tools/call"},
        "llm": {
          "operation": "classify",
          "prompt": "You are a safety classifier...",
          "connection": "local-llama"
        },
        "processor": {"path": "/wasm/safety-gate.wasm"}
      }
    ]
  }'

What happens: The engine:

  1. Parses the config
  2. Validates that llm.connection names a connection already loaded from wanaku.yaml — an unknown name rejects the whole update with 422 and leaves the previous evaluators active
  3. Compiles all WASM files (expensive — do this at startup or infrequently)
  4. Replaces the active evaluators
  5. Returns the new config in {"data": [...], "error": null}

If a WASM file fails to compile: That evaluator is skipped, the rest are loaded, and you get a warning in the response.

Legacy payloads: a request with model/url/api_key inline under llm (the pre-connection shape) is rejected with 400 rather than silently dropping those fields.

List LLM Connections ​

bash
curl http://localhost:8080/api/v1/evaluators/llm-connections

Response:

json
{
  "data": ["local-llama"],
  "error": null
}

Read-only. Lists connection names loaded from wanaku.yaml at startup, for display/selection purposes. Returns names only — never model, url, or api_key — so this endpoint has nothing about your LLM backend worth leaking. There is no endpoint to create, update, or delete a connection; edit wanaku.yaml and restart the server.

List Namespace Bindings ​

bash
curl http://localhost:8080/api/v1/evaluators/namespaces

Response:

json
{
  "data": {
    "finance-team": "conv-finance-2024",
    "engineering": "conv-eng-0815"
  },
  "error": null
}

Bind Namespace to Conversation ​

bash
curl -X PUT http://localhost:8080/api/v1/evaluators/namespaces/finance-team \
  -H "Content-Type: application/json" \
  -d '{"conversation_id": "conv-finance-2024"}'

Use case: You want the evaluator to retrieve conversation history for a specific namespace. When a request arrives for the finance-team namespace, the evaluator looks up the binding, resolves the conversation ID, and fetches the matching interaction history to include in the LLM prompt.

Unbind Namespace ​

bash
curl -X DELETE http://localhost:8080/api/v1/evaluators/namespaces/finance-team

Testing Action Scripts ​

Quick Local Test ​

This script configures an evaluator, makes a tool call against a tool discovered from a forwarded MCP server, and verifies the response.

Prerequisites: You need an upstream MCP server that exposes at least one tool. Register it as a forward before running this script:

bash
curl -X POST http://localhost:8080/api/v1/forwards \
  -H "Content-Type: application/json" \
  -d '{"name":"my-mcp-server","address":"http://<your-mcp-server>/mcp"}'

Replace restart-database in the script below with a tool name discovered from your forward (curl http://localhost:8080/api/v1/tools to see available tools).

bash
#!/usr/bin/env bash
set -euo pipefail

MGMT=http://localhost:8080
MCP=http://localhost:8081
WASM="$(pwd)/actions/dist/safety_review_action.wasm"

echo "Configuring evaluator with WASM action..."
curl -sf -X PUT $MGMT/api/v1/evaluators -H "Content-Type: application/json" \
  -d '{
    "evaluators": [{
      "name": "js-safety",
      "trigger": {"method": "tools/call"},
      "llm": {
        "operation": "classify",
        "prompt": "You are a safety classifier. You MUST classify every tool call as exactly one of: green, yellow, or red. Restarting any database is ALWAYS red. Respond with ONLY a JSON object, no other text: {\"level\": \"green|yellow|red\", \"reason\": \"brief\"}",
        "model": "llama3.2",
        "url": "http://localhost:11434/v1"
      },
      "processor": {"path": "'"$WASM"'"}
    }]
  }' > /dev/null

echo "Making tool call..."
curl -sf -X POST $MCP/default/mcp -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"restart-database","arguments":{"target":"prod"}}}' | jq .

echo "Cleaning up evaluator..."
curl -sf -X PUT $MGMT/api/v1/evaluators -H "Content-Type: application/json" -d '{"evaluators":[]}' > /dev/null
echo "Done."

Expected output: JSON-RPC error with code -32001 and message containing "blocked by safety classification".

Rust Integration Tests (no server needed) ​

The evaluator crate has integration tests that run in the process. The test runner skips WASM engine tests if the action .wasm files do not exist.

bash
# Run all integration tests
cargo test -p wanaku-feature-evaluator

# Build WASM actions so engine tests run (instead of skipping)
cd actions/safety-review && cargo component build --release && cp target/wasm32-wasip1/release/*.wasm ../dist/

Tests cover: config parsing (including result_schema), trigger matching, evaluator state management, schema validation, action result variants, and WASM action execution with hardcoded llm_result (no LLM needed).

Rust E2E Tests (server required, no LLM needed) ​

End-to-end tests send requests to the management API and MCP endpoint on a running server. These tests replace the shell scripts and use the #[ignore] attribute.

bash
# Start wanaku-server first, then:
cargo test -p wanaku-feature-evaluator --test e2e -- --ignored

Tests cover: evaluator API CRUD, WASM block/warn actions, clearing evaluators, result_schema config, namespace-scoped evaluators, and LLM classification (classification tests also need Ollama on localhost:11434).

What to look for in server logs:

INFO  evaluator triggered method="tools/call" namespace="default" evaluator="safety-gate"
INFO  LLM operation result llm_result="{\"level\":\"red\",\"reason\":\"database restart\"}"
INFO  processor result action=Block("Tool call blocked by safety classification: red")

These traces confirm:

  1. The filter matched the trigger
  2. The LLM returned a classification
  3. The WASM action executed and returned a decision

How It Works (Internals) ​

You do not need these internal details to use the engine. Use them to diagnose execution problems.

Execution Flow ​

  1. MCP request arrives → praxis-ai MCP filter parses JSON-RPC, sets mcp.method metadata
  2. Namespace filter sets wanaku.namespace metadata from URL path
  3. Evaluator filter (wanaku_evaluator in the pipeline):
    • Reads metadata to get method, namespace, tool name, arguments
    • Finds matching evaluators (trigger.method matches, optional namespace/binding checks)
    • For each match:
      • Builds context prompt (conversation history + request details)
      • Calls LLM with your system prompt + context
      • Extracts the raw LLM output
      • If result_schema is configured, validates the output; on mismatch, retries once with a correction prompt that includes the specific validation error
      • Loads pre-compiled WASM processor module
      • Instantiates WASM with fresh host state (registry, interactions, result accumulator)
      • Calls evaluate(ctx) export
      • WASM calls host imports (e.g., block(), copyToolToNamespace())
      • Host state accumulates the action result
      • Returns action (Pass, Block, RejectMalformed, Warn, FilterTools, SetMetadata)
    • Applies the action: Block → JSON-RPC error (-32001), RejectMalformed → JSON-RPC error (-32002), FilterTools → synthetic response, etc.

WASM Compilation ​

Happens at startup and on PUT /api/v1/evaluators. The engine uses wasmtime with the component model:

  • Loads the .wasm file
  • Compiles it to native code (JIT)
  • Links WASI + custom host imports
  • Stores the compiled module

Each invocation gets a fresh instance — no state leaks between calls.

LLM Context Building ​

The engine builds the user prompt from:

  • Conversation history (last 10 interactions from the intercept filter's in-memory store)
  • Request details (method, namespace, tool name, arguments)
  • Available tools (for tools/list triggers)

Your system prompt tells the LLM what to extract from that context. The engine sanitizes all user input (truncates long strings, strips control characters) to prevent prompt injection.

Error Handling ​

  • LLM schema validation failure: If you set result_schema and the LLM output does not match, the engine retries once with a correction prompt. If the retry fails, the engine passes the raw result to the WASM guest. The guest can use verifyLlmResult() and rejectMalformed().
  • LLM failure (network error, timeout, invalid response): logged, evaluator skipped, request proceeds
  • WASM compile failure: logged at hot-reload time, evaluator disabled
  • WASM runtime failure: depends on on_error:
    • continue (default): logged, action is treated as Pass, request proceeds
    • block: logged, request blocked with JSON-RPC error

Philosophy: Fail open for operational resilience. A safety gate must not cause an availability failure. To fail closed, use on_error: block.

Common Patterns ​

Fail-Open Safety Gate ​

yaml
on_error: continue  # LLM/WASM failures do not block production
processor:
  path: "/wasm/safety-gate.wasm"

If the LLM is down, requests proceed. The WASM script decides whether to block based on the LLM output — if it never runs, the request continues.

Fail-Closed Safety Gate ​

yaml
on_error: block  # Any failure blocks the request
processor:
  path: "/wasm/safety-gate.wasm"

If the LLM is down or WASM crashes, the request is blocked. Use this for high-security environments where availability is secondary to safety.

Namespace-Scoped Evaluation ​

Restrict evaluators to specific namespaces:

yaml
trigger:
  method: "tools/list"
  namespace: "production"

This evaluator only fires for requests to /production/mcp. The WASM action can use filterTools() to curate the tool catalog for that namespace.

Namespace-Conversation Binding ​

Bind a namespace to a conversation ID so the evaluator can retrieve relevant interaction history:

bash
curl -X PUT http://localhost:8080/api/v1/evaluators/namespaces/finance-team \
  -H "Content-Type: application/json" \
  -d '{"conversation_id": "conv-finance-2024"}'

When a request arrives for the finance-team namespace, the evaluator resolves the conversation ID from the binding and includes the matching interaction history in the LLM prompt. This gives the LLM context about what the user has been doing in this session.

Metadata Propagation ​

Set metadata in your WASM action:

javascript
import { setMetadata } from 'wanaku:evaluator/response';

export function evaluate(ctx) {
  const risk = JSON.parse(ctx.llmResult).risk_level;
  setMetadata("wanaku.risk_level", risk);
  setMetadata("wanaku.evaluator", "safety-gate");
}

Downstream filters can read this metadata and make decisions (e.g., extra logging for high-risk calls, rate limiting).

Troubleshooting ​

"WASM component failed to compile" ​

Cause: The .wasm file is invalid or does not match the WIT interface.

Fix:

  1. Verify you compiled with the correct --wit path and --world-name evaluator-action
  2. Check that your imports match the WIT exactly (namespace, function names, signatures)
  3. For JavaScript, use jco componentize. Do not use jco transpile.
  4. For Rust, verify cargo.toml has the correct [package.metadata.component.target] path and world

"Evaluator did not trigger" ​

Cause: The trigger does not match the request.

Fix:

  1. Check server logs for evaluator filter trace messages — they show what the filter sees
  2. Verify trigger.method matches mcp.method metadata (e.g., tools/call not tool/call)
  3. If you use trigger.namespace, verify that the request URL uses /{namespace}/mcp. A bare /mcp path is invalid.

Enable trace logs: RUST_LOG=wanaku_feature_evaluator=trace

"LLM returned invalid output" ​

Cause: The LLM did not follow the prompt format.

Fix:

  1. Add result_schema to the LLM configuration. The host validates the LLM output. If the output does not match, the host retries once with a correction prompt:
    yaml
    llm:
  result_schema:
    type: object
    properties:
      level: { type: string, enum: ["green", "yellow", "red"] }
      reason: { type: string }
    required: ["level", "reason"]
  2. Use verifyLlmResult() in your WASM script — even with host-side validation, the guest can double-check and call rejectMalformed() instead of silently proceeding with bad data:
    javascript
    const validated = verifyLlmResult(ctx.llmResult);
if (validated.tag === 'err') {
  rejectMalformed(`Cannot assess: ${validated.val}`);
  return;
}
  3. Make your prompt more explicit: "Respond with ONLY a JSON object, no other text"
  4. Use a smaller, faster model for classification tasks — they need less context

"WASM action did nothing" ​

Cause: The action did not call a response function. The default pass() action applied.

Fix: Call block(), warn(), filterTools(), or setMetadata() in the action. To pass explicitly, call pass(). This call is optional.

"Hot-reload did not apply my WASM changes" ​

Cause: You changed the WASM file but did not call PUT /api/v1/evaluators.

Fix: The engine only compiles WASM at hot-reload time. After building a new .wasm file, trigger a reload:

bash
curl -X PUT http://localhost:8080/api/v1/evaluators \
  -H "Content-Type: application/json" \
  -d @current-config.json

This re-compiles all WASM files referenced in the config.

Performance Notes ​

  • LLM calls are slow (100-500ms per classification). Evaluators add latency to the request path. Use them sparingly, or only for high-stakes operations (e.g., tools/call, not tools/list).
  • WASM compilation is expensive (10-50ms per module). The engine caches compiled modules, so only the first load (or hot-reload) is slow.
  • WASM execution is fast (sub-millisecond for simple logic). A compiled action has minimal overhead.
  • Conversation history is capped at 10 interactions to keep LLM context bounded. If you need more, fetch it explicitly in your WASM action via getHistory().

Security Considerations ​

  • WASM is sandboxed — actions cannot access the filesystem, network, or system calls beyond what the WIT interface exposes.
  • LLM prompts can be attacked — sanitize user input, use system prompts that are robust to injection, and fail open if unsure.
  • WASM actions are deterministic — the same input always produces the same output. Use this property to test your logic thoroughly.
  • Namespace bindings are ephemeral — they live in memory, not persisted. If the server restarts, you lose bindings.
  • Evaluator revisions persist when persistence is enabled — revision history and the active revision survive a restart. The server writes evaluator-revisions.json after every change and restores the active revision at startup. See Evaluator Revision Persistence. When persistence is disabled, revisions live only in memory and the server rebuilds the active revision from wanaku.yaml on each restart.
  • LLM credentials never transit the management API. model, url, and api_key live only in llm_connections in wanaku.yaml, loaded once at startup. Evaluators refer to a connection by name. No route can set, update, or read an api_key, not on evaluator create or update, and not in evaluator revision history. To rotate a credential, edit wanaku.yaml and restart the server.

Next Steps ​

  • Start simple: Safety classification with a single rule is the easiest on-ramp.
  • Iterate on prompts: The LLM is the cognitive core — spend time tuning your system prompt.
  • Test offline: Write unit tests for your WASM logic before deploying.
  • Monitor logs: The evaluator filter emits detailed traces — watch for LLM responses, WASM decisions, and action results.
  • Read the WIT: features/evaluator/wit/evaluator.wit is the authoritative API reference.