vakra · code engine
4 / 4 capabilities live on Code Engine
Enterprise tool-calling benchmark · IBM Code Engine + COS

The VAKRA benchmark environment, running in the cloud.

Four capability servers on IBM Code Engine expose their tools over MCP-HTTP, backed by ~35 GB of data in IBM COS. Try it in the browser, benchmark against it with one config flag, or deploy your own — start below.

What this is

VAKRA is an enterprise benchmark: an agent is given a set of domain tools (SQL/REST APIs, retrievers, BPO services) and must answer questions by calling them. This deployment moves that tool environment onto Code Engine and its data into COS, so no one has to run containers or download 35 GB locally. Two things get bridged for the cloud, and nothing about the local setup changes:

Per-domain scoping is preserved. A connection to /mcp/california_schools only ever sees that domain's tools — the bridge pins MCP_DOMAIN per request, exactly like the local docker exec -e path.

Choose your path

Pick the row that matches you. Each links to the full guide.

Try it — zero setup

live

Open the hosted explorer, pick a capability and a domain, and browse or invoke the tools right in your browser. No clone, no install, no keys.

Open the explorer → · first click: cap 2 → california_schools

Benchmark against it

dev

Run the usual VAKRA runner — the only change from a local run is the MCP config. Add a provider key (OpenAI or watsonx) for a real agent run.

# after: source deploy/ce/.ce_urls.env
python benchmark_runner.py --capability_id 2 \
  --domain california_schools \
  --mcp-config benchmark/mcp_connection_config.ce.yaml
Full instructions → USING.md

Deploy your own

admin

Push data to COS, build the image, and roll out the four apps. Admin-gated, idempotent, and fully additive — existing deployments are untouched.

python deploy/ce/0_push_data_to_cos.py --stream
cd deploy/ce
VAKRA_CE_ADMIN=1 YES=1 ./1_create_bucket_and_pds.sh
VAKRA_CE_ADMIN=1 YES=1 ./2_build_push_image.sh
VAKRA_CE_ADMIN=1 YES=1 ./3_deploy_apps.sh
Full runbook → RUNBOOK.md

Understand it

how

How the stdio→HTTP bridge keeps tools domain-scoped, how COS is mounted, and every issue we hit (and its fix) while standing this up.

Architecture + gotchas → RUNBOOK.md · quick reference → CHEATSHEET.md

The four capabilities

Each capability is its own app and MCP endpoint. Tool counts below are from the live smoke test.

AppCapabilityServesExample domainsTools
vakra-cap1bi_apis — slot-fill / selectionRouterMCP toolsaddress, …9
vakra-cap2dashboard_apis — SQL / RESTM3 REST toolscalifornia_schools, financial, formula_1, movie139
vakra-cap3multihop_reasoningBPO or M3 REST (by domain)bpo, …32
vakra-cap4multiturn — REST + retrieverM3 REST + semantic retrieveraddress, …144

Valid --domain values are the task-file names under data/test/capability_<id>_*/input/. Browse them per capability in the explorer.

The endpoint contract

Any MCP client can talk to a deployed app over streamable HTTP. Get the real URLs from deploy/ce/.ce_urls.env (an admin has them).

basehttps://vakra-cap<1–4>.<hash>.us-east.codeengine.appdomain.cloud
MCP<base>/mcp/<domain>  — streamable HTTP; scoped to one domain
health<base>/healthz  → {"status":"ok","capability_id":"N"}
! These endpoints are public and unauthenticated. Keep the URLs within the team; add a bearer-token gate in mcp_http_bridge.py if you need to lock them down.

At a glance

Region · Project
us-east · ce-project-routing
Image
icr.io/routing_namespace/ vakra-benchmark:latest
Data
COS mount (default) · ~35 GB
MCP
mcp==1.27.0 · /mcp/<domain>

The docs