onix/benchmarks/README.md

beckn-onix Adapter Benchmarks

End-to-end performance benchmarks for the beckn-onix ONIX adapter, using Go's native testing.B framework and net/http/httptest. No Docker, no external services — everything runs in-process.

---

Quick Start

# From the repo root
go mod tidy                        # fetch miniredis + benchstat checksums
bash benchmarks/run_benchmarks.sh  # compile plugins, run all scenarios, generate report

Runtime output lands in benchmarks/results/<timestamp>/ (gitignored). Committed reports live in benchmarks/reports/.

---

What Is Being Benchmarked

The benchmarks target the bapTxnCaller handler — the primary outbound path a BAP takes when initiating a Beckn transaction. Every request travels through the full production pipeline:

Benchmark goroutine(s)
        │  HTTP POST /bap/caller/<action>
        ▼
httptest.Server  ←  ONIX adapter (real compiled .so plugins)
        │
        ├── addRoute      router plugin      resolve BPP URL from routing config
        ├── sign          signer + simplekeymanager  Ed25519 / BLAKE-512 signing
        └── validateSchema  schemav2validator  Beckn OpenAPI spec validation
        │
        └──▶ httptest mock BPP  (instant ACK — no network)

Mock services replace all external dependencies so results reflect adapter-internal latency only:

Dependency	Replaced by
Redis	miniredis (in-process)
BPP backend	httptest mock — returns {"message":{"ack":{"status":"ACK"}}}
Beckn registry	httptest mock — returns the dev key pair for signature verification

---

Benchmark Scenarios

Benchmark	What it measures
BenchmarkBAPCaller_Discover	Baseline single-goroutine latency for /discover
BenchmarkBAPCaller_Discover_Parallel	Throughput under concurrent load; run with -cpu=1,2,4,8,16
BenchmarkBAPCaller_AllActions	Per-action latency: discover, select, init, confirm
BenchmarkBAPCaller_Discover_Percentiles	p50 / p95 / p99 latency via b.ReportMetric
BenchmarkBAPCaller_CacheWarm	Latency when the Redis key cache is already populated
BenchmarkBAPCaller_CacheCold	Latency on a cold cache — full key-derivation round-trip
BenchmarkBAPCaller_RPS	Requests-per-second under parallel load (req/s custom metric)

---

How It Works

Startup (TestMain)

Before any benchmark runs, TestMain in e2e/setup_test.go:

1. Compiles all required plugins to a temporary directory using go build -buildmode=plugin. The first run takes 60–90 s (cold Go build cache); subsequent runs are near-instant.
2. Starts miniredis — an in-process Redis server used by the cache plugin (no external Redis needed).
3. Starts mock servers — an instant-ACK BPP and a registry mock that returns the dev signing public key.
4. Starts the adapter — wires all plugins programmatically (no YAML parsing) and wraps it in an httptest.Server.

Per-iteration (buildSignedRequest)

Each benchmark iteration:

1. Loads the JSON fixture for the requested Beckn action (testdata/<action>_request.json).
2. Substitutes sentinel values (BENCH_TIMESTAMP, BENCH_MESSAGE_ID, BENCH_TRANSACTION_ID) with fresh values, ensuring unique message IDs per iteration.
3. Signs the body using the Beckn Ed25519/BLAKE-512 spec (same algorithm as the production signer plugin).
4. Sends the signed POST to the adapter and validates a 200 OK response.

Validation test (TestSignBecknPayload)

A plain Test* function runs before the benchmarks and sends one signed request end-to-end. If the signing helper is mis-implemented, this fails fast before any benchmark time is wasted.

---

Directory Layout

benchmarks/
├── README.md                        ← you are here
├── run_benchmarks.sh                ← one-shot runner script
├── e2e/
│   ├── bench_test.go                ← benchmark functions
│   ├── setup_test.go                ← TestMain, startAdapter, signing helper
│   ├── mocks_test.go                ← mock BPP and registry servers
│   ├── keys_test.go                 ← dev key pair constants
│   └── testdata/
│       ├── routing-BAPCaller.yaml   ← routing config (BENCH_BPP_URL placeholder)
│       ├── discover_request.json    ← Beckn search payload fixture
│       ├── select_request.json
│       ├── init_request.json
│       └── confirm_request.json
├── tools/
│   ├── parse_results.go             ← CSV exporter for latency + throughput data
│   └── generate_report.go           ← fills REPORT_TEMPLATE.md with run data
├── reports/                         ← committed benchmark reports and template
│   ├── REPORT_TEMPLATE.md           ← template used to generate each run's report
│   └── REPORT_ONIX_v150.md          ← baseline report (Apple M5, Beckn v2.0.0)
└── results/                         ← gitignored; created by run_benchmarks.sh
    └── <timestamp>/
        ├── BENCHMARK_REPORT.md            — generated human-readable report
        ├── run1.txt, run2.txt, run3.txt   — raw go test -bench output
        ├── parallel_cpu*.txt              — concurrency sweep
        ├── benchstat_summary.txt          — statistical aggregation
        ├── latency_report.csv             — per-benchmark latency (from parse_results.go)
        └── throughput_report.csv          — RPS vs GOMAXPROCS (from parse_results.go)

---

Reports

Committed reports are stored in benchmarks/reports/. Each report documents the environment, raw numbers, and analysis for a specific run and adapter version.

File	Platform	Adapter version
REPORT_ONIX_v150.md	Apple M5 · darwin/arm64 · GOMAXPROCS=10	beckn-onix v1.5.0

The script auto-generates BENCHMARK_REPORT.md in each results directory using REPORT_TEMPLATE.md. To permanently record a run:

1. Run bash benchmarks/run_benchmarks.sh — BENCHMARK_REPORT.md is generated automatically.
2. Review it, fill in the B5 bottleneck analysis section.
3. Copy it to benchmarks/reports/REPORT_<tag>.md and commit.
4. benchmarks/results/ stays gitignored; only the curated report goes in.

---

Running Individual Benchmarks

# Single benchmark, 10 s
go test ./benchmarks/e2e/... \
  -bench=BenchmarkBAPCaller_Discover \
  -benchtime=10s -benchmem -timeout=30m

# All actions in one shot
go test ./benchmarks/e2e/... \
  -bench=BenchmarkBAPCaller_AllActions \
  -benchtime=5s -benchmem -timeout=30m

# Concurrency sweep at 1, 4, and 16 goroutines
go test ./benchmarks/e2e/... \
  -bench=BenchmarkBAPCaller_Discover_Parallel \
  -benchtime=30s -cpu=1,4,16 -timeout=30m

# Race detector check (no data races)
go test ./benchmarks/e2e/... \
  -bench=BenchmarkBAPCaller_Discover_Parallel \
  -benchtime=5s -race -timeout=30m

# Percentile metrics (p50/p95/p99 in µs)
go test ./benchmarks/e2e/... \
  -bench=BenchmarkBAPCaller_Discover_Percentiles \
  -benchtime=10s -benchmem -timeout=30m

Comparing Two Runs with benchstat

go test ./benchmarks/e2e/... -bench=. -benchtime=10s -count=6 > before.txt
# ... make your change ...
go test ./benchmarks/e2e/... -bench=. -benchtime=10s -count=6 > after.txt
go tool benchstat before.txt after.txt

---

Dependencies

Package	Purpose
github.com/alicebob/miniredis/v2	In-process Redis for the cache plugin
golang.org/x/perf/cmd/benchstat	Statistical benchmark comparison (CLI tool)

Both are declared in go.mod. Run go mod tidy once to fetch their checksums.