Observability Lab

5 Patterns for Debuggable APIs

You can't test what you can't observe. Before asserting on an API's behaviour, learn to see its behaviour — request IDs, structured logs, distributed traces, RED metrics, and aggregation. Each pattern with curl, Playwright, and OTEL examples.

1. Correlation IDs — every request, every log

Attach an X-Request-Id header to every API call. Propagate it through every log line the server writes.

When to reach for it: Any time you need to answer: 'which backend logs correspond to this frontend failure?'

Outcome you want: Given a request ID, a developer can grep every service's log and reconstruct the full call graph in under a minute.

curl

# Generate a request ID and include it in every call
REQ_ID=$(uuidgen)
curl -H "X-Request-Id: $REQ_ID" \
  https://api.qa-practice.dev/api/v1/users

# Read it back from the response
curl -sI -H "X-Request-Id: $REQ_ID" \
  https://api.qa-practice.dev/api/v1/users | grep -i 'x-request-id'

# Then grep server logs for that ID:
# journalctl -u api | grep $REQ_ID

Playwright

import { test, expect } from '@playwright/test';
import { randomUUID } from 'node:crypto';

test('request ID round-trips through every layer', async ({ request }) => {
  const reqId = randomUUID();

  const res = await request.get('/api/v1/users', {
    headers: { 'X-Request-Id': reqId },
  });

  // Echo-back: the server must NOT generate a new ID if client provided one
  expect(res.headers()['x-request-id']).toBe(reqId);

  // Body (if your API surfaces the ID for debugging)
  const body = await res.json();
  expect(body.meta?.requestId).toBe(reqId);
});

OpenTelemetry

// OpenTelemetry SDK — instrument fetch and auto-propagate
import { trace, context, propagation } from '@opentelemetry/api';
import { NodeSDK } from '@opentelemetry/sdk-node';

const sdk = new NodeSDK({ /* collector endpoint */ });
await sdk.start();

const tracer = trace.getTracer('practice-api');

await tracer.startActiveSpan('test-request', async (span) => {
  const headers: Record<string, string> = {};
  propagation.inject(context.active(), headers);
  // headers now contains traceparent + tracestate
  await fetch('https://api.qa-practice.dev/api/v1/users', { headers });
  span.end();
});

2. Structured logging — JSON, not console.log

Log as structured JSON (pino/winston/zap) with consistent fields: level, timestamp, request_id, user_id, duration_ms.

When to reach for it: Always. Text logs are write-only. Structured logs are queryable.

Outcome you want: Given a field name (e.g. user_id), you can filter across every service in seconds — no grep-fu required.

curl

# A good log line looks like this:
# {"level":"info","ts":"2026-04-21T09:12:03.412Z","req_id":"abc-123",
#  "user_id":"u-42","route":"GET /users","duration_ms":47,"status":200}
#
# Test that your API produces it:
curl -sS https://api.qa-practice.dev/api/v1/users | head -5

# Verify NO logs contain raw PII (email, password) — a spot-check:
journalctl -u api --since '5 min ago' | jq 'select(.email != null)'

Playwright

test('error responses include request_id for debugging', async ({ request }) => {
  const res = await request.get('/api/v1/users/99999'); // expected 404

  const body = await res.json();
  expect(res.status()).toBe(404);

  // Error envelope must expose request_id so a support ticket is actionable
  expect(body.error?.requestId ?? body.meta?.requestId).toMatch(
    /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/,
  );
});

OpenTelemetry

// Pino + OpenTelemetry correlation — every log line carries trace/span IDs
import pino from 'pino';
import { trace, context } from '@opentelemetry/api';

const logger = pino({
  mixin() {
    const span = trace.getSpan(context.active());
    if (!span) return {};
    const { traceId, spanId } = span.spanContext();
    return { trace_id: traceId, span_id: spanId };
  },
});

logger.info({ user_id: 'u-42', route: 'GET /users' }, 'handled request');
// Output: {"level":30,"trace_id":"4bf...","span_id":"00f...","user_id":"u-42",...}

3. Distributed traces — W3C traceparent

One header (traceparent) carries the trace identity across every service hop. Visualise as a waterfall.

When to reach for it: The moment you have more than one service. A 300ms p95 on the edge can be 50ms compute + 250ms waiting on a downstream.

Outcome you want: Given a slow request, a developer sees which span dominates the latency — without running a profiler.

curl

# Traceparent format: version-traceid(16B)-spanid(8B)-flags
# e.g. 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
VERSION=00
TRACE_ID=$(openssl rand -hex 16)
SPAN_ID=$(openssl rand -hex 8)
FLAGS=01

curl -H "traceparent: $VERSION-$TRACE_ID-$SPAN_ID-$FLAGS" \
  https://api.qa-practice.dev/api/v1/tracing

# Server should propagate the trace_id but generate its OWN span_id
curl -sI https://api.qa-practice.dev/api/v1/tracing | grep -i traceparent

Playwright

test('traceparent is propagated, span-id changes', async ({ request }) => {
  const traceId = 'a'.repeat(32);
  const parentSpan = 'b'.repeat(16);
  const inboundTraceparent = `00-${traceId}-${parentSpan}-01`;

  const res = await request.get('/api/v1/tracing', {
    headers: { traceparent: inboundTraceparent },
  });

  const outbound = res.headers()['traceparent'];
  expect(outbound, 'Server must echo a traceparent').toBeTruthy();

  const [, outTraceId, outSpanId] = outbound.split('-');
  expect(outTraceId, 'trace_id must be preserved').toBe(traceId);
  expect(outSpanId, 'span_id must be fresh').not.toBe(parentSpan);
});

OpenTelemetry

// Automatic trace propagation via instrumentation
import { registerInstrumentations } from '@opentelemetry/instrumentation';
import { HttpInstrumentation } from '@opentelemetry/instrumentation-http';

registerInstrumentations({
  instrumentations: [
    new HttpInstrumentation({
      requestHook: (span, req) => {
        span.setAttribute('http.user_agent', req.getHeader('user-agent'));
      },
    }),
  ],
});

// Every outbound fetch now auto-creates a child span with proper traceparent

4. RED metrics — Rate, Errors, Duration

Per endpoint, track: requests/sec, error ratio, latency distribution (p50/p95/p99).

When to reach for it: You want to answer: 'is the API healthy right now?' in 10 seconds without reading logs.

Outcome you want: Alerts fire on RED burn-rate (e.g. p95 > 500ms for 5 min) — not on raw log patterns.

curl

# Scrape a Prometheus metrics endpoint
curl -sS https://api.qa-practice.dev/metrics | grep http_request

# Expected output:
# http_requests_total{route="/api/v1/users",method="GET",status="200"} 12345
# http_request_duration_seconds_bucket{route="/api/v1/users",le="0.3"} 11500
# http_request_duration_seconds_bucket{route="/api/v1/users",le="0.5"} 12300

Playwright

test('metrics endpoint exposes RED fundamentals', async ({ request }) => {
  // Prime the counter with a few requests
  for (let i = 0; i < 5; i++) await request.get('/api/v1/users');

  const metrics = await request.get('/metrics').then((r) => r.text());

  // Rate: http_requests_total must increase
  expect(metrics).toMatch(/http_requests_total\{[^}]*route="\/api\/v1\/users"[^}]*\} \d+/);

  // Errors: 5xx counter must be exposed (even at 0)
  expect(metrics).toMatch(/http_requests_total\{[^}]*status="5\d\d"/);

  // Duration: histogram buckets must be present
  expect(metrics).toMatch(/http_request_duration_seconds_bucket/);
});

OpenTelemetry

// OpenTelemetry metrics — standard names, auto-exported
import { metrics } from '@opentelemetry/api';

const meter = metrics.getMeter('practice-api');
const requestCounter = meter.createCounter('http_requests_total', {
  description: 'Total HTTP requests',
});
const latencyHistogram = meter.createHistogram('http_request_duration_seconds', {
  unit: 's',
});

// In your handler:
const start = Date.now();
try {
  // ... process request
  requestCounter.add(1, { route, method, status: '200' });
} finally {
  latencyHistogram.record((Date.now() - start) / 1000, { route, method });
}

5. Log aggregation — ELK / Loki / Datadog

Ship structured logs to a central place queryable by field. Give every developer read access.

When to reach for it: The moment you have more than 3 pods — tailing them by hand no longer scales.

Outcome you want: Any engineer can answer 'find all 500s from user u-42 in the last hour' in seconds, via UI or CLI.

curl

# Query Loki via LogCLI for errors in the last 15 minutes
logcli query '{app="practice-api"} |= "error" | json' --since=15m

# Aggregation: count errors by route
logcli query 'sum by (route) (count_over_time({app="practice-api"} | json | level="error" [5m]))'

# Correlation: find logs for a specific request ID across all services
logcli query '{app=~".+"} | json | req_id="abc-123-def"' --limit 1000

Playwright

test('logs are indexed and queryable by request_id', async ({ request }) => {
  const reqId = `test-${Date.now()}`;

  await request.get('/api/v1/users', { headers: { 'X-Request-Id': reqId } });

  // Wait a few seconds for log pipeline to ingest
  await new Promise((r) => setTimeout(r, 3000));

  const logs = await request.get(
    `http://loki:3100/loki/api/v1/query_range?query={app="practice-api"}%20|%20json%20|%20req_id="${reqId}"`,
  );

  const body = await logs.json();
  expect(body.data?.result?.length, 'Log must appear in aggregator').toBeGreaterThan(0);
});

OpenTelemetry

// Forward logs via OpenTelemetry Collector — one config for many backends
// otel-collector.yaml:
receivers:
  otlp:
    protocols: { grpc: {}, http: {} }

processors:
  batch: {}

exporters:
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
  # OR swap for elasticsearch, datadog, clickhouse, etc.

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [loki]

5 Patterns for Debuggable APIs

1. Correlation IDs — every request, every log

2. Structured logging — JSON, not console.log

3. Distributed traces — W3C traceparent

4. RED metrics — Rate, Errors, Duration

5. Log aggregation — ELK / Loki / Datadog

Related Labs

Mock Server Lab

Contract Testing Lab

Security Testing Lab

Performance Testing Lab