> ## Documentation Index
> Fetch the complete documentation index at: https://docs.neuralcleave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

> Metrics, structured logs, and the Observability dashboard for monitoring NeuralCleave in production.

NeuralCleave exposes Prometheus-compatible metrics, structured JSON logs, and a real-time Observability dashboard page.

## Metrics endpoints

```bash theme={null}
curl http://localhost:7432/api/v1/metrics             # Prometheus text exposition
curl http://localhost:7432/api/v1/metrics/snapshot     # JSON, for the web UI
```

`/metrics/snapshot` returns one entry per registered metric:

```json theme={null}
{
  "tokens_total": {
    "type": "counter",
    "description": "LLM tokens consumed, by model and direction (input/output)",
    "values": { "direction=input,model=claude-opus-4-8": 4210 }
  },
  "cost_usd_total": {
    "type": "counter",
    "description": "Estimated USD cost of LLM generations, by provider and model",
    "values": { "model=claude-opus-4-8,provider=anthropic": 0.1642 }
  },
  "process_cpu_percent": { "type": "gauge", "values": { "": 3.2 } },
  "process_memory_rss_bytes": { "type": "gauge", "values": { "": 84213760 } },
  "approval_decisions_total": {
    "type": "counter",
    "description": "Exec-approval gate outcomes, by decision (auto_approved/prompted/denied_outright)",
    "values": { "decision=auto_approved": 12 }
  }
}
```

Key metrics: `messages_total`/`messages_sent_total`/`messages_errors_total` (by channel), `generation_requests_total`/`generation_errors_total`/`generation_latency_ms` (by model), `tokens_total` and `cost_usd_total` (by model/provider — see [Cost Tracking](#cost-tracking) below), `memory_entries_total`, `approval_decisions_total`, `approval_notifications_total` (channel-forwarded approval sends, by `outcome=sent|failed`), voice metrics (`voice_transcriptions_total`, `voice_synthesis_total`, PTT/VAD/wake-word counters), and host resource gauges (below).

## Host resource metrics

`GET /api/v1/status` includes a live CPU/memory/disk snapshot (also mirrored into the gauges above):

```json theme={null}
{
  "host": {
    "cpu_percent": 3.2,
    "memory_rss_bytes": 84213760,
    "disk_usage_percent": 41.7,
    "disk_free_bytes": 128849018880
  }
}
```

Fields are `null` — not `0` — when `psutil` isn't installed or a read fails, so you can tell "unavailable" from "genuinely zero." `cpu_percent` reflects usage since the *previous* read in this process (always `0.0` on the very first call — a `psutil` characteristic, not a bug).

## Readiness vs liveness

```bash theme={null}
curl http://localhost:7432/health   # always 200 once the process is up
curl http://localhost:7432/ready    # 200 once startup fully completes, 503 while initializing
```

`/ready`'s response includes a `checks` breakdown (`phase`, `runtime`, and `channel_connected` when at least one channel is configured) — all must be true for `ready: true`. It does not make a live network call to any LLM provider (that would make a frequently-polled endpoint slow and flaky); a total provider/router construction failure already surfaces via the `runtime` check.

## Cost tracking

Token counts are already captured per generation; `cost_usd_total` turns them into an estimated dollar figure from a static per-model pricing table.

```bash theme={null}
neuralcleave usage
curl http://localhost:7432/api/v1/usage
```

Unpriced providers (OpenRouter, Azure, Bedrock, or any unrecognized model) are omitted rather than reported as `$0.00`; Ollama is always `$0.00` (self-hosted, no per-token charge). This is a live, in-process view — it resets on restart. Scrape the metrics endpoints into a real time-series store for long-term history.

## Logs

The gateway logs via Python's standard `logging` module to stdout/stderr (module-scoped loggers, e.g. `neuralcleave.agent.runtime`, `neuralcleave.models.router`) — redirect/capture as your process supervisor prefers. There's no built-in file rotation or a dedicated JSON log format yet.

## Observability dashboard

The `/observability` page in the dashboard shows live charts for:

* Request rate (requests/min)
* Token usage (in/out, rolling 60 min)
* Memory chunk count
* Tool call count and max chain depth
* Voice session count
* Connected channels

## Prometheus integration

```yaml theme={null}
scrape_configs:
  - job_name: neuralcleave
    static_configs:
      - targets: ['localhost:7432']
    metrics_path: /api/v1/metrics
```

## Alerting

Combine with Grafana for dashboards and alerting. Key signals to alert on:

* `messages_total` rate drops to 0 (gateway or channel down)
* `generation_errors_total` rate spike (provider outage or bad credentials — cross-check with `neuralcleave models status --live`)
* `process_memory_rss_bytes` / `host_disk_usage_percent` trending toward exhaustion
* `cost_usd_total` rate spike (runaway loop or unexpectedly expensive model)