Skip to main content
NeuralCleave exposes Prometheus-compatible metrics, structured JSON logs, and a real-time Observability dashboard page.

Metrics endpoints

/metrics/snapshot returns one entry per registered metric:
Key metrics: messages_total/messages_sent_total/messages_errors_total (by channel), generation_requests_total/generation_errors_total/generation_latency_ms (by model), tokens_total and cost_usd_total (by model/provider — see Cost Tracking below), memory_entries_total, approval_decisions_total, approval_notifications_total (channel-forwarded approval sends, by outcome=sent|failed), voice metrics (voice_transcriptions_total, voice_synthesis_total, PTT/VAD/wake-word counters), and host resource gauges (below).

Host resource metrics

GET /api/v1/status includes a live CPU/memory/disk snapshot (also mirrored into the gauges above):
Fields are null — not 0 — when psutil isn’t installed or a read fails, so you can tell “unavailable” from “genuinely zero.” cpu_percent reflects usage since the previous read in this process (always 0.0 on the very first call — a psutil characteristic, not a bug).

Readiness vs liveness

/ready’s response includes a checks breakdown (phase, runtime, and channel_connected when at least one channel is configured) — all must be true for ready: true. It does not make a live network call to any LLM provider (that would make a frequently-polled endpoint slow and flaky); a total provider/router construction failure already surfaces via the runtime check.

Cost tracking

Token counts are already captured per generation; cost_usd_total turns them into an estimated dollar figure from a static per-model pricing table.
Unpriced providers (OpenRouter, Azure, Bedrock, or any unrecognized model) are omitted rather than reported as $0.00; Ollama is always $0.00 (self-hosted, no per-token charge). This is a live, in-process view — it resets on restart. Scrape the metrics endpoints into a real time-series store for long-term history.

Logs

The gateway logs via Python’s standard logging module to stdout/stderr (module-scoped loggers, e.g. neuralcleave.agent.runtime, neuralcleave.models.router) — redirect/capture as your process supervisor prefers. There’s no built-in file rotation or a dedicated JSON log format yet.

Observability dashboard

The /observability page in the dashboard shows live charts for:
  • Request rate (requests/min)
  • Token usage (in/out, rolling 60 min)
  • Memory chunk count
  • Tool call count and max chain depth
  • Voice session count
  • Connected channels

Prometheus integration

Alerting

Combine with Grafana for dashboards and alerting. Key signals to alert on:
  • messages_total rate drops to 0 (gateway or channel down)
  • generation_errors_total rate spike (provider outage or bad credentials — cross-check with neuralcleave models status --live)
  • process_memory_rss_bytes / host_disk_usage_percent trending toward exhaustion
  • cost_usd_total rate spike (runaway loop or unexpectedly expensive model)