RPC Provider Health Metrics
Metrics, alerts, and dashboard grouping for chain/provider reliability
RPC Provider Health Metrics
Use this page for BDT-321 on the M6 Observability + Settlement Ops milestone.
Plaidly depends on chain RPC providers for payment detection, confirmation tracking, sweeps, and future agent spend flows. Health signals must distinguish a provider outage from chain congestion so operators can fail over, pause risky flows, or explain delayed settlement.
Required Metrics
| Metric | Type | Labels | Purpose |
|---|---|---|---|
plaidly_rpc_call_total | Counter | provider, chain, network, method, status | Request volume and error rate per provider and method. |
plaidly_rpc_call_duration_seconds | Histogram | provider, chain, network, method | Provider latency and tail behavior. |
plaidly_rpc_breaker_state | Gauge | provider, chain, network | Circuit breaker state: 0=closed, 1=half_open, 2=open. |
plaidly_chain_confirmation_lag_blocks | Gauge | chain, network, token | Distance between observed head and required confirmation depth. |
plaidly_chain_confirmation_lag_seconds | Gauge | chain, network, token | Wall-clock lag for pending confirmations. |
plaidly_payment_detection_lag_seconds | Histogram | chain, network, token | Time from payer transfer to Plaidly detection. |
plaidly_queue_lag_seconds | Gauge | queue, chain, network | Event or worker queue delay for balance checks, sweeps, webhooks, and finalization. |
plaidly_provider_failover_total | Counter | from_provider, to_provider, chain, network, reason | Tracks automatic provider failover events. |
Alert Rules
| Alert | Severity | Condition | Interpretation |
|---|---|---|---|
| Provider hard outage | P1 | rpc_error_rate > 50% for one provider and one chain/network for 5 minutes | Provider-specific outage. Fail over if another provider is healthy. |
| Chain-wide RPC outage | P1 | rpc_error_rate > 50% across all providers for one chain/network for 5 minutes | Chain endpoint or network-wide issue. Do not blame a single provider. |
| High RPC latency | P2 | p95 duration above 5 seconds for 10 minutes | Payments may detect slowly; review provider and queue lag together. |
| Circuit breaker open | P2, P1 if all providers open | Any plaidly_rpc_breaker_state == 2 for 5 minutes | Provider is removed from active routing. |
| Confirmation lag high | P2 | confirmation_lag_seconds > 900 for 15 minutes | Chain congestion or indexer lag. |
| Payment detection lag high | P2 | p95 detection lag above 10 minutes for 15 minutes | Balance checker/indexer is delayed. |
| Queue lag high | P2 | queue lag above 5 minutes for balance or finalization workers | Internal processing is delayed even if chains are healthy. |
Triage
-
Compare provider error rate by
provider,chain, andnetwork.- One provider bad and others healthy: fail over or lower its priority.
- All providers bad for one chain: treat as chain/network incident.
- All chains bad for one provider: treat as provider account/API incident.
-
Compare
rpc_call_duration_seconds,confirmation_lag_seconds, andqueue_lag_seconds.- RPC slow but confirmation lag normal: provider issue.
- Confirmation lag high across providers: chain congestion.
- Queue lag high but RPC/chain normal: internal worker capacity issue.
-
Check affected payment sessions.
- Pending sessions need detection-lag status.
- Paid/finalizing sessions need finalization and webhook queue status.
- Merchant-facing diagnostics should say whether the recommended action is wait, retry webhook, or contact support.
-
Keep payloads safe.
- Include
provider,chain,network,method,status, queue names, request IDs, and session IDs. - Do not include RPC credentials, webhook secrets, private keys, raw signed payloads, or API keys.
- Include
Dashboard Grouping
Group health by:
- chain/network
- provider priority order
- current breaker state
- p50/p95/p99 RPC latency
- error rate
- confirmation lag
- queue lag
- active incident link
Use the chain/network group as the primary dashboard row. Provider details belong inside the row so operators can tell provider outage from chain congestion without opening logs.