Overview
Effective monitoring is critical for maintaining a healthy lending protocol. This guide covers key metrics, refresh operations, oracle monitoring, and recommended tooling.Key Metrics to Monitor
Reserve-Level Metrics
Utilization Rate
Utilization measures how much of the available liquidity is currently borrowed:state/reserve.rs:977-984 for calculation.
Monitoring thresholds:
- < 50%: Low utilization, consider lowering rates
- 50-80%: Target range for most reserves
- 80-95%: High utilization, rates rising
- > 95%: Critical - very little liquidity available
Total Deposits and Borrows
- Rapid deposit/withdrawal spikes
- Approaching deposit or borrow limits
- Unusual patterns suggesting manipulation
Current Borrow Rate
Calculate the current interest rate:state/reserve.rs:160-166 for implementation.
Alert if:
- Rates exceed expected maximum
- Rates are 0 at high utilization (curve misconfiguration)
- Sharp rate changes indicate potential issues
Available Liquidity
state/reserve.rs:206-214 for calculation.
Alert if:
- Available liquidity < 5% of total supply
- Freely available < total available (withdraw queue building up)
- Sudden drops suggesting large withdrawals
Protocol Fees
state/reserve.rs:684-689.
Monitor:
- Fee accumulation rate
- Fee collection frequency
- Uncollected fees vs available liquidity
Obligation-Level Metrics
Unhealthy Obligations
Identify at-risk positions:- LTV > 90% of liquidation threshold (close to liquidation)
- Any obligations exceeding liquidation threshold (liquidatable)
- Rapid LTV increases suggesting price movements
Total Borrowed Value
- Global borrow limit in lending market
- Historical trends and growth rate
- Concentration in specific reserves
Market-Level Metrics
Global Borrow Limit
- Utilization > 80% of global limit
- Rapid increases suggesting demand surge
Emergency Mode Status
state/lending_market.rs:297-299.
Immediate alerts:
- Emergency mode activated
- Borrowing disabled
Refresh Operations
Kamino Lending requires periodic refresh operations to update interest accrual and prices.Refresh Reserve
Update reserve interest accrual and price:handler_refresh_reserve.rs:11-63.
Refresh frequency:
- Minimum: Every 1-2 hours to accrue interest
- Price updates: When price age >
priceRefreshTriggerToMaxAgePctthreshold - Before operations: Automatically called before borrow/liquidate/etc.
Refresh Obligation
Update obligation’s collateral and debt values:handler_refresh_obligation.rs:10-68.
Refresh frequency:
- After reserve price updates
- Before liquidations
- Before borrowing more
- When checking position health
Batch Refresh
Refresh multiple reserves in one transaction:- Efficient refresh of all reserves
- Scheduled maintenance operations
- Before market-wide analysis
Price Oracle Monitoring
Oracle Types
Kamino supports multiple oracle sources:- Pyth Network - High-frequency price feeds
- Switchboard - Decentralized oracle network with TWAP
- Scope - Aggregated pricing from multiple sources
Price Staleness
Monitor price feed freshness:- Age > 50% of max age (warning)
- Age > max age (critical - price will be rejected)
- Price not updating (oracle offline)
Price Divergence
For tokens with multiple oracle sources, monitor divergence:- Divergence > configured threshold
- Prices moving in opposite directions
- One feed static while others move
Oracle Failures
Handle oracle outages gracefully:Querying On-Chain State
Using RPC
Using getProgramAccounts
Subscription for Real-Time Updates
Monitoring Tools and Strategies
Recommended Stack
- Metrics Database: Prometheus or InfluxDB for time-series data
- Dashboards: Grafana for visualization
- Alerting: PagerDuty, Opsgenie, or Grafana alerts
- Logging: CloudWatch, Datadog, or self-hosted ELK stack
- On-Chain Data: Custom indexer or The Graph subgraph
Alert Priorities
P0 - Critical (Immediate Response)- Emergency mode activated
- Oracle complete failure
- Protocol exploit detected
- Insolvency risk
- Liquidation threshold exceeded (no liquidations occurring)
- Price staleness exceeding limits
- Utilization > 95%
- Unusual borrowing patterns
- Approaching deposit/borrow limits
- Sustained high utilization (> 90%)
- Fee accumulation anomalies
- Withdrawal cap hits
- Suboptimal utilization rates
- Minor price feed issues
- Performance degradation
Automation Recommendations
- Periodic Refresh: Cron job to refresh reserves every hour
- Health Checks: Monitor all obligations every 5-10 minutes
- Price Monitoring: Check oracle feeds every minute
- Limit Checks: Verify limits and caps every 15 minutes
- Emergency Actions: Automated emergency mode trigger for critical events
Dashboard Metrics
Key dashboard panels:- Total Value Locked (TVL) by reserve
- Overall utilization rate
- Total borrowed value vs global limit
- Number of liquidatable obligations
- Protocol fee accumulation rate
- Average LTV across all obligations
- Borrow rate trends
- Oracle price trends with divergence
Best Practices
- Redundancy: Monitor from multiple vantage points and RPC providers
- Historical Data: Retain metrics for trend analysis and forensics
- Dry Runs: Test alert systems and runbooks regularly
- Documentation: Maintain runbooks for all alert types
- Escalation: Clear escalation paths for different severity levels
- Rate Limiting: Respect RPC rate limits, use multiple providers
- Graceful Degradation: Handle RPC failures and partial data
- Security: Protect monitoring infrastructure, use read-only keys
Related Operations
- Market Management - Configure market-level parameters
- Reserve Configuration - Configure reserve risk parameters
- Deployment - Initial setup and deployment