Skip to main content

​
System Design

The Hridaya Steam Market Tracker is a sophisticated async Python data platform designed to continuously track Steam Community Market data while respecting API rate limits. The architecture follows a coordinated scheduler pattern with shared rate limiting.

​
Core Components

Orchestrator

Central coordinator that initializes and manages all schedulers with graceful shutdown

Rate Limiter

Sliding window algorithm ensuring API compliance across all concurrent requests

Schedulers

Dual scheduling system: urgency-based (snoozer) and fixed-interval (clockwork)

API Client

Async HTTP client with automatic data validation and error handling

​
Architecture Diagram

​
Data Flow

​
1. Initialization Phase

# cerebro.py:196-214
async def run(self):
    # Load and validate configuration
    self.load_and_validate_config()
    
    # Setup schedulers with shared rate limiter
    self.setup_schedulers()
    
    # Setup signal handlers for graceful shutdown
    self.setup_signal_handlers()
Key validations:
  • Required fields per endpoint type
  • Rate limit feasibility calculation
  • Startup burst warnings

​
2. Runtime Phase

Both schedulers run concurrently, sharing a single RateLimiter instance: 
# Execute ALL overdue items immediately
for item in self.live_items:
    urgency = self.calculate_urgency(item)
    if urgency >= 1.0:
        await self.execute_item(item)  # Rate limited

# Sleep until next item becomes urgent
if not executed_any:
    sleep_duration = self.calculate_min_sleep_duration()
    await asyncio.sleep(sleep_duration)

​
3. Request Execution

Every API call follows this pattern: 
# steamAPIclient.py:72-73 (example from fetch_price_overview)
async def fetch_price_overview(...):
    # CRITICAL: Acquire rate limit token before making request
    await self.rate_limiter.acquire_token()
    
    # Make HTTP request
    async with self.session.get(url, params=params) as response:
        response.raise_for_status()
        return PriceOverviewData(**await response.json())
The rate limiter ensures:
  • Maximum N requests per window (configured in config.yaml)
  • No request proceeds without a token
  • Automatic queuing when limit is reached

​
Configuration-Driven Design

The system is entirely controlled by config.yaml: 
LIMITS:
  REQUESTS: 100          # Max requests per window
  WINDOW_SECONDS: 300    # 5-minute window

TRACKING_ITEMS:
  - market_hash_name: "AK-47 | Redline (Field-Tested)"
    apiid: priceoverview
    appid: 730
    polling-interval-in-seconds: 60
    
  - market_hash_name: "AWP | Asiimov (Field-Tested)"
    apiid: itemordershistogram
    appid: 730
    item_nameid: 176240541
    polling-interval-in-seconds: 120
Feasibility validation prevents impossible configurations: 
# cerebro.py:112-139
def validate_config_feasibility(self, rate_limit, window_seconds, items):
    total_reqs = sum(
        window_seconds // item['polling-interval-in-seconds']
        for item in items
    )
    
    if total_reqs > rate_limit:
        print(f"❌ CONFIG ERROR: Infeasible configuration")
        exit(1)

​
Endpoint Scheduling Strategy

EndpointSchedulerReason
priceoverviewsnoozerSchedulerReal-time price updates (every 5-60s)
itemordershistogramsnoozerSchedulerLive order book changes
itemordersactivitysnoozerSchedulerRecent transaction feed
pricehistoryClockworkSchedulerSteam updates hourly, no benefit to polling faster

​
Error Handling & Resilience

​
Exponential Backoff (snoozerScheduler)

Transient errors (429, 5xx, network) trigger exponential backoff: 
# snoozerScheduler.py:129-159
def apply_exponential_backoff(self, item, error_code):
    item['consecutive_backoffs'] = item.get('consecutive_backoffs', 0) + 1
    
    # Skip N polling intervals, where N = 2^(consecutive - 1), capped at 8
    skip_multiplier = min(2 ** (item['consecutive_backoffs'] - 1), 8)
    skip_seconds = item['polling-interval-in-seconds'] * skip_multiplier
    
    item['skip_until'] = datetime.now() + timedelta(seconds=skip_seconds)
Backoff progression:
  • 1st error: skip 1 interval
  • 2nd consecutive: skip 2 intervals
  • 3rd consecutive: skip 4 intervals
  • 4th+ consecutive: skip 8 intervals (capped)

​
Fixed Retry Logic (ClockworkScheduler)

Historical data uses fixed retry delays: 
# clockworkScheduler.py:124
backoff_seconds = [30, 60, 120, 240]  # 30s → 1m → 2m → 4m

​
Graceful Shutdown

Signal handlers ensure clean termination: 
# cerebro.py:181-194
def setup_signal_handlers(self):
    loop = asyncio.get_event_loop()
    for sig in (signal.SIGINT, signal.SIGTERM):
        loop.add_signal_handler(
            sig,
            lambda: asyncio.create_task(self.shutdown())
        )

async def shutdown(self):
    print("\n\nShutdown signal received. Stopping schedulers...")
    self.shutdown_event.set()
All tasks are cancelled gracefully when shutdown is triggered.

​
Database Storage

All fetched data is stored in SQLite (data/market_data.db) via the SQLinserts class: 
# Both schedulers follow this pattern
await self.data_wizard.store_data(result, item)
The database layer is abstracted from scheduling concerns, enabling easy migration to PostgreSQL or other backends.

​
Performance Characteristics

All items fire immediately on first run (urgency = ∞). The rate limiter queues them automatically.Example: 50 items configured → all 50 attempt to execute → rate limiter serializes them.
Items spread out naturally based on urgency scores. High-frequency items (30s polling) execute more often than low-frequency items (300s polling).The system sleeps when no items are urgent, waking only when the next item becomes overdue.
The orchestrator calculates capacity usage:
✓ Config feasible: 87 req/300s (87.0% capacity)
This assumes worst-case (all items synchronized). Real utilization is typically lower due to urgency-based spreading.

​
Next Steps

Orchestrator Deep Dive

Learn about configuration validation and scheduler coordination

Rate Limiter Algorithm

Understand the sliding window log implementation

Scheduler Strategies

Explore urgency-based vs fixed-interval scheduling

Configuration Guide

Set up your tracking configuration

Build docs developers (and LLMs) love

Get started for freeTalk to us