Skip to main content

​
Overview

Garage provides S3-compatible object storage for the observability stack. It serves as the backend for Loki (log chunks) and Tempo (trace blocks), enabling persistent storage without external dependencies.

​
Configuration

​
Nixidy Module (nixidy/env/local/garage.nix)

applications.garage = {
  namespace = "storage";
  createNamespace = true;
  
  resources = {
    configMaps.garage-config = {
      data."garage.toml" = ''
        metadata_dir = "/var/lib/garage/meta"
        data_dir = "/var/lib/garage/data"
        db_engine = "sqlite"
        replication_factor = 1
        
        rpc_bind_addr = "[::]:3901"
        rpc_public_addr = "127.0.0.1:3901"
        rpc_secret = "0000...0000"  # 64-char hex string
        
        [s3_api]
        s3_region = "garage"
        api_bind_addr = "[::]:3900"
        root_domain = ".s3.garage.localhost"
        
        [admin]
        api_bind_addr = "[::]:3903"
        admin_token = "admin"
        metrics_token = "metrics"
      '';
    };
  };
};

​
Storage Configuration

  • Database: SQLite (single-node deployment)
  • Replication factor: 1 (no redundancy in local dev)
  • Metadata dir: /var/lib/garage/meta (PV)
  • Data dir: /var/lib/garage/data (PV)

​
Deployment

Garage runs as a StatefulSet: 
image: dxflrs/garage:v1.1.0
replicas: 1
volumeClaims:
  - meta: 512Mi
  - data: 4Gi

​
Persistent Volumes

VolumeSizePurpose
meta512MiCluster metadata and object index
data4GiObject data (chunks and blocks)

​
Service Ports

Garage exposes three services:
PortProtocolPurpose
3900TCPS3 API (object operations)
3901TCPRPC (cluster communication)
3903TCPAdmin API (management, metrics)

​
Service Endpoint

  • S3 API: http://garage.storage:3900
  • Admin API: http://garage.storage:3903
  • Namespace: storage

​
Setup Process

Garage requires initial configuration via garage-setup.sh:

​
1. Apply Cluster Layout

NODE_ID=$(garage_exec status | grep -oE '[a-f0-9]{16}' | head -1)
garage_exec layout assign "$NODE_ID" -z dc1 -c 4G
garage_exec layout apply --version "$LAYOUT_VER"
This assigns 4GB capacity to the single node in zone dc1.

​
2. Create Access Key

garage_exec key create garage-o11y-key
Outputs:
  • Access Key: GK... (20+ chars)
  • Secret Key: Random string

​
3. Create Buckets

for BUCKET in loki-chunks tempo-traces; do
  garage_exec bucket create "$BUCKET"
  garage_exec bucket allow "$BUCKET" --read --write --key "$ACCESS_KEY"
done
Buckets:
  • loki-chunks - Loki log storage
  • tempo-traces - Tempo trace storage

​
4. Create Kubernetes Secrets

kubectl create secret generic garage-s3-credentials \
  --namespace=observability \
  --from-literal=AWS_ACCESS_KEY_ID="$ACCESS_KEY" \
  --from-literal=AWS_SECRET_ACCESS_KEY="$SECRET_KEY"
Secret is created in:
  • storage namespace (for Garage itself)
  • observability namespace (for Loki and Tempo)

​
5. Encrypt Credentials (SOPS)

sops --encrypt --in-place secrets/garage.yaml
Credentials are encrypted with SOPS for GitOps reproducibility.

​
Health Checks

​
Readiness Probe

readinessProbe:
  httpGet:
    path: /health
    port: 3903
  initialDelaySeconds: 5
  periodSeconds: 10

​
Liveness Probe

livenessProbe:
  httpGet:
    path: /health
    port: 3903
  initialDelaySeconds: 10
  periodSeconds: 30
Both probes use the admin API health endpoint.

​
Resource Limits

requests:
  cpu: 50m
  memory: 128Mi
limits:
  cpu: 500m
  memory: 512Mi
Garage is lightweight and suitable for local development workloads.

​
Integration

​
Loki

Loki uses Garage for log chunk storage: 
storage:
  type: s3
  bucketNames:
    chunks: loki-chunks
  s3:
    endpoint: http://garage.storage:3900
    region: garage
    s3forcepathstyle: true
Credentials: garage-s3-credentials secret injected via extraEnvFrom

​
Tempo

Tempo uses Garage for trace block storage: 
storage:
  trace:
    backend: s3
    s3:
      endpoint: garage.storage:3900
      bucket: tempo-traces
      region: garage
      forcepathstyle: true
Credentials: garage-s3-credentials secret injected via extraEnvFrom

​
S3 Compatibility

Garage implements S3 API:
  • PutObject - Upload objects
  • GetObject - Download objects
  • DeleteObject - Remove objects
  • ListObjects - List bucket contents
  • HeadObject - Get object metadata
  • MultipartUpload - Large object uploads
Path style: Required (s3forcepathstyle: true)

​
Admin Operations

​
Check Cluster Status

kubectl exec -n storage garage-0 -- /garage status

​
List Buckets

kubectl exec -n storage garage-0 -- /garage bucket list

​
View Key Info

kubectl exec -n storage garage-0 -- /garage key info <access-key>

​
Check Metrics

curl -H "Authorization: Bearer metrics" http://garage.storage:3903/metrics
Metrics are Prometheus-compatible.

​
Configuration Details

​
RPC Settings

rpc_bind_addr = "[::]:3901"      # Listen on all interfaces
rpc_public_addr = "127.0.0.1:3901"  # Single-node cluster
rpc_secret = "0000...0000"       # Cluster authentication

​
S3 Settings

s3_region = "garage"
api_bind_addr = "[::]:3900"
root_domain = ".s3.garage.localhost"

​
Admin Settings

api_bind_addr = "[::]:3903"
admin_token = "admin"       # Admin API auth
metrics_token = "metrics"   # Metrics endpoint auth

​
Storage Backend

Garage stores objects in two layers:
  1. Metadata - SQLite database with object index
  2. Data blocks - Raw object data on disk
This provides efficient lookups and streaming.
  • Loki - Log chunk storage client
  • Tempo - Trace block storage client

Build docs developers (and LLMs) love

Get started for freeTalk to us