Self-Managed Materialize Overview

​

What is Self-Managed Materialize?

• Infrastructure: Deploy on any Kubernetes cluster (AWS EKS, GCP GKE, Azure AKS, or on-premises)
• Data residency: Keep all data within your own environment
• Network configuration: Full control over network policies and connectivity
• Compliance: Meet strict regulatory requirements for data handling
• Resource management: Direct control over compute, memory, and storage resources

​

Architecture

┌─────────────────────────────────────────────────────┐
│              Kubernetes Cluster                      │
│                                                       │
│  ┌──────────────────────────────────────────┐      │
│  │     Materialize Operator                 │      │
│  │  (orchestratord)                         │      │
│  └──────────────┬───────────────────────────┘      │
│                 │                                    │
│  ┌──────────────▼───────────────────────────┐      │
│  │   Materialize Environment                │      │
│  │                                           │      │
│  │  ┌─────────────┐  ┌────────────────┐    │      │
│  │  │ environmentd │  │   balancerd    │    │      │
│  │  │ (SQL Layer)  │  │ (Load Balancer)│    │      │
│  │  └─────────────┘  └────────────────┘    │      │
│  │                                           │      │
│  │  ┌─────────────────────────────────┐    │      │
│  │  │      clusterd (Workers)          │    │      │
│  │  │  ┌──────┐ ┌──────┐ ┌──────┐     │    │      │
│  │  │  │ Pod  │ │ Pod  │ │ Pod  │     │    │      │
│  │  │  └──────┘ └──────┘ └──────┘     │    │      │
│  │  └─────────────────────────────────┘    │      │
│  │                                           │      │
│  │  ┌─────────────┐                         │      │
│  │  │   console   │                         │      │
│  │  │ (Web UI)    │                         │      │
│  │  └─────────────┘                         │      │
│  └───────────────────────────────────────────┘      │
│                                                       │
│  ┌──────────────┐  ┌─────────────────┐              │
│  │  PostgreSQL  │  │   Object Store  │              │
│  │  (Metadata)  │  │  (S3/Minio)     │              │
│  └──────────────┘  └─────────────────┘              │
└─────────────────────────────────────────────────────┘

​

Components

​

Materialize Operator (orchestratord)

• Custom Resource Definitions (CRDs): Define Materialize environments declaratively
• Reconciliation: Ensures actual state matches desired state
• Upgrades: Manages rolling upgrades with minimal downtime
• Scaling: Handles cluster scaling and replica management

​

environmentd

• Accepts PostgreSQL wire protocol connections on port 6875
• HTTP API on port 6876
• Coordinates query execution across compute clusters
• Manages catalog and metadata

• CPU: 1 core
• Memory: 4 GiB

​

clusterd

• Execute data processing and maintain materialized views
• Run as pods on Kubernetes nodes
• Scale horizontally based on workload
• Require local NVMe storage for optimal performance

​

balancerd

• Distributes connections across environmentd replicas
• Provides health checking and failover
• Exposes external SQL endpoint

• CPU: 500m
• Memory: 256 MiB

​

console

• Environment monitoring and metrics
• Query history and performance analysis
• Cluster management UI
• User and role management

• CPU: 500m
• Memory: 256 MiB

​

Prerequisites

​

Kubernetes Requirements

• Kubernetes version: 1.29 or later
• Helm: 3.2.0 or later
• Storage: Local NVMe storage with provisioner (OpenEBS recommended)
• Node requirements:
  • ARM-based CPU (AWS Graviton, GCP Tau, or equivalent)
  • 1:8 ratio of vCPU to GiB memory
  • Local instance storage (for spill-to-disk)

​

Recommended Instance Types

​

AWS

• r7g.xlarge to r7g.16xlarge (with local NVMe)
• r7gd.xlarge to r7gd.16xlarge (with instance store)
• r6g, r6gd families also supported

​

GCP

• n2-standard-* series with local SSDs
• c2-standard-* series with local SSDs
• tau-* series (Arm-based)

​

Azure

• D-series v5 with local NVMe
• E-series v5 with local NVMe

​

External Dependencies

​

PostgreSQL or CockroachDB (Metadata Store)

• PostgreSQL: Version 13 or later
• CockroachDB: Version 23.1 or later (recommended)

postgres://materialize_user:[email protected]:5432/materialize_db?sslmode=require

​

Object Storage (Persist Backend)

• AWS S3: Recommended for AWS deployments
• GCS: Recommended for GCP deployments
• Azure Blob Storage: Recommended for Azure deployments
• MinIO: For on-premises or testing

s3://bucket-name/path?endpoint=https://s3.region.amazonaws.com&region=us-east-1

​

Deployment Options

​

Kubernetes Deployment

• Install Materialize Operator via Helm
• Deploy Materialize environments using Custom Resources
• Manage via kubectl and Kubernetes APIs

​

Docker Deployment

• Run Materialize containers with Docker Compose
• Suitable for local development
• Not recommended for production

​

Licensing

• Trial licenses: Available for evaluation (contact Materialize)
• Commercial licenses: Required for production use
• License enforcement: Newer versions enforce license key checks

apiVersion: v1
kind: Secret
metadata:
  name: materialize-backend
stringData:
  license_key: "your-license-key-here"

​

Support

• Documentation: Comprehensive deployment guides
• GitHub: Source code and issue tracking
• Enterprise support: Available for commercial customers
• Community: Slack channel for community support

​

Support Scope

• Materialize software issues and bugs
• Configuration guidance
• Upgrade assistance
• Performance tuning recommendations

• Kubernetes cluster management
• Infrastructure provisioning and scaling
• Backup and disaster recovery operations
• Monitoring and alerting setup

​

Comparison: Cloud vs Self-Managed

​

Next Steps

​

Additional Resources

• Materialize GitHub Repository
• Helm Chart Documentation
• Kubernetes Operator Guide
• OpenEBS Documentation