Cloud Deployment (GKE) - Gnosis Prediction Market Agent

Overview

Cloud deployment uses Google Kubernetes Engine (GKE) to run agents in a managed Kubernetes environment. This provides high availability, automatic scaling, and seamless integration with Google Cloud services.

Architecture

Prerequisites

Google Cloud Account

Active GCP project with billing enabled

GKE Cluster

Running Kubernetes cluster in your GCP project

kubectl

Kubernetes command-line tool configured for your cluster

Docker Image

Built and pushed to container registry (GHCR)

Deployment Architecture

Kubernetes Resources

The application runs in the agents namespace with multiple deployments:

pma-agent

Deployment

Main prediction market agent deployment running configured agents

pma-agent-monitoring

Deployment

Monitoring and observability services for agent performance tracking

autonomous-trader-agent

Deployment

Autonomous trading agent with microchain framework (deployed here)

deployed-general-agent-viewer

Deployment

Web interface for viewing deployed agent status and performance

treasury

Deployment

Treasury management service for fund allocation and tracking

Setup Instructions

Configure kubectl

Connect kubectl to your GKE cluster:

gcloud container clusters get-credentials <cluster-name> \
  --region <region> \
  --project <project-id>

Verify connection:

kubectl cluster-info
kubectl get nodes

Create Namespace

Create the agents namespace if it doesn’t exist:

kubectl create namespace agents

Set as default namespace:

kubectl config set-context --current --namespace=agents

Configure Secrets

Create Kubernetes secrets for sensitive environment variables:

kubectl create secret generic agent-secrets \
  --from-literal=BET_FROM_PRIVATE_KEY="your_private_key" \
  --from-literal=OPENAI_API_KEY="your_openai_key" \
  --from-literal=SERPER_API_KEY="your_serper_key" \
  --from-literal=TAVILY_API_KEY="your_tavily_key" \
  --namespace=agents

Never commit secrets to version control. Use secret management tools like Google Secret Manager.

Create Deployment

Create a deployment manifest:

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pma-agent
  namespace: agents
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pma-agent
  template:
    metadata:
      labels:
        app: pma-agent
    spec:
      containers:
      - name: agent
        image: ghcr.io/gnosis/prediction-market-agent:main
        env:
        - name: runnable_agent_name
          value: "prophet_gpt4o"
        - name: market_type
          value: "omen"
        - name: BET_FROM_PRIVATE_KEY
          valueFrom:
            secretKeyRef:
              name: agent-secrets
              key: BET_FROM_PRIVATE_KEY
        - name: OPENAI_API_KEY
          valueFrom:
            secretKeyRef:
              name: agent-secrets
              key: OPENAI_API_KEY
        resources:
          requests:
            memory: "2Gi"
            cpu: "1000m"
          limits:
            memory: "4Gi"
            cpu: "2000m"
        imagePullPolicy: Always
      imagePullSecrets:
      - name: ghcr-secret

Apply the deployment:

kubectl apply -f deployment.yaml

Verify Deployment

Check deployment status:

kubectl get deployments -n agents
kubectl get pods -n agents
kubectl logs -f <pod-name> -n agents

Deployment Configuration

Environment Variables

Configure agents via environment variables in your deployment:

env:
- name: runnable_agent_name
  value: "prophet_gpt4o"  # Agent to run
- name: market_type
  value: "omen"  # Target market
- name: BET_FROM_PRIVATE_KEY
  valueFrom:
    secretKeyRef:
      name: agent-secrets
      key: BET_FROM_PRIVATE_KEY

Resource Limits

Set appropriate resource requests and limits:

resources:
  requests:
    memory: "1Gi"
    cpu: "500m"
  limits:
    memory: "2Gi"
    cpu: "1000m"

Image Pull Policy

For production, use specific tags instead of latest:

containers:
- name: agent
  image: ghcr.io/gnosis/prediction-market-agent:v1.2.3
  imagePullPolicy: IfNotPresent  # Only pull if not cached

For development, always pull latest:

containers:
- name: agent
  image: ghcr.io/gnosis/prediction-market-agent:main
  imagePullPolicy: Always  # Always pull latest

Managing Deployments

Restarting Deployments

The restart_app_deployments.sh script restarts all agent deployments to pick up new container images:

restart_app_deployments.sh

#!/bin/bash
kubectl rollout restart deploy pma-agent -n agents
kubectl rollout restart deploy pma-agent-monitoring  -n agents
kubectl rollout restart deploy autonomous-trader-agent -n agents
kubectl rollout restart deploy deployed-general-agent-viewer -n agents
kubectl rollout restart deploy treasury -n agents

Run the script:

bash restart_app_deployments.sh

This script is currently used until CI/CD pipeline integration with GKE is complete. DevOps is working on securing the connection between GitHub and GKE for automated deployments.

Rolling Updates

Kubernetes performs rolling updates automatically when you change the deployment:

# Update image
kubectl set image deployment/pma-agent \
  agent=ghcr.io/gnosis/prediction-market-agent:v1.2.3 \
  -n agents

# Check rollout status
kubectl rollout status deployment/pma-agent -n agents

Scaling

Scale deployments up or down:

# Scale to 3 replicas
kubectl scale deployment/pma-agent --replicas=3 -n agents

# Auto-scale based on CPU
kubectl autoscale deployment/pma-agent \
  --min=1 --max=5 --cpu-percent=80 \
  -n agents

Rollback

Rollback to a previous deployment version:

# View rollout history
kubectl rollout history deployment/pma-agent -n agents

# Rollback to previous version
kubectl rollout undo deployment/pma-agent -n agents

# Rollback to specific revision
kubectl rollout undo deployment/pma-agent --to-revision=3 -n agents

Multiple Agent Deployment

Run multiple agents simultaneously with different configurations:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pma-agent-prophet-gpt4o
  namespace: agents
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pma-agent
      agent-type: prophet-gpt4o
  template:
    metadata:
      labels:
        app: pma-agent
        agent-type: prophet-gpt4o
    spec:
      containers:
      - name: agent
        image: ghcr.io/gnosis/prediction-market-agent:main
        env:
        - name: runnable_agent_name
          value: "prophet_gpt4o"
        - name: market_type
          value: "omen"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pma-agent-microchain
  namespace: agents
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pma-agent
      agent-type: microchain
  template:
    metadata:
      labels:
        app: pma-agent
        agent-type: microchain
    spec:
      containers:
      - name: agent
        image: ghcr.io/gnosis/prediction-market-agent:main
        env:
        - name: runnable_agent_name
          value: "microchain"
        - name: market_type
          value: "omen"

Monitoring and Observability

Viewing Logs

# Tail logs from a specific pod
kubectl logs -f <pod-name> -n agents

# Logs from all pods in a deployment
kubectl logs -f deployment/pma-agent -n agents

# Previous container logs (if crashed)
kubectl logs <pod-name> -n agents --previous

# Last 100 lines
kubectl logs <pod-name> -n agents --tail=100

Pod Status

# Get pod status
kubectl get pods -n agents

# Detailed pod information
kubectl describe pod <pod-name> -n agents

# Watch pod status in real-time
kubectl get pods -n agents -w

Resource Usage

# CPU and memory usage
kubectl top pods -n agents

# Node resource usage
kubectl top nodes

Deployment Dashboard

View agent activity on the Dune dashboard:

AI Agents Overview

Track on-chain activity of deployed agents from this repo

Configuration Management

ConfigMaps

Use ConfigMaps for non-sensitive configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: agent-config
  namespace: agents
data:
  market_type: "omen"
  log_level: "INFO"
  free_for_everyone: "1"

Reference in deployment:

envFrom:
- configMapRef:
    name: agent-config

Secrets Management

Integrate with Google Secret Manager:

# Enable Secret Manager API
gcloud services enable secretmanager.googleapis.com

# Create secret
echo -n "your-api-key" | \
  gcloud secrets create openai-api-key --data-file=-

# Grant access to GKE service account
gcloud secrets add-iam-policy-binding openai-api-key \
  --member="serviceAccount:<gke-service-account>" \
  --role="roles/secretmanager.secretAccessor"

Use with Workload Identity or External Secrets Operator for automatic secret injection.

CI/CD Pipeline

Current State

Images are automatically built and pushed to GitHub Container Registry on:

Pushes to main branch
Pull requests with “build please” in description

Deployment Process

Code Push

Developer pushes code to GitHub

CI Build

GitHub Actions builds Docker image and pushes to GHCR

Manual Deployment

Currently, deployments are manually triggered using restart_app_deployments.sh

Future: Automated CD

DevOps is working on automating deployment after successful builds

The deployment process will be fully automated once the connection between GitHub Actions and GKE is secured.

High Availability

Multiple Replicas

Run multiple replicas for redundancy:

spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1

Pod Disruption Budget

Ensure minimum availability during updates:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: pma-agent-pdb
  namespace: agents
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: pma-agent

Health Checks

Add liveness and readiness probes:

livenessProbe:
  exec:
    command:
    - python
    - -c
    - "import sys; sys.exit(0)"
  initialDelaySeconds: 30
  periodSeconds: 10
  
readinessProbe:
  exec:
    command:
    - python
    - -c
    - "import sys; sys.exit(0)"
  initialDelaySeconds: 10
  periodSeconds: 5

Cost Optimization

Right-size Resources

Start with smaller resource requests and scale up based on actual usage:

kubectl top pods -n agents

Adjust resource limits based on observed consumption.

Use Preemptible Nodes

For non-critical workloads, use GKE’s preemptible nodes for ~80% cost savings:

nodeSelector:
  cloud.google.com/gke-preemptible: "true"

Horizontal Pod Autoscaling

Scale pods based on actual demand:

kubectl autoscale deployment/pma-agent \
  --min=1 --max=10 --cpu-percent=70 \
  -n agents

Cluster Autoscaling

Enable GKE cluster autoscaling to add/remove nodes based on pod requirements.

Troubleshooting

ImagePullBackOff

Pod can’t pull the Docker image:

# Check image exists
docker pull ghcr.io/gnosis/prediction-market-agent:main

# Verify image pull secret
kubectl get secret ghcr-secret -n agents -o yaml

# Recreate secret if needed
kubectl create secret docker-registry ghcr-secret \
  --docker-server=ghcr.io \
  --docker-username=<username> \
  --docker-password=<token> \
  -n agents

CrashLoopBackOff

Pod keeps crashing:

# Check logs
kubectl logs <pod-name> -n agents --previous

# Describe pod for events
kubectl describe pod <pod-name> -n agents

# Common causes:
# - Missing environment variables
# - Invalid configuration
# - Resource constraints

Pending Pods

Pods stuck in Pending state:

# Check pod events
kubectl describe pod <pod-name> -n agents

# Common causes:
# - Insufficient cluster resources
# - Node selector/affinity issues
# - Persistent volume claims not bound

Resource Exhaustion

Out of memory or CPU:

# Check resource usage
kubectl top pods -n agents

# Increase resource limits in deployment
# or scale horizontally

Next Steps

Environment Config

Complete environment variable reference for all agents

Docker Deployment

Learn about the Docker build and container setup

Local Development

Test agents locally before deploying to cloud

Monitoring Dashboard

View live agent performance and activity

Deployment Options

​Overview

​Architecture

​Prerequisites

Google Cloud Account

GKE Cluster

kubectl

Docker Image

​Deployment Architecture

​Kubernetes Resources

​Setup Instructions

​Deployment Configuration

​Environment Variables

​Resource Limits

​Image Pull Policy

​Managing Deployments

​Restarting Deployments

​Rolling Updates

​Scaling

​Rollback

​Multiple Agent Deployment

​Monitoring and Observability

​Viewing Logs

​Pod Status

​Resource Usage

​Deployment Dashboard

AI Agents Overview

​Configuration Management

​ConfigMaps

​Secrets Management

​CI/CD Pipeline

​Current State

​Deployment Process

​High Availability

​Multiple Replicas

​Pod Disruption Budget

​Health Checks

​Cost Optimization

​Troubleshooting

​Next Steps

Environment Config

Docker Deployment

Local Development

Monitoring Dashboard

Build docs developers (and LLMs) love

Overview

Architecture

Prerequisites

Deployment Architecture

Kubernetes Resources

Setup Instructions

Deployment Configuration

Environment Variables

Resource Limits

Image Pull Policy

Managing Deployments

Restarting Deployments

Rolling Updates

Scaling

Rollback

Multiple Agent Deployment

Monitoring and Observability

Viewing Logs

Pod Status

Resource Usage

Deployment Dashboard

Configuration Management

ConfigMaps

Secrets Management

CI/CD Pipeline

Current State

Deployment Process

High Availability

Multiple Replicas

Pod Disruption Budget

Health Checks

Cost Optimization

Troubleshooting

Next Steps