Health Checks - Argo CD

Argo CD continuously monitors the health of application resources and surfaces their status in the UI and CLI. Understanding health checks is crucial for ensuring your applications are truly ready.

Overview

Argo CD provides built-in health assessments for standard Kubernetes resources and allows custom health checks for CRDs and special cases. Application health is determined by the worst health of its immediate child resources:

Healthy > Suspended > Progressing > Missing > Degraded > Unknown

For example, if an app has a Missing resource and a Degraded resource, the app is marked Degraded.

Resource health is not inherited from child resources. A Deployment’s health is based only on its own status, not its Pods.

Built-in Health Checks

Deployment, ReplicaSet, StatefulSet, DaemonSet

Healthy when:

Observed generation equals desired generation
Number of updated replicas equals desired replicas

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  # ...
status:
  observedGeneration: 5
  replicas: 3
  updatedReplicas: 3  # Must equal replicas
  readyReplicas: 3

Service (LoadBalancer)

Healthy when:

Service type is LoadBalancer
status.loadBalancer.ingress list has at least one IP or hostname

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 203.0.113.10  # Health check passes

Ingress

Healthy when:

status.loadBalancer.ingress list has at least one IP or hostname

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
status:
  loadBalancer:
    ingress:
    - hostname: app.example.com  # Health check passes

PersistentVolumeClaim

Healthy when:

status.phase is Bound

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-volume
status:
  phase: Bound  # Health check passes

Job

Suspended: If spec.suspended is true
Healthy: If job completed successfully
Degraded: If job failed

apiVersion: batch/v1
kind: Job
metadata:
  name: migration
spec:
  suspend: false
status:
  succeeded: 1  # Health check passes

CronJob

Degraded: If last scheduled job failed
Progressing: If last scheduled job is running
Healthy: Otherwise

apiVersion: batch/v1
kind: CronJob
metadata:
  name: backup
status:
  lastScheduleTime: "2024-03-04T10:00:00Z"
  # Health depends on status of last job

Health Status Types

Status	Meaning
`Healthy`	Resource is functioning correctly
`Progressing`	Resource is not healthy yet but making progress
`Degraded`	Resource is not functioning correctly
`Suspended`	Resource is suspended (e.g., paused Deployment, suspended CronJob)
`Missing`	Resource is not present in the cluster
`Unknown`	Health status cannot be determined

Custom Health Checks

Define custom health checks in Lua for CRDs or to override built-in checks.

Method 1: ConfigMap Configuration

Add custom health checks to the argocd-cm ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.health.cert-manager.io_Certificate: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.conditions ~= nil then
        for i, condition in ipairs(obj.status.conditions) do
          if condition.type == "Ready" and condition.status == "False" then
            hs.status = "Degraded"
            hs.message = condition.message
            return hs
          end
          if condition.type == "Ready" and condition.status == "True" then
            hs.status = "Healthy"
            hs.message = condition.message
            return hs
          end
        end
      end
    end
    
    hs.status = "Progressing"
    hs.message = "Waiting for certificate"
    return hs

Wildcard Health Checks

Apply a single health check to multiple resources:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations: |
    ec2.aws.crossplane.io/*:
      health.lua: |
        hs = {}
        if obj.status ~= nil then
          if obj.status.conditions ~= nil then
            for i, condition in ipairs(obj.status.conditions) do
              if condition.type == "Ready" then
                if condition.status == "True" then
                  hs.status = "Healthy"
                  hs.message = "Resource is ready"
                else
                  hs.status = "Progressing"
                  hs.message = condition.message or "Resource is not ready"
                end
                return hs
              end
            end
          end
        end
        hs.status = "Progressing"
        hs.message = "Waiting for status"
        return hs

Wildcard patterns only work with the resource.customizations key format, not resource.customizations.health.<group>_<kind>.

Example: Crossplane Resources

data:
  resource.customizations: |
    "*.aws.crossplane.io/*":
      health.lua: |
        hs = {}
        if obj.status ~= nil and obj.status.conditions ~= nil then
          for i, condition in ipairs(obj.status.conditions) do
            if condition.type == "Ready" then
              if condition.status == "True" then
                hs.status = "Healthy"
              else
                hs.status = "Degraded"
              end
              hs.message = condition.reason or ""
              return hs
            end
          end
        end
        hs.status = "Progressing"
        hs.message = "Provisioning resource"
        return hs

Example: Custom Application CRD

data:
  resource.customizations.health.app.example.com_MyApp: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.phase == "Running" then
        hs.status = "Healthy"
        hs.message = "Application is running"
      elseif obj.status.phase == "Failed" then
        hs.status = "Degraded"
        hs.message = obj.status.message or "Application failed"
      else
        hs.status = "Progressing"
        hs.message = "Application is starting"
      end
      return hs
    end
    hs.status = "Progressing"
    hs.message = "Waiting for status"
    return hs

Enabling Lua Standard Libraries

By default, standard Lua libraries are disabled for security. Enable them per resource:

data:
  resource.customizations.useOpenLibs.cert-manager.io_Certificate: "true"
  resource.customizations.health.cert-manager.io_Certificate: |
    -- Lua standard libraries available here
    hs = {}
    -- ... health check logic
    return hs

Only enable standard libraries if absolutely necessary, as they can pose security risks.

Method 2: Contribute Built-in Health Check

Contribute health checks directly to Argo CD’s codebase:

argo-cd/
└── resource_customizations/
    └── cert-manager.io/
        └── Certificate/
            ├── health.lua
            ├── health_test.yaml
            └── testdata/
                ├── healthy.yaml
                └── degraded.yaml

health.lua:

hs = {}
if obj.status ~= nil then
  if obj.status.conditions ~= nil then
    for i, condition in ipairs(obj.status.conditions) do
      if condition.type == "Ready" and condition.status == "True" then
        hs.status = "Healthy"
        hs.message = condition.message
        return hs
      end
    end
  end
end
hs.status = "Progressing"
hs.message = "Waiting for certificate"
return hs

health_test.yaml:

tests:
- healthStatus:
    status: Healthy
    message: Certificate is ready
  inputPath: testdata/healthy.yaml
- healthStatus:
    status: Progressing
    message: Waiting for certificate
  inputPath: testdata/degraded.yaml

Test with: go test -v ./util/lua/

Overriding Built-in Health Checks

Override Go-based health checks with Lua:

data:
  resource.customizations.health.apps_Deployment: |
    hs = {}
    -- Custom logic to override built-in Deployment health check
    if obj.status.availableReplicas == obj.spec.replicas then
      hs.status = "Healthy"
    else
      hs.status = "Progressing"
    end
    return hs

Resources with Go-based health checks:

PersistentVolumeClaim
Pod
Service
apiregistration.k8s.io/APIService
apps/DaemonSet
apps/Deployment
apps/ReplicaSet
apps/StatefulSet
argoproj.io/Workflow
autoscaling/HorizontalPodAutoscaler
batch/Job
extensions/Ingress
networking.k8s.io/Ingress

Ignoring Resource Health

Exclude specific resources from affecting application health:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: optional-component
  annotations:
    argocd.argoproj.io/ignore-healthcheck: "true"
spec:
  replicas: 1
  # ...

The application will not be affected by this Deployment’s health status.

Argo CD Application Health Check

Argo CD removed the built-in health check for argoproj.io/Application in v1.8. Restore it for App-of-Apps patterns:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.health.argoproj.io_Application: |
    hs = {}
    hs.status = "Progressing"
    hs.message = ""
    if obj.status ~= nil then
      if obj.status.health ~= nil then
        hs.status = obj.status.health.status
        if obj.status.health.message ~= nil then
          hs.message = obj.status.health.message
        end
      end
    end
    return hs

Health Check Examples

Example: Kafka Topic (Strimzi)

data:
  resource.customizations.health.kafka.strimzi.io_KafkaTopic: |
    hs = {}
    if obj.status ~= nil and obj.status.conditions ~= nil then
      for i, condition in ipairs(obj.status.conditions) do
        if condition.type == "Ready" then
          if condition.status == "True" then
            hs.status = "Healthy"
            hs.message = "Topic is ready"
          else
            hs.status = "Degraded"
            hs.message = condition.message or "Topic is not ready"
          end
          return hs
        end
      end
    end
    hs.status = "Progressing"
    hs.message = "Waiting for topic to be ready"
    return hs

Example: Sealed Secret

data:
  resource.customizations.health.bitnami.com_SealedSecret: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.conditions ~= nil then
        for i, condition in ipairs(obj.status.conditions) do
          if condition.type == "Synced" and condition.status == "True" then
            hs.status = "Healthy"
            hs.message = "SealedSecret synced successfully"
            return hs
          elseif condition.type == "Synced" and condition.status == "False" then
            hs.status = "Degraded"
            hs.message = condition.message or "Failed to sync SealedSecret"
            return hs
          end
        end
      end
    end
    hs.status = "Progressing"
    hs.message = "Waiting for SealedSecret to sync"
    return hs

Example: External Secrets

data:
  resource.customizations.health.external-secrets.io_ExternalSecret: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.conditions ~= nil then
        for i, condition in ipairs(obj.status.conditions) do
          if condition.type == "Ready" and condition.status == "True" then
            hs.status = "Healthy"
            hs.message = "ExternalSecret is ready"
            return hs
          elseif condition.type == "Ready" and condition.status == "False" then
            hs.status = "Degraded"
            hs.message = condition.message
            return hs
          end
        end
      end
    end
    hs.status = "Progressing"
    hs.message = "Waiting for ExternalSecret"
    return hs

Testing Health Checks

Test health checks locally before deploying:

# Install Argo CD locally
git clone https://github.com/argoproj/argo-cd.git
cd argo-cd

# Add your health check
mkdir -p resource_customizations/mygroup.io/MyKind
cat > resource_customizations/mygroup.io/MyKind/health.lua <<EOF
hs = {}
-- your health check logic
return hs
EOF

# Run tests
go test -v ./util/lua/

Troubleshooting

Resource Shows as Progressing Forever

Check if a custom health check exists:

kubectl get configmap argocd-cm -n argocd -o yaml | grep -A 20 customizations

Inspect resource status:
```
kubectl get <resource> <name> -o yaml
```

Check Argo CD application controller logs:

kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller

Custom Health Check Not Working

Verify ConfigMap format (key should be resource.customizations.health.<group>_<kind>)
Check for Lua syntax errors in controller logs
Ensure the resource group and kind match exactly

Restart application controller:

kubectl rollout restart deployment argocd-application-controller -n argocd

Application Shows Wrong Health

Remember: Application health is the worst health of immediate child resources:

# Check all resource health
argocd app get my-app

# Check specific resource
argocd app get my-app --resource <group>:<kind>:<namespace>/<name>

Best Practices

Start Simple

Begin with built-in checks before writing custom logic

Use Conditions

Leverage Kubernetes status conditions for consistent checks

Return Messages

Always provide meaningful health messages for debugging

Test Thoroughly

Create test cases for all health states

Next Steps

Sync Waves

Wait for resources to be healthy between waves

Resource Hooks

Run validation hooks after resources are healthy

Sync Options

Configure sync behavior

Creating Apps

Learn how to create applications

Get Started

Installation

Core Features

Configuration

Application Management

ApplicationSet

Manifest Sources

Operations

Security

​Overview

​Built-in Health Checks

​Deployment, ReplicaSet, StatefulSet, DaemonSet

​Service (LoadBalancer)

​Ingress

​PersistentVolumeClaim

​Job

​CronJob

​Health Status Types

​Custom Health Checks

​Method 1: ConfigMap Configuration

​Wildcard Health Checks

​Example: Crossplane Resources

​Example: Custom Application CRD

​Enabling Lua Standard Libraries

​Method 2: Contribute Built-in Health Check

​Overriding Built-in Health Checks

​Ignoring Resource Health

​Argo CD Application Health Check

​Health Check Examples

​Example: Kafka Topic (Strimzi)

​Example: Sealed Secret

​Example: External Secrets

​Testing Health Checks

​Troubleshooting

​Resource Shows as Progressing Forever

​Custom Health Check Not Working

​Application Shows Wrong Health

​Best Practices

Start Simple

Use Conditions

Return Messages

Test Thoroughly

​Next Steps

Sync Waves

Resource Hooks

Sync Options

Creating Apps

Build docs developers (and LLMs) love

Overview

Built-in Health Checks

Deployment, ReplicaSet, StatefulSet, DaemonSet

Service (LoadBalancer)

Ingress

PersistentVolumeClaim

Job

CronJob

Health Status Types

Custom Health Checks

Method 1: ConfigMap Configuration

Wildcard Health Checks

Example: Crossplane Resources

Example: Custom Application CRD

Enabling Lua Standard Libraries

Method 2: Contribute Built-in Health Check

Overriding Built-in Health Checks

Ignoring Resource Health

Argo CD Application Health Check

Health Check Examples

Example: Kafka Topic (Strimzi)

Example: Sealed Secret

Example: External Secrets

Testing Health Checks

Troubleshooting

Resource Shows as Progressing Forever

Custom Health Check Not Working

Application Shows Wrong Health

Best Practices

Next Steps