Automated Performance Testing

Automated performance testing establishes a repeatable and consistent process for checking reliability across different stages of your development and release cycle. This guide helps you plan and implement a comprehensive automation strategy.

Why Automate Performance Tests

Automation transforms performance testing from reactive to proactive:

Continuous Confidence

Expand test coverage, improve maintenance, and increase confidence in testing outcomes through consistent execution

Early Detection

Catch performance issues earlier in the SDLC when they’re cheaper and easier to fix

Cross-Team Collaboration

Foster shared practices and accountability for reliability across engineering teams

Efficient Process

Create a more efficient and effective testing process through standardization

Automation doesn’t eliminate manual testing. It complements it by establishing routines for consistent performance validation.

Beyond CI/CD

While CI/CD integration is important, it’s not the only way to automate tests:

CI/CD Pipelines
Scheduled Tests
Cloud Scheduling
Manual Triggers

Run tests on code changes with pass/fail criteria:

.github/workflows/performance.yml

name: Performance Tests
on: [pull_request]
jobs:
  smoke-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run k6 smoke test
        uses: grafana/[email protected]
        with:
          filename: tests/smoke.js

Use cron jobs for regular testing:

.github/workflows/scheduled-tests.yml

name: Scheduled Load Tests
on:
  schedule:
    - cron: '0 2 * * *'  # Daily at 2 AM
jobs:
  load-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run k6 load test
        uses: grafana/[email protected]
        with:
          filename: tests/load.js

Use Grafana Cloud k6 for managed scheduling:

// Schedule via k6 Cloud
k6 cloud tests/load.js --schedule "0 */6 * * *"

Planning Your Automation Strategy

Step 1: Define Test Purposes

Determine what each test should accomplish:

Performance baselines

Compare current performance against established baselines

Trend analysis

Observe performance metrics over time to detect gradual degradation

Regression detection

Catch performance regressions in new releases

SLO validation

Verify Service Level Objectives on a regular basis

Quality gates

Set pass/fail criteria in CI/CD pipelines

Step 2: Choose Tests to Automate

Start simple and iterate:

// Example: Modular test design
// scenarios/checkout.js
export function checkoutFlow() {
  // Reusable checkout scenario
}

// tests/smoke-checkout.js
import { checkoutFlow } from '../scenarios/checkout.js';
export const options = {
  vus: 3,
  duration: '1m',
};
export default checkoutFlow;

// tests/load-checkout.js
import { checkoutFlow } from '../scenarios/checkout.js';
export const options = {
  stages: [
    { duration: '5m', target: 50 },
    { duration: '10m', target: 50 },
  ],
};
export default checkoutFlow;

Modularize scenarios and workloads separately. This lets you reuse logic across different test types (smoke, load, stress).

Step 3: Model Scenarios and Workload

Create different test types for each scenario:

Scenario	Smoke	Load	Stress	Spike	Soak
GET /api/products	1 iteration	100 req/s, 3m	1500 req/s, 5m	-	-
Checkout flow	3 iterations	50 VUs, 5m	-	200 VUs, 1m	-
User registration	1 iteration	20 VUs, 5m	50 VUs, 10m	-	100 VUs, 8h

Always create smoke tests for script validation and load tests for baseline comparisons.

Environment-Based Automation

Development Environment

Purpose: Validate test scripts and basic functionality

// smoke-local.js
import http from 'k6/http';
import { check } from 'k6';

export const options = {
  vus: 1,
  iterations: 1,
};

export default function () {
  const res = http.get(__ENV.BASE_URL || 'http://localhost:3000');
  check(res, {
    'status is 200': (r) => r.status === 200,
    'response time < 500ms': (r) => r.timings.duration < 500,
  });
}

When to run: Before committing code, during development

QA Environment

Purpose: Functional validation with smoke tests

.github/workflows/qa-tests.yml

name: QA Smoke Tests
on:
  push:
    branches: [develop]
jobs:
  smoke-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run all smoke tests
        run: |
          k6 run tests/smoke-api.js
          k6 run tests/smoke-checkout.js
          k6 run tests/smoke-auth.js

When to run: On every push to develop branch

Pre-Release Environment

Purpose: Comprehensive testing with quality gates

// load-test-with-thresholds.js
export const options = {
  stages: [
    { duration: '5m', target: 100 },
    { duration: '10m', target: 100 },
    { duration: '5m', target: 0 },
  ],
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(90)<600'],
    'http_req_duration{type:api}': ['p(95)<800'],
  },
};

Strategy:

Run all test types (load, stress, spike)
Execute each test at least twice
Schedule tests every 4-8 hours during pre-release period
Validate before release approval

Staging Environment

Purpose: Track performance trends

.github/workflows/staging-baseline.yml

name: Staging Baseline Tests
on:
  schedule:
    - cron: '0 0 * * 2,5'  # Tuesday and Friday
jobs:
  baseline-tests:
    runs-on: ubuntu-latest
    steps:
      - name: Run baseline tests
        run: |
          k6 run --out experimental-prometheus-rw tests/load-api.js
          k6 run --out experimental-prometheus-rw tests/load-checkout.js

When to run: 2-3 times per week

Production Environment

Purpose: Continuous monitoring and validation

Synthetic Monitoring
Low-Load Testing
Canary Testing

// synthetic-monitor.js
import http from 'k6/http';
import { check } from 'k6';

export const options = {
  vus: 1,
  duration: '30s',
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(95)<1000'],
  },
};

export default function () {
  const res = http.get('https://api.example.com/health');
  check(res, {
    'API available': (r) => r.status === 200,
    'response time OK': (r) => r.timings.duration < 500,
  });
}

Schedule to run every 5 minutes with alerting.

// production-baseline.js
export const options = {
  stages: [
    { duration: '2m', target: 25 },  // 50% of average load
    { duration: '5m', target: 25 },
    { duration: '2m', target: 0 },
  ],
};

Run weekly during off-peak hours.

// canary-validation.js
export const options = {
  scenarios: {
    production_canary: {
      executor: 'constant-vus',
      vus: 50,
      duration: '10m',
      env: { TARGET: 'https://canary.example.com' },
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(95)<600'],
  },
};

Run during canary deployments.

Complete Automation Plan Example

Test	Environment	Type	Workload	Automation	Frequency
API smoke	QA	Smoke	1 iteration	CI pipeline	Every commit
API load	Pre-release	Load	100 req/s, 3m	Scheduled	3x/day during pre-release
API stress	Pre-release	Stress	1500 req/s, 5m	Scheduled	3x/day during pre-release
API baseline	Staging	Load	100 req/s, 3m	Scheduled	2x/week
Checkout smoke	QA	Smoke	3 iterations	CI pipeline	Every commit
Checkout load	Pre-release	Load	50 VUs, 5m	Scheduled	3x/day during pre-release
Checkout spike	Pre-release	Spike	200 VUs, 1m	Scheduled	3x/day during pre-release
Production health	Production	Synthetic	1 VU, continuous	Scheduled	Every 5 min
Production baseline	Production	Load (50%)	50 req/s, 3m	Scheduled	Weekly

Result Analysis Strategy

Store Results

Prometheus
Grafana Cloud k6
InfluxDB
JSON

// k6 run --out experimental-prometheus-rw test.js
export const options = {
  // Test configuration
};

Configure Prometheus endpoint:

export K6_PROMETHEUS_RW_SERVER_URL=http://prometheus:9090/api/v1/write

# Set cloud token
export K6_CLOUD_TOKEN=your-token

# Run test with cloud output
k6 run --out cloud test.js

k6 run --out influxdb=http://localhost:8086/k6 test.js

k6 run --out json=results.json test.js

Define Key Metrics

import { Trend, Rate, Counter } from 'k6/metrics';

const apiLatency = new Trend('api_latency', true);
const errorRate = new Rate('error_rate');
const apiCalls = new Counter('api_calls');

export const options = {
  thresholds: {
    'api_latency': ['p(95)<500', 'p(99)<1000'],
    'error_rate': ['rate<0.01'],
    'http_req_duration{endpoint:critical}': ['p(90)<300'],
  },
};

export default function () {
  const start = Date.now();
  const res = http.get('https://api.example.com');
  apiLatency.add(Date.now() - start);
  errorRate.add(res.status !== 200);
  apiCalls.add(1);
}

Set Up Alerts

prometheus-alerts.yml

groups:
  - name: k6_performance
    interval: 30s
    rules:
      - alert: HighErrorRate
        expr: rate(k6_http_req_failed[5m]) > 0.01
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value }} (threshold: 0.01)"
      
      - alert: SlowResponseTime
        expr: histogram_quantile(0.95, k6_http_req_duration_bucket) > 1000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Slow response times"
          description: "P95 latency is {{ $value }}ms"

CI/CD Integration Examples

GitHub Actions
GitLab CI
Jenkins

.github/workflows/performance.yml

name: Performance Tests
on:
  pull_request:
  push:
    branches: [main]

jobs:
  smoke-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Run k6 smoke test
        uses: grafana/[email protected]
        with:
          filename: tests/smoke.js
          flags: --out json=results.json
      
      - name: Upload results
        uses: actions/upload-artifact@v3
        if: always()
        with:
          name: k6-results
          path: results.json
  
  load-test:
    runs-on: ubuntu-latest
    if: github.event_name == 'push'
    steps:
      - uses: actions/checkout@v3
      
      - name: Run k6 load test
        uses: grafana/[email protected]
        env:
          K6_CLOUD_TOKEN: ${{ secrets.K6_CLOUD_TOKEN }}
        with:
          filename: tests/load.js
          flags: --out cloud

.gitlab-ci.yml

stages:
  - test
  - performance

smoke_test:
  stage: test
  image: grafana/k6:latest
  script:
    - k6 run tests/smoke.js
  artifacts:
    reports:
      junit: results.xml

load_test:
  stage: performance
  image: grafana/k6:latest
  only:
    - main
  script:
    - k6 run --out cloud tests/load.js
  variables:
    K6_CLOUD_TOKEN: $K6_CLOUD_TOKEN

Jenkinsfile

pipeline {
  agent any
  
  stages {
    stage('Smoke Test') {
      steps {
        sh 'k6 run tests/smoke.js'
      }
    }
    
    stage('Load Test') {
      when {
        branch 'main'
      }
      steps {
        withCredentials([string(credentialsId: 'k6-cloud-token', variable: 'K6_CLOUD_TOKEN')]) {
          sh 'k6 run --out cloud tests/load.js'
        }
      }
    }
  }
  
  post {
    always {
      archiveArtifacts artifacts: '*.json', allowEmptyArchive: true
    }
  }
}

Best Practices

Start Simple

Begin with a few tests across different environments. Expand coverage gradually as you learn.

Maintain Consistency

Always run identical tests. Don’t change workload or scenario between runs you want to compare.

Run Tests Twice

Schedule each test to run twice consecutively for better comparison and to identify unreliable tests.

Control Risky Tests

Don’t fully automate heavy-load tests that might cause outages. Keep them manual with supervision.

Use Thresholds Wisely

Start with warning thresholds, not blocking ones. Mature your criteria before blocking releases.

Correlate Data

Connect test results with observability data to find root causes faster.

Stopping Tests Automatically

Stop tests when critical conditions are met:

export const options = {
  thresholds: {
    // Abort if error rate exceeds 5%
    http_req_failed: [
      { threshold: 'rate<0.05', abortOnFail: true },
    ],
    // Abort if p95 latency exceeds 3 seconds
    http_req_duration: [
      { threshold: 'p(95)<3000', abortOnFail: true },
    ],
  },
};

Use abortOnFail carefully. It stops the test immediately when the threshold is crossed, which may prevent gathering complete data.

Complete Automation Example

Here’s a full automation setup:

tests/api-load-test.js

import http from 'k6/http';
import { check, sleep } from 'k6';
import { Rate, Trend } from 'k6/metrics';

const errorRate = new Rate('errors');
const apiDuration = new Trend('api_duration');

export const options = {
  stages: [
    { duration: '2m', target: 50 },
    { duration: '5m', target: 50 },
    { duration: '2m', target: 0 },
  ],
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(95)<500'],
    errors: ['rate<0.02'],
    api_duration: ['p(99)<800'],
  },
  tags: {
    test_type: 'load',
    environment: __ENV.ENVIRONMENT || 'staging',
  },
};

const BASE_URL = __ENV.BASE_URL || 'https://api.staging.example.com';

export default function () {
  const start = Date.now();
  
  const res = http.get(`${BASE_URL}/api/products`, {
    tags: { endpoint: 'products' },
  });
  
  const success = check(res, {
    'status is 200': (r) => r.status === 200,
    'has products': (r) => r.json('products').length > 0,
  });
  
  errorRate.add(!success);
  apiDuration.add(Date.now() - start);
  
  sleep(1);
}

.github/workflows/automated-testing.yml

name: Automated Performance Testing

on:
  push:
    branches: [main]
  schedule:
    - cron: '0 */8 * * *'  # Every 8 hours
  workflow_dispatch:

jobs:
  performance-tests:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        test: [smoke, load, stress]
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Run ${{ matrix.test }} test
        uses: grafana/[email protected]
        env:
          K6_CLOUD_TOKEN: ${{ secrets.K6_CLOUD_TOKEN }}
          BASE_URL: https://api.staging.example.com
          ENVIRONMENT: staging
        with:
          filename: tests/${{ matrix.test }}-test.js
          flags: --out cloud
      
      - name: Notify on failure
        if: failure()
        uses: actions/github-script@v6
        with:
          script: |
            github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: `Performance test failed: ${{ matrix.test }}`,
              body: `The ${{ matrix.test }} test failed. Check the workflow run for details.`,
              labels: ['performance', 'automated-test']
            })

This comprehensive setup provides continuous performance validation across your entire development lifecycle.

Using k6

Testing Guides

Browser Testing

Results & Visualization

Automated Performance Testing

Why Automate Performance Tests

Continuous Confidence

Early Detection

Cross-Team Collaboration

Efficient Process

Beyond CI/CD

Planning Your Automation Strategy

Step 1: Define Test Purposes

Step 2: Choose Tests to Automate

Step 3: Model Scenarios and Workload

Environment-Based Automation

Development Environment

QA Environment

Pre-Release Environment

Staging Environment

Production Environment

Complete Automation Plan Example

Result Analysis Strategy

Store Results

Define Key Metrics

Set Up Alerts

CI/CD Integration Examples

Best Practices

Start Simple

Maintain Consistency

Run Tests Twice

Control Risky Tests

Use Thresholds Wisely

Correlate Data

Stopping Tests Automatically

Complete Automation Example

Build docs developers (and LLMs) love

Using k6

Testing Guides

Browser Testing

Results & Visualization

​Why Automate Performance Tests

Continuous Confidence

Early Detection

Cross-Team Collaboration

Efficient Process

​Beyond CI/CD

​Planning Your Automation Strategy

​Step 1: Define Test Purposes

​Step 2: Choose Tests to Automate

​Step 3: Model Scenarios and Workload

​Environment-Based Automation

​Development Environment

​QA Environment

​Pre-Release Environment

​Staging Environment

​Production Environment

​Complete Automation Plan Example

​Result Analysis Strategy

​Store Results

​Define Key Metrics

​Set Up Alerts

​CI/CD Integration Examples

​Best Practices

Start Simple

Maintain Consistency

Run Tests Twice

Control Risky Tests

Use Thresholds Wisely

Correlate Data

​Stopping Tests Automatically

​Complete Automation Example

Build docs developers (and LLMs) love

Why Automate Performance Tests

Beyond CI/CD

Planning Your Automation Strategy

Step 1: Define Test Purposes

Step 2: Choose Tests to Automate

Step 3: Model Scenarios and Workload

Environment-Based Automation

Development Environment

QA Environment

Pre-Release Environment

Staging Environment

Production Environment

Complete Automation Plan Example

Result Analysis Strategy

Store Results

Define Key Metrics

Set Up Alerts

CI/CD Integration Examples

Best Practices

Stopping Tests Automatically

Complete Automation Example