Skip to main content
You can store logs from your Aiven services in an external Elasticsearch service, enabling you to search, analyze, and archive logs using your own Elasticsearch infrastructure.

​
Prerequisites

Before setting up the integration, collect these connection details:
VariableDescription
ELASTICSEARCH_USERUsername to access the Elasticsearch service
ELASTICSEARCH_PASSWORDPassword to access the Elasticsearch service
ELASTICSEARCH_HOSTHTTPS service host of your external Elasticsearch service
ELASTICSEARCH_PORTPort to use for the connection
CA_CERTIFICATECA certificate in PEM format (optional, if necessary)
CONNECTION_NAMEName of this external connection to be used with Aiven services

​
Create external Elasticsearch integration

Start by setting up an external service integration for Elasticsearch.
1

Access integration endpoints

Log in to the Aiven Console and navigate to your project.
2

Select External Elasticsearch

Click Integration endpoints and select External Elasticsearch from the list.
3

Add new endpoint

Select Add new endpoint to create a new integration.
4

Configure endpoint name

Set a preferred endpoint name (this will be your CONNECTION_NAME).
5

Configure connection URL

In the connection URL field, set the connection string:
https://ELASTICSEARCH_USER:ELASTICSEARCH_PASSWORD@ELASTICSEARCH_HOST:ELASTICSEARCH_PORT
Replace the placeholders with your actual values.Example:
https://admin:[email protected]:9200
6

Set index prefix

Configure the desired index prefix. This should not overlap with any existing indexes in your Elasticsearch service.
Logs are split per day with index names consisting of your prefix and a date in year-month-day format (e.g., logs-2022-08-30).
7

Add CA certificate (optional)

If your Elasticsearch service uses a custom CA certificate, add the certificate body in PEM format.This is typically needed if:
  • Your Elasticsearch server uses a self-signed certificate
  • Your server’s certificate is issued by an internal CA
8

Create endpoint

Click Create to save the endpoint configuration.

​
Send service logs to Elasticsearch

After creating the integration endpoint, enable it for your Aiven services.
1

Navigate to service

Click Services and open the service whose logs you want to send to Elasticsearch.
2

Access integrations

On the sidebar, click Integrations.
3

Select Elasticsearch Logs

Select Elasticsearch Logs from the list of available integrations.
4

Enable integration

Select the Endpoint name you created earlier and click Enable.
5

Verify logs

Logs will begin flowing to your Elasticsearch service. Check your Elasticsearch cluster to verify the indices are being created.

​
Index naming convention

Logs are automatically organized into daily indices:
  • Format: {prefix}-{year}-{month}-{day}
  • Example: If your prefix is aiven-logs, indices will be named:
    • aiven-logs-2024-03-01
    • aiven-logs-2024-03-02
    • aiven-logs-2024-03-03
This daily index pattern:
  • Facilitates easier log retention management
  • Improves search performance for time-based queries
  • Simplifies index lifecycle management

​
Using Aiven CLI

You can also set up the integration using the Aiven CLI: 
avn service integration-endpoint-create \
  --project your-project \
  --endpoint-name CONNECTION_NAME \
  --endpoint-type external_elasticsearch_logs \
  --user-config-json '{
    "url": "https://ELASTICSEARCH_USER:ELASTICSEARCH_PASSWORD@ELASTICSEARCH_HOST:ELASTICSEARCH_PORT",
    "index_prefix": "aiven-logs"
  }'

​
CLI commands reference

​
Log structure

Logs sent to Elasticsearch include:
  • Timestamp - When the log event occurred
  • Hostname - Service host that generated the log
  • Service name - Aiven service name
  • Log level - Severity level (INFO, WARNING, ERROR, etc.)
  • Message - Log message content
  • Additional fields - Service-specific metadata

​
Troubleshooting

​
Logs not appearing

  • Verify the connection URL format is correct with proper URL encoding
  • Check that Elasticsearch credentials have write permissions
  • Confirm the Elasticsearch service is accessible from Aiven infrastructure
  • Verify the index prefix doesn’t conflict with existing indices
  • Check Elasticsearch cluster health and available disk space

​
Authentication failures

  • Verify username and password are correct
  • Check that the user has appropriate permissions in Elasticsearch
  • Confirm the Elasticsearch service allows connections from Aiven
  • Review any IP allowlists or firewall rules

​
Certificate errors

  • Ensure the CA certificate is in proper PEM format
  • Verify the certificate matches your Elasticsearch server
  • Check that the certificate hasn’t expired
  • Confirm the certificate chain is complete

​
Index creation failures

  • Check that the index prefix doesn’t conflict with existing indices
  • Verify Elasticsearch has sufficient resources to create new indices
  • Review Elasticsearch logs for any error messages
  • Confirm index templates or ILM policies aren’t blocking creation

​
Connection timeout

  • Verify the Elasticsearch host and port are correct
  • Check that the Elasticsearch service is running
  • Confirm network connectivity between Aiven and Elasticsearch
  • Review firewall rules and security groups

​
Index management

​
Retention policies

To manage log retention:
  1. Use Elasticsearch Index Lifecycle Management (ILM) policies
  2. Set up automated deletion of old indices
  3. Configure index rollover based on size or age
  4. Archive indices to cold storage before deletion

​
Search optimization

For better search performance:
  • Use index patterns for time-based queries
  • Configure appropriate shard and replica counts
  • Set up index templates for consistent field mappings
  • Consider using data tiers for older logs

​
Storage considerations

  • Monitor Elasticsearch disk usage regularly
  • Set up alerts for high disk usage
  • Plan capacity based on log volume and retention period
  • Use compression for older indices

​
Security best practices

  • Always use HTTPS for the connection URL
  • Store credentials securely
  • Use strong passwords for Elasticsearch users
  • Limit Elasticsearch user permissions to only what’s needed
  • Regularly rotate credentials
  • Monitor access logs for suspicious activity

Build docs developers (and LLMs) love

Get started for freeTalk to us