Automation and Build Processes

Terra Packages implements a sophisticated automated build system that handles package detection, dependency resolution, compilation, caching, and deployment. This page explains how the automation works and how to leverage it effectively.

Build System Architecture

The Terra Packages build system consists of several interconnected components:

┌─────────────────┐
│  Git Changes    │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  anda ci        │  Detects changed packages
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Build Matrix   │  JSON matrix with pkg, arch, labels
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Runner Select  │  Chooses appropriate runner
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  anda build     │  Compiles package
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Subatomic      │  Uploads to repository
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Madoguchi      │  Tracks build status
└─────────────────┘

Change Detection

The anda ci command analyzes git history to determine which packages need to be built.

How It Works

Compares current commit with base branch (e.g., frawhide)
Identifies modified files under anda/
Maps files to package definitions
Generates a build matrix including:
- Package path
- Target architecture
- Package labels
- Dependencies

Build Matrix Format

The build matrix is a JSON array of objects: Each entry represents one build job with specific architecture and configuration.

Package Labels

Labels control various aspects of the build process. They’re defined in the package’s anda.hcl file.

Common Labels

Label	Value	Effect
`large`	`1`	Use large runner with more resources
`sccache`	`0`	Disable compiler caching
`mock`	`1`	Build in mock chroot (isolated)
`subrepo`	`name`	Upload to terra- repository
`no_upload_srpms`	`1`	Skip source package upload
`nightly`	`1`	Exclude from automatic updates
`weekly`	`1`	Update weekly instead of continuous
`updbranch`	`1`	Update from specific branch

Example Label Usage

Runner Selection Logic

Terra Packages uses dynamic runner selection based on package requirements: The selection priority:

Custom builder - If specified via workflow input
arm64-lg - Large ARM64 packages (e.g., Chromium on aarch64)
ubuntu-22.04-arm - Standard ARM64 packages
cirun-x86-64-lg - Large x86_64 packages (dynamic runner)
ubuntu-22.04 - Standard x86_64 packages

Large runners are ephemeral and created on-demand using Cirun for x86_64 builds.

Build Container Environment

All builds run in privileged containers based on the Terra builder image: The builder container includes:

Fedora Rawhide base system
Andaman (anda) build tool
Mock for isolated builds
DNF5 package manager
Rust toolchain
Common build dependencies

Containers run with --privileged and SYS_ADMIN capability to support mock’s filesystem operations and systemd-nspawn containers.

Compiler Caching with sccache

sccache provides distributed compilation caching to speed up builds.

Cache Configuration

Caching is configured per package:

Cache Keys

Caches are keyed by:

Version - Fedora version (rawhide, f44, etc.)
Architecture - x86_64 or aarch64
Package path - Full path to package definition

This ensures cache hits for repeated builds of the same package while avoiding cross-contamination.

Cache Statistics

After each build, cache statistics are reported: The summary shows:

Cache hit rate
Compilation time saved
Cache size

Build Process Stages

Each package build goes through several stages:

1. CI Setup Script

Optional pre-build setup for complex packages: The ci_setup.rhai script can:

Download additional sources
Generate files
Configure build environment
Install extra dependencies

2. Dependency Installation

Automatic build dependency installation:

Packages using mock (isolated builds) skip this step as mock handles dependencies internally.

3. Package Compilation

The actual build using Andaman: Build flags:

-D "vendor Terra" - Set vendor to Terra
-D "__python %{__python3}" - Use Python 3
-c terra-{version}-{arch} - Use Terra mock config
-rrpmbuild - Use rpmbuild runner (non-mock builds)

4. Artifact Generation

Built packages are organized and uploaded: Artifact naming:

Forward slashes replaced with @
Format: {package}@{subpackage}@pkg-{arch}-{version}
Example: anda@apps@firefox@pkg-x86_64-rawhide

Repository Upload

Packages are uploaded to Subatomic, Terra’s package repository server.

Binary Package Upload

Source Package Upload

Repository Structure

Packages can be uploaded to different repositories:

Repository	Purpose
`terrarawhide`	Main rolling-release repository
`terrarawhide-source`	Source packages
`terrarawhide-browser`	Sub-repository for browsers (example)
`terraf44`	Fedora 44 stable repository

The --prune flag removes older versions of the same package automatically.

Build Status Tracking

Madoguchi tracks build status for all packages.

Success Notification

Failure Notification

Madoguchi tracks:

Build success/failure status
Build duration
Package versions
Historical build data
Architecture-specific builds

Automatic Updates

The update workflow automatically checks for new upstream versions.

Update Frequency

Updates run every 10 minutes, checking all packages without exclusion labels.

Update Excludes

Excluded packages:

nightly=1 - Packages tracking nightly builds (updated separately)
weekly=1 - Packages updated weekly (not continuous)
updbranch=1 - Packages tracking specific branches

Update Detection Methods

Andaman supports multiple update detection strategies:

GitHub Releases - Checks latest release tag
Git Tags - Monitors git repository tags
HTTP HEAD - Checks file modification time
Custom Scripts - Runs update.rhai scripts
Version Comparison - Parses version strings

Multi-Branch Updates

Updates are applied to all active release branches: The process:

Commit update to main branch (frawhide)
Generate patch file
Apply patch to each stable branch
Commit with same message
Push all branches simultaneously

All update commits are SSH-signed for security and auditability.

AppStream Generation

After building, AppStream metadata is generated for package discovery in software centers.

Generation Process

Veto Reporting

Packages that fail AppStream validation are reported: Common veto reasons:

Missing AppStream metadata file
Invalid desktop file
Missing icon
Malformed XML

Bootstrap Process

The bootstrap workflow builds the build system from scratch.

Bootstrap Stages

Base System Setup

SRPM Macros

Andaman

Terra Infrastructure

Subatomic

The bootstrap must build packages in a specific order due to dependencies. Later packages depend on earlier ones being installed.

Manual Build Workflow

Maintainers can manually trigger builds with custom parameters.

Build Trigger

Package Selection

The workflow uses a clever trick to force specific packages to build: By touching .build files and creating a temporary commit, the workflow tricks anda ci into detecting those packages as changed.

Architecture Filtering

This allows building specific architectures:

Input: x86_64 aarch64
Result: Build for both architectures
Input: x86_64
Result: Build for x86_64 only

Best Practices

For Package Maintainers

Use appropriate labels
- Mark large packages with large=1
- Disable sccache for small packages with sccache=0
- Use sub-repositories for themed package groups
Optimize builds
- Enable compiler caching for C/C++ projects
- Use mock for packages with complex dependencies
- Write ci_setup.rhai for complex pre-build steps
Version management
- Use nightly=1 for packages tracking development versions
- Use weekly=1 for large packages that change frequently
- Implement update.rhai for custom update logic

For CI/CD Efficiency

Minimize rebuilds
- Use fail-fast: false to build all architectures even if one fails
- Leverage sccache for incremental builds
- Cache downloaded sources
Resource optimization
- Reserve large runners for truly large packages
- Use compression-level 0 for pre-compressed artifacts
- Prune old packages during upload
Monitoring
- Review AppStream veto reports
- Check Madoguchi for build trends
- Monitor cache hit rates

The entire build system is designed for transparency and reproducibility. All builds run in clean containers with declared dependencies.

Package Management

CI/CD

Resources

​Automation and Build Processes

​Build System Architecture

​Change Detection

​How It Works

​Build Matrix Format

​Package Labels

​Common Labels

​Example Label Usage

​Runner Selection Logic

​Build Container Environment

​Compiler Caching with sccache

​Cache Configuration

​Cache Keys

​Cache Statistics

​Build Process Stages

​1. CI Setup Script

​2. Dependency Installation

​3. Package Compilation

​4. Artifact Generation

​Repository Upload

​Binary Package Upload

​Source Package Upload

​Repository Structure

​Build Status Tracking

​Success Notification

​Failure Notification

​Automatic Updates

​Update Frequency

​Update Excludes

​Update Detection Methods

​Multi-Branch Updates

​AppStream Generation

​Generation Process

​Veto Reporting

​Bootstrap Process

​Bootstrap Stages

​Manual Build Workflow

​Build Trigger

​Package Selection

​Architecture Filtering

​Best Practices

​For Package Maintainers

​For CI/CD Efficiency

Build docs developers (and LLMs) love

Automation and Build Processes

Build System Architecture

Change Detection

How It Works

Build Matrix Format

Package Labels

Common Labels

Example Label Usage

Runner Selection Logic

Build Container Environment

Compiler Caching with sccache

Cache Configuration

Cache Keys

Cache Statistics

Build Process Stages

1. CI Setup Script

2. Dependency Installation

3. Package Compilation

4. Artifact Generation

Repository Upload

Binary Package Upload

Source Package Upload

Repository Structure

Build Status Tracking

Success Notification

Failure Notification

Automatic Updates

Update Frequency

Update Excludes

Update Detection Methods

Multi-Branch Updates

AppStream Generation

Generation Process

Veto Reporting

Bootstrap Process

Bootstrap Stages

Manual Build Workflow

Build Trigger

Package Selection

Architecture Filtering

Best Practices

For Package Maintainers

For CI/CD Efficiency