fedora-39-x86_64). This guide covers chroot lifecycle management for Copr administrators.
Commands Overview
Copr provides several frontend commands for chroot management:All commands must be run as the
copr-fe user on the Copr Frontend server.Fedora Branching Process
When a new Fedora version branches from Rawhide, Copr needs to create corresponding chroots and migrate builds.Why Branch Before Official Release?
Branching Workflow
Run branch-fedora Command
Before Fedora branches to version 31:This command:
- Creates new
fedora-31-*chroots fromfedora-rawhide-*chroots - Copies/forks latest successful Rawhide builds to new chroots
- Creates chroots in deactivated state
Wait for Actions to Process
Monitor the FE statistics page for the Actions peak to be processed.During this time:
- Don’t activate the new chroots yet
- New Rawhide builds won’t be copied (known limitation)
Verify Mock Configs
Check that mock configs are available:
- Visit mock-core-configs repository
- Verify configs for the new Fedora version exist
- Ensure
mock-core-configspackage is updated on builders
Activate New Chroots
Activate the chroots as soon as possible after processing:
The sooner you activate, the fewer Rawhide builds will be missing from the branched chroot.
Announce to Mailing List
Send announcement email to
[email protected]:Manual Branching (Per Architecture)
For individual architectures:Follow Fedora Branching
Projects can opt into automatic chroot creation:- New Fedora chroots are automatically enabled for the project
- Rawhide builds are forked to the new chroot
- Happens during the
branch-fedoracommand execution
EOL Deactivation Process
When a Fedora version reaches End of Life, disable its chroots:Mark Chroots as EOL
Wait a few weeks after official EOL, then:
No data is removed. Repositories are preserved. Users simply cannot submit new builds.
Send Announcement
Email
[email protected] with EOL notice.Use the disable chroots template.Disable Chroots Template
Email Template
Email Template
Rolling Chroot EOL
For rolling release chroots (Rawhide, CentOS Stream, etc.):Consider running this daily in the
copr-frontend-optional cron job.- Identifies rolling chroots with no recent builds
- Marks them for removal/deactivation
- Applies the same notification policy as standard EOL
Keeping Rolling Chroots Alive
Users can prevent EOL by:- Triggering a new build in the chroot
- Manually prolonging the chroot lifetime (if supported)
Chroot Comments
Add informational messages to specific chroots:Single Chroot
HTML formatting is supported in chroot comments.
Bulk Comments with Template
For managing comments across many chroots:Chroot Lifecycle States
Active
Active
Chroot is available for new builds.
Preserved
Preserved
Chroot is EOL but not deleted. Cannot submit new builds.
Deleted
Deleted
Chroot is completely removed.
Project Chroot Access
Projects can query available chroots:Outdated Chroots
Chroots scheduled for deletion:Multilib Chroots
For 32-bit multilib support on 64-bit architectures:Multilib requires both 64-bit and 32-bit chroots to be active.
Chroot Modifications
Track per-chroot configuration changes:Best Practices
- Branch Early: Run
branch-fedorabefore official Fedora branching - Monitor Actions: Watch the FE statistics page during branching
- Activate Quickly: Minimize time between branching and activation
- Wait Before EOL: Give users a few weeks after official EOL
- Announce Changes: Always email the mailing list
- Check Dependencies: Verify no services depend on chroots before EOL
- Test Mock Configs: Ensure mock-core-configs is updated before activation
Common Issues
Builds Missing from Branched Chroot
Builds Missing from Branched Chroot
Problem: Some Rawhide builds aren’t in the new chroot.Cause: New builds submitted to Rawhide after
branch-fedora but before activation aren’t copied.Solution: Activate new chroots quickly. Users can rebuild missing packages.Mock Config Not Found
Mock Config Not Found
Problem: Builds fail with “mock config not found.”Cause:
mock-core-configs not updated on builders.Solution: Update mock packages on all builder machines.Action Queue Backlog
Action Queue Backlog
Problem: Actions take too long to process.Cause: Many projects with
follow_fedora_branching enabled.Solution: Normal. Be patient, it processes eventually (~1 day).Mailing List Notifications
After chroot changes, send email to:- Address:
[email protected] - When: After enabling new chroots OR disabling EOL chroots
- Combined: Can describe both actions in one email if done together