Skip to main content
Warewulf supplements provisioned OS images with an overlay system. Overlays are collections of files and templates that are rendered per-node and applied over the image during provisioning. Overlays are the primary mechanism for adding functionality to Warewulf. Much of even core functionality is implemented as distribution overlays, and this flexibility is also available for local, custom overlays. By combining templates with tags, network tags, and resources, the node registry (nodes.conf) becomes an expressive metadata store for arbitrary cluster node configuration.

​
System overlays vs. runtime overlays

A node or profile can configure overlays in two ways:
  • System overlays are applied only during boot, alongside the node image. Use them for configuration that is baked in at provisioning time.
  • Runtime overlays are also applied periodically while the node is running. Use them for configuration that must stay current without a reboot (for example, SSH authorized keys or /etc/hosts).
wwctl profile set default \
  --system-overlays="wwinit,wwclient,fstab,hostname,ssh.host_keys,systemd.netname,NetworkManager" \
  --runtime-overlays="hosts,ssh.authorized_keys"
Multiple overlays can be applied to a single node. Overlays from multiple profiles are appended together.

​
Distribution overlays vs. site overlays

Warewulf distinguishes between two kinds of overlays:
  • Distribution overlays are shipped with Warewulf and stored at /usr/share/warewulf/overlays/ (path may vary by distribution and configuration).
  • Site overlays are created or added locally and stored at /var/lib/warewulf/overlays/. A site overlay always takes precedence over a distribution overlay with the same name.
Any modification to a distribution overlay using wwctl automatically creates a site overlay cloned from the distribution overlay. Distribution overlays themselves are never modified.

​
Overlay directory structure

An overlay is a directory containing a single rootfs/ subdirectory that represents the root of the cluster node’s file system. 
/usr/share/warewulf/overlays/issue
└── rootfs
    └── etc
        └── issue.ww
Files ending in .ww are Go templates. When an overlay is built, template files are rendered per-node and the .ww suffix is dropped — so etc/issue.ww becomes /etc/issue on the node.

​
Listing overlays

# List all available overlays
wwctl overlay list

# List all overlays and their files
wwctl overlay list --all

# List files in a specific overlay
wwctl overlay list --all fstab
Example output: 
OVERLAY NAME  FILES/DIRS    SITE
------------  ----------    ----
fstab         etc/          false
fstab         etc/fstab.ww  false

​
Adding overlays to nodes and profiles

Assign overlays to a profile or node using the --system-overlays and --runtime-overlays flags: 
wwctl profile set default \
  --system-overlays="wwinit,wwclient,fstab,hostname,ssh.host_keys,systemd.netname,NetworkManager" \
  --runtime-overlays="hosts,ssh.authorized_keys"
You can also set overlays on individual nodes: 
wwctl node set n1 \
  --system-overlays="wwinit,wwclient,fstab,hostname,ssh.host_keys,NetworkManager" \
  --runtime-overlays="hosts,ssh.authorized_keys"

​
Building overlays

Overlays are compiled into compressed images for distribution to cluster nodes. Each node typically has two overlay images — one for system overlays and one for runtime overlays. 
wwctl overlay build
Example output: 
Building system overlay image for n1
Created image for n1 system overlay: /var/lib/warewulf/provision/overlays/n1/__SYSTEM__.img
Compressed image for n1 system overlay: /var/lib/warewulf/provision/overlays/n1/__SYSTEM__.img.gz
Building runtime overlay image for n1
Created image for n1 runtime overlay: /var/lib/warewulf/provision/overlays/n1/__RUNTIME__.img
Compressed image for n1 runtime overlay: /var/lib/warewulf/provision/overlays/n1/__RUNTIME__.img.gz
Overlay images for multiple nodes are built in parallel. By default, one worker per CPU is used. Specify --workers to control parallelism: 
wwctl overlay build --workers 4

​
Overlay autobuild

Warewulf can automatically rebuild overlays when changes are detected. This is controlled by the autobuild overlays setting in warewulf.conf: 
warewulf:
  autobuild overlays: true
Autobuild is not 100% reliable. Manual overlay builds are often necessary, particularly after changes to overlay templates or node configuration.

​
Creating and modifying overlays

​
Creating a new overlay

wwctl overlay create myoverlay
A new overlay is an empty directory. Add files to make it useful.

​
Importing files

Import files from the Warewulf server into an overlay: 
wwctl overlay import --parents issue /etc/issue
This imports /etc/issue from the server into the issue overlay, recreating its parent directory structure.

​
Editing files

Edit an overlay file interactively: 
wwctl overlay edit issue /etc/issue

​
Showing overlay contents

Inspect the raw content of an overlay file: 
wwctl overlay show issue /etc/issue
For template files (.ww), render the output as it would appear for a specific node: 
wwctl overlay show issue /etc/issue.ww --render=n1
It is not possible to delete files using an overlay. Overlay files can only be added or modified.

​
Deleting and re-importing files

To replace a plain file with a template: 
wwctl overlay delete issue /etc/issue
wwctl overlay import issue /etc/issue /etc/issue.ww
wwctl overlay show issue /etc/issue.ww --render=n1

​
File permissions

Overlay files are distributed to cluster nodes with the same owner, group, and mode they have on the Warewulf server. Adjust permissions as needed: 
wwctl overlay chown issue /etc/issue.ww root root
wwctl overlay chmod issue /etc/issue.ww 0644

Build docs developers (and LLMs) love

Get started for freeTalk to us