File Deduplication

Overview

copyparty can detect duplicate files based on their content and avoid storing multiple copies. This saves significant disk space, especially when multiple users upload the same files.

How Deduplication Works

When deduplication is enabled:

File Upload: User uploads a file
Hash Calculation: copyparty calculates the file’s hash (checksum)
Duplicate Check: Compares hash against indexed files
Link Creation: If duplicate found, creates a link instead of copying

The result: only one physical copy exists on disk, with multiple references to it.

Deduplication Methods

copyparty supports three types of deduplication:

Method	Safety	Compatibility	OS Support
Symlinks	Medium	Good	All
Hardlinks	Medium	Excellent	All
Reflinks	High	Good	Linux 5.3+, limited FS

Symlinks (Default)

Symbolic links point to the original file. ✅ Advantages:

Each link can have its own timestamp
Clearly visible that it’s not a regular file
Works on all systems

⚠️ Disadvantages:

If you delete the original, symlinks break
Some software doesn’t handle symlinks well
Renaming files requires copyparty to update links

Hardlinks

Hard links are indistinguishable from regular files. ✅ Advantages:

Compatible with all software
Deleting one copy doesn’t affect others
Can be moved/renamed safely with any tool

⚠️ Disadvantages:

All copies share the same timestamp
Editing one copy modifies all copies
Less obvious that deduplication is happening

Reflinks (Copy-on-Write)

Reflinks use filesystem-level copy-on-write. ✅ Advantages:

Safest option: editing one copy doesn’t affect others
Each copy is fully independent
Automatic copy-on-write when modified
Most space-efficient

⚠️ Disadvantages:

Requires Python 3.14+ and Linux kernel 5.3+
Limited filesystem support (btrfs, maybe XFS)
Not available on ZFS yet

Basic Deduplication Setup

Enable file indexing

Deduplication requires the file database:

python copyparty-sfx.py -e2dsa --dedup \
  -v /mnt/uploads:/uploads:w

-e2dsa - Scan and index all files

--dedup - Enable symlink-based deduplication

Using configuration file

[global]
  e2dsa    # scan and index all files
  dedup    # enable deduplication

[/uploads]
  /mnt/uploads
  accs:
    w: *
    r: admin

Verify deduplication

After uploading duplicate files:

# Check for symlinks
ls -la /mnt/uploads/

# Count physical vs logical size
du -sh /mnt/uploads/        # physical size
du -sh --apparent-size /mnt/uploads/  # logical size

Choosing Deduplication Method

Symlinks (Default)
Hardlinks
Reflinks

Best for most use cases:

python copyparty-sfx.py -e2dsa --dedup \
  -v /mnt/uploads:/uploads:w

Or in config:

[global]
  e2dsa
  dedup

Better compatibility with other software:

python copyparty-sfx.py -e2dsa --dedup --hardlink-only \
  -v /mnt/uploads:/uploads:w

Or in config:

[global]
  e2dsa
  dedup
  hardlink-only

Safest option (requires compatible system):

python copyparty-sfx.py -e2dsa --dedup --reflink \
  -v /mnt/uploads:/uploads:w

Or in config:

[global]
  e2dsa
  dedup
  reflink

Per-Volume Deduplication

Enable deduplication for specific volumes only:

[global]
  e2dsa    # global indexing

[/uploads]
  /mnt/uploads
  accs:
    w: *
  flags:
    dedup           # enable for this volume
    hardlinkonly    # use hardlinks instead of symlinks

[/media]
  /mnt/media
  accs:
    r: *
  # no dedup flag = no deduplication

Cross-Volume Deduplication

Deduplicate files across different volumes:

python copyparty-sfx.py -e2dsa --dedup --xlink \
  -v /mnt/uploads1:/up1:w \
  -v /mnt/uploads2:/up2:w

--xlink - Enable cross-volume linking

⚠️ Warning: Cross-volume deduplication is experimental and may have bugs.

Deduplication Statistics

View disk space saved:

python copyparty-sfx.py --stats -e2dsa --dedup \
  -a admin:password \
  -v /mnt/uploads:/uploads:A,admin

Access metrics at:

http://your-server:3923/.cpr/metrics

Metrics include:

cpp_dupe_bytes - Disk space saved
cpp_dupe_files - Number of duplicate files
cpp_vol_bytes - Total volume size
cpp_vol_files - Total file count

Advanced Configuration

Safe Deduplication Mode

If you have other software modifying files, use this:

python copyparty-sfx.py -e2dsa --dedup --safe-dedup=1 \
  -v /mnt/uploads:/uploads:w

--safe-dedup=1 - Verify file integrity before deduplicating

This is slower but prevents deduplication of modified files.

Disable Clone Detection

If using S3 or similar storage where reading is expensive:

[/uploads]
  /mnt/uploads
  accs:
    w: *
  flags:
    noclone    # disable duplicate detection entirely

Database Location

Move the deduplication database to faster storage:

[global]
  e2dsa
  dedup
  hist: /mnt/ssd/copyparty-db  # put database on SSD

[/uploads]
  /mnt/hdd/uploads  # data stays on HDD
  accs:
    w: *

Skip Hashing for Large Files

Exclude large files from deduplication:

[/videos]
  /mnt/videos
  accs:
    w: *
  flags:
    e2dsa
    dedup
    nohash: \.(mkv|mp4|avi)$  # don't hash video files

Saves indexing time
Disables deduplication for matched files
Files are still indexed by path/size/date

Deduplication with Uploads

Reject Duplicate Uploads

Prevent users from uploading duplicates:

[/uploads]
  /mnt/uploads
  accs:
    w: *
  flags:
    e2dsa
    # no dedup flag = reject duplicate uploads

Without dedup, upload fails if file already exists.

Allow Duplicates with Links

[/uploads]
  /mnt/uploads
  accs:
    w: *
  flags:
    e2dsa
    dedup    # create link instead of rejecting

Upload succeeds but creates a link instead of a new copy.

Randomize Duplicate Filenames

Combine with filename randomization:

python copyparty-sfx.py -e2dsa --dedup \
  -v /mnt/uploads:/uploads:wG:c,fk=8

Uploaders get unique filenames even for duplicate content.

Filesystem Compatibility

Symlinks and Hardlinks

Supported on all major filesystems:

ext4, ext3, ext2 ✅
Btrfs ✅
XFS ✅
ZFS ✅
NTFS ✅
exFAT ⚠️ (symlinks may not work)
FAT32 ❌ (no symlinks or hardlinks)

Reflinks

Only works on:

Btrfs ✅ (fully supported)
XFS ⚠️ (maybe, needs testing)
ZFS ❌ (not yet, known bugs)
ext4/NTFS/others ❌

Example: Complete Deduplication Setup

complete-dedup.conf

[global]
  e2dsa              # scan all files on startup
  dedup              # enable deduplication
  hardlink-only      # use hardlinks instead of symlinks
  hist: /mnt/ssd/cpp # database on SSD for performance
  stats              # enable prometheus metrics

[accounts]
  uploader: pass1
  admin: admin-secret

[/uploads]
  /mnt/hdd/uploads
  accs:
    w: uploader
    A: admin
  flags:
    fk: 8              # filekeys for duplicate links
    sz: 1k-1g          # limit file sizes
    vmaxb: 500g        # max 500GB total
    maxn: 100,3600     # rate limit uploads

[/videos]
  /mnt/hdd/videos
  accs:
    w: uploader  
    r: admin
  flags:
    dedup
    nohash: \.(mkv|mp4|avi)$  # skip hashing large videos

Monitoring Deduplication

Check Symlink Destinations

# Find all symlinks
find /mnt/uploads -type l

# Show symlink targets
find /mnt/uploads -type l -ls

# Find broken symlinks
find /mnt/uploads -type l ! -exec test -e {} \; -print

Calculate Space Savings

# Physical disk usage
du -sh /mnt/uploads

# Logical size (if all files were real)
du -sh --apparent-size /mnt/uploads

# Difference = space saved

Use Prometheus Metrics

Enable metrics and monitor:

[global]
  e2dsa
  dedup
  stats

[accounts]
  monitoring: metrics-secret

[/]
  /mnt/uploads
  accs:
    w: *
    a: monitoring  # admin access for metrics

Query metrics:

curl -u :metrics-secret http://your-server:3923/.cpr/metrics

Look for:

cpp_dupe_bytes{vol="/"} - Space saved
cpp_dupe_files{vol="/"} - Duplicate count

Important Warnings

Do not edit deduplicated files in-place!With symlinks or hardlinks, editing one file edits ALL copies.Safe editing methods:

Delete and re-upload
Copy the file first, then edit the copy
Use an editor that creates a new file (like vim’s “backup” mode)

Database corruption = broken symlinksIf the .hist/up2k.db database becomes corrupted or deleted:

Symlinks may point to wrong files
Some files may become inaccessible

Prevention:

Regular database backups
Use --hist to store DB on reliable storage
Consider reflinks if your filesystem supports them

Cross-volume deduplication is experimentalThe --xlink option may have bugs. Use at your own risk:

Test thoroughly before production use
Keep backups
Monitor for broken links

Troubleshooting

Deduplication not working

Check these requirements:

Indexing enabled: -e2dsa or -e2d
Dedup flag set: --dedup or volflag dedup
Database exists: .hist/up2k.db in volume
Files are actually identical (same hash)

Verify with:

python copyparty-sfx.py -e2dsa --dedup -v /test::w
# Upload same file twice and check with ls -la

Broken symlinks after moving files

Symlinks break if you move/rename the original file outside of copyparty.Solutions:

Use hardlinks instead: --hardlink-only
Only move files through copyparty’s web UI
Rebuild the database: restart with -e2dsa

Hardlinks editing all copies

This is how hardlinks work. All hardlinks point to the same data.Solutions:

Delete before editing (removes the hardlink)
Copy the file first, then edit the copy
Use reflinks if possible: --reflink

Reflinks not working

Reflinks require:

Python 3.14 or newer
Linux kernel 5.3 or newer
Btrfs filesystem (XFS maybe)

Check your setup:

python3 --version  # needs 3.14+
uname -r           # needs 5.3+
df -T /mnt/uploads # needs btrfs

Database growing too large

The up2k.db database can grow large with many files.Solutions:

Move to SSD: --hist /mnt/ssd/cpp
Exclude large files: nohash: \.(mkv|iso)$
Compress: use XZ filesystem compression
Clean old entries (no built-in tool yet)

Next Steps

Set up write-only folders for uploads
Configure file sharing with deduplication
Learn about media server with space-efficient storage
Set up authentication to track uploaders

Get Started

Core Features

Configuration

Deployment

Guides

File Deduplication

Overview

How Deduplication Works

Deduplication Methods

Symlinks (Default)

Hardlinks

Reflinks (Copy-on-Write)

Basic Deduplication Setup

Choosing Deduplication Method

Per-Volume Deduplication

Cross-Volume Deduplication

Deduplication Statistics

Advanced Configuration

Safe Deduplication Mode

Disable Clone Detection

Database Location

Skip Hashing for Large Files

Deduplication with Uploads

Reject Duplicate Uploads

Allow Duplicates with Links

Randomize Duplicate Filenames

Filesystem Compatibility

Symlinks and Hardlinks

Reflinks

Example: Complete Deduplication Setup

Monitoring Deduplication

Check Symlink Destinations

Calculate Space Savings

Use Prometheus Metrics

Important Warnings

Troubleshooting

Next Steps

Build docs developers (and LLMs) love

Get Started

Core Features

Configuration

Deployment

Guides

​Overview

​How Deduplication Works

​Deduplication Methods

​Symlinks (Default)

​Hardlinks

​Reflinks (Copy-on-Write)

​Basic Deduplication Setup

​Choosing Deduplication Method

​Per-Volume Deduplication

​Cross-Volume Deduplication

​Deduplication Statistics

​Advanced Configuration

​Safe Deduplication Mode

​Disable Clone Detection

​Database Location

​Skip Hashing for Large Files

​Deduplication with Uploads

​Reject Duplicate Uploads

​Allow Duplicates with Links

​Randomize Duplicate Filenames

​Filesystem Compatibility

​Symlinks and Hardlinks

​Reflinks

​Example: Complete Deduplication Setup

​Monitoring Deduplication

​Check Symlink Destinations

​Calculate Space Savings

​Use Prometheus Metrics

​Important Warnings

​Troubleshooting

​Next Steps

Build docs developers (and LLMs) love

Overview

How Deduplication Works

Deduplication Methods

Symlinks (Default)

Hardlinks

Reflinks (Copy-on-Write)

Basic Deduplication Setup

Choosing Deduplication Method

Per-Volume Deduplication

Cross-Volume Deduplication

Deduplication Statistics

Advanced Configuration

Safe Deduplication Mode

Disable Clone Detection

Database Location

Skip Hashing for Large Files

Deduplication with Uploads

Reject Duplicate Uploads

Allow Duplicates with Links

Randomize Duplicate Filenames

Filesystem Compatibility

Symlinks and Hardlinks

Reflinks

Example: Complete Deduplication Setup

Monitoring Deduplication

Check Symlink Destinations

Calculate Space Savings

Use Prometheus Metrics

Important Warnings

Troubleshooting

Next Steps