Skip to main content

​
Overview

The update process of blackweb.txt consists of several steps executed in sequence by the bwupdate.sh script. This automated process downloads public blocklists, validates domains, performs DNS lookups, and integrates the final list with Squid-Cache.
Before You Continue: This section explains how the update and optimization process works. It is not necessary for users to run it. This process can take significant time and consume substantial hardware and bandwidth resources. It is strongly recommended to use test equipment.

​
Process Steps

The update script performs the following operations in sequence:
1

Capture Public Blocklists

Downloads domains from public blocklists and unifies them into a single file. See the complete list of sources in the project documentation.
2

Domain Debugging

Removes overlapping domains, performs homologation to Squid-Cache format, and excludes false positives using an allowlist (debugwl.txt).
3

TLD Validation

Validates domains against a comprehensive list of Public and Private Suffix TLDs (ccTLD, ccSLD, sTLD, uTLD, gSLD, gTLD, eTLD) up to 4th level domains.
4

Punycode/IDN Debugging

Converts international characters (non-ASCII) to Punycode/IDNA format and removes hostnames larger than 63 characters (RFC 1035).
5

Non-ASCII Character Cleanup

Removes entries with invalid encoding, non-printable characters, whitespace, and disallowed symbols, ensuring clean ASCII format.
6

DNS Lookup (2-Step Validation)

Performs parallel DNS validation to exclude invalid or nonexistent domains. This is the most resource-intensive step.
7

Government TLD Exclusion

Removes government-related domains (.gov, .mil, etc.) from the final blocklist.
8

Squid-Cache Integration

Tests the final list with Squid-Cache and logs any errors for review.

​
Running the Update Script

To download and execute the update script: 
wget -q -N https://raw.githubusercontent.com/maravento/blackweb/master/bwupdate/bwupdate.sh && chmod +x bwupdate.sh && ./bwupdate.sh
The script will request elevated privileges when required for Squid operations.

​
Important Considerations

The default path for BlackWeb is /etc/acl. You can modify this in the script to match your preferences.
If you interrupt bwupdate.sh execution (Ctrl + C) during the DNS Lookup phase, the script will restart from that point on the next run. If stopped earlier, you must start from the beginning or manually modify the script to resume from a specific point.
If you use aufs, temporarily change it to ufs during the upgrade to avoid the error:
ERROR: Can't change type of existing cache_dir aufs /var/spool/squid to ufs. Restart required

​
Completion Verification

After successful completion, check your system log: 
tail /var/log/syslog
You should see: 
BlackWeb: Done 06/05/2023 15:47:14
Any Squid errors during the process are saved to SquidError.txt for review.

​
Next Steps

Install Dependencies

Set up required packages and Squid installation

Run Update Script

Execute the bwupdate.sh script

DNS Lookup Process

Understand parallel DNS validation

Debugging Features

Learn about domain validation and cleaning

Build docs developers (and LLMs) love

Get started for freeTalk to us