The Downloader class is an abstract base class that NewPipe Extractor uses to download resources from streaming services. Implementations must provide the HTTP client logic while the base class provides convenient methods for common request patterns.
Overview The Downloader handles all HTTP communication with streaming services, supporting GET, HEAD, and POST requests with automatic localization header management. It works in conjunction with Request and Response objects to provide a clean API for HTTP operations.
Abstract Method execute The request object containing URL, method, headers, and optional data
The response object containing status code, headers, and body
Executes a request using the specified Request object. Implementations must override this method to provide the actual HTTP client logic.
public abstract Response execute (@ Nonnull Request request) throws IOException, ReCaptchaException;
Throws:
IOException - If a network error occursReCaptchaException - If the service requires CAPTCHA verification
GET Requests get(String url) The URL pointing to the wanted resource
The result of the GET request
Performs a GET request using the default preferred localization. Only use this when the resource won’t be affected by localization.
Response response = downloader . get ( "https://example.com/api/data" );
get(String url, Localization localization) The URL pointing to the wanted resource
Source of the Accept-Language header value
The result of the GET request
Performs a GET request with the specified localization. Sets the Accept-Language header automatically.
Localization locale = new Localization ( "en" , "US" ); Response response = downloader . get (url, locale);
The URL pointing to the wanted resource
Headers to use in the request (overrides defaults)
The result of the GET request
Performs a GET request with custom headers using the default localization.
Map < String , List < String >> headers = new HashMap <>(); headers . put ( "User-Agent" , Collections . singletonList ( "CustomAgent/1.0" )); Response response = downloader . get (url, headers);
The URL pointing to the wanted resource
Headers to use in the request (overrides defaults)
Source of the Accept-Language header value
The result of the GET request
Performs a GET request with custom headers and localization. This is the most flexible GET method.
Map < String , List < String >> headers = new HashMap <>(); headers . put ( "Authorization" , Collections . singletonList ( "Bearer token" )); Localization locale = new Localization ( "fr" , "FR" ); Response response = downloader . get (url, headers, locale);
HEAD Requests head(String url) The URL pointing to the wanted resource
The result of the HEAD request
Performs a HEAD request to retrieve headers without downloading the body.
Response response = downloader . head (url); String contentType = response . getHeader ( "Content-Type" );
The URL pointing to the wanted resource
Headers to use in the request (overrides defaults)
The result of the HEAD request
Performs a HEAD request with custom headers.
Map < String , List < String >> headers = new HashMap <>(); headers . put ( "Range" , Collections . singletonList ( "bytes=0-1023" )); Response response = downloader . head (url, headers);
POST Requests The URL pointing to the wanted resource
Headers to use in the request (overrides defaults)
Byte array to send in the request body
The result of the POST request
Performs a POST request with the default localization.
byte [] data = "{ \" key \" : \" value \" }" . getBytes ( StandardCharsets . UTF_8 ); Response response = downloader . post (url, null , data);
The URL pointing to the wanted resource
Headers to use in the request (overrides defaults)
Byte array to send in the request body
Source of the Accept-Language header value
The result of the POST request
Performs a POST request with custom localization and sets the Accept-Language header.
postWithContentType The URL pointing to the wanted resource
Headers to use in the request (overrides defaults)
Byte array to send in the request body
MIME type to set as the Content-Type header value
The result of the POST request
Convenient method to send a POST request with a specific Content-Type header. Automatically adds the Content-Type to the headers.
byte [] xmlData = "<root><item>value</item></root>" . getBytes (); Response response = downloader . postWithContentType ( url, null , xmlData, "application/xml" );
With localization: Localization locale = new Localization ( "de" , "DE" ); Response response = downloader . postWithContentType ( url, null , data, locale, "application/xml" );
postWithContentTypeJson The URL pointing to the wanted resource
Headers to use in the request (overrides defaults)
Byte array to send in the request body
The result of the POST request
Convenient method to send a POST request with Content-Type: application/json. This is a specialized version of postWithContentType.
byte [] jsonData = "{ \" query \" : \" search term \" }" . getBytes ( StandardCharsets . UTF_8 ); Response response = downloader . postWithContentTypeJson (url, null , jsonData);
With localization: Localization locale = new Localization ( "ja" , "JP" ); Response response = downloader . postWithContentTypeJson ( url, null , jsonData, locale );
Request Object The Request class encapsulates all request parameters. Use the builder pattern to create requests:
Request request = Request . newBuilder () . get (url) . localization ( new Localization ( "en" , "US" )) . setHeader ( "Custom-Header" , "value" ) . build (); Response response = downloader . execute (request);
Request Properties HTTP method (GET, POST, HEAD)
Target URL for the request
Request headers (overrides defaults)
Optional data to send in request body
Localization for Accept-Language header
Request Builder Methods Sets HTTP method to GET and the target URL
Sets HTTP method to HEAD and the target URL
Show post(String url, byte[] data)
Sets HTTP method to POST, the target URL, and data to send
Show headers(Map headers)
Replaces all headers with the provided map
Show localization(Localization localization)
Sets the localization for the request
Show setHeader(String name, String value)
Sets a single header value, replacing any existing values
Show addHeader(String name, String value)
Adds a header value to existing values
Show automaticLocalizationHeader(boolean enabled)
Controls whether Accept-Language header is automatically added (default: true)
Response Object The Response class contains the results of an HTTP request:
Response response = downloader . get (url); int status = response . responseCode (); String body = response . responseBody (); String contentType = response . getHeader ( "Content-Type" );
Response Properties HTTP status code (e.g., 200, 404, 500)
HTTP status message (e.g., “OK”, “Not Found”)
Response body as a string (empty string if null)
The final URL after any redirects
Response Methods Show getHeader(String name)
Returns the first value of the specified header (case-insensitive). Useful for headers that typically don’t repeat. String contentType = response . getHeader ( "Content-Type" );
Example Implementation import org.schabi.newpipe.extractor.downloader.Downloader; import org.schabi.newpipe.extractor.downloader.Request; import org.schabi.newpipe.extractor.downloader.Response; import org.schabi.newpipe.extractor.exceptions.ReCaptchaException; import java.io.IOException; public class CustomDownloader extends Downloader { private final HttpClient client ; public CustomDownloader ( HttpClient client ) { this . client = client; } @ Override public Response execute (@ Nonnull Request request ) throws IOException , ReCaptchaException { // Build HTTP request from Request object HttpRequest httpRequest = HttpRequest . newBuilder () . uri ( URI . create ( request . url ())) . method ( request . httpMethod (), request . dataToSend () != null ? BodyPublishers . ofByteArray ( request . dataToSend ()) : BodyPublishers . noBody ()) . build (); // Add headers request . headers (). forEach ((name, values) -> { values . forEach (value -> httpRequest . headers (). add (name, value)); }); // Execute request HttpResponse < String > httpResponse = client . send (httpRequest, BodyHandlers . ofString ()); // Check for CAPTCHA if ( httpResponse . statusCode () == 429 ) { throw new ReCaptchaException ( "CAPTCHA required" , request . url ()); } // Build Response object return new Response ( httpResponse . statusCode (), httpResponse . headers (). firstValue ( ":status" ) . orElse ( "Unknown" ), httpResponse . headers (). map (), httpResponse . body (), httpResponse . uri (). toString () ); } }
// In a service extractor public class YouTubeExtractor { private final Downloader downloader ; public String fetchChannelData ( String channelId ) throws IOException , ExtractionException { String url = "https://youtube.com/channel/" + channelId; // Simple GET request Response response = downloader . get (url); // With localization Localization locale = new Localization ( "en" , "US" ); response = downloader . get (url, locale); // POST with JSON String jsonBody = buildApiRequest (channelId); byte [] data = jsonBody . getBytes ( StandardCharsets . UTF_8 ); response = downloader . postWithContentTypeJson ( "https://youtube.com/youtubei/v1/browse" , null , data ); return response . responseBody (); } }
Localization - For setting language and country preferencesExceptions - Error handling including ReCaptchaException