Download Plugins

Download plugins must inherit the following class and implement download():

class eodag.plugins.download.base.Download(provider: str, config: PluginConfig)

Base Download Plugin.

A Download plugin has two download methods that it must implement:

• download: download a single EOProduct

• download_all: download multiple products from a SearchResult

They must:

• download data in the outputs_prefix folder defined in the plugin’s configuration or passed through kwargs

• extract products from their archive (if relevant) if extract is set to True (True by default)

• save a product in an archive/directory (in outputs_prefix) whose name must be the product’s title property

• update the product’s location attribute once its data is downloaded (and eventually after it’s extracted) to the product’s location given as a file URI (e.g. ‘file:///tmp/product_folder’ on Linux or ‘file:///C:/Users/username/AppData/LOcal/Temp’ on Windows)

• save a record file in the directory outputs_prefix/.downloaded whose name is built on the MD5 hash of the product’s product_type and properties['id'] attributes (hashlib.md5((product.product_type+"-"+product.properties['id']).encode("utf-8")).hexdigest()) and whose content is the product’s remote_location attribute itself.

• not try to download a product whose location attribute already points to an existing file/directory

• not try to download a product if its record file exists as long as the expected product’s file/directory. If the record file only is found, it must be deleted (it certainly indicates that the download didn’t complete)

Parameters:

• provider (dict) – An eodag providers configuration dictionary

• config (str) – Path to the user configuration file

download(product: EOProduct, auth: AuthBase | Dict[str, str] | None = None, progress_callback: ProgressCallback | None = None, wait: int = 2, timeout: int = 20, **kwargs: Unpack[DownloadConf]) → str | None

Base download method. Not available, it must be defined for each plugin.

Parameters:

• product (EOProduct) – The EO product to download

• auth (Optional[Union[AuthBase, Dict[str, str]]]) – (optional) authenticated object

• progress_callback (ProgressCallback) – (optional) A progress callback

• wait (int) – (optional) If download fails, wait time in minutes between two download tries

• timeout (int) – (optional) If download fails, maximum time in minutes before stop retrying to download

• kwargs (Union[str, bool, dict]) – outputs_prefix (str), extract (bool), delete_archive (bool) and dl_url_params (dict) can be provided as additional kwargs and will override any other values defined in a configuration file or with environment variables.

Returns:

The absolute path to the downloaded product in the local filesystem (e.g. ‘/tmp/product.zip’ on Linux or ‘C:\Users\username\AppData\Local\Temp\product.zip’ on Windows)

Return type:

str

download_all(products: SearchResult, auth: AuthBase | Dict[str, str] | None = None, downloaded_callback: DownloadedCallback | None = None, progress_callback: ProgressCallback | None = None, wait: int = 2, timeout: int = 20, **kwargs: Unpack[DownloadConf]) → List[str]

Base download_all method.

This specific implementation uses the eodag.plugins.download.base.Download.download() method implemented by the plugin to sequentially attempt to download products.

Parameters:

• products (SearchResult) – Products to download

• auth (Optional[Union[AuthBase, Dict[str, str]]]) – (optional) authenticated object

• downloaded_callback (Callable[[EOProduct], None] or None) – (optional) A method or a callable object which takes as parameter the product. You can use the base class DownloadedCallback and override its __call__ method. Will be called each time a product finishes downloading

• progress_callback (ProgressCallback) – (optional) A progress callback

• wait (int) – (optional) If download fails, wait time in minutes between two download tries

• timeout (int) – (optional) If download fails, maximum time in minutes before stop retrying to download

• kwargs (Union[str, bool, dict]) – outputs_prefix (str), extract (bool), delete_archive (bool) and dl_url_params (dict) can be provided as additional kwargs and will override any other values defined in a configuration file or with environment variables.

Returns:

List of absolute paths to the downloaded products in the local filesystem (e.g. ['/tmp/product.zip'] on Linux or ['C:\Users\username\AppData\Local\Temp\product.zip'] on Windows)

Return type:

list

generate_record_hash(product: EOProduct) → str

Generate the record hash of the given product.

The MD5 hash is built from the product’s product_type and properties['id'] attributes (hashlib.md5((product.product_type+"-"+product.properties['id']).encode("utf-8")).hexdigest())

Parameters:

product (EOProduct) – The product to calculate the record hash

Returns:

The MD5 hash

Return type:

str

This table lists all the download plugins currently available:

eodag.plugins.download.http.HTTPDownload(...)	HTTPDownload plugin.
eodag.plugins.download.aws.AwsDownload(...)	Download on AWS using S3 protocol.
eodag.plugins.download.s3rest.S3RestDownload(...)	Http download on S3-like object storage location for example using Mundi REST API (free account) https://mundiwebservices.com/keystoneapi/uploads/documents/CWS-DATA-MUT-087-EN-Mundi_Download_v1.1.pdf#page=13

Download methods call graph

Here is a graph showing how download methods are called from API to plugin:

Progress bars

While downloading, progress bars are generated using ProgressCallback. They are instantiated and closed in EOProduct.download() which means that plugins do not need to handle this part. Using download_all(), one progress bar will be created and closed for each product.

The same progress bar will always be used to track both individual product download and extraction.

If a custom ProgressCallback is passed as argument of download() or download_all(), it will be used and duplicated to create bars to track download_all/download. The same bar will be used for all products, and will not be closed by eodag.

ProgressCallback handling

	plugin.download_all()	product.download()	Output
download(product)		Default ProgressCallback created for product download/extraction, then closed
download(product, progress_callback=bar)		Passed ProgressCallback used for product download/extraction, kept open
download_all(products)	Default ProgressCallback created for products count, then closed	Default ProgressCallback created per product for download/extraction, then closed
download_all(products, progress_callback=bar)	Passed ProgressCallback used for products count, copied for product.download(), then closed.	Passed ProgressCallback used for all products download/extraction, kept open