Download Plugins#

Download plugins must inherit the following class and implement download():

class eodag.plugins.download.base.Download(provider, config)[source]#

Base Download Plugin.

A Download plugin has two download methods that it must implement:

They must:

  • download data in the outputs_prefix folder defined in the plugin’s configuration or passed through kwargs

  • extract products from their archive (if relevant) if extract is set to True (True by default)

  • save a product in an archive/directory (in outputs_prefix) whose name must be the product’s title property

  • update the product’s location attribute once its data is downloaded (and eventually after it’s extracted) to the product’s location given as a file URI (e.g. ‘file:///tmp/product_folder’ on Linux or ‘file:///C:/Users/username/AppData/LOcal/Temp’ on Windows)

  • save a record file in the directory outputs_prefix/.downloaded whose name is built on the MD5 hash of the product’s remote_location attribute (hashlib.md5(remote_location.encode("utf-8")).hexdigest()) and whose content is the product’s remote_location attribute itself.

  • not try to download a product whose location attribute already points to an existing file/directory

  • not try to download a product if its record file exists as long as the expected product’s file/directory. If the record file only is found, it must be deleted (it certainly indicates that the download didn’t complete)

Parameters

  • provider (dict) – An eodag providers configuration dictionary

  • config (str) – Path to the user configuration file

download(product, auth=None, progress_callback=None, wait=2, timeout=20, **kwargs)[source]#

Base download method. Not available, it must be defined for each plugin.

Parameters

  • product (EOProduct) – The EO product to download

  • auth (PluginConfig) – (optional) The configuration of a plugin of type Authentication

  • progress_callback (ProgressCallback) – (optional) A progress callback

  • wait (int) – (optional) If download fails, wait time in minutes between two download tries

  • timeout (int) – (optional) If download fails, maximum time in minutes before stop retrying to download

  • kwargs (Union[str, bool, dict]) – outputs_prefix (str), extract (bool), delete_archive (bool) and dl_url_params (dict) can be provided as additional kwargs and will override any other values defined in a configuration file or with environment variables.

Returns

The absolute path to the downloaded product in the local filesystem (e.g. ‘/tmp/product.zip’ on Linux or ‘C:\Users\username\AppData\Local\Temp\product.zip’ on Windows)

Return type

str

download_all(products, auth=None, downloaded_callback=None, progress_callback=None, wait=2, timeout=20, **kwargs)[source]#

Base download_all method.

This specific implementation uses the eodag.plugins.download.base.Download.download() method implemented by the plugin to sequentially attempt to download products.

Parameters

  • products (SearchResult) – Products to download

  • auth (PluginConfig) – (optional) The configuration of a plugin of type Authentication

  • downloaded_callback (Callable[[EOProduct], None] or None) – (optional) A method or a callable object which takes as parameter the product. You can use the base class DownloadedCallback and override its __call__ method. Will be called each time a product finishes downloading

  • progress_callback (ProgressCallback) – (optional) A progress callback

  • wait (int) – (optional) If download fails, wait time in minutes between two download tries

  • timeout (int) – (optional) If download fails, maximum time in minutes before stop retrying to download

  • kwargs (Union[str, bool, dict]) – outputs_prefix (str), extract (bool), delete_archive (bool) and dl_url_params (dict) can be provided as additional kwargs and will override any other values defined in a configuration file or with environment variables.

Returns

List of absolute paths to the downloaded products in the local filesystem (e.g. ['/tmp/product.zip'] on Linux or ['C:\Users\username\AppData\Local\Temp\product.zip'] on Windows)

Return type

list

Here is a graph showing how download methods are called from API to plugin:

Download methods call graph

This table lists all the download plugins currently available:

eodag.plugins.download.http.HTTPDownload(...)

HTTPDownload plugin.

eodag.plugins.download.aws.AwsDownload(...)

Download on AWS using S3 protocol.

eodag.plugins.download.s3rest.S3RestDownload(...)

Http download on S3-like object storage location for example using Mundi REST API (free account) https://mundiwebservices.com/keystoneapi/uploads/documents/CWS-DATA-MUT-087-EN-Mundi_Download_v1.1.pdf#page=13

Progress bars#

While downloading, progress bars are generated using ProgressCallback. They are instantiated and closed in EOProduct.download() which means that plugins do not need to handle this part. Using download_all(), one progress bar will be created and closed for each product.

The same progress bar will always be used to track both individual product download and extraction.

If a custom ProgressCallback is passed as argument of download() or download_all(), it will be used and duplicated to create bars to track download_all/download. The same bar will be used for all products, and will not be closed by eodag.

ProgressCallback handling#

plugin.download_all()

product.download()

Output

download(product)

Default ProgressCallback created for product download/extraction, then closed
../_images/progress_1_none.png

download(product, progress_callback=bar)

Passed ProgressCallback used for product download/extraction, kept open
../_images/progress_1.png

download_all(products)

Default ProgressCallback created for products count, then closed

Default ProgressCallback created per product for download/extraction, then closed
../_images/progress_2_none.png

download_all(products, progress_callback=bar)

Passed ProgressCallback used for products count, copied for product.download(), then closed.

Passed ProgressCallback used for all products download/extraction, kept open
../_images/progress_2.png