Download
Contents
Hint
You can run this notebook in a live session with 
.
Download#
Downloading products is after searching one of the most important features of eodag. This page describes the different methods available to download products and the parameters that these methods accept.
Warning
Downloading products from a provider whose storage is based on AWS may incur some cost.
Setup#
Results obtained from PEPS after a search of Sentinel 2 Level-1C products over France in March 2021 will be loaded in a SearchResult. But first, the credentials need to be set in order to be able to download anything.
[1]:
import os
# os.environ["EODAG__PEPS__AUTH__CREDENTIALS__USERNAME"] = "PLEASE_CHANGE_ME"
# os.environ["EODAG__PEPS__AUTH__CREDENTIALS__PASSWORD"] = "PLEASE_CHANGE_ME"
A workspace directory is created to store the downloaded products.
[2]:
workspace = 'eodag_workspace_download'
if not os.path.isdir(workspace):
os.mkdir(workspace)
By default eodag saves products in the directory set by outputs_prefix which is by default the system temporary folder (/tmp on Linux) and quicklooks in a quicklookssubfolder of outputs_prefix (tmp/quicklooks on Linux). Here eodag is configured to download products in this workspace directory.
[3]:
os.environ["EODAG__PEPS__DOWNLOAD__OUTPUTS_PREFIX"] = os.path.abspath(workspace)
Another setting that could be defined here is whether or not products need to be automatically extracted from their archive. They are extracted by default, this setting is not going to be altered here. The search result is finally loaded with deserialize_and_register().
[4]:
from eodag import EODataAccessGateway
dag = EODataAccessGateway()
search_results = dag.deserialize_and_register("data/download_search_results.geojson")
print(f"This SearchResult stores {len(search_results)} {search_results[0].product_type} products.")
This SearchResult stores 10 S2_MSI_L1C products.
Logging is set to see more about what eodag does when it downloads products.
[5]:
from eodag import setup_logging
setup_logging(2)
Progress bar#
eodag displays a progress bar every time it downloads products or quicklooks. It uses tqdm.auto to create a progress bar adapted to the context (Notebook, terminal).
Cutomize progress bar#
Progress bars can be customized, using the progress_callback parameter of the download methods. Create your own instance of ProgressCallback class, customize it to pass it later to download methods:
[7]:
from eodag.utils import ProgressCallback
with ProgressCallback(total=3, unit="carrots", unit_scale=False, desc="Eating carrots") as progress_callback:
progress_callback(1)
progress_callback(2)
Keep another progress bar opened, and customize it changing ProgressCallback attributes inherited from tqdm:
[8]:
another_bar = ProgressCallback(dynamic_ncols=False, ncols=300)
another_bar.colour = "purple"
[9]:
another_bar(1)
Mute progress bar#
Progress bars can be muted using setup_logging and 0 as verbosity level, or no_progress_bar parameter.
[10]:
with ProgressCallback(total=2) as progress_callback:
progress_callback.desc = "This will be displayed"
progress_callback.position = 0
progress_callback(1)
setup_logging(verbose=0)
with ProgressCallback() as progress_callback:
progress_callback.desc = "This will be hidden"
progress_callback.position = 1
progress_callback(1)
setup_logging(verbose=2, no_progress_bar=True)
with ProgressCallback() as progress_callback:
progress_callback.desc = "This will be hidden"
progress_callback.position = 2
progress_callback(1)
setup_logging(verbose=2)
with ProgressCallback() as progress_callback:
progress_callback.desc = "This will be also displayed"
progress_callback.position = 3
progress_callback(1)
with ProgressCallback(disable=True) as progress_callback:
progress_callback.desc = "This will be hidden"
progress_callback.position = 4
progress_callback(1)
Already downloaded product#
An EOProduct has a location attribute that stores either its remote location (before being downloaded) or its local location (after being downloaded). The download methods of eodag checks first wheter the product/quicklook to download has been already downloaded. If so, it just returns its path and does not try to download it again.
Download quicklooks#
EO products usually offer a quicklook image, a low resolution by-product of the original data. An EOProduct has a get_quicklook() method that takes care of downloading the quicklook image and returns its path. It accepts a base_dir (optional) parameter to specify an output directory and a filename (optional) parameter to
specify an output filename. By default, eodag saves a quicklook with its ID as filename.
get_quicklook() returns the local absolute path to the image downloaded. This is useful to precess it afterwards. Here matplotlib is used here to display 4 quicklooks.
[11]:
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
fig = plt.figure(figsize=(10, 8))
for i, product in enumerate(search_results, start=1):
# This line takes care of downloading the quicklook
quicklook_path = product.get_quicklook()
# Plot the quicklook
img = mpimg.imread(quicklook_path)
ax = fig.add_subplot(3, 4, i)
ax.set_title(i - 1)
plt.imshow(img)
plt.tight_layout()
2021-05-18 15:53:36,297-15s eodag.api.product [INFO ] Download recorded in /home/sylvain/workspace/eodag/docs/notebooks/api_user_guide/eodag_workspace_download/quicklooks/S2B_MSIL1C_20201231T105349_N0209_R051_T31TCK_20201231T120402
2021-05-18 15:53:36,860-15s eodag.api.product [INFO ] Download recorded in /home/sylvain/workspace/eodag/docs/notebooks/api_user_guide/eodag_workspace_download/quicklooks/S2B_MSIL1C_20201231T105349_N0209_R051_T31TCL_20201231T120402
2021-05-18 15:53:37,628-15s eodag.api.product [INFO ] Download recorded in /home/sylvain/workspace/eodag/docs/notebooks/api_user_guide/eodag_workspace_download/quicklooks/S2A_MSIL1C_20201229T110451_N0209_R094_T31TCL_20201229T131620
2021-05-18 15:53:38,276-15s eodag.api.product [INFO ] Download recorded in /home/sylvain/workspace/eodag/docs/notebooks/api_user_guide/eodag_workspace_download/quicklooks/S2A_MSIL1C_20201226T105451_N0209_R051_T31TCK_20201226T130209
2021-05-18 15:53:38,843-15s eodag.api.product [INFO ] Download recorded in /home/sylvain/workspace/eodag/docs/notebooks/api_user_guide/eodag_workspace_download/quicklooks/S2A_MSIL1C_20201226T105451_N0209_R051_T31TCL_20201226T130209
2021-05-18 15:53:39,354-15s eodag.api.product [INFO ] Download recorded in /home/sylvain/workspace/eodag/docs/notebooks/api_user_guide/eodag_workspace_download/quicklooks/S2A_MSIL1C_20201119T110341_N0209_R094_T31TCL_20201119T131255
2021-05-18 15:53:40,024-15s eodag.api.product [INFO ] Download recorded in /home/sylvain/workspace/eodag/docs/notebooks/api_user_guide/eodag_workspace_download/quicklooks/S2A_MSIL1C_20201116T105331_N0209_R051_T31TCK_20201116T130215
2021-05-18 15:53:40,657-15s eodag.api.product [INFO ] Download recorded in /home/sylvain/workspace/eodag/docs/notebooks/api_user_guide/eodag_workspace_download/quicklooks/S2A_MSIL1C_20201116T105331_N0209_R051_T31TCL_20201116T130215
2021-05-18 15:53:41,151-15s eodag.api.product [INFO ] Download recorded in /home/sylvain/workspace/eodag/docs/notebooks/api_user_guide/eodag_workspace_download/quicklooks/S2B_MSIL1C_20201114T110319_N0209_R094_T31TCL_20201114T120840
2021-05-18 15:53:41,709-15s eodag.api.product [INFO ] Download recorded in /home/sylvain/workspace/eodag/docs/notebooks/api_user_guide/eodag_workspace_download/quicklooks/S2B_MSIL1C_20201111T105259_N0209_R051_T31TCK_20201111T130651
Download EO products#
Dynamically configure some download options#
The 3 download methods introduced below accept the following optional kwargs that temporarily override the global configuration:
outputs_prefix(str): absolute path to a folder where the products should be savedextract(bool): whether to automatically extract or not the downloaded product archivedl_url_params(dict): additional parameters to pass over to the download url as an url parameterdelete_archive(bool): whether to delete the downloaded archives
Order OFFLINE products#
As introduced in the getting started guide an EO product may not be available for download immediately. If the product status is OFFLINE, the download methods will request an order of the product and, by default, retry to download it every 2 minutes during 20 minutes. These two durations can be set with the wait (in minutes) and timeout (in minutes) optional parameters of all the download methods.
The storage status of a product can be obtained from its storageStatus field. The status of an OFFLINE product is updated by eodag to STAGING when ordered and to ONLINE when found available.
[12]:
[p.properties["storageStatus"] for p in search_results]
[12]:
['ONLINE',
'ONLINE',
'ONLINE',
'ONLINE',
'ONLINE',
'OFFLINE',
'OFFLINE',
'OFFLINE',
'OFFLINE',
'OFFLINE']
A FilterProperty can be used to filter out OFFLINE products to avoid triggering any product order.
[14]:
online_search_results = search_results.filter_property(
storageStatus="ONLINE"
)
[p.properties["storageStatus"] for p in online_search_results]
2021-05-18 15:53:55,220-15s eodag.plugins.crunch.filter_property [INFO ] Finished filtering products. 5 resulting products
[14]:
['ONLINE', 'ONLINE', 'ONLINE', 'ONLINE', 'ONLINE']
Download multiple products at once#
EODataAccessGateway offers a download_all() method that takes a SearchResult argument and will try to download each EOProduct it contains. It returns a list of absolute paths to the downloaded products. For the purpose of this user guide only 2 products will be downloaded.
[15]:
products_to_download = online_search_results[2:4]
paths = dag.download_all(products_to_download)
2021-05-18 15:53:57,558-15s eodag.core [INFO ] Downloading 2 products
2021-05-18 15:53:57,642-15s eodag.plugins.download.base [INFO ] Download url: https://peps.cnes.fr/resto/collections/S2ST/86440880-7f7d-50f0-87a7-ac1bc2bd8fbd/download
2021-05-18 15:56:13,034-15s eodag.plugins.download.base [INFO ] Extraction activated
2021-05-18 15:56:15,056-15s eodag.api.product [INFO ] Remote location of the product is still available through its 'remote_location' property: https://peps.cnes.fr/resto/collections/S2ST/86440880-7f7d-50f0-87a7-ac1bc2bd8fbd/download
2021-05-18 15:56:15,103-15s eodag.plugins.download.base [INFO ] Download url: https://peps.cnes.fr/resto/collections/S2ST/c51639e0-18ab-55a4-bc7c-7757a64911fa/download
2021-05-18 16:00:46,615-15s eodag.plugins.download.base [INFO ] Extraction activated
2021-05-18 16:00:50,326-15s eodag.api.product [INFO ] Remote location of the product is still available through its 'remote_location' property: https://peps.cnes.fr/resto/collections/S2ST/c51639e0-18ab-55a4-bc7c-7757a64911fa/download
[16]:
paths
[16]:
['/home/sylvain/workspace/eodag/docs/notebooks/api_user_guide/eodag_workspace_download/S2A_MSIL1C_20201229T110451_N0209_R094_T31TCL_20201229T131620/S2A_MSIL1C_20201229T110451_N0209_R094_T31TCL_20201229T131620.SAFE',
'/home/sylvain/workspace/eodag/docs/notebooks/api_user_guide/eodag_workspace_download/S2A_MSIL1C_20201226T105451_N0209_R051_T31TCK_20201226T130209/S2A_MSIL1C_20201226T105451_N0209_R051_T31TCK_20201226T130209.SAFE']
The command below shows that the products have been downloaded as archived files (Sentinel products are distributed in the SAFE format). eodag then took care of extracting them to their own directory.
[17]:
!ls {workspace}
quicklooks
S2A_MSIL1C_20201226T105451_N0209_R051_T31TCK_20201226T130209
S2A_MSIL1C_20201226T105451_N0209_R051_T31TCK_20201226T130209.zip
S2A_MSIL1C_20201229T110451_N0209_R094_T31TCL_20201229T131620
S2A_MSIL1C_20201229T110451_N0209_R094_T31TCL_20201229T131620.zip
[18]:
!ls {paths[0]}
AUX_DATA GRANULE INSPIRE.xml MTD_MSIL1C.xml
DATASTRIP HTML manifest.safe rep_info
eodag saves a file in the outputs_prefix folder that contains a hash list of the products downloaded. Before downloading products, this file is read and it is checked whether the products to download were already downloaded or not. If they were, their download is skipped.
[19]:
products_to_download = online_search_results[2:4]
dag.download_all(products_to_download)
2021-05-18 16:10:58,354-15s eodag.core [INFO ] Downloading 2 products
2021-05-18 16:10:58,438-15s eodag.plugins.download.base [INFO ] Product already present on this platform. Identifier: /home/sylvain/workspace/eodag/docs/notebooks/api_user_guide/eodag_workspace_download/S2A_MSIL1C_20201229T110451_N0209_R094_T31TCL_20201229T131620/S2A_MSIL1C_20201229T110451_N0209_R094_T31TCL_20201229T131620.SAFE
2021-05-18 16:10:58,440-15s eodag.api.product [INFO ] Remote location of the product is still available through its 'remote_location' property: https://peps.cnes.fr/resto/collections/S2ST/86440880-7f7d-50f0-87a7-ac1bc2bd8fbd/download
2021-05-18 16:10:58,479-15s eodag.plugins.download.base [INFO ] Product already present on this platform. Identifier: /home/sylvain/workspace/eodag/docs/notebooks/api_user_guide/eodag_workspace_download/S2A_MSIL1C_20201226T105451_N0209_R051_T31TCK_20201226T130209/S2A_MSIL1C_20201226T105451_N0209_R051_T31TCK_20201226T130209.SAFE
2021-05-18 16:10:58,481-15s eodag.api.product [INFO ] Remote location of the product is still available through its 'remote_location' property: https://peps.cnes.fr/resto/collections/S2ST/c51639e0-18ab-55a4-bc7c-7757a64911fa/download
[19]:
['/home/sylvain/workspace/eodag/docs/notebooks/api_user_guide/eodag_workspace_download/S2A_MSIL1C_20201229T110451_N0209_R094_T31TCL_20201229T131620/S2A_MSIL1C_20201229T110451_N0209_R094_T31TCL_20201229T131620.SAFE',
'/home/sylvain/workspace/eodag/docs/notebooks/api_user_guide/eodag_workspace_download/S2A_MSIL1C_20201226T105451_N0209_R051_T31TCK_20201226T130209/S2A_MSIL1C_20201226T105451_N0209_R051_T31TCK_20201226T130209.SAFE']
Download a single product#
EODataAccessGateway also offers a download() method that takes an EOProduct argument and will try to download it. It is a wrapper around the download() method that each EOProduct has. Its advantage over the latter is that, similarly to what does deserialize_and_register(), it adds the information required for a product to download itself if it was missing. It does so by extracting the product’s provider and getting the right authentication and download plugins. Both methods return the absolute path to the downloaded product.
Download an OFFLINE product#
This examples shows what logs eodag emits when it tries to download a product that is originally OFFLINE. It is set to retry downloading the product every 1 minute after ordering it, and to stop trying (skip it if download_all() is used) after 5 minutes.
[20]:
offline_product = search_results.crunch(
FilterProperty(dict(storageStatus="OFFLINE"))
)[1]
offline_product.properties["storageStatus"]
2021-05-18 16:11:17,493-15s eodag.plugins.crunch.filter_property [INFO ] Finished filtering products. 5 resulting products
[20]:
'OFFLINE'
[21]:
path = dag.download(
offline_product,
wait=1,
timeout=5,
)
2021-05-18 16:11:18,974-15s eodag.plugins.download.base [INFO ] Download url: https://peps.cnes.fr/resto/collections/S2ST/0b57ac26-d487-5367-aa86-9101dac07271/download
2021-05-18 16:15:39,370-15s eodag.plugins.download.base [INFO ] Extraction activated
2021-05-18 16:15:43,101-15s eodag.api.product [INFO ] Remote location of the product is still available through its 'remote_location' property: https://peps.cnes.fr/resto/collections/S2ST/0b57ac26-d487-5367-aa86-9101dac07271/download
[22]:
path
[22]:
'/home/sylvain/workspace/eodag/docs/notebooks/api_user_guide/eodag_workspace_download/S2A_MSIL1C_20201116T105331_N0209_R051_T31TCK_20201116T130215/S2A_MSIL1C_20201116T105331_N0209_R051_T31TCK_20201116T130215.SAFE'
Now the location attribute rightfully points to its local path.
[23]:
offline_product.location
[23]:
'file:///home/sylvain/workspace/eodag/docs/notebooks/api_user_guide/eodag_workspace_download/S2A_MSIL1C_20201116T105331_N0209_R051_T31TCK_20201116T130215/S2A_MSIL1C_20201116T105331_N0209_R051_T31TCK_20201116T130215.SAFE'
Download assets#
Searching on some providers (mainly STAC providers) returns products having an assets attribute listing single files that can be individually donwloaded.
[5]:
search_results, total_count = dag.search(
productType="S2_MSI_L2A",
provider="planetary_computer"
)
[6]:
# view assets keys
[*search_results[0].assets]
[6]:
['AOT',
'B01',
'B02',
'B03',
'B04',
'B05',
'B06',
'B07',
'B08',
'B09',
'B11',
'B12',
'B8A',
'SCL',
'WVP',
'visual',
'preview',
'safe-manifest',
'granule-metadata',
'inspire-metadata',
'product-metadata',
'datastrip-metadata',
'tilejson',
'rendered_preview']
[7]:
# view single asset content
search_results[0].assets["B05"]
[7]:
{'href': 'https://sentinel2l2a01.blob.core.windows.net/sentinel2-l2/04/V/DM/2023/11/30/S2B_MSIL2A_20231130T221859_N0509_R115_T04VDM_20231130T231119.SAFE/GRANULE/L2A_T04VDM_A035176_20231130T221857/IMG_DATA/R20m/T04VDM_20231130T221859_B05_20m.tif', 'proj:bbox': [399960.0, 6590220.0, 509760.0, 6700020.0], 'proj:shape': [5490, 5490], 'proj:transform': [20.0, 0.0, 399960.0, 0.0, -20.0, 6700020.0], 'gsd': 20.0, 'type': 'image/tiff; application=geotiff; profile=cloud-optimized', 'roles': ['data'], 'title': 'Band 5 - Vegetation red edge 1 - 20m', 'eo:bands': [{'name': 'B05', 'common_name': 'rededge', 'description': 'Band 5 - Vegetation red edge 1', 'center_wavelength': 0.704, 'full_width_half_max': 0.019}]}
Configure provider for download, if not already done
[8]:
dag.update_providers_config(f"""
planetary_computer:
auth:
credentials:
apikey: PLEASE_CHANGE_ME
download:
outputs_prefix: {os.path.abspath(workspace)}
""")
Download several assets using core download() method, allowing regex in asset parameter to identify assets to download
[9]:
path = dag.download(search_results[0], asset=r"B0[23]")
path
[9]:
'/home/sylvain/workspace/eodag/docs/notebooks/api_user_guide/eodag_workspace_download/S2B_MSIL2A_20231130T221859_R115_T04VDM_20231130T231119'
[10]:
! ls {path}
T04VDM_20231130T221859_B02_10m.tif T04VDM_20231130T221859_B03_10m.tif
Download a single asset using asset.download() method
[11]:
path = search_results[0].assets["B05"].download()
path
[11]:
'/home/sylvain/workspace/eodag/docs/notebooks/api_user_guide/eodag_workspace_download/S2B_MSIL2A_20231130T221859_R115_T04VDM_20231130T231119'
[12]:
! ls {path}
T04VDM_20231130T221859_B02_10m.tif T04VDM_20231130T221859_B05_20m.tif
T04VDM_20231130T221859_B03_10m.tif