Overview
Contents
Overview#
This overview demonstrates the basic capabilities of eodag
in a simple worflow, whose steps are introduced in more details in the following pages.
The workflow consists of the following steps:
Configure:
eodag
is configured to use the provider PEPS (registering to PEPS is required to download the products, see how to register to a provider)Search: Sentinel 2 Level-1C products (i.e. images) are searched for over an area in France in March 2021.
Crunch: A decicated filter - that we call cruncher - provided by
eodag
is used to select products with a cloud cover less than 10%.Serialize: Save the filtered products as a GeoJSON file.
Download: The products are downloaded.
Post-process: eodag-cube is an external package that is used to access a product’s data, it is going to be used to calculate the NDVI of a product.
Configure#
Let us first create a workspace directory to save products downloaded during this workflow.
[1]:
import os
workspace = 'eodag_workspace_overview'
if not os.path.isdir(workspace):
os.mkdir(workspace)
Now we will configure eodag
to be able to download using PEPS. For that we need to fill our credentials:
in the user configuration file
~/.config/eodag.eodag.yml
:
peps:
priority: # Lower value means lower priority (Default: 1)
search: # Search parameters configuration
download:
extract: # whether to extract the downloaded products (true or false).
outputs_prefix: # where to store downloaded products.
dl_url_params: # additional parameters to pass over to the download url as an url parameter
delete_archive: # whether to delete the downloaded archives (true or false, Default: true).
auth:
credentials:
username: PLEASE_CHANGE_ME
password: PLEASE_CHANGE_ME
or using environment variables: (we also set
outputs_prefix
, the directory where to store downloaded products)
[2]:
os.environ["EODAG__PEPS__AUTH__CREDENTIALS__USERNAME"] = "PLEASE_CHANGE_ME"
os.environ["EODAG__PEPS__AUTH__CREDENTIALS__PASSWORD"] = "PLEASE_CHANGE_ME"
os.environ["EODAG__PEPS__DOWNLOAD__OUTPUTS_PREFIX"] = os.path.abspath(workspace)
Logging is then activated with the setup_logging() method. It’s a useful way to see what eodag
does under the hood, e.g. requesting the provider, adapting the response. It’s also useful to detect when things go wrong, and, if relevant, create an issue on GitHub with the log messages.
[3]:
from eodag import setup_logging
setup_logging(2) # 3 for even more information
The next object to import, and this is certainly one of the most important objects provided by eodag
, is the EODataAccessGateway class. The creation of a single instance of this class is enough in a workflow, it is going to take care of configuring the providers, exposing the products configured off-the-shelf by eodag
, and many more things.
[4]:
from eodag import EODataAccessGateway
dag = EODataAccessGateway()
2023-01-13 10:54:32,231 eodag.config [INFO ] Loading user configuration from: /home/sylvain/.config/eodag/eodag.yml
2023-01-13 10:54:32,461 eodag.core [INFO ] Locations configuration loaded from /home/sylvain/.config/eodag/locations.yml
eodag
stores an internal catalog of products it makes easily accessible. Sentinel 2 Level-1C products are designated with the identifier S2_MSI_L1C. It’s possible to check that it’s made available by PEPS with the method list_product_types() which returns a list of dictionnaries, each one of them containing general metadata (eodag
’s product type ID, platform, sensor type, etc.).
[5]:
[product_type["ID"] for product_type in dag.list_product_types("peps")]
[5]:
['S1_SAR_GRD', 'S1_SAR_OCN', 'S1_SAR_SLC', 'S2_MSI_L1C', 'S2_MSI_L2A']
The method available_providers() allows to get the list of providers that make available a given product.
[6]:
dag.available_providers("S2_MSI_L1C")
[6]:
['astraea_eod',
'aws_eos',
'creodias',
'earth_search',
'earth_search_gcs',
'mundi',
'onda',
'peps',
'sara',
'usgs']
Finally PEPS is declared as the preferred provider, which means that eodag
will look for products with this provider (all the providers are pre-configured).
[7]:
dag.set_preferred_provider("peps")
Search#
The EODataAccessGateway class provides three search methods which have a similar signature but behave in different ways:
search() returns a tuple with:
a SearchResult that stores the products obtained from a given page (default:
page=1
) and a given maximum number of items per page (default:items_per_page=20
)an integer that is the estimated total number of products matching the search criteria
Note
search() returns a paginated result. Pagination is the act of serving the result of a search page by page, each page contaning a maximum number of items. This is ubiquitous among search engines, e.g. a Google search displays in the browser 10 results per page. Pagination is built deep into the implementation of eodag
since most providers return a paginated result.
Note
A search made with the search() method with its default parameter often ends up with a SearchResult containing 20 products and a significantly greater estimated total number of products available. The parameter items_per_page
can be set to a higher value, or, the
search_all() method can be used instead.
Warning
The second element returned by search() is the estimated total number of products matching the search criteria, since, unfortunately, all the providers do not return the exact total number.
search_all() returns a SearchResult that contains all the products matching the search criteria. It takes the pain away from thinking about pagination.
search_iter_page() is a generator that returns a SearchResult page per page.
To get the S2_MSI_L1C products looked for, search() is used first as it’s handy to obtain some products and inspect them. Then search_all() is used to make all the products offered by PEPS are collected.
[9]:
search_criteria = {
"productType": "S2_MSI_L1C",
"start": "2021-03-01",
"end": "2021-03-31",
"geom": {"lonmin": 1, "latmin": 43, "lonmax": 2, "latmax": 44}
}
The search criteria in this example consists of:
productType
:eodag
’s product typestart
andend
: the start and end sensing datetimes (UTC)geom
: the region of interest, which can be defined in different ways (list
,dict
, WKT string,shapely.geometry
).
[10]:
products_first_page, estimated_total_number = dag.search(**search_criteria)
2021-04-20 18:02:40,944-15s eodag.core [INFO ] Searching product type 'S2_MSI_L1C' on provider: peps
2021-04-20 18:02:40,945-15s eodag.plugins.search.qssearch [INFO ] Sending count request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=1&page=1
2021-04-20 18:02:41,529-15s eodag.plugins.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=20&page=1
2021-04-20 18:02:42,499-15s eodag.core [INFO ] Found 48 result(s) on provider 'peps'
[11]:
print(f"Got {len(products_first_page)} products and an estimated total number of {estimated_total_number} products.")
Got 20 products and an estimated total number of 48 products.
A SearchResult contains a number of EOProduct. Each one of them has a properties
attribute that is a dictionnary storing the product’s metadata.
[12]:
products_first_page
[12]:
[EOProduct(id=S2B_MSIL1C_20210328T103629_N0209_R008_T31TCJ_20210328T124650, provider=peps), EOProduct(id=S2B_MSIL1C_20210328T103629_N0209_R008_T31TCH_20210328T124650, provider=peps), EOProduct(id=S2B_MSIL1C_20210328T103629_N0209_R008_T31TDH_20210328T124650, provider=peps), EOProduct(id=S2B_MSIL1C_20210328T103629_N0209_R008_T31TDJ_20210328T124650, provider=peps), EOProduct(id=S2A_MSIL1C_20210326T105031_N0209_R051_T31TDH_20210326T125540, provider=peps), EOProduct(id=S2A_MSIL1C_20210326T105031_N0209_R051_T31TDJ_20210326T125540, provider=peps), EOProduct(id=S2A_MSIL1C_20210326T105031_N0209_R051_T31TCJ_20210326T125540, provider=peps), EOProduct(id=S2A_MSIL1C_20210326T105031_N0209_R051_T31TCH_20210326T125540, provider=peps), EOProduct(id=S2A_MSIL1C_20210323T104021_N0209_R008_T31TCJ_20210323T141236, provider=peps), EOProduct(id=S2A_MSIL1C_20210323T104021_N0209_R008_T31TDJ_20210323T141236, provider=peps), EOProduct(id=S2A_MSIL1C_20210323T104021_N0209_R008_T31TDH_20210323T141236, provider=peps), EOProduct(id=S2A_MSIL1C_20210323T104021_N0209_R008_T31TCH_20210323T141236, provider=peps), EOProduct(id=S2B_MSIL1C_20210321T104639_N0209_R051_T31TCJ_20210321T130616, provider=peps), EOProduct(id=S2B_MSIL1C_20210321T104639_N0209_R051_T31TDJ_20210321T130616, provider=peps), EOProduct(id=S2B_MSIL1C_20210321T104639_N0209_R051_T31TCH_20210321T130616, provider=peps), EOProduct(id=S2B_MSIL1C_20210321T104639_N0209_R051_T31TDH_20210321T130616, provider=peps), EOProduct(id=S2B_MSIL1C_20210318T103649_N0209_R008_T31TDH_20210318T143809, provider=peps), EOProduct(id=S2B_MSIL1C_20210318T103649_N0209_R008_T31TCH_20210318T143809, provider=peps), EOProduct(id=S2B_MSIL1C_20210318T103649_N0209_R008_T31TCJ_20210318T143809, provider=peps), EOProduct(id=S2B_MSIL1C_20210318T103649_N0209_R008_T31TDJ_20210318T143809, provider=peps)]
[13]:
one_product = products_first_page[0]
one_product.properties.keys()
[13]:
dict_keys(['abstract', 'instrument', 'platform', 'platformSerialIdentifier', 'processingLevel', 'sensorType', 'license', 'missionStartDate', 'title', 'bands', 'productType', 'uid', 'keyword', 'resolution', 'organisationName', 'publicationDate', 'parentIdentifier', 'orbitNumber', 'orbitDirection', 'cloudCover', 'snowCover', 'creationDate', 'modificationDate', 'sensorMode', 'startTimeFromAscendingNode', 'completionTimeFromAscendingNode', 'id', 'quicklook', 'downloadLink', 'storageStatus', 'thumbnail', 'resourceSize', 'resourceChecksum', 'visible', 'newVersion', 'isNrt', 'realtime', 'relativeOrbitNumber', 's2TakeId', 'mgrs', 'bareSoil', 'highProbaClouds', 'mediumProbaClouds', 'lowProbaClouds', 'snowIce', 'vegetation', 'water', 'services', 'links', 'storage'])
A product also has a location
attribute. After being searched, it is equal to the remote location of the product, that will be used later by eodag
to download it.
[14]:
one_product.location
[14]:
'https://peps.cnes.fr/resto/collections/S2ST/387c7327-9a71-5a34-9163-0dfdeb024522/download'
All the products are now obtained from PEPS.
[15]:
all_products = dag.search_all(**search_criteria)
2021-04-20 18:02:42,532-15s eodag.core [INFO ] Searching product type 'S2_MSI_L1C' on provider: peps
2021-04-20 18:02:42,533-15s eodag.core [INFO ] Searching product type 'S2_MSI_L1C' on provider: peps
2021-04-20 18:02:42,537-15s eodag.plugins.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=500&page=1
2021-04-20 18:02:44,422-15s eodag.core [INFO ] Found 48 result(s) on provider 'peps'
[16]:
print(f"Got a total number of {len(all_products)} products.")
Got a total number of 48 products.
Crunch#
Crunching as defined in eodag
is a way to filter the products contained in a SearchResult object. Several crunchers/filters are available (e.g. FilterProperty to filter products products according to one of their properties). They can be called by passing the cruncher
object to SearchResult.crunch() or by running a dedicated method (e.g. SearchResult.filter_property()). The following example shows how to filter products to keep only those whose cloud cover is less than 10%.
[18]:
filtered_products = all_products.filter_property(cloudCover=10, operator="lt")
2021-04-20 18:02:44,440-15s eodag.plugins.crunch.filter_property [INFO ] Finished filtering products. 10 resulting products
[19]:
print(f"Got now {len(filtered_products)} products after filtering.")
Got now 10 products after filtering.
Serialize#
The method serialize() allows to save a SearchResult as a GeoJSON file.
[20]:
filtered_prods_filepath = dag.serialize(filtered_products, filename=os.path.join(workspace, "filtered_products.geojson"))
This method can come in handy to save the state of a search and restore it later with deserialize_and_register().
[21]:
# restored_filtered_prods = dag.deserialize_and_register(filtered_prods_filepath)
Download#
Before downloading any product, it can be useful to have a quick look at them. EO products usually offer a quicklook image, a low resolution by-product of the original data. An EOProduct
has a get_quicklook method that takes care of downloading the quicklook image and returns its path. matplotlib
can be used here to display the images collected.
[23]:
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
quicklooks_dir = os.path.join(workspace, "quicklooks")
if not os.path.isdir(quicklooks_dir):
os.mkdir(quicklooks_dir)
fig = plt.figure(figsize=(10, 8))
for i, product in enumerate(filtered_products, start=1):
# This line takes care of downloading the quicklook
quicklook_path = product.get_quicklook(base_dir=quicklooks_dir)
img = mpimg.imread(quicklook_path)
ax = fig.add_subplot(3, 4, i)
ax.set_title(i)
plt.imshow(img)
plt.tight_layout()
2021-04-20 18:02:45,068-15s eodag.api.product [INFO ] Download recorded in /home/maxime/TRAVAIL/06_EODAG/01_eodag/eodag/docs/notebooks/api_user_guide/eodag_workspace_overview/quicklooks/S2B_MSIL1C_20210328T103629_N0209_R008_T31TCJ_20210328T124650
2021-04-20 18:02:45,335-15s eodag.api.product [INFO ] Download recorded in /home/maxime/TRAVAIL/06_EODAG/01_eodag/eodag/docs/notebooks/api_user_guide/eodag_workspace_overview/quicklooks/S2B_MSIL1C_20210328T103629_N0209_R008_T31TCH_20210328T124650
2021-04-20 18:02:45,636-15s eodag.api.product [INFO ] Download recorded in /home/maxime/TRAVAIL/06_EODAG/01_eodag/eodag/docs/notebooks/api_user_guide/eodag_workspace_overview/quicklooks/S2B_MSIL1C_20210328T103629_N0209_R008_T31TDH_20210328T124650
2021-04-20 18:02:45,947-15s eodag.api.product [INFO ] Download recorded in /home/maxime/TRAVAIL/06_EODAG/01_eodag/eodag/docs/notebooks/api_user_guide/eodag_workspace_overview/quicklooks/S2B_MSIL1C_20210328T103629_N0209_R008_T31TDJ_20210328T124650
2021-04-20 18:02:46,255-15s eodag.api.product [INFO ] Download recorded in /home/maxime/TRAVAIL/06_EODAG/01_eodag/eodag/docs/notebooks/api_user_guide/eodag_workspace_overview/quicklooks/S2A_MSIL1C_20210323T104021_N0209_R008_T31TCJ_20210323T141236
2021-04-20 18:02:46,530-15s eodag.api.product [INFO ] Download recorded in /home/maxime/TRAVAIL/06_EODAG/01_eodag/eodag/docs/notebooks/api_user_guide/eodag_workspace_overview/quicklooks/S2A_MSIL1C_20210323T104021_N0209_R008_T31TDJ_20210323T141236
2021-04-20 18:02:46,802-15s eodag.api.product [INFO ] Download recorded in /home/maxime/TRAVAIL/06_EODAG/01_eodag/eodag/docs/notebooks/api_user_guide/eodag_workspace_overview/quicklooks/S2A_MSIL1C_20210323T104021_N0209_R008_T31TDH_20210323T141236
2021-04-20 18:02:47,165-15s eodag.api.product [INFO ] Download recorded in /home/maxime/TRAVAIL/06_EODAG/01_eodag/eodag/docs/notebooks/api_user_guide/eodag_workspace_overview/quicklooks/S2B_MSIL1C_20210301T104859_N0209_R051_T31TCH_20210301T115849
2021-04-20 18:02:47,455-15s eodag.api.product [INFO ] Download recorded in /home/maxime/TRAVAIL/06_EODAG/01_eodag/eodag/docs/notebooks/api_user_guide/eodag_workspace_overview/quicklooks/S2B_MSIL1C_20210301T104859_N0209_R051_T31TDJ_20210301T115849
2021-04-20 18:02:47,739-15s eodag.api.product [INFO ] Download recorded in /home/maxime/TRAVAIL/06_EODAG/01_eodag/eodag/docs/notebooks/api_user_guide/eodag_workspace_overview/quicklooks/S2B_MSIL1C_20210301T104859_N0209_R051_T31TCJ_20210301T115849

EOProducts can either be downloaded individually with download() or together with download_all() from a SearchResult. The last image is going to be downloaded, it is cloud-free and has no no-data pixel.
[24]:
product_to_download = filtered_products[-1]
product_path = dag.download(product_to_download)
product_path
2021-04-20 18:02:48,884-15s eodag.plugins.download.base [INFO ] Download url: https://peps.cnes.fr/resto/collections/S2ST/f913081f-eead-5709-ab65-cbaa1884150c/download
2021-04-20 18:04:11,370-15s eodag.plugins.download.base [INFO ] Extraction activated
2021-04-20 18:04:12,745-15s eodag.api.product [INFO ] Remote location of the product is still available through its 'remote_location' property: https://peps.cnes.fr/resto/collections/S2ST/f913081f-eead-5709-ab65-cbaa1884150c/download
[24]:
'file:///home/maxime/TRAVAIL/06_EODAG/01_eodag/eodag/docs/notebooks/api_user_guide/eodag_workspace_overview/S2B_MSIL1C_20210301T104859_N0209_R051_T31TCJ_20210301T115849/S2B_MSIL1C_20210301T104859_N0209_R051_T31TCJ_20210301T115849.SAFE'
The location
property of this product now points to a local path.
[25]:
product_to_download.location
[25]:
'file:///home/maxime/TRAVAIL/06_EODAG/01_eodag/eodag/docs/notebooks/api_user_guide/eodag_workspace_overview/S2B_MSIL1C_20210301T104859_N0209_R051_T31TCJ_20210301T115849/S2B_MSIL1C_20210301T104859_N0209_R051_T31TCJ_20210301T115849.SAFE'
Post-process#
Now the product is downloaded, it can be post-processed with softwares such as Sen2Cor or SNAP.
At some point eodag
had some capabilities to directly post-process a product, i.e. to access its data. These capabilities, which relied on rasterio, have been ported to the Python package eodag-cube to avoid the heavy dependencies associated with GDAL in particular. Installing this package is enough to benefit from its capabilities, it is going to extend
EOProduct with a get_data()
method which returns a product’s image band as a xarray.DataArray
.
The capabilities of eodag-cube are used hereafter to compute the NDVI of the downloaded product over a sub-extent of the original product (this is actually Toulouse, France).
Warning
eodag-cube needs to be installed to run correcly the following code.
[26]:
crs = "epsg:4326"
resolution = 0.0001
sub_extent = (1.435905, 43.586857, 1.458907, 43.603827)
VIR = product_to_download.get_data(band="B04", extent=sub_extent, crs=crs, resolution=resolution)
NIR = product_to_download.get_data(band="B08", extent=sub_extent, crs=crs, resolution=resolution)
NDVI = (NIR - VIR * 1.) / (NIR + VIR)
The NDVI can quickly be plotted directly with xarray
which uses matplotlib
under the hood.
[27]:
NDVI.plot(cmap="RdYlGn", center=False)
[27]:
<matplotlib.collections.QuadMesh at 0x7fb6e5f7da90>
