Hint
You can run this notebook in a live session with 
.
Search#
Searching for products is one of the most important features of eodag. This page describes the different methods available to search for products and the parameters that these methods accept.
eodag is set here to search for Sentinel 2 Level-1C products with PEPS.
[1]:
from eodag import EODataAccessGateway
from eodag import setup_logging
setup_logging(2)
dag = EODataAccessGateway()
dag.set_preferred_provider("peps")
2024-11-27 11:09:08,569 eodag.config [INFO ] Loading user configuration from: /home/julia/.config/eodag/eodag.yml
2024-11-27 11:09:08,606 eodag.core [INFO ] aws_eos: provider needing auth for search has been pruned because no credentials could be found
2024-11-27 11:09:08,607 eodag.core [INFO ] hydroweb_next: provider needing auth for search has been pruned because no credentials could be found
2024-11-27 11:09:08,608 eodag.core [INFO ] dedl: provider needing auth for search has been pruned because no credentials could be found
2024-11-27 11:09:08,621 eodag.core [INFO ] Locations configuration loaded from /home/julia/.config/eodag/locations.yml
A default set of search criteria is defined, the area of interest is in the South-West of France, and count is enabled.
[2]:
default_search_criteria = {
"productType": "S2_MSI_L1C",
"start": "2021-03-01",
"end": "2021-03-31",
"geom": {"lonmin": 1, "latmin": 43, "lonmax": 2, "latmax": 44},
"count": True
}
Pagination#
A Google search displays the first result page, which has 10 items (i.e. URLs). An action is required to get the next results, i.e. the second page, which would also contain 10 items. This process has the advantage of sending less data through the web, as a consequence the results are displayed faster. And after all, in most cases the URLs obtained from the first page are enough.
Pagination is what is described above. Most EO product providers operate this way, they return by default the first page result with a given number of items (i.e. products). Since pagination is ubiquitous among providers, it is built deep into eodag too and its search capabilities.
Search methods#
The three search methods introduced below accept pretty much the same parameters, which are described further down the page.
search()#
search() was the first search method implemented in eodag. It returns a SearchResult that stores the products obtained from a given page and a given maximum number of items per page. The attribute SearchResult.number_matched optionally stores the
estimated total number of products matching the search criteria.
By default, search() returns the products from the first page with a maximum of 20 products. This means that it is often expected to obtain a SearchResult that contains 20 products and a much larger estimated total number of products available.
Warning
The SearchResult.number_matched attribute is the estimated total number of products matching the search criteria, since, unfortunately, all the providers do not return the exact total number. For example, theia returns the number of products available in the whole collection instead of the number of products that match the search criteria.
Pagination can be controlled with two optional parameters: page and items_per_page.
[3]:
products_first_page = dag.search(**default_search_criteria)
2024-11-22 17:29:43,637 eodag.core [INFO ] Searching on provider peps
2024-11-22 17:29:43,639 eodag.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=20&page=1
2024-11-22 17:29:47,470 eodag.core [INFO ] Found 48 result(s) on provider 'peps'
[4]:
print(f"Got a hand on {len(products_first_page)} products and an estimated total number of {products_first_page.number_matched} products available.")
Got a hand on 20 products and an estimated total number of 48 products available.
[5]:
products_another_second_page = dag.search(page=2, items_per_page=10, **default_search_criteria)
2024-11-22 17:29:47,486 eodag.core [INFO ] Searching on provider peps
2024-11-22 17:29:47,487 eodag.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=10&page=2
2024-11-22 17:29:49,428 eodag.core [INFO ] Found 48 result(s) on provider 'peps'
[6]:
print(f"Got a hand on {len(products_another_second_page)} products and an estimated total number of {products_another_second_page.number_matched} products available.")
Got a hand on 10 products and an estimated total number of 48 products available.
Warning
To get all the products available, it would seem natural to set items_per_page to a very high value (e.g. 10000). However, the providers usually have set a maximum number of products/items that can be requested in a single query. If items_per_page is set to a value higher than this provider’s limit, the search may either return an empty SearchResult or fail and raise an error.
The raise_errors parameter controls how errors raised internally during a search are propagated to the user. By default this parameter is set to False, which means that errors are not raised. Instead, errors are logged and a null result is returned (empty SearchResult and 0). The use of the provider kwarg and the error raised in the example below are explained in the id and provider
sub-section and the fallback section respectively.
[7]:
bad_search_criteria = default_search_criteria.copy()
bad_search_criteria["start"] = "malformed_start_date"
[8]:
products_first_page = dag.search(provider="peps", **bad_search_criteria)
2024-11-22 17:29:49,455 eodag.core [INFO ] Searching on provider peps
2024-11-22 17:29:49,458 eodag.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=malformed_start_date&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=20&page=1
2024-11-22 17:29:49,757 eodag.search.qssearch [ERROR ] Skipping error while searching for peps QueryStringSearch instance
Traceback (most recent call last):
File "/home/julia/workspace/DEDL/eodag/venv/lib/python3.11/site-packages/eodag/plugins/search/qssearch.py", line 1248, in _request
response.raise_for_status()
File "/home/julia/workspace/DEDL/eodag/venv/lib/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=malformed_start_date&completionDate=2021-03-31&geometry=POLYGON%20((1.0000%2043.0000,%201.0000%2044.0000,%202.0000%2044.0000,%202.0000%2043.0000,%201.0000%2043.0000))&productType=S2MSI1C&maxRecords=20&page=1
2024-11-22 17:29:49,760 eodag.core [ERROR ] Error while searching on provider peps (ignored):
Traceback (most recent call last):
File "/home/julia/workspace/DEDL/eodag/venv/lib/python3.11/site-packages/eodag/plugins/search/qssearch.py", line 1248, in _request
response.raise_for_status()
File "/home/julia/workspace/DEDL/eodag/venv/lib/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=malformed_start_date&completionDate=2021-03-31&geometry=POLYGON%20((1.0000%2043.0000,%201.0000%2044.0000,%202.0000%2044.0000,%202.0000%2043.0000,%201.0000%2043.0000))&productType=S2MSI1C&maxRecords=20&page=1
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/julia/workspace/DEDL/eodag/venv/lib/python3.11/site-packages/eodag/api/core.py", line 1862, in _do_search
res, nb_res = search_plugin.query(prep, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/julia/workspace/DEDL/eodag/venv/lib/python3.11/site-packages/eodag/plugins/search/qssearch.py", line 791, in query
provider_results = self.do_search(prep, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/julia/workspace/DEDL/eodag/venv/lib/python3.11/site-packages/eodag/plugins/search/qssearch.py", line 929, in do_search
response = self._request(single_search_prep)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/julia/workspace/DEDL/eodag/venv/lib/python3.11/site-packages/eodag/plugins/search/qssearch.py", line 1263, in _request
raise RequestError.from_error(err, exception_message) from err
eodag.utils.exceptions.RequestError: ('Skipping error while searching for peps QueryStringSearch instance', ' {"ErrorMessage":"Value for \\"startDate\\" must follow the pattern ^[0-9]{4}-[0-9]{2}-[0-9]{2}(T[0-9]{2}:[0-9]{2}:[0-9]{2}(\\\\.[0-9]+)?(|Z|[\\\\+\\\\-][0-9]{2}:[0-9]{2}))?$","ErrorCode":400}')
[9]:
print(f"Got a hand on {len(products_first_page)} products and an estimated total number of {products_first_page.number_matched} products available.")
Got a hand on 0 products and an estimated total number of 0 products available.
Setting this parameter to True does propagate errors.
[10]:
products_first_page = dag.search(**bad_search_criteria, raise_errors=True)
2024-11-22 17:29:49,787 eodag.core [INFO ] Searching on provider peps
2024-11-22 17:29:49,790 eodag.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=malformed_start_date&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=20&page=1
2024-11-22 17:29:50,043 eodag.search.qssearch [ERROR ] Skipping error while searching for peps QueryStringSearch instance
Traceback (most recent call last):
File "/home/julia/workspace/DEDL/eodag/venv/lib/python3.11/site-packages/eodag/plugins/search/qssearch.py", line 1248, in _request
response.raise_for_status()
File "/home/julia/workspace/DEDL/eodag/venv/lib/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=malformed_start_date&completionDate=2021-03-31&geometry=POLYGON%20((1.0000%2043.0000,%201.0000%2044.0000,%202.0000%2044.0000,%202.0000%2043.0000,%201.0000%2043.0000))&productType=S2MSI1C&maxRecords=20&page=1
---------------------------------------------------------------------------
HTTPError Traceback (most recent call last)
File ~/workspace/DEDL/eodag/venv/lib/python3.11/site-packages/eodag/plugins/search/qssearch.py:1248, in QueryStringSearch._request(self, prep)
1241 response = session.get(
1242 url,
1243 timeout=timeout,
(...)
1246 **kwargs,
1247 )
-> 1248 response.raise_for_status()
1249 except requests.exceptions.Timeout as exc:
File ~/workspace/DEDL/eodag/venv/lib/python3.11/site-packages/requests/models.py:1024, in Response.raise_for_status(self)
1023 if http_error_msg:
-> 1024 raise HTTPError(http_error_msg, response=self)
HTTPError: 400 Client Error: Bad Request for url: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=malformed_start_date&completionDate=2021-03-31&geometry=POLYGON%20((1.0000%2043.0000,%201.0000%2044.0000,%202.0000%2044.0000,%202.0000%2043.0000,%201.0000%2043.0000))&productType=S2MSI1C&maxRecords=20&page=1
The above exception was the direct cause of the following exception:
RequestError Traceback (most recent call last)
Cell In[10], line 1
----> 1 products_first_page = dag.search(**bad_search_criteria, raise_errors=True)
File ~/workspace/DEDL/eodag/venv/lib/python3.11/site-packages/eodag/api/core.py:1218, in EODataAccessGateway.search(self, page, items_per_page, raise_errors, start, end, geom, locations, provider, count, **kwargs)
1216 for i, search_plugin in enumerate(search_plugins):
1217 search_plugin.clear()
-> 1218 search_results = self._do_search(
1219 search_plugin,
1220 count=count,
1221 raise_errors=raise_errors,
1222 **search_kwargs,
1223 )
1224 errors.extend(search_results.errors)
1225 if len(search_results) == 0 and i < len(search_plugins) - 1:
File ~/workspace/DEDL/eodag/venv/lib/python3.11/site-packages/eodag/api/core.py:1862, in EODataAccessGateway._do_search(self, search_plugin, count, raise_errors, **kwargs)
1859 prep.page = kwargs.pop("page", None)
1860 prep.items_per_page = kwargs.pop("items_per_page", None)
-> 1862 res, nb_res = search_plugin.query(prep, **kwargs)
1864 if not isinstance(res, list):
1865 raise PluginImplementationError(
1866 "The query function of a Search plugin must return a list of "
1867 "results, got {} instead".format(type(res))
1868 )
File ~/workspace/DEDL/eodag/venv/lib/python3.11/site-packages/eodag/plugins/search/qssearch.py:791, in QueryStringSearch.query(self, prep, **kwargs)
788 del prep.total_items_nb
789 del prep.need_count
--> 791 provider_results = self.do_search(prep, **kwargs)
792 if count and total_items is None and hasattr(prep, "total_items_nb"):
793 total_items = prep.total_items_nb
File ~/workspace/DEDL/eodag/venv/lib/python3.11/site-packages/eodag/plugins/search/qssearch.py:929, in QueryStringSearch.do_search(self, prep, **kwargs)
922 single_search_prep.info_message = "Sending search request: {}".format(
923 search_url
924 )
925 single_search_prep.exception_message = (
926 f"Skipping error while searching for {self.provider}"
927 f" {self.__class__.__name__} instance"
928 )
--> 929 response = self._request(single_search_prep)
930 next_page_url_key_path = self.config.pagination.get(
931 "next_page_url_key_path", None
932 )
933 next_page_query_obj_key_path = self.config.pagination.get(
934 "next_page_query_obj_key_path", None
935 )
File ~/workspace/DEDL/eodag/venv/lib/python3.11/site-packages/eodag/plugins/search/qssearch.py:1263, in QueryStringSearch._request(self, prep)
1255 else:
1256 logger.exception(
1257 "Skipping error while requesting: %s (provider:%s, plugin:%s): %s",
1258 url,
(...)
1261 err_msg,
1262 )
-> 1263 raise RequestError.from_error(err, exception_message) from err
1264 return response
RequestError: ('Skipping error while searching for peps QueryStringSearch instance', ' {"ErrorMessage":"Value for \\"startDate\\" must follow the pattern ^[0-9]{4}-[0-9]{2}-[0-9]{2}(T[0-9]{2}:[0-9]{2}:[0-9]{2}(\\\\.[0-9]+)?(|Z|[\\\\+\\\\-][0-9]{2}:[0-9]{2}))?$","ErrorCode":400}')
search_all()#
search_all() takes the pain away from thinking about pagination. It returns a SearchResult that contains all the products matching the search criteria. It does so by iterating over the pages of a search result (with search_iter_page()) and gathering products. Compared to search():
The attribute SearchResult.number_matched is
None. The estimate of total number of products available isn’t required here, since they all get collected anyway. This also spares some requests to be sent, since the estimate is usually obtained by sending an additional request.It tries to optimize the number of items/products requested per page. The limit of most providers has been configured in
eodag, it is used if available (e.g. 500 products per page). If not available, a default value of 50 is used. An arbitrary value can also be used.It has no
raise_errorsparameter, errors are not caught.countis not available for this method, as it will be directly SearchResult length.
[11]:
# remove count parameter from search criteria
default_search_criteria.pop("count", None)
all_products = dag.search_all(**default_search_criteria)
2024-11-22 17:31:06,308 eodag.core [INFO ] Searching for all the products with provider peps and a maximum of 500 items per page.
2024-11-22 17:31:06,310 eodag.core [INFO ] Iterate search over multiple pages: page #1
2024-11-22 17:31:06,310 eodag.core [INFO ] Searching on provider peps
2024-11-22 17:31:06,312 eodag.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=500&page=1
2024-11-22 17:31:10,425 eodag.core [INFO ] Found 48 result(s) on provider 'peps'
[12]:
print(f"Got a hand on a total number of {len(all_products)} products.")
Got a hand on a total number of 48 products.
The parameter items_per_page controls the maximum number of products than can be retrieved at each iteration internally.
[13]:
all_products = dag.search_all(**default_search_criteria, items_per_page=30)
2024-11-22 17:31:19,692 eodag.core [INFO ] Searching for all the products with provider peps and a maximum of 30 items per page.
2024-11-22 17:31:19,694 eodag.core [INFO ] Iterate search over multiple pages: page #1
2024-11-22 17:31:19,695 eodag.core [INFO ] Searching on provider peps
2024-11-22 17:31:19,698 eodag.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=30&page=1
2024-11-22 17:31:23,145 eodag.core [INFO ] Iterate search over multiple pages: page #2
2024-11-22 17:31:23,146 eodag.core [INFO ] Searching on provider peps
2024-11-22 17:31:23,148 eodag.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=30&page=2
2024-11-22 17:31:27,738 eodag.core [INFO ] Found 48 result(s) on provider 'peps'
The logs show that two requests have been sent to gather all the products, while only one was required in the previous case where search_all() used internally a limit higher than 48.
search_iter_page()#
search_iter_page() is a generator that returns a SearchResult page per page. Compared to search() and search_all(), it is certainly dedicated to be used by advanced users for some particular application.
As with search_all(), search_iter_page() doesn’t have a raise_errors parameter, it doesn’t catch errors. While search_all() optimizes the number of items per page requested by iteration,
search_iter_page() uses a default value of 20, which can be set to any arbitrary value.
[14]:
all_results = []
for i, page_results in enumerate(dag.search_iter_page(**default_search_criteria, items_per_page=30)):
print(f"Got a hand on {len(page_results)} products on page {i+1}")
all_results.extend(page_results)
2024-11-22 17:31:41,878 eodag.core [INFO ] Iterate search over multiple pages: page #1
2024-11-22 17:31:41,879 eodag.core [INFO ] Searching on provider peps
2024-11-22 17:31:41,881 eodag.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=30&page=1
2024-11-22 17:31:48,065 eodag.core [INFO ] Iterate search over multiple pages: page #2
2024-11-22 17:31:48,066 eodag.core [INFO ] Searching on provider peps
2024-11-22 17:31:48,069 eodag.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=30&page=2
Got a hand on 30 products on page 1
Got a hand on 18 products on page 2
[15]:
print(f"Got {len(all_products)} products after iterating over {i+1} pages.")
Got 48 products after iterating over 2 pages.
SearchResult and EOProduct#
Each search method returns an instance of the SearchResult class. This object is a sequence that stores a number of EOProduct instances. A SearchResult supports some of the capabilities of a classic Python list object.
[16]:
all_products[:2]
[16]:
| SearchResult (2) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
0 EOProduct(id=S2B_MSIL1C_20210328T103629_N0500_R008_T31TDH_20230602T033834, provider=peps)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
1 EOProduct(id=S2B_MSIL1C_20210328T103629_N0500_R008_T31TCJ_20230602T033834, provider=peps)
|
An EOProduct is the representation of an EO product for eodag, it stores enough information about how it was obtained (search criteria, provider) and about how to download itself. Most importantly it stores all the metadata that have been acquired by eodag during the search made
[17]:
one_product = all_products[0]
one_product
[17]:
| EOProduct | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
properties: (54){
assets: (0) |
geometry |
An EOProduct has the following attributes:
properties: dictionary of the product’s metadatageometry: the product’s geometry as ashapely.geometryobjectprovider: the provider used to obtain this productproduct_type:eodag’s identifier of its product typesearch_kwargs: a dictionary of the search criteria used to find the product.remote_location: the URL to the product’s download linklocation: it is equal toremote_locationbefore the product is downloaded. Once downloaded, it is updated to the absolute local path to the product.
[18]:
one_product.geometry
[18]:
[19]:
one_product.provider, one_product.product_type
[19]:
('peps', 'S2_MSI_L1C')
[20]:
one_product.search_kwargs
[20]:
{'productType': 'S2_MSI_L1C',
'startTimeFromAscendingNode': '2021-03-01',
'completionTimeFromAscendingNode': '2021-03-31',
'geometry': <POLYGON ((1 43, 1 44, 2 44, 2 43, 1 43))>}
[21]:
one_product.remote_location, one_product.location
[21]:
('https://peps.cnes.fr/resto/collections/S2ST/2d85d4c8-34c1-55dc-b43c-0ab0fe4c2b97/download',
'https://peps.cnes.fr/resto/collections/S2ST/2d85d4c8-34c1-55dc-b43c-0ab0fe4c2b97/download')
[22]:
one_product.properties.keys()
[22]:
dict_keys(['abstract', 'instrument', 'platform', 'platformSerialIdentifier', 'processingLevel', 'keywords', 'sensorType', 'license', 'missionStartDate', 'title', '_id', 'productType', 'uid', 'keyword', 'resolution', 'organisationName', 'publicationDate', 'parentIdentifier', 'orbitNumber', 'orbitDirection', 'cloudCover', 'snowCover', 'creationDate', 'modificationDate', 'sensorMode', 'startTimeFromAscendingNode', 'completionTimeFromAscendingNode', 'id', 'quicklook', 'downloadLink', 'tileIdentifier', 'storageStatus', 'thumbnail', 'resourceSize', 'resourceChecksum', 'visible', 'newVersion', 'isNrt', 'realtime', 'relativeOrbitNumber', 'useDatalake', 'bucket', 'prefix', 's2TakeId', 'bareSoil', 'highProbaClouds', 'mediumProbaClouds', 'lowProbaClouds', 'snowIce', 'vegetation', 'water', 'isRefined', 'nrtResource', 'storage'])
An EOProduct has an as_dict() to convert it into a GeoJSON-like dictionary and a from_geojson() method to create an EOProduct from a GeoJSON dictionary.
[23]:
from eodag import EOProduct
product_geojson_structure = one_product.as_dict()
recreated_product = EOProduct.from_geojson(product_geojson_structure)
recreated_product
[23]:
| EOProduct | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
properties: (57){
assets: (0) |
geometry |
Since EOProducts have a way to convert themselves to a GeoJSON dictionary, it is natural to be able to convert a SearchResult object to a GeoJSON FeatureCollection (as_geojson_object()). It is also possible to create a SearchResult from a dictionary structured as a FeatureCollection with from_geojson().
[24]:
from eodag import SearchResult
feature_collection = all_products.as_geojson_object()
recreated_search_result = SearchResult.from_geojson(feature_collection)
recreated_search_result[:2]
[24]:
| SearchResult (2) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
0 EOProduct(id=S2B_MSIL1C_20210328T103629_N0500_R008_T31TDH_20230602T033834, provider=peps)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
1 EOProduct(id=S2B_MSIL1C_20210328T103629_N0500_R008_T31TCJ_20230602T033834, provider=peps)
|
You can also convert a SearchResult object to other formats, like shapely.geometry.GeometryCollection (as_shapely_geometry_object()), and WKT (as_wkt_object()).
Note
EOProduct and SearchResult objects have more methods and attributes than the ones listed aboved. They are gradually introduced in the next pages of this user guide.
Note
When eodag-cube is installed the EOProduct class used by eodag is the one provided by eodag-cube, which extends it with a few more capabilities, including a get_data() method to directly access the product’s data.
Search parameters#
The search methods have exposed in their signature a common set of search parameters:
productTypestartandendgeomlocationscount
More parameters can be passed through kwargs, they are also described below.
Note that if you wish to know which parameters are available to use in a search, you may use the queryables API.
Product type#
The productType parameter allows to specify which kind of product is searched for. It should be one of eodag’s product type ID, such as S2_MSI_L1C. They are discoverable with the method list_product_types() or can be viewed here.
Warning
The feature described below is intended for advanced users.
The product types exposed by eodag for a given provider are hard-coded in a configuration file. This means that a provider can add a new product type to its catalog, which would then not be listed in eodag’s configuration until it is updated. To cover this case, it has been made possible to search for a product type not referenced by eodag.
Internally, a generic product type has been created (GENERIC_PRODUCT_TYPE) to emulate a non-referenced product type. The parameters required to search for a non-referenced product type are saved in eodag’s internal provider configuration file. For example creodias is configured with this generic produc type:
GENERIC_PRODUCT_TYPE:
productType: '{productType}'
collection: '{collection}'
The above configuration indicates that for this provider a non-referenced product type can be searched for by passing the productType and collection parameters to a search method, e.g. dag.search(collection="foo", productType="bar"). The values these parameters should take must be found by the user, eodag has just no clue about them.
To provide a more concrete example, eodag allows to search for S2_MSI_L1C product types with creodias. This product type configuration is the following:
S2_MSI_L1C:
productType: L1C
collection: Sentinel2
This means that dag.search(productType="L1C", collection="Sentinel2") should return the same products as dag.search(productType="S2_MSI_L1C").
Time period#
start and end parameters are optional. They are used to search for products that have been sensed between these two times. Dates or datetimes must be passed as strings in UTC ISO8601 format (e.g. yyyy-MM-dd, yyyy-MM-ddThh:mm:ss.SSSZ).
The string representation of dates will be interpreted as UTC datetime values, for instance, 2020-01-01 will be interpreted as 2020-01-01T00:00:00Z. Consequently, when time is not explicitly specified:
The start date will be inclusive, while the end date will be exclusive. In other words, results will include dates satisfying the condition:
start ≤ results_datetime < end.To search for products within a single day, the start date can be specified as
2020-01-01and the end date as2020-01-02.
Area of interest#
eodag provides multiple ways to define the area over which products should be searched for. The products returned are those that intersect with the area of interest, which means that their geometry can sometimes only partially overlap with the search geometry.
Warning
eodag transforms the area of interest passed by the user (e.g. a shapely polygon) to a geometry representation accepted by the targeted provider (e.g. a WKT string). Providers can set a limit to the size of its query parameters. In the case of a very large and detailed geometry passed by the user, its representation might exceed the provider’s limit (e.g. a very long WKT string). eodag tries internally to detect too long WKT strings, and simplifies them if so, by reducing the
resolution of the input geometry iteratively. However, this algorithm doesn’t enforce the output WKT string to be lower than a certain limit, eodag doesn’t want to change too much the geometry passed by the user. As a consequence, the request sent might still contain a WKT string that is above the provider’s limit (a WARNING is emitted), which would then end up in an error. In that case, it’s the user’s responsability to pass to eodag a simplified geometry, either by generating a
convex hull or by splitting it (and executing multiple searches).
folium is used in this section to display the search area and the extent of the products found on an interactive map.
[25]:
import folium
geom parameter#
The first way to define an area of interest is to define the optinal geom parameter which accepts the following different inputs:
a
Shapelygeometry object (any kind of geometry: point, line, polygon, multipolygon…)a Well-Known Text (WKT) string (any kind of geometry: point, line, polygon, multipolygon…)
a bounding box as a dictionary with keys:
"lonmin","latmin","lonmax","latmax"a bounding box as a list with elements provided in the order [lonmin, latmin, lonmax, latmax]
The coordinates must be provided in the WGS84 projection (EPSG: 4326).
[26]:
# Only a subset of the products is used not to overload the map
prods_to_map = all_products[::5]
print(f"{len(prods_to_map)} products are going to be mapped.")
10 products are going to be mapped.
In the previous searches made geom was defined as a bounding box expressed as a dictionary.
[27]:
geometry = default_search_criteria["geom"]
geometry
[27]:
{'lonmin': 1, 'latmin': 43, 'lonmax': 2, 'latmax': 44}
This is equivalent to:
[28]:
geom = [1, 43, 2, 44] # or geom = (1, 43, 1, 44)
[29]:
from shapely.geometry import Polygon
geom = Polygon([[1, 43], [2, 43], [2, 44], [1, 44], [1, 43]])
[30]:
geom = "POLYGON ((1 43, 2 43, 2 44, 1 44, 1 43))" # WKT string
[31]:
# Create a map zoomed over the search area
fmap = folium.Map([43.5, 1.5], zoom_start=7)
# Create a layer that represents the search area in red
folium.Rectangle(
bounds=[[geometry["latmin"], geometry["lonmin"]], [geometry["latmax"], geometry["lonmax"]]],
color="red",
tooltip="Search extent"
).add_to(fmap)
# Create a layer that maps the products found
folium.GeoJson(
data=prods_to_map, # SearchResult has a __geo_interface__ interface used by folium to get its GeoJSON representation
tooltip=folium.GeoJsonTooltip(fields=["title"])
).add_to(fmap)
fmap
[31]:
Make this Notebook Trusted to load map: File -> Trust Notebook
Locations search#
Locations search is a powerful feature that greatly simplifies the setting of an area of interest.
When the EODataAcessGateway instance was created the logs showed that a locations configuration was automatically loaded by eodag from its local configuration directory. A locations configuration is a YAML file that contains a shapefile list associated to a name and an attribute. A minimal example of such a file is provided below:
shapefiles:
- name: continent
path: /path/to/continents.shp
attr: fullname
Where:
nameis the argument name that can be used in a search() to refer to this specific location.pathis the absolute path to the shapefileattris the field of the shapefile that can be used to select features from it
For example, a continents.shp shapefile is set as a location in this file. The path entry is set to its absolute filepath, the name entry is set to continent. The shapefile contains continent’s areas (polygons) and a field fullname (it may have other fields, they just won’t be of any use here). The following search uses the geometry of the features of continents.shp that have fullname equal to Europe:
products = dag.search(
productType="S2_MSI_L1C",
locations=dict(continent='Europe')
)
The location query (continent="Europe") is passed as a dictionary to the locations parameter. It accepts regular expressions which can come in handy when the query field has an underlying structure (e.g. see this tutorial dedicated to search for products by tile(s)).
The locations configuration is stored in the locations_config attribute of the EODataAcessGateway once instantiated. eodag provides a default location which is a Natural Earth Countries shapefile whose ADM0_A3_US field can be used to query specific countries by a short code such as FRA for France or JPN for Japan.
[32]:
dag.locations_config
[32]:
[{'name': 'country',
'path': '/home/julia/.config/eodag/shp/ne_110m_admin_0_map_units.shp',
'attr': 'ADM0_A3_US'}]
[33]:
# Get the shapefile filepath and the field used as query parameter
countries_shpfile = dag.locations_config[0]["path"]
attr = dag.locations_config[0]["attr"]
# pyshp is imported to read the shapefile and display the values taken
# by the features for the field ADM0_A3_US.
import shapefile
with shapefile.Reader(countries_shpfile) as shp:
shaperecs = shp.shapeRecords()
countries_adm0 = sorted(set(shprec.record[attr] for shprec in shaperecs))
print(f"Values taken by `country` ({attr}):\n\n{' '.join(countries_adm0)}")
Values taken by `country` (ADM0_A3_US):
AFG AGO ALB ARE ARG ARM ATA ATF AUS AUT AZE BDI BEL BEN BFA BGD BGR BHS BIH BLR BLZ BOL BRA BRN BTN BWA CAF CAN CHE CHL CHN CIV CMR COD COG COL CRI CUB CYP CZE DEU DJI DNK DOM DZA ECU EGY ERI ESP EST ETH FIN FJI FLK FRA GAB GBR GEO GHA GIN GMB GNB GNQ GRC GRL GTM GUY HND HRV HTI HUN IDN IND IRL IRN IRQ ISL ISR ITA JAM JOR JPN KAZ KEN KGZ KHM KOR KOS KWT LAO LBN LBR LBY LKA LSO LTU LUX LVA MAR MDA MDG MEX MKD MLI MMR MNE MNG MOZ MRT MWI MYS NAM NCL NER NGA NIC NLD NOR NPL NZL OMN PAK PAN PER PHL PNG POL PRI PRK PRT PRY PSX QAT ROU RUS RWA SAH SAU SDN SDS SEN SLB SLE SLV SOM SRB SUR SVK SVN SWE SWZ SYR TCD TGO THA TJK TKM TLS TTO TUN TUR TWN TZA UGA UKR URY USA UZB VEN VNM VUT YEM ZAF ZMB ZWE
A search will be made over Switzerland (CHE) and Belgium (BEL).
[34]:
location_search_criteria = default_search_criteria.copy()
del location_search_criteria["geom"]
location_search_criteria["locations"] = dict(country="BEL|CHE") # This regex means: BEL or CHE
location_search_criteria
[34]:
{'productType': 'S2_MSI_L1C',
'start': '2021-03-01',
'end': '2021-03-31',
'locations': {'country': 'BEL|CHE'}}
[35]:
locations_products = dag.search(**location_search_criteria, items_per_page=50)
2024-11-22 17:32:30,158 eodag.core [INFO ] Searching on provider peps
2024-11-22 17:32:30,159 eodag.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=MULTIPOLYGON (((6.1567 50.8037, 6.0431 50.1281, 5.7824 50.0903, 5.6741 49.5295, 4.7992 49.9854, 4.2860 49.9075, 3.5882 50.3790, 3.1233 50.7804, 2.6584 50.7968, 2.5136 51.1485, 3.3150 51.3458, 3.3150 51.3458, 3.3150 51.3458, 4.0471 51.2673, 4.9740 51.4750, 5.6070 51.0373, 6.1567 50.8037)), ((9.5942 47.5251, 9.6329 47.3476, 9.4800 47.1028, 9.9324 46.9207, 10.4427 46.8935, 10.3634 46.4836, 9.9228 46.3149, 9.1829 46.4402, 8.9663 46.0369, 8.4900 46.0052, 8.3166 46.1636, 7.7560 45.8245, 7.2739 45.7769, 6.8436 45.9911, 6.5001 46.4297, 6.0226 46.2730, 6.0374 46.7258, 6.7687 47.2877, 6.7366 47.5418, 7.1922 47.4498, 7.4668 47.6206, 8.3173 47.6136, 8.5226 47.8308, 9.5942 47.5251)))&productType=S2MSI1C&maxRecords=50&page=1
The results obtained are displayed on a map in addition to the countries of the default location shapefile.
[36]:
# Create a map zoomed over the search area
fmap = folium.Map([48, 1.5], zoom_start=5)
# Create a layer that maps the countries in green
folium.GeoJson(
data=shaperecs,
tooltip=folium.GeoJsonTooltip(fields=["ADM0_A3_US"]),
style_function=lambda x: {"color": "green"}
).add_to(fmap)
# Create a layer that maps the products found
folium.GeoJson(
data=locations_products,
tooltip=folium.GeoJsonTooltip(fields=["title"])
).add_to(fmap)
fmap
[36]:
Make this Notebook Trusted to load map: File -> Trust Notebook