{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Search for products by tile" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Built-in search by tile\n", "\n", "Many providers already support search by tile (*Sentinel 2 MGRS tiling grid*):\n", "`peps`, `theia`, `onda`, `creodias`, `cop_dataspace`, `planetary_computer`, `earth_search`.\n", "\n", "For these providers, you can use `tileIdentifier` as search parameter using EODAG:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[(EOProduct(id=S2A_MSIL1C_20180614T103021_N0206_R108_T31TFK_20180614T124154, provider=peps),\n", " '31TFK'),\n", " (EOProduct(id=S2B_MSIL1C_20180612T104019_N0206_R008_T31TFK_20180612T124717, provider=peps),\n", " '31TFK'),\n", " (EOProduct(id=S2B_MSIL1C_20180609T103019_N0206_R108_T31TFK_20180609T131037, provider=peps),\n", " '31TFK'),\n", " (EOProduct(id=S2A_MSIL1C_20180607T104021_N0206_R008_T31TFK_20180607T124742, provider=peps),\n", " '31TFK'),\n", " (EOProduct(id=S2A_MSIL1C_20180604T103021_N0206_R108_T31TFK_20180604T141551, provider=peps),\n", " '31TFK'),\n", " (EOProduct(id=S2B_MSIL1C_20180602T104019_N0206_R008_T31TFK_20180602T132118, provider=peps),\n", " '31TFK')]\n" ] } ], "source": [ "from pprint import pprint\n", "from eodag import EODataAccessGateway, setup_logging\n", "\n", "dag = EODataAccessGateway()\n", "dag.set_preferred_provider(\"peps\")\n", "products, _ = dag.search(\n", " productType=\"S2_MSI_L1C\", \n", " start=\"2018-06-01\", \n", " end=\"2018-06-15\", \n", " tileIdentifier=\"31TFK\"\n", ")\n", "pprint([(product, product.properties[\"tileIdentifier\"]) for product in products])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Search by tile using custom locations\n", "`eodag` allows to search for products by geometric features that match a *location query*, see the [API user guide](../api_user_guide/4_search.ipynb#locations-search) for an introduction to this concept.\n", "\n", "In this tutorial we will use a shapefile that represents the Sentinel 2 tiling grid to search for *Sentinel 2 Level-1C* products with *PEPS* **at a specific tile**. In this shapefile each tile is defined by its centroid and a `tile_id` attribute (e.g. *29PMT*). This shapefile was created by downloading first the Sentinel 2 tiling grid (MGRS) provided [by ESA as a KML file](https://web.archive.org/web/20200907072744/https://sentinel.esa.int/web/sentinel/missions/sentinel-2/data-products). It was then converted as a shapefile and processed to compute the centroids. We use the tile's centroid here as `eodag` returns products that intersects the user defined search area. Since tiles overlap with each other, using the polygons instead of the centroids would return more tiles than just the one we target. " ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import os\n", "from zipfile import ZipFile\n", "\n", "# Interactive mapping\n", "import folium\n", "from folium.plugins import TimestampedGeoJson\n", "# pyshp: to read shapefiles\n", "import shapefile" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Setup" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A workspace directory is created to store the files that will be generated." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "workspace = \"eodag_workspace_locations_tiles\"\n", "if not os.path.isdir(workspace):\n", " os.mkdir(workspace)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You should have an `auxdata` folder next to this tutorial's file. It contains a shapefile that is needed to run this tutorial correctly." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "sentinel2_grid_zip = os.path.join(\"auxdata\", \"sentinel2_tiling_grid_centroids.zip\")\n", "if not os.path.isfile(sentinel2_grid_zip):\n", " raise FileNotFoundError(\"Auxdata not found, please check your configuration.\")" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "# We unzip the archived shapefile.\n", "with ZipFile(sentinel2_grid_zip, \"r\") as fzip:\n", " fzip.extractall(\"auxdata\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this tutorial products will just be searched for, not downloaded. We don't need to set up PEPS credentials to search for products. If you wish to download them, you should set the credentials beforehand, using these two environment variables for instance." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# os.environ[\"EODAG__PEPS__AUTH__CREDENTIALS__USERNAME\"] = \"PLEASE_CHANGE_ME\"\n", "# os.environ[\"EODAG__PEPS__AUTH__CREDENTIALS__PASSWORD\"] = \"PLEASE_CHANGE_ME\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Logging is activated to better inspect what `eodag` does internally." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "setup_logging(2) # INFO level" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The default search criteria consists of a time period in June 2018 and `eodag`'s product type identifier for *Sentinel 2 Level-1C* products." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "default_search_criteria = dict(\n", " productType=\"S2_MSI_L1C\",\n", " start=\"2018-06-01\",\n", " end=\"2018-06-15\"\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Add a locations configuration" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We check and store the content of this shapefile." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "shapefile Reader\n", " 56686 shapes (type 'POINT')\n", " 56686 records (2 fields) \n", "\n", "fields: [('DeletionFlag', 'C', 1, 0), ['tile_id', 'C', 5, 0]]\n" ] } ], "source": [ "sentinel2_shp = os.path.join('auxdata', 'sentinel2_tiling_grid_centroids.shp')\n", "with shapefile.Reader(sentinel2_shp) as shp:\n", " print(shp, \"\\n\")\n", " print(\"fields:\", shp.fields)\n", " shaperecs = shp.shapeRecords()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It has about 57 000 tiles/polygons and a field `tile_id`.\n", "\n", "We create a YAML file to configure this new location selector, we will refer to it with `s2_tile_centroid`." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "# Save the locations configuration file.\n", "locations_yaml_content = \"\"\"\n", "shapefiles:\n", " - name: s2_tile_centroid\n", " path: {}\n", " attr: tile_id\n", "\"\"\".format(os.path.abspath(sentinel2_shp))\n", "\n", "locations_filepath = os.path.abspath(os.path.join(workspace, \"custom_locations.yml\"))\n", "\n", "with open(locations_filepath, \"w\") as f_yml:\n", " f_yml.write(locations_yaml_content.strip())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "An instance of an [EODataAccessGateway](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway) class is created, it makes use of this location configuration file." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-10-17 15:39:30,945 eodag.config [INFO ] Loading user configuration from: /home/sylvain/.config/eodag/eodag.yml\n", "2023-10-17 15:39:31,063 eodag.core [INFO ] Locations configuration loaded from /home/sylvain/workspace/eodag/docs/notebooks/tutos/eodag_workspace_locations_tiles/custom_locations.yml\n" ] } ], "source": [ "dag = EODataAccessGateway(locations_conf_path=locations_filepath)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We want to look for *Sentinel 2 Level-1C* products. We can check whether this product type is offered by *PEPS* (as configured in `eodag`). If so, *PEPS* is set as the provider used to search for products." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "\"peps\" in dag.available_providers(\"S2_MSI_L1C\")" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "dag.set_preferred_provider(\"peps\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Search" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### A single tile" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Our target tile is `31TFK` and is located in the South-East of France. Its feature is retrieved from the shapefil to be displayed later on an interactive map." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "targeted_tile_name = \"31TFK\"\n", "\n", "# Get the targeted tile feature\n", "targeted_tile = [\n", " sr\n", " for sr in shaperecs\n", " if sr.record[\"tile_id\"] == \"31TFK\"\n", "][0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We search for all the products that intersect with the centroid of this tile." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-10-17 15:39:41,948 eodag.core [INFO ] Searching product type 'S2_MSI_L1C' on provider: peps\n", "2023-10-17 15:39:41,949 eodag.core [INFO ] Iterate search over multiple pages: page #1\n", "2023-10-17 15:39:41,950 eodag.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2018-06-01&completionDate=2018-06-15&geometry=POINT (4.9533 44.6422)&productType=S2MSI1C&maxRecords=500&page=1\n", "2023-10-17 15:39:42,796 eodag.core [INFO ] Found 6 result(s) on provider 'peps'\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "6 were found given the above search criteria\n" ] } ], "source": [ "products = dag.search_all(\n", " locations=dict(s2_tile_centroid=\"31TFK\"),\n", " **default_search_criteria\n", ")\n", "print(f\"{len(products)} were found given the above search criteria\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The products found are displayed on an interactive map along with the centroid of the targeted tile. A time player allows to see when the products were sensed." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Make this Notebook Trusted to load map: File -> Trust Notebook
" ], "text/plain": [ "" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# The GeoJSON representation has to be slightly adapted for the time slider\n", "adapted_prods = products.as_geojson_object()\n", "for feature in adapted_prods[\"features\"]:\n", " feature[\"properties\"][\"time\"] = feature[\"properties\"][\"startTimeFromAscendingNode\"]\n", "\n", "# Create a map zoomed over the search area\n", "fmap = folium.Map([44.5, 5], zoom_start=8)\n", "# Add a layer that map the tile's centroid\n", "folium.GeoJson(\n", " data=targeted_tile,\n", " tooltip = targeted_tile_name,\n", ").add_to(fmap)\n", "# Add layer that temporally maps the products found\n", "TimestampedGeoJson(\n", " adapted_prods,\n", " transition_time=50, # Transition duration in ms\n", " period=\"PT3H\", # Array of times, here every 3 hours\n", " duration=\"PT12H\", # Feature display duragion, here 6 hours\n", " time_slider_drag_update=True, # Update the map when the slider is dragged\n", " auto_play=False, # Don't auto play the animation\n", ").add_to(fmap)\n", "fmap" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "Note\n", "\n", "Instead of using the tile's centroid we could have directly used its extent and filter the returned products to keep only those fully contained within the tile. Check out the section dedicated to [filtering products](../api_user_guide/6_crunch.ipynb) in the API user guide.\n", "\n", "
\n", "\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Multiple tiles" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can search for products that overlap with several tiles using a **regular expression**. We use the expression `\"31T[CDE][MLK]\"` to look for products over 9 different tiles (*31TCM*, *31TCL*, *31TCK*, *31TDM*, etc.) over France." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-10-17 15:39:55,883 eodag.core [INFO ] Searching product type 'S2_MSI_L1C' on provider: peps\n", "2023-10-17 15:39:55,884 eodag.core [INFO ] Iterate search over multiple pages: page #1\n", "2023-10-17 15:39:55,885 eodag.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2018-06-01&completionDate=2018-06-15&geometry=MULTIPOINT (3.7147 46.4567, 3.7032 45.5565, 3.6922 44.6568, 2.4122 46.4574, 2.4216 45.5572, 1.1109 46.4433, 1.1413 45.5436, 1.1703 44.6442, 2.4306 44.6575)&productType=S2MSI1C&maxRecords=500&page=1\n", "2023-10-17 15:39:57,428 eodag.core [INFO ] Found 32 result(s) on provider 'peps'\n" ] } ], "source": [ "products = dag.search_all(\n", " locations=dict(s2_tile_centroid=\"31T[CDE][MLK]\"),\n", " **default_search_criteria\n", ")" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "32 were found given the above search criteria\n" ] } ], "source": [ "print(f\"{len(products)} were found given the above search criteria\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The products are displayed on an interactive map. By hovering over them you can observe that the MGRS number of the tiles match with the regular expressions we used." ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Make this Notebook Trusted to load map: File -> Trust Notebook
" ], "text/plain": [ "" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Create a map zoomed over the search area\n", "fmap = folium.Map([44.5, 1.5], zoom_start=6)\n", "# Create a layer that maps the products found\n", "folium.GeoJson(\n", " data=products,\n", " tooltip=folium.GeoJsonTooltip(\n", " fields=[\n", " \"title\", # The product's title\n", " \"tileIdentifier\", # The tile number on the MGRS grid\n", " ]\n", " ),\n", ").add_to(fmap)\n", "fmap" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This example has demonstrated the possibilities offered by `eodag` to easily select products from a tile grid by using regular expressions over their identifier." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 4 }