{ "cells": [ { "cell_type": "markdown", "id": "weird-cleveland", "metadata": {}, "source": [ "# EODAG as STAC client" ] }, { "cell_type": "markdown", "id": "reverse-sailing", "metadata": {}, "source": [ "## STAC API\n", "EODAG can perform an item search over a STAC compliant API. Found STAC items are returned as [EOProduct](../../api_reference/eoproduct.rst#eodag.api.product._product.EOProduct) objects with STAC metadata mapped to OGC OpenSearch Extension for Earth Observation.\n", "\n", "EODAG comes with already configured providers, but you can also add new ones dynamically." ] }, { "cell_type": "code", "execution_count": 1, "id": "pending-circumstances", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2021-03-08 18:07:00,193-15s eodag.config [INFO ] Loading user configuration from: /home/sylvain/.config/eodag/eodag.yml\n", "2021-03-08 18:07:00,405-15s eodag.core [INFO ] Locations configuration loaded from /home/sylvain/.config/eodag/locations.yml\n" ] } ], "source": [ "import os\n", "# To have some basic feedback on what eodag is doing, we configure logging to output minimum information\n", "from eodag import setup_logging\n", "setup_logging(verbose=2)\n", "\n", "from eodag.api.core import EODataAccessGateway\n", "\n", "dag = EODataAccessGateway()" ] }, { "cell_type": "markdown", "id": "signal-finland", "metadata": {}, "source": [ "List already configured providers providing a STAC API compliant service:" ] }, { "cell_type": "code", "execution_count": 2, "id": "naughty-speaker", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['astraea_eod', 'earth_search', 'usgs_satapi_aws']" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "[p.name for p in dag.providers_config.values() if hasattr(p, \"search\") and p.search.type == 'StacSearch']" ] }, { "cell_type": "markdown", "id": "suspended-wrapping", "metadata": {}, "source": [ "Then, let's update EODAG's configuration with a new STAC provider" ] }, { "cell_type": "code", "execution_count": 3, "id": "romance-midnight", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2021-03-08 18:07:00,425-15s eodag.config [INFO ] tamn: unknown provider found in user conf, trying to use provided configuration\n" ] } ], "source": [ "dag.update_providers_config(\"\"\"\n", " tamn:\n", " search:\n", " type: StacSearch\n", " api_endpoint: https://tamn.snapplanet.io/search\n", " products:\n", " S2_MSI_L1C:\n", " productType: S2\n", " GENERIC_PRODUCT_TYPE:\n", " productType: '{productType}'\n", " download:\n", " type: AwsDownload\n", " base_uri: https://tamn.snapplanet.io\n", " flatten_top_dirs: True\n", " auth:\n", " type: AwsAuth\n", " credentials:\n", " aws_access_key_id: PLEASE_CHANGE_ME\n", " aws_secret_access_key: PLEASE_CHANGE_ME\n", "\"\"\")\n", "dag.set_preferred_provider(\"tamn\")" ] }, { "cell_type": "markdown", "id": "accredited-heath", "metadata": {}, "source": [ "Search some S2_MSI_L1C products over Luxembourg:" ] }, { "cell_type": "code", "execution_count": 4, "id": "annual-scale", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2021-03-08 18:07:00,602-15s eodag.core [INFO ] Searching product type 'S2_MSI_L1C' on provider: tamn\n", "2021-03-08 18:07:00,603-15s eodag.plugins.search.qssearch [INFO ] Sending count request: https://tamn.snapplanet.io/search?datetime=2020-05-01T00:00:00.000Z/2020-05-15T00:00:00.000Z&bbox=5.674051954784829,49.44266714130711,6.242751092156993,50.128051662794235&collections=S2&limit=1&page=1\n", "2021-03-08 18:07:00,821-15s eodag.plugins.search.qssearch [INFO ] Sending search request: https://tamn.snapplanet.io/search?datetime=2020-05-01T00:00:00.000Z/2020-05-15T00:00:00.000Z&bbox=5.674051954784829,49.44266714130711,6.242751092156993,50.128051662794235&collections=S2&limit=50&page=1\n", "2021-03-08 18:07:01,525-15s eodag.core [INFO ] Found 35 result(s) on provider 'tamn'\n" ] } ], "source": [ "prods_S2L1C, _ = dag.search(productType=\"S2_MSI_L1C\", locations=dict(country=\"LUX\"), start=\"2020-05-01\", end=\"2020-05-15\", items_per_page=50)" ] }, { "cell_type": "markdown", "id": "surgical-condition", "metadata": {}, "source": [ "Filter over any item property using crunchers:" ] }, { "cell_type": "code", "execution_count": 5, "id": "widespread-january", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2021-03-08 18:07:01,528-15s eodag.plugins.crunch.filter_property [INFO ] Finished filtering products. 11 resulting products\n" ] }, { "data": { "text/plain": [ "11" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from eodag.plugins.crunch.filter_property import FilterProperty\n", "\n", "prods_S2L1C_filtered = prods_S2L1C.crunch(FilterProperty({\"cloudCover\": 10, \"operator\": \"lt\"}))\n", "len(prods_S2L1C_filtered)" ] }, { "cell_type": "markdown", "id": "sound-collective", "metadata": {}, "source": [ "List available assets from the first retrieved product" ] }, { "cell_type": "code", "execution_count": 6, "id": "chief-destination", "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "[('thumbnail',\n", " 'https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/31/U/GQ/2020/5/14/0/preview.jpg'),\n", " ('metadata',\n", " 'https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/31/U/GQ/2020/5/14/0/metadata.xml'),\n", " ('tileInfo',\n", " 'https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/31/U/GQ/2020/5/14/0/tileInfo.json'),\n", " ('productInfo',\n", " 'https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/31/U/GQ/2020/5/14/0/productInfo.json'),\n", " ('B1', 's3://sentinel-s2-l1c/tiles/31/U/GQ/2020/5/14/0/B1.jp2'),\n", " ('B2', 's3://sentinel-s2-l1c/tiles/31/U/GQ/2020/5/14/0/B2.jp2'),\n", " ('B3', 's3://sentinel-s2-l1c/tiles/31/U/GQ/2020/5/14/0/B3.jp2'),\n", " ('B4', 's3://sentinel-s2-l1c/tiles/31/U/GQ/2020/5/14/0/B4.jp2'),\n", " ('B5', 's3://sentinel-s2-l1c/tiles/31/U/GQ/2020/5/14/0/B5.jp2'),\n", " ('B6', 's3://sentinel-s2-l1c/tiles/31/U/GQ/2020/5/14/0/B6.jp2'),\n", " ('B7', 's3://sentinel-s2-l1c/tiles/31/U/GQ/2020/5/14/0/B7.jp2'),\n", " ('B8', 's3://sentinel-s2-l1c/tiles/31/U/GQ/2020/5/14/0/B8.jp2'),\n", " ('B8A', 's3://sentinel-s2-l1c/tiles/31/U/GQ/2020/5/14/0/B8A.jp2'),\n", " ('B9', 's3://sentinel-s2-l1c/tiles/31/U/GQ/2020/5/14/0/B9.jp2'),\n", " ('B10', 's3://sentinel-s2-l1c/tiles/31/U/GQ/2020/5/14/0/B10.jp2'),\n", " ('B11', 's3://sentinel-s2-l1c/tiles/31/U/GQ/2020/5/14/0/B11.jp2'),\n", " ('B12', 's3://sentinel-s2-l1c/tiles/31/U/GQ/2020/5/14/0/B12.jp2')]" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "[(key, asset[\"href\"]) for key, asset in prods_S2L1C[0].assets.items()]" ] }, { "cell_type": "markdown", "id": "american-rolling", "metadata": {}, "source": [ "Same thing with an unconfigured product type (should match available collections). \n", "\n", "For Tamn Landsat-8 products are available in L8 Collection. Let's search them over Spain:" ] }, { "cell_type": "code", "execution_count": 7, "id": "alpha-denver", "metadata": { "scrolled": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2021-03-08 18:07:01,562-15s eodag.plugins.manager [INFO ] UnsupportedProductType: L8, using generic settings\n", "2021-03-08 18:07:01,562-15s eodag.core [INFO ] Searching product type 'L8' on provider: tamn\n", "2021-03-08 18:07:01,563-15s eodag.plugins.search.qssearch [INFO ] Sending count request: https://tamn.snapplanet.io/search?datetime=2020-05-01T00:00:00.000Z/2020-05-15T00:00:00.000Z&bbox=-9.392883673530648,35.946850083961465,3.0394840836805486,43.74833771420099&collections=L8&limit=1&page=1\n", "2021-03-08 18:07:01,830-15s eodag.plugins.search.qssearch [INFO ] Sending search request: https://tamn.snapplanet.io/search?datetime=2020-05-01T00:00:00.000Z/2020-05-15T00:00:00.000Z&bbox=-9.392883673530648,35.946850083961465,3.0394840836805486,43.74833771420099&collections=L8&limit=20&page=1\n", "2021-03-08 18:07:02,393-15s eodag.core [INFO ] Found 98 result(s) on provider 'tamn'\n" ] } ], "source": [ "prods_L8, _ = dag.search(productType=\"L8\", country=\"ESP\", start=\"2020-05-01\", end=\"2020-05-15\")" ] }, { "cell_type": "code", "execution_count": 8, "id": "racial-statement", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[('thumbnail',\n", " 'https://landsat-pds.s3.amazonaws.com/c1/L8/196/035/LC08_L1TP_196035_20200514_20200527_01_T1/LC08_L1TP_196035_20200514_20200527_01_T1_thumb_large.jpg'),\n", " ('metadata',\n", " 'https://landsat-pds.s3.amazonaws.com/c1/L8/196/035/LC08_L1TP_196035_20200514_20200527_01_T1/LC08_L1TP_196035_20200514_20200527_01_T1_MTL.txt'),\n", " ('B1',\n", " 'https://landsat-pds.s3.amazonaws.com/c1/L8/196/035/LC08_L1TP_196035_20200514_20200527_01_T1/LC08_L1TP_196035_20200514_20200527_01_T1_B1.TIF'),\n", " ('B2',\n", " 'https://landsat-pds.s3.amazonaws.com/c1/L8/196/035/LC08_L1TP_196035_20200514_20200527_01_T1/LC08_L1TP_196035_20200514_20200527_01_T1_B2.TIF'),\n", " ('B3',\n", " 'https://landsat-pds.s3.amazonaws.com/c1/L8/196/035/LC08_L1TP_196035_20200514_20200527_01_T1/LC08_L1TP_196035_20200514_20200527_01_T1_B3.TIF'),\n", " ('B4',\n", " 'https://landsat-pds.s3.amazonaws.com/c1/L8/196/035/LC08_L1TP_196035_20200514_20200527_01_T1/LC08_L1TP_196035_20200514_20200527_01_T1_B4.TIF'),\n", " ('B5',\n", " 'https://landsat-pds.s3.amazonaws.com/c1/L8/196/035/LC08_L1TP_196035_20200514_20200527_01_T1/LC08_L1TP_196035_20200514_20200527_01_T1_B5.TIF'),\n", " ('B6',\n", " 'https://landsat-pds.s3.amazonaws.com/c1/L8/196/035/LC08_L1TP_196035_20200514_20200527_01_T1/LC08_L1TP_196035_20200514_20200527_01_T1_B6.TIF'),\n", " ('B7',\n", " 'https://landsat-pds.s3.amazonaws.com/c1/L8/196/035/LC08_L1TP_196035_20200514_20200527_01_T1/LC08_L1TP_196035_20200514_20200527_01_T1_B7.TIF'),\n", " ('B8',\n", " 'https://landsat-pds.s3.amazonaws.com/c1/L8/196/035/LC08_L1TP_196035_20200514_20200527_01_T1/LC08_L1TP_196035_20200514_20200527_01_T1_B8.TIF'),\n", " ('B9',\n", " 'https://landsat-pds.s3.amazonaws.com/c1/L8/196/035/LC08_L1TP_196035_20200514_20200527_01_T1/LC08_L1TP_196035_20200514_20200527_01_T1_B9.TIF'),\n", " ('B10',\n", " 'https://landsat-pds.s3.amazonaws.com/c1/L8/196/035/LC08_L1TP_196035_20200514_20200527_01_T1/LC08_L1TP_196035_20200514_20200527_01_T1_B10.TIF'),\n", " ('B11',\n", " 'https://landsat-pds.s3.amazonaws.com/c1/L8/196/035/LC08_L1TP_196035_20200514_20200527_01_T1/LC08_L1TP_196035_20200514_20200527_01_T1_B11.TIF')]" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "[(key, asset[\"href\"]) for key, asset in prods_L8[0].assets.items()]" ] }, { "cell_type": "markdown", "id": "offshore-stereo", "metadata": {}, "source": [ "## STAC Static catalog" ] }, { "cell_type": "markdown", "id": "seasonal-internet", "metadata": {}, "source": [ "EODAG can search for items over a STAC static catalog. Path to the catalog must be set as `provider.search.api_endpoint` with `provider.search.type=StaticStacSearch`. A download plugin must also be set, and depends of the provider.\n", "\n", "Here is an example with a catalog from https://stacindex.org/catalogs/spot-orthoimages-canada-2005, which will use \n", " `HTTPDownload` as download plugin, without credentials as no authentication needed for download.\n", " \n", "See [download plugins documentation](../../plugins_reference/download.rst) for other available plugins.\n", "\n", "
\n", "\n", "Warning\n", "\n", "Please note that `StaticStacSearch` plugin development is still at an early stage. If search is too slow using this plugin, please use a catalog with less elements.\n", "\n", "
" ] }, { "cell_type": "code", "execution_count": 9, "id": "greek-seminar", "metadata": {}, "outputs": [], "source": [ "# Decrease logging level\n", "setup_logging(verbose=1)\n", "\n", "# create a workspace\n", "workspace = 'eodag_workspace_stac_client'\n", "if not os.path.isdir(workspace):\n", " os.mkdir(workspace)\n", "\n", "# add the provider\n", "dag.update_providers_config(\"\"\"\n", "stac_http_provider:\n", " search:\n", " type: StaticStacSearch\n", " api_endpoint: https://canada-spot-ortho.s3.amazonaws.com/canada_spot_orthoimages/canada_spot5_orthoimages/S5_2007/catalog.json\n", " products:\n", " GENERIC_PRODUCT_TYPE:\n", " productType: '{productType}'\n", " download:\n", " type: HTTPDownload\n", " base_uri: https://fake-endpoint\n", " flatten_top_dirs: True\n", " outputs_prefix: %s\n", "\"\"\" % os.path.abspath(workspace))\n", "\n", "dag.set_preferred_provider(\"stac_http_provider\")" ] }, { "cell_type": "markdown", "id": "changing-tooth", "metadata": {}, "source": [ "Let's perform search :" ] }, { "cell_type": "code", "execution_count": 10, "id": "elder-convenience", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "3 product(s) found\n" ] } ], "source": [ "from shapely.geometry import Polygon\n", "\n", "search_polygon = Polygon([(-70, 45), (-75, 47), (-80, 47), (-80, 44)])\n", "query_args = {\"start\": \"2007-05-01\", \"end\": \"2007-05-06\" , \"geom\": search_polygon}\n", "\n", "products, found = dag.search(**query_args)\n", "print(\"%s product(s) found\" % found)" ] }, { "cell_type": "markdown", "id": "aerial-contrast", "metadata": {}, "source": [ "Before downloading, make some cleanup in the products as `canada-spot-ortho.s3.amazonaws.com` thumbnails are not available for download :\n", "- remove thumbnail assets\n", "- remove products with no assets" ] }, { "cell_type": "code", "execution_count": 11, "id": "executive-reserve", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2 product(s) with valid assets\n" ] } ], "source": [ "for idx, product in enumerate(products):\n", " # remove thumbnail\n", " if \"thumbnail\" in product.assets:\n", " del products[idx].assets[\"thumbnail\"]\n", " # remove items with empty assets\n", " if not product.assets:\n", " del products[idx]\n", "print(\"%s product(s) with valid assets\" % len(products))" ] }, { "cell_type": "code", "execution_count": 12, "id": "specialized-excuse", "metadata": { "scrolled": false }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "83fc933de3794e3db81b3da0a087541c", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Map(center=[45, -75], controls=(ZoomControl(options=['position', 'zoom_in_text', 'zoom_in_title', 'zoom_out_te…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# plot products and search polygon on map\n", "import ipyleaflet as ipyl\n", "\n", "m = ipyl.Map(center=(45, -75), zoom=5)\n", "\n", "polygon_layer = ipyl.GeoJSON(data=search_polygon.__geo_interface__, style=dict(color='blue'))\n", "m.add_layer(polygon_layer)\n", "\n", "items_layer = ipyl.GeoJSON(data=products.as_geojson_object(), style=dict(color='green'))\n", "m.add_layer(items_layer)\n", "m" ] }, { "cell_type": "markdown", "id": "optional-flower", "metadata": {}, "source": [ "Download items from the filtered search results:" ] }, { "cell_type": "code", "execution_count": 13, "id": "becoming-mason", "metadata": { "scrolled": false }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "24be9c35efb7433ea14b3fae9112305f", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloaded products: 0%| | 0/2 [00:00