{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Providers and products" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from eodag import EODataAccessGateway\n", "dag = EODataAccessGateway()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Providers available" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The method [available_providers()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.available_providers) returns a list of the pre-configured providers." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['astraea_eod',\n", " 'aws_eos',\n", " 'cop_ads',\n", " 'cop_cds',\n", " 'cop_dataspace',\n", " 'creodias',\n", " 'earth_search',\n", " 'earth_search_cog',\n", " 'earth_search_gcs',\n", " 'ecmwf',\n", " 'hydroweb_next',\n", " 'meteoblue',\n", " 'onda',\n", " 'peps',\n", " 'planetary_computer',\n", " 'sara',\n", " 'theia',\n", " 'usgs',\n", " 'usgs_satapi_aws',\n", " 'wekeo']" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "available_providers = dag.available_providers()\n", "available_providers" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "eodag has 20 providers already configured.\n" ] } ], "source": [ "print(f\"eodag has {len(available_providers)} providers already configured.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It can take a product type as an argument and will return the providers known to `eodag` that offer this product." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['astraea_eod',\n", " 'aws_eos',\n", " 'cop_dataspace',\n", " 'creodias',\n", " 'earth_search',\n", " 'earth_search_gcs',\n", " 'onda',\n", " 'peps',\n", " 'sara',\n", " 'usgs',\n", " 'wekeo']" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dag.available_providers(\"S2_MSI_L1C\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "Note\n", "\n", "If a provider is configured to need authentication for search, and has no crendentials set, it will be pruned on EODAG initialization, and will not appear in available providers list.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Product types available" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The method [list_product_types()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.list_product_types) returns a dictionary that represents `eodag`'s internal product type catalog if used with `fetch_providers=False`. It will fetch providers for new product types and return an extended list if used with `fetch_providers=True` (default behavior)." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "EODAG has 149 product types stored in its internal catalog.\n" ] } ], "source": [ "internal_catalog = dag.list_product_types(fetch_providers=False)\n", "print(f\"EODAG has {len(internal_catalog)} product types stored in its internal catalog.\")" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "EODAG has 295 product types stored in its extended catalog, after having fetched providers.\n" ] } ], "source": [ "extended_catalog = dag.list_product_types()\n", "print(f\"EODAG has {len(extended_catalog)} product types stored in its extended catalog, after having fetched providers.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When providers are fetched for new product types, `eodag`'s product types configuration is updated in `EODataAccessGateway` instance. Extended product types list is then returned independantly of `fetch_providers` option in [list_product_types()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.list_product_types):" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "list_product_types() keeps returning 295 product types.\n" ] } ], "source": [ "called_again_catalog = dag.list_product_types(fetch_providers=False)\n", "print(f\"list_product_types() keeps returning {len(called_again_catalog)} product types.\")" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'ID': 'CAMS_EAC4',\n", " 'abstract': 'CAMS (Copernicus Atmosphere Monitoring Service) ECMWF Atmospheric Composition Reanalysis 4\\nfrom Copernicus ADS\\n',\n", " 'instrument': None,\n", " 'platform': 'CAMS',\n", " 'platformSerialIdentifier': 'CAMS',\n", " 'processingLevel': None,\n", " 'keywords': 'Copernicus,Atmosphere,Atmospheric,Reanalysis,CAMS,EAC4,ADS,ECMWF',\n", " 'sensorType': 'ATMOSPHERIC',\n", " 'license': 'proprietary',\n", " 'title': 'CAMS ECMWF Atmospheric Composition Reanalysis 4',\n", " 'missionStartDate': '2003-01-01T00:00:00Z'}" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "internal_catalog[0]" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['CAMS_EAC4',\n", " 'CAMS_GACF_AOT',\n", " 'CAMS_GACF_MR',\n", " 'CAMS_GACF_RH',\n", " 'CBERS4_AWFI_L2',\n", " 'CBERS4_AWFI_L4',\n", " 'CBERS4_MUX_L2',\n", " 'CBERS4_MUX_L4',\n", " 'CBERS4_PAN10M_L2',\n", " 'CBERS4_PAN10M_L4',\n", " 'CBERS4_PAN5M_L2',\n", " 'CBERS4_PAN5M_L4',\n", " 'CLMS_CORINE',\n", " 'CLMS_GLO_DMP_333M',\n", " 'CLMS_GLO_FAPAR_333M',\n", " 'CLMS_GLO_FCOVER_333M',\n", " 'CLMS_GLO_GDMP_333M',\n", " 'CLMS_GLO_LAI_333M',\n", " 'CLMS_GLO_NDVI_1KM_LTS',\n", " 'CLMS_GLO_NDVI_333M',\n", " 'COP_DEM_GLO30_DGED',\n", " 'COP_DEM_GLO30_DTED',\n", " 'COP_DEM_GLO90_DGED',\n", " 'COP_DEM_GLO90_DTED',\n", " 'EEA_DAILY_SSM_1KM',\n", " 'EEA_DAILY_SWI_1KM',\n", " 'EEA_DAILY_VI',\n", " 'EFAS_FORECAST',\n", " 'EFAS_HISTORICAL',\n", " 'EFAS_REFORECAST',\n", " 'EFAS_SEASONAL',\n", " 'EFAS_SEASONAL_REFORECAST',\n", " 'ERA5_LAND',\n", " 'ERA5_LAND_MONTHLY',\n", " 'ERA5_PL',\n", " 'ERA5_PL_MONTHLY',\n", " 'ERA5_SL',\n", " 'ERA5_SL_MONTHLY',\n", " 'FIRE_HISTORICAL',\n", " 'GLACIERS_DIST_RANDOLPH',\n", " 'GLACIERS_ELEVATION_AND_MASS_CHANGE',\n", " 'GLOFAS_FORECAST',\n", " 'GLOFAS_HISTORICAL',\n", " 'GLOFAS_REFORECAST',\n", " 'GLOFAS_SEASONAL',\n", " 'GLOFAS_SEASONAL_REFORECAST',\n", " 'L57_REFLECTANCE',\n", " 'L8_OLI_TIRS_C1L1',\n", " 'L8_REFLECTANCE',\n", " 'LANDSAT_C2L1',\n", " 'LANDSAT_C2L2',\n", " 'LANDSAT_C2L2ALB_BT',\n", " 'LANDSAT_C2L2ALB_SR',\n", " 'LANDSAT_C2L2ALB_ST',\n", " 'LANDSAT_C2L2ALB_TA',\n", " 'LANDSAT_C2L2_SR',\n", " 'LANDSAT_C2L2_ST',\n", " 'LANDSAT_ETM_C1',\n", " 'LANDSAT_ETM_C2L1',\n", " 'LANDSAT_ETM_C2L2',\n", " 'LANDSAT_TM_C1',\n", " 'LANDSAT_TM_C2L1',\n", " 'LANDSAT_TM_C2L2',\n", " 'MODIS_MCD43A4',\n", " 'NAIP',\n", " 'NEMSAUTO_TCDC',\n", " 'NEMSGLOBAL_TCDC',\n", " 'OSO',\n", " 'PLD_BUNDLE',\n", " 'PLD_PAN',\n", " 'PLD_PANSHARPENED',\n", " 'PLD_XS',\n", " 'S1_SAR_GRD',\n", " 'S1_SAR_OCN',\n", " 'S1_SAR_RAW',\n", " 'S1_SAR_SLC',\n", " 'S2_MSI_L1C',\n", " 'S2_MSI_L2A',\n", " 'S2_MSI_L2A_COG',\n", " 'S2_MSI_L2A_MAJA',\n", " 'S2_MSI_L2B_MAJA_SNOW',\n", " 'S2_MSI_L2B_MAJA_WATER',\n", " 'S2_MSI_L3A_WASP',\n", " 'S3_EFR',\n", " 'S3_ERR',\n", " 'S3_LAN',\n", " 'S3_OLCI_L2LFR',\n", " 'S3_OLCI_L2LRR',\n", " 'S3_OLCI_L2WFR',\n", " 'S3_OLCI_L2WRR',\n", " 'S3_RAC',\n", " 'S3_SLSTR_L1RBT',\n", " 'S3_SLSTR_L2',\n", " 'S3_SLSTR_L2AOD',\n", " 'S3_SLSTR_L2FRP',\n", " 'S3_SLSTR_L2LST',\n", " 'S3_SLSTR_L2WST',\n", " 'S3_SRA',\n", " 'S3_SRA_A',\n", " 'S3_SRA_BS',\n", " 'S3_SY_AOD',\n", " 'S3_SY_SYN',\n", " 'S3_SY_V10',\n", " 'S3_SY_VG1',\n", " 'S3_SY_VGP',\n", " 'S3_WAT',\n", " 'S5P_L1B2_IR_ALL',\n", " 'S5P_L1B_IR_SIR',\n", " 'S5P_L1B_IR_UVN',\n", " 'S5P_L1B_RA_BD1',\n", " 'S5P_L1B_RA_BD2',\n", " 'S5P_L1B_RA_BD3',\n", " 'S5P_L1B_RA_BD4',\n", " 'S5P_L1B_RA_BD5',\n", " 'S5P_L1B_RA_BD6',\n", " 'S5P_L1B_RA_BD7',\n", " 'S5P_L1B_RA_BD8',\n", " 'S5P_L2_AER_AI',\n", " 'S5P_L2_AER_LH',\n", " 'S5P_L2_CH4',\n", " 'S5P_L2_CLOUD',\n", " 'S5P_L2_CO',\n", " 'S5P_L2_HCHO',\n", " 'S5P_L2_NO2',\n", " 'S5P_L2_NP_BD3',\n", " 'S5P_L2_NP_BD6',\n", " 'S5P_L2_NP_BD7',\n", " 'S5P_L2_O3',\n", " 'S5P_L2_O3_PR',\n", " 'S5P_L2_O3_TCL',\n", " 'S5P_L2_SO2',\n", " 'SATELLITE_CARBON_DIOXIDE',\n", " 'SATELLITE_METHANE',\n", " 'SATELLITE_SEA_LEVEL_BLACK_SEA',\n", " 'SEASONAL_MONTHLY_PL',\n", " 'SEASONAL_MONTHLY_SL',\n", " 'SEASONAL_ORIGINAL_PL',\n", " 'SEASONAL_ORIGINAL_SL',\n", " 'SEASONAL_POSTPROCESSED_PL',\n", " 'SEASONAL_POSTPROCESSED_SL',\n", " 'SIS_HYDRO_MET_PROJ',\n", " 'SPOT5_SPIRIT',\n", " 'SPOT_SWH',\n", " 'SPOT_SWH_OLD',\n", " 'TIGGE_CF_SFC',\n", " 'UERRA_EUROPE_SL',\n", " 'VENUS_L1C',\n", " 'VENUS_L2A_MAJA',\n", " 'VENUS_L3A_MAJA']" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "products_id = [p[\"ID\"] for p in internal_catalog]\n", "products_id" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The method can take a provider name as an argument and will return the product types known to `eodag` that are offered by this provider." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['S1_SAR_GRD', 'S1_SAR_OCN', 'S1_SAR_SLC', 'S2_MSI_L1C', 'S2_MSI_L2A']" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "peps_products = dag.list_product_types(\"peps\")\n", "[p[\"ID\"] for p in peps_products]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Combine these two methods" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "These two methods can be combined to find which product type is the most common in `eodag`'s catalog among all the providers." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The most common product type is 'S2_MSI_L1C' with 11 providers offering it.\n" ] } ], "source": [ "availability_per_product = []\n", "for product in products_id:\n", " providers = dag.available_providers(product)\n", " availability_per_product.append((product, len(providers)))\n", "availability_per_product = sorted(availability_per_product, key=lambda x: x[1], reverse=True)\n", "most_common_p_type, nb_providers = availability_per_product[0]\n", "print(f\"The most common product type is '{most_common_p_type}' with {nb_providers} providers offering it.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "These can be also used to find out which provider (as configured by `eodag`) offers the hights number of different product types." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The provider with the largest number of product types is 'planetary_computer' with 122.\n" ] } ], "source": [ "availability_per_provider = []\n", "for provider in dag.available_providers():\n", " provider_products_id = [\n", " p[\"ID\"]\n", " for p in dag.list_product_types(provider, fetch_providers=False)\n", " ]\n", " availability_per_provider.append(\n", " (provider, len(provider_products_id))\n", " )\n", "availability_per_provider = sorted(availability_per_provider, key=lambda x: x[1], reverse=True)\n", "provider, nb_p_types = availability_per_provider[0]\n", "print(f\"The provider with the largest number of product types is '{provider}' with {nb_p_types}.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Product types discovery" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "EODAG comes with a large list of pre-configured product types. Some others are available from providers catalogs but will not be configured, or are not yet configured in EODAG.\n", "\n", "Some providers, like STAC providers, come in EODAG with a configuration describing how to discover these not-already-configured product types.\n", "\n", "With the method [discover_product_types()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.discover_product_types) \n", "or CLI command [eodag discover](../../cli_user_guide.rst) we can obtain a JSON configuration file that will be used as *EODAG \n", "external product types configuration file*.\n", "\n", "In EODAG, the discovered *EODAG external product types configuration file* can be set to:\n", "\n", "* a file automatically built from github actions and stored in [eodag/resources/ext_product_types.json](https://raw.githubusercontent.com/CS-SI/eodag/develop/eodag/resources/ext_product_types.json) (default settings)\n", "* a custom remote or local file by setting its path in `EODAG_EXT_PRODUCT_TYPES_CFG_FILE` environment variable (if the file is not readable, only user-modified providers will be fetched).\n", "\n", "Then, when listing product types using [list_product_types(fetch_providers=True)](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.list_product_types), EODAG will first read the content of the *EODAG external product types configuration file* using [fetch_product_types_list()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.fetch_product_types_list) \n", "then update [EODataAccessGateway](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway) instance product types configuration, if needed.\n", "\n", "The obtained product types list will contain both pre-configured and discovered product types.\n", "\n", "![Fetch product types schema](../../_static/eodag_fetch_product_types.png \"Fetch product types schema\")" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" }, "nbsphinx": { "execute": "always" }, "vscode": { "interpreter": { "hash": "946c0635d97db016b4c650c0f14014bdbd70e0aaae6a8f4b82f75907772dbc16" } }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 4 }