Hint

You can run this notebook in a live session with Binder.

Providers and products#

[1]:
from eodag import EODataAccessGateway
dag = EODataAccessGateway()

Providers available#

The method available_providers() returns a list of the pre-configured providers.

[2]:
available_providers = dag.available_providers()
available_providers
[2]:
['astraea_eod',
 'cop_ads',
 'cop_cds',
 'cop_dataspace',
 'creodias',
 'earth_search',
 'earth_search_cog',
 'earth_search_gcs',
 'ecmwf',
 'onda',
 'peps',
 'planetary_computer',
 'sara',
 'theia',
 'usgs_satapi_aws']
[3]:
print(f"eodag has {len(available_providers)} providers already configured.")
eodag has 15 providers already configured.

It can take a product type as an argument and will return the providers known to eodag that offer this product.

[4]:
dag.available_providers("S2_MSI_L1C")
[4]:
['astraea_eod',
 'cop_dataspace',
 'creodias',
 'earth_search',
 'earth_search_gcs',
 'onda',
 'peps',
 'sara']

Note

If a provider is configured to need authentication for search, and has no crendentials set, it will be pruned on EODAG initialization, and will not appear in available providers list.

Product types available#

The method list_product_types() returns a dictionary that represents eodag’s internal product type catalog if used with fetch_providers=False. It will fetch providers for new product types and return an extended list if used with fetch_providers=True (default behavior).

[5]:
internal_catalog = dag.list_product_types(fetch_providers=False)
print(f"EODAG has {len(internal_catalog)} product types stored in its internal catalog.")
EODAG has 128 product types stored in its internal catalog.
[6]:
extended_catalog = dag.list_product_types()
print(f"EODAG has {len(extended_catalog)} product types stored in its extended catalog, after having fetched providers.")
EODAG has 270 product types stored in its extended catalog, after having fetched providers.

When providers are fetched for new product types, eodag’s product types configuration is updated in EODataAccessGateway instance. Extended product types list is then returned independantly of fetch_providers option in list_product_types():

[7]:
called_again_catalog = dag.list_product_types(fetch_providers=False)
print(f"list_product_types() keeps returning {len(called_again_catalog)} product types.")
list_product_types() keeps returning 270 product types.
[8]:
internal_catalog[0]
[8]:
{'ID': 'CAMS_EAC4',
 'abstract': 'EAC4 (ECMWF Atmospheric Composition Reanalysis 4) is the fourth generation ECMWF global reanalysis of atmospheric composition. Reanalysis combines model data with observations from across the world into a globally complete and consistent dataset using a model of the atmosphere based on the laws of physics and chemistry.\nThis principle, called data assimilation, is based on the method used by numerical weather prediction centres and air quality forecasting centres, where every so many hours (12 hours at ECMWF) a previous forecast is combined with newly available observations in an optimal way to produce a new best estimate of the state of the atmosphere, called analysis, from which an updated, improved forecast is issued.\nReanalysis works in the same way to allow for the provision of a dataset spanning back more than a decade.\nReanalysis does not have the constraint of issuing timely forecasts, so there is more time to collect observations, and when going further back in time, to allow for the ingestion of improved versions of the original observations, which all benefit the quality of the reanalysis product.\nThe assimilation system is able to estimate biases between observations and to sift good-quality data from poor data.\nThe atmosphere model allows for estimates at locations where data coverage is low or for atmospheric pollutants for which no direct observations are available.\nThe provision of estimates at each grid point around the globe for each regular output time, over a long period, always using the same format, makes reanalysis a very convenient and popular dataset to work with.\nThe observing system has changed drastically over time, and although the assimilation system can resolve data holes, the initially much sparser networks will lead to less accurate estimates.\nFor this reason, EAC4 is only available from 2003 onwards.\nAlthough the analysis procedure considers chunks of data in a window of 12 hours in one go, EAC4 provides estimates every 3 hours, worldwide. This is made possible by the 4D-Var assimilation method, which takes account of the exact timing of the observations and model evolution within the assimilation window.\n',
 'instrument': None,
 'platform': 'CAMS',
 'platformSerialIdentifier': 'CAMS',
 'processingLevel': None,
 'keywords': 'Copernicus,ADS,CAMS,Atmosphere,Atmospheric,EWMCF,EAC4',
 'sensorType': 'ATMOSPHERIC',
 'license': 'proprietary',
 'title': 'CAMS global reanalysis (EAC4)',
 'missionStartDate': '2003-01-01T00:00:00Z'}
[9]:
products_id = [p["ID"] for p in internal_catalog]
products_id
[9]:
['CAMS_EAC4',
 'CAMS_EAC4_MONTHLY',
 'CAMS_EU_AIR_QUALITY_FORECAST',
 'CAMS_EU_AIR_QUALITY_RE',
 'CAMS_GAC_FORECAST',
 'CAMS_GFE_GFAS',
 'CAMS_GLOBAL_EMISSIONS',
 'CAMS_GREENHOUSE_EGG4',
 'CAMS_GREENHOUSE_EGG4_MONTHLY',
 'CAMS_GREENHOUSE_INVERSION',
 'CAMS_GRF',
 'CAMS_GRF_AUX',
 'CAMS_SOLAR_RADIATION',
 'COP_DEM_GLO30_DGED',
 'COP_DEM_GLO30_DTED',
 'COP_DEM_GLO90_DGED',
 'COP_DEM_GLO90_DTED',
 'EFAS_FORECAST',
 'EFAS_HISTORICAL',
 'EFAS_REFORECAST',
 'EFAS_SEASONAL',
 'EFAS_SEASONAL_REFORECAST',
 'ERA5_LAND',
 'ERA5_LAND_MONTHLY',
 'ERA5_PL',
 'ERA5_PL_MONTHLY',
 'ERA5_SL',
 'ERA5_SL_MONTHLY',
 'FIRE_HISTORICAL',
 'GLACIERS_DIST_RANDOLPH',
 'GLOFAS_FORECAST',
 'GLOFAS_HISTORICAL',
 'GLOFAS_REFORECAST',
 'GLOFAS_SEASONAL',
 'GLOFAS_SEASONAL_REFORECAST',
 'L57_REFLECTANCE',
 'L8_OLI_TIRS_C1L1',
 'L8_REFLECTANCE',
 'LANDSAT_C2L1',
 'LANDSAT_C2L2',
 'LANDSAT_C2L2ALB_BT',
 'LANDSAT_C2L2ALB_SR',
 'LANDSAT_C2L2ALB_ST',
 'LANDSAT_C2L2ALB_TA',
 'LANDSAT_C2L2_SR',
 'LANDSAT_C2L2_ST',
 'MODIS_MCD43A4',
 'NAIP',
 'OSO',
 'PLD_BUNDLE',
 'PLD_PAN',
 'PLD_PANSHARPENED',
 'PLD_XS',
 'S1_SAR_GRD',
 'S1_SAR_OCN',
 'S1_SAR_RAW',
 'S1_SAR_SLC',
 'S2_MSI_L1C',
 'S2_MSI_L2A',
 'S2_MSI_L2A_COG',
 'S2_MSI_L2A_MAJA',
 'S2_MSI_L2B_MAJA_SNOW',
 'S2_MSI_L2B_MAJA_WATER',
 'S2_MSI_L3A_WASP',
 'S3_EFR',
 'S3_ERR',
 'S3_LAN',
 'S3_OLCI_L2LFR',
 'S3_OLCI_L2LRR',
 'S3_OLCI_L2WFR',
 'S3_OLCI_L2WRR',
 'S3_RAC',
 'S3_SLSTR_L1RBT',
 'S3_SLSTR_L2AOD',
 'S3_SLSTR_L2FRP',
 'S3_SLSTR_L2LST',
 'S3_SLSTR_L2WST',
 'S3_SRA',
 'S3_SRA_A',
 'S3_SRA_BS',
 'S3_SY_AOD',
 'S3_SY_SYN',
 'S3_SY_V10',
 'S3_SY_VG1',
 'S3_SY_VGP',
 'S3_WAT',
 'S5P_L1B_IR_SIR',
 'S5P_L1B_IR_UVN',
 'S5P_L1B_RA_BD1',
 'S5P_L1B_RA_BD2',
 'S5P_L1B_RA_BD3',
 'S5P_L1B_RA_BD4',
 'S5P_L1B_RA_BD5',
 'S5P_L1B_RA_BD6',
 'S5P_L1B_RA_BD7',
 'S5P_L1B_RA_BD8',
 'S5P_L2_AER_AI',
 'S5P_L2_AER_LH',
 'S5P_L2_CH4',
 'S5P_L2_CLOUD',
 'S5P_L2_CO',
 'S5P_L2_HCHO',
 'S5P_L2_NO2',
 'S5P_L2_NP_BD3',
 'S5P_L2_NP_BD6',
 'S5P_L2_NP_BD7',
 'S5P_L2_O3',
 'S5P_L2_O3_PR',
 'S5P_L2_O3_TCL',
 'S5P_L2_SO2',
 'SATELLITE_CARBON_DIOXIDE',
 'SATELLITE_METHANE',
 'SATELLITE_SEA_LEVEL_BLACK_SEA',
 'SEASONAL_MONTHLY_PL',
 'SEASONAL_MONTHLY_SL',
 'SEASONAL_ORIGINAL_PL',
 'SEASONAL_ORIGINAL_SL',
 'SEASONAL_POSTPROCESSED_PL',
 'SEASONAL_POSTPROCESSED_SL',
 'SIS_HYDRO_MET_PROJ',
 'SPOT5_SPIRIT',
 'SPOT_SWH',
 'SPOT_SWH_OLD',
 'TIGGE_CF_SFC',
 'UERRA_EUROPE_SL',
 'VENUS_L1C',
 'VENUS_L2A_MAJA',
 'VENUS_L3A_MAJA']

The method can take a provider name as an argument and will return the product types known to eodag that are offered by this provider.

[10]:
peps_products = dag.list_product_types("peps")
[p["ID"] for p in peps_products]
[10]:
['S1_SAR_GRD', 'S1_SAR_OCN', 'S1_SAR_SLC', 'S2_MSI_L1C']

Combine these two methods#

These two methods can be combined to find which product type is the most common in eodag’s catalog among all the providers.

[11]:
availability_per_product = []
for product in products_id:
    providers = dag.available_providers(product)
    availability_per_product.append((product, len(providers)))
availability_per_product = sorted(availability_per_product, key=lambda x: x[1], reverse=True)
most_common_p_type, nb_providers = availability_per_product[0]
print(f"The most common product type is '{most_common_p_type}' with {nb_providers} providers offering it.")
The most common product type is 'S1_SAR_GRD' with 8 providers offering it.

These can be also used to find out which provider (as configured by eodag) offers the hights number of different product types.

[12]:
availability_per_provider = []
for provider in dag.available_providers():
    provider_products_id = [
        p["ID"]
        for p in dag.list_product_types(provider, fetch_providers=False)
    ]
    availability_per_provider.append(
        (provider, len(provider_products_id))
    )
availability_per_provider = sorted(availability_per_provider, key=lambda x: x[1], reverse=True)
provider, nb_p_types = availability_per_provider[0]
print(f"The provider with the largest number of product types is '{provider}' with {nb_p_types}.")
The provider with the largest number of product types is 'planetary_computer' with 123.

Product types discovery#

EODAG comes with a large list of pre-configured product types. Some others are available from providers catalogs but will not be configured, or are not yet configured in EODAG.

Some providers, like STAC providers, come in EODAG with a configuration describing how to discover these not-already-configured product types.

With the method discover_product_types() or CLI command eodag discover we can obtain a JSON configuration file that will be used as EODAG external product types configuration file.

In EODAG, the discovered EODAG external product types configuration file can be set to:

  • a file automatically built from github actions and stored in eodag/resources/ext_product_types.json (default settings)

  • a custom remote or local file by setting its path in EODAG_EXT_PRODUCT_TYPES_CFG_FILE environment variable (if the file is not readable, only user-modified providers will be fetched).

Then, when listing product types using list_product_types(fetch_providers=True), EODAG will first read the content of the EODAG external product types configuration file using fetch_product_types_list() then update EODataAccessGateway instance product types configuration, if needed.

The obtained product types list will contain both pre-configured and discovered product types.

Fetch product types schema