Add a Provider#

eodag provides a set of plugins which don’t know anything about the providers themselves, they just implement generic methods required to talk to different kinds of data catalog. For instance:

Configure a new provider#

The simplest way to add a new provider is to configure existing plugins. This approach requires to provide the new provider’s configuration in a YAML format. The following example, extracted from the STAC client page, shows how to add a new STAC provider:

tamn:
   search:
      type: StacSearch
      api_endpoint: https://tamn.snapplanet.io/search
      need_auth: false
   products:
      S2_MSI_L1C:
            productType: S2
      GENERIC_PRODUCT_TYPE:
            productType: '{productType}'
   download:
      type: AwsDownload
      base_uri: https://tamn.snapplanet.io
      flatten_top_dirs: True
   auth:
      type: AwsAuth
      credentials:
            aws_access_key_id: PLEASE_CHANGE_ME
            aws_secret_access_key: PLEASE_CHANGE_ME

It configures the following existing plugins: StacSearch (search), AwsAuth (authentication) and AwsDownload (download).

Of course, it is also necessary to know how to configure these plugins (which parameters they take, what values they can have, etc.). You can get some inspiration from the Providers pre-configuration section by analysing how eodag configures the providers it comes installed with.

Add more plugins#

eodag is a plugin-oriented framework which means it can be easily extended. If the plugins it offers are not sufficient for your own needs (i.e. getting data from a provider not supported by eodag), you should then write your own plugins (possibly by extending one of provided by eodag) and configure them. What you are the most likely to be willing to do is either to develop a new Search plugin or an Api plugin (e.g. to create an interface with another program).

eodag-sentinelsat is a good example of an Api plugin. It creates an interface with the sentinalsat library to search and download products from SciHub.

See more details about how to create a new plugin in this dedicated section.

Providers pre-configuration#

All the providers are pre-configured in eodag in a YAML file. Click on the link below to display its full content.

providers.yml
---
!provider
  name: usgs
  priority: 0
  description: U.S geological survey catalog for Landsat products
  roles:
    - host
  url: https://earthexplorer.usgs.gov/
  api: !plugin
    type: UsgsApi
    need_auth: true
    google_base_url: 'http://storage.googleapis.com/earthengine-public/landsat/'
    pagination:
      max_items_per_page: 5000
      total_items_nb_key_path: '$.totalHits'
    metadata_mapping:
      id: '$.displayId'
      geometry: '$.spatialBounds'
      productType: '$.productType'
      title: '$.displayId'
      abstract: '$.summary'
      cloudCover: '$.cloudCover'
      startTimeFromAscendingNode: '$.temporalCoverage.startDate'
      completionTimeFromAscendingNode: '$.temporalCoverage.endDate'
      publicationDate: '$.publishDate'
      thumbnail: '$.browse[0].thumbnailPath'
      quicklook: '$.browse[0].browsePath'
      storageStatus: '{$.available#get_group_name((?P<ONLINE>True)|(?P<OFFLINE>False))}'
      downloadLink: 'https://earthexplorer.usgs.gov/download/external/options/{productType}/{entityId}/M2M/'
      # metadata needed for download
      entityId: '$.entityId'
      productId: '$.id'
    extract: True
    order_enabled: true
  products:
    # datasets list http://kapadia.github.io/usgs/_sources/reference/catalog/ee.txt may be outdated
    # see also https://dds.cr.usgs.gov/ee-data/coveragemaps/shp/ee/
    LANDSAT_C2L1:
      dataset: landsat_ot_c2_l1
      outputs_extension: .tar.gz
    LANDSAT_C2L2:
      dataset: landsat_ot_c2_l2
      outputs_extension: .tar.gz
    LANDSAT_TM_C1:
      dataset: landsat_tm_c1
      outputs_extension: .tar.gz
    LANDSAT_TM_C2L1:
      dataset: landsat_tm_c2_l1
      outputs_extension: .tar.gz
    LANDSAT_TM_C2L2:
      dataset: landsat_tm_c2_l2
      outputs_extension: .tar.gz
    LANDSAT_ETM_C1:
      dataset: landsat_etm_c1
      outputs_extension: .tar.gz
    LANDSAT_ETM_C2L1:
      dataset: landsat_etm_c2_l1
      outputs_extension: .tar.gz
    LANDSAT_ETM_C2L2:
      dataset: landsat_etm_c2_l2
      outputs_extension: .tar.gz
    S2_MSI_L1C:
      dataset: SENTINEL_2A
      outputs_extension: .zip
    GENERIC_PRODUCT_TYPE:
      dataset: '{productType}'

---
!provider
  name: aws_eos
  priority: 0
  description: EOS search for Amazon public datasets
  roles:
    - host
  url: https://developers.eos.com/datasets_description.html
  search: !plugin
    type: PostJsonSearch
    api_endpoint: 'https://gate.eos.com/api/lms/search/v2/{collection}?api_key={apikey}'
    need_auth: true
    auth_error_code: 403
    results_entry: 'results'
    pagination:
      next_page_query_obj: '{{"limit":{items_per_page},"page":{page}}}'
      total_items_nb_key_path: '$.meta.found'
      # 2021/04/28: aws_eos doesn't specify a limit in its docs. It says that the default
      # value is 500 (https://doc.eos.com/search.api/#single-dataset-search).
      # Let's set it to this value for now
      max_items_per_page: 500
    query_params_key: 'search'
    discover_metadata:
      auto_discovery: true
      metadata_pattern: '^[a-zA-Z0-9_]+$'
      search_param: '{{{{"search":{{{{"{metadata}":"{{{metadata}}}" }}}} }}}}'
      metadata_path: '$.*'
    metadata_mapping:
      # landsat8_downloadLink : 's3://landsat-pds/c{storedInCollection}/L8/{path}/{row}/{productID}'
      geometry:
        - '{{"search":{{"shape": {geometry#to_geojson} }} }}'
        - '$.dataGeometry'
      # storageStatus set to ONLINE for consistency between providers
      storageStatus: '{$.null#replace_str("Not Available","ONLINE")}'
  products:
    CBERS4_PAN10M_L2:
      instrument: PAN10M
      collection: cbers4
      processingLevel: 2
      metadata_mapping:
        # OpenSearch Parameters for Collection Search (Table 3)
        productType: '$.null'
        platformSerialIdentifier:
          - '{{"search":{{"satelliteName":"{platformSerialIdentifier}" }} }}'
          - '$.satelliteName'
        instrument:
          - '{{"search":{{"sensor":"{instrument}" }} }}'
          - '$.sensor'
        processingLevel:
          - '{{"search":{{"processingLevel":"{processingLevel}" }} }}'
          - '$.processingLevel'
        # INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
        title:
          - '{{"search":{{"sceneID":"{title}" }} }}'
          - '$.sceneID'
        # OpenSearch Parameters for Acquistion Parameters Search (Table 6)
        startTimeFromAscendingNode:
          - '{{"search":{{"date":{{"from":"{startTimeFromAscendingNode}"}} }} }}'
          - '$.date'
        completionTimeFromAscendingNode:
          - '{{"search":{{"date":{{"to":"{completionTimeFromAscendingNode}"}} }} }}'
          - '$.date'
        illuminationAzimuthAngle:
          - '{{"search":{{"sunAzimuth":"{illuminationAzimuthAngle}" }} }}'
          - '$.sunAzimuth'
        illuminationElevationAngle:
          - '{{"search":{{"sunElevation":"{illuminationElevationAngle}" }} }}'
          - '$.sunElevation'
        # Custom parameters (not defined in the base document referenced above)
        awsPath:
          - '{{"search":{{"downloadUrl":"{awsPath}" }} }}'
          - '$.downloadUrl'
        downloadLink: 's3://cbers-pds/{awsPath}'
        metaDownloadLink: 's3://cbers-meta-pds/{awsPath}'
        previewBaseName: '{$.sceneID#replace_str("_L2","")}'
        thumbnail: 'https://s3.amazonaws.com/cbers-meta-pds/{awsPath}/{previewBaseName}_small.jpeg'
        quicklook: 'https://s3.amazonaws.com/cbers-meta-pds/{awsPath}/{previewBaseName}.jpg'
        id:
          - '{{"search":{{"sceneID":"{id}" }} }}'
          - '{title}'
    CBERS4_PAN10M_L4:
      instrument: PAN10M
      collection: cbers4
      processingLevel: 4
      metadata_mapping_from_product: CBERS4_PAN10M_L2
      metadata_mapping:
        # Custom parameters (not defined in the base document referenced above)
        previewBaseName: '{$.sceneID#replace_str("_L4","")}'
        thumbnail: 'https://s3.amazonaws.com/cbers-meta-pds/{awsPath}/{previewBaseName}_small.jpeg'
        quicklook: 'https://s3.amazonaws.com/cbers-meta-pds/{awsPath}/{previewBaseName}.jpg'
    CBERS4_PAN5M_L2:
      instrument: PAN5M
      collection: cbers4
      processingLevel: 2
      metadata_mapping_from_product: CBERS4_PAN10M_L2
    CBERS4_PAN5M_L4:
      instrument: PAN5M
      collection: cbers4
      processingLevel: 4
      metadata_mapping_from_product: CBERS4_PAN10M_L2
      metadata_mapping:
        # Custom parameters (not defined in the base document referenced above)
        previewBaseName: '{$.sceneID#replace_str("_L4","")}'
        thumbnail: 'https://s3.amazonaws.com/cbers-meta-pds/{awsPath}/{previewBaseName}_small.jpeg'
        quicklook: 'https://s3.amazonaws.com/cbers-meta-pds/{awsPath}/{previewBaseName}.jpg'
    CBERS4_MUX_L2:
      instrument: MUX
      collection: cbers4
      processingLevel: 2
      metadata_mapping_from_product: CBERS4_PAN10M_L2
    CBERS4_MUX_L4:
      instrument: MUX
      collection: cbers4
      processingLevel: 4
      metadata_mapping_from_product: CBERS4_PAN10M_L2
      metadata_mapping:
        # Custom parameters (not defined in the base document referenced above)
        previewBaseName: '{$.sceneID#replace_str("_L4","")}'
        thumbnail: 'https://s3.amazonaws.com/cbers-meta-pds/{awsPath}/{previewBaseName}_small.jpeg'
        quicklook: 'https://s3.amazonaws.com/cbers-meta-pds/{awsPath}/{previewBaseName}.jpg'
    CBERS4_AWFI_L2:
      instrument: AWFI
      collection: cbers4
      processingLevel: 2
      metadata_mapping_from_product: CBERS4_PAN10M_L2
    CBERS4_AWFI_L4:
      instrument: AWFI
      collection: cbers4
      processingLevel: 4
      metadata_mapping_from_product: CBERS4_PAN10M_L2
      metadata_mapping:
        # Custom parameters (not defined in the base document referenced above)
        previewBaseName: '{$.sceneID#replace_str("_L4","")}'
        thumbnail: 'https://s3.amazonaws.com/cbers-meta-pds/{awsPath}/{previewBaseName}_small.jpeg'
        quicklook: 'https://s3.amazonaws.com/cbers-meta-pds/{awsPath}/{previewBaseName}.jpg'
    L8_OLI_TIRS_C1L1:
      collection: landsat8
      onAmazon: true
      metadata_mapping:
        # OpenSearch Parameters for Collection Search (Table 3)
        productType: '$.null'
        platform:
          - '{{"search":{{"satelliteName":"{platform}" }} }}'
          - '$.satelliteName'
        instrument:
          - '{{"search":{{"sensor":"{instrument}" }} }}'
          - '$.sensor'
        # INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
        title:
          - '{{"search":{{"productID":"{title}" }} }}'
          - '$.productID'
        # OpenSearch Parameters for Product Search (Table 5)
        cloudCover:
          - '{{"search":{{"cloudCoverage":"{cloudCover}" }} }}'
          - '$.cloudCoverage'
        # OpenSearch Parameters for Acquistion Parameters Search (Table 6)
        availabilityTime:
          - '{{"search":{{"IndexingTime":{{"from":"{availabilityTime}"}} }} }}'
          - '$.IndexingTime'
        acquisitionStation:
          - '{{"search":{{"receivingStation":{{"from":"{acquisitionStation}"}} }} }}'
          - '$.receivingStation'
        startTimeFromAscendingNode:
          - '{{"search":{{"date":{{"from":"{startTimeFromAscendingNode}"}} }} }}'
          - '$.sceneStartTime'
        completionTimeFromAscendingNode:
          - '{{"search":{{"date":{{"to":"{completionTimeFromAscendingNode}"}} }} }}'
          - '$.sceneStopTime'
        illuminationAzimuthAngle:
          - '{{"search":{{"sunAzimuth":"{illuminationAzimuthAngle}" }} }}'
          - '$.sunAzimuth'
        illuminationElevationAngle:
          - '{{"search":{{"sunElevation":"{illuminationElevationAngle}" }} }}'
          - '$.sunElevation'
        # Custom parameters (not defined in the base document referenced above)
        onAmazon:
          - '{{"search":{{"onAmazon":"{onAmazon}" }} }}'
          - '$.onAmazon'
        path:
          - '{{"search":{{"path":"{path}" }} }}'
          - '$.path'
        row:
          - '{{"search":{{"row":"{row}" }} }}'
          - '$.row'
        downloadLink: 's3://landsat-pds/c1/L8/{path:03.0f}/{row:03.0f}/{title}/'
        thumbnail: '$.thumbnail'
        quicklook: 'https://landsat-pds.s3.amazonaws.com/c1/L8/{path:03.0f}/{row:03.0f}/{title}/{title}_thumb_large.jpg'
        id:
          - '{{"search":{{"productID":"{id}" }} }}'
          - '{title}'
    MODIS_MCD43A4:
      collection: modis
      metadata_mapping:
        # OpenSearch Parameters for Collection Search (Table 3)
        productType: '$.null'
        instrument:
          - '{{"search":{{"satelliteName":"{instrument}" }} }}'
          - '$.satelliteName'
        # INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
        title:
          - '{{"search":{{"sceneID":"{title}" }} }}'
          - '$.sceneID'
        # OpenSearch Parameters for Product Search (Table 5)
        cloudCover:
          - '{{"search":{{"cloudCoverage":"{cloudCover}" }} }}'
          - '$.cloudCoverage'
        # OpenSearch Parameters for Acquistion Parameters Search (Table 6)
        startTimeFromAscendingNode:
          - '{{"search":{{"date":{{"from":"{startTimeFromAscendingNode}"}} }} }}'
          - '$.BeginningDateTime'
        completionTimeFromAscendingNode:
          - '{{"search":{{"date":{{"to":"{completionTimeFromAscendingNode}"}} }} }}'
          - '$.EndingDateTime'
        # Custom parameters (not defined in the base document referenced above)
        verticalTileNumber:
          - '{{"search":{{"verticalTileNumber":"{verticalTileNumber}" }} }}'
          - '$.verticalTileNumber'
        horizontalTileNumber:
          - '{{"search":{{"horizontalTileNumber":"{horizontalTileNumber}" }} }}'
          - '$.horizontalTileNumber'
        doyDate: '{$.sceneID#slice_str(9,16,1)}'
        downloadLink: 's3://modis-pds/MCD43A4.006/{horizontalTileNumber:02.0f}/{verticalTileNumber:02.0f}/{doyDate}/'
        thumbnail: '$.thumbnail'
        quicklook: '$.thumbnail'
        id:
          - '{{"search":{{"sceneID":"{id}" }} }}'
          - '{title}'
    NAIP:
      collection: naip
      metadata_mapping:
        # OpenSearch Parameters for Collection Search (Table 3)
        productType: '$.null'
        platformSerialIdentifier:
          - '{{"search":{{"satelliteName":"{platformSerialIdentifier}" }} }}'
          - '$.satelliteName'
        # INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
        title:
          - '{{"search":{{"sceneID":"{title}" }} }}'
          - '$.sceneID'
        # OpenSearch Parameters for Acquistion Parameters Search (Table 6)
        startTimeFromAscendingNode:
          - '{{"search":{{"date":{{"from":"{startTimeFromAscendingNode}"}} }} }}'
          - '$.date'
        completionTimeFromAscendingNode:
          - '{{"search":{{"date":{{"to":"{completionTimeFromAscendingNode}"}} }} }}'
          - '$.date'
        # Custom parameters (not defined in the base document referenced above)
        awsPath:
          - '{{"search":{{"awsPath":"{awsPath}" }} }}'
          - '$.awsPath'
        downloadLink: 's3://naip-analytic/{awsPath}'
        id:
          - '{{"search":{{"sceneID":"{id}" }} }}'
          - '{title}'
    S1_SAR_GRD:
      productType: GRD
      collection: sentinel1
      metadata_mapping:
        cloudCover: '$.null'
        # OpenSearch Parameters for Collection Search (Table 3)
        platformSerialIdentifier:
          - '{{"search":{{"missionId":"{platformSerialIdentifier}" }} }}'
          - '$.missionId'
        # INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
        title:
          - '{{"search":{{"sceneID":"{title}" }} }}'
          - '$.sceneID'
        # OpenSearch Parameters for Product Search (Table 5)
        orbitNumber:
          - '{{"search":{{"absoluteOrbitNumber":"{orbitNumber}" }} }}'
          - '$.absoluteOrbitNumber'
        orbitDirection:
          - '{{"search":{{"passDirection":"{orbitDirection}" }} }}'
          - '$.passDirection'
        sensorMode:
          - '{{"search":{{"mode":"{sensorMode}" }} }}'
          - '$.mode'
        # OpenSearch Parameters for Acquistion Parameters Search (Table 6)
        startTimeFromAscendingNode:
          - '{{"search":{{"date":{{"from":"{startTimeFromAscendingNode}"}} }} }}'
          - '$.date'
        completionTimeFromAscendingNode:
          - '{{"search":{{"date":{{"to":"{completionTimeFromAscendingNode}"}} }} }}'
          - '$.date'
        polarizationMode:
          - '{{"search":{{"polarization":"{polarizationMode}" }} }}'
          - '$.polarization'
        # Custom parameters (not defined in the base document referenced above)
        awsPath:
          - '{{"search":{{"awsPath":"{awsPath}" }} }}'
          - '$.awsPath'
        downloadLink: 's3://sentinel-s1-l1c/{awsPath}'
        thumbnail: 'https://render.eosda.com/S1/thumb/{title}.png'
        quicklook: 'https://render.eosda.com/S1/thumb/{title}.png'
        id:
          - '{{"search":{{"sceneID":"{id}" }} }}'
          - '{title}'
    S2_MSI_L1C:
      collection: sentinel2
      metadata_mapping:
        # OpenSearch Parameters for Collection Search (Table 3)
        productType: '$.null'
        platformSerialIdentifier:
          - '{{"search":{{"satelliteName":"{platformSerialIdentifier}" }} }}'
          - '$.satelliteName'
        # INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
        title:
          - '{{"search":{{"productName":"{title}" }} }}'
          - '$.productName'
        # OpenSearch Parameters for Product Search (Table 5)
        cloudCover:
          - '{{"search":{{"cloudCoverage":"{cloudCover}" }} }}'
          - '$.cloudCoverage'
        # OpenSearch Parameters for Acquistion Parameters Search (Table 6)
        startTimeFromAscendingNode:
          - '{{"search":{{"date":{{"from":"{startTimeFromAscendingNode}"}} }} }}'
          - '$.timestamp'
        completionTimeFromAscendingNode:
          - '{{"search":{{"date":{{"to":"{completionTimeFromAscendingNode}"}} }} }}'
          - '$.timestamp'
        illuminationAzimuthAngle:
          - '{{"search":{{"azimuthAngle":"{illuminationAzimuthAngle}" }} }}'
          - '$.azimuthAngle'
        illuminationZenithAngle:
          - '{{"search":{{"zenithAngle":"{illuminationZenithAngle}" }} }}'
          - '$.zenithAngle'
        # Custom parameters (not defined in the base document referenced above)
        originalTileMeta: '{$.originalTileMeta#replace_str("sentinel-s2-l1c.s3.amazonaws.com","roda.sentinel-hub.com/sentinel-s2-l1c")}'
        thumbnail: '{$.thumbnail#replace_str("sentinel-s2-l1c.s3.amazonaws.com","roda.sentinel-hub.com/sentinel-s2-l1c")}'
        awsPath:
          - '{{"search":{{"awsPath":"{awsPath}" }} }}'
          - '$.awsPath'
        downloadLink: 's3://sentinel-s2-l1c/{awsPath}'
        productMetaLink: '{$.productMetaLink#replace_str("sentinel-s2-l1c.s3.amazonaws.com","roda.sentinel-hub.com/sentinel-s2-l1c")}'
        productPath:
          - '{{"search":{{"productPath":"{productPath}" }} }}'
          - '$.productPath'
        id:
          - '{{"search":{{"productName":"{id}" }} }}'
          - '{title}'
        processedL2A: '$.null'
        awsPathL2A: '$.null'
    S2_MSI_L2A:
      collection: sentinel2
      processedL2A: true
      # specific QueryStringSearch usage for these parameters (replaces current query)
      specific_qssearch:
        parameters:
          - title
          - id
        results_entry: ''
        collection:
          - tileInfo
          - productInfo
        merge_responses: true
        metadata_mapping:
          title:
            - 'title'
            - '$.name'
          id:
            - 'title'
            - '{title}'
          awsPathL2A: '$.tiles[0].path'
          downloadLink: 's3://sentinel-s2-l2a/{awsPathL2A}'
          productPath: '$.path'
          startTimeFromAscendingNode: '$.timestamp'
          completionTimeFromAscendingNode: '$.timestamp'
          geometry: '$.tileDataGeometry'
          productInfo: 'https://roda.sentinel-hub.com/sentinel-s2-l2a/{awsPathL2A}/productInfo.json'
          originalSceneID: '$.tiles[0].datastrip.id'
      metadata_mapping:
        # OpenSearch Parameters for Collection Search (Table 3)
        productType: '$.null'
        platformSerialIdentifier:
          - '{{"search":{{"satelliteName":"{platformSerialIdentifier}" }} }}'
          - '$.satelliteName'
        # INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
        title: '{$.productName#fake_l2a_title_from_l1c}'
        # OpenSearch Parameters for Product Search (Table 5)
        cloudCover:
          - '{{"search":{{"cloudCoverage":"{cloudCover}" }} }}'
          - '$.cloudCoverage'
        # OpenSearch Parameters for Acquistion Parameters Search (Table 6)
        startTimeFromAscendingNode:
          - '{{"search":{{"date":{{"from":"{startTimeFromAscendingNode}"}} }} }}'
          - '$.timestamp'
        completionTimeFromAscendingNode:
          - '{{"search":{{"date":{{"to":"{completionTimeFromAscendingNode}"}} }} }}'
          - '$.timestamp'
        illuminationAzimuthAngle:
          - '{{"search":{{"azimuthAngle":"{illuminationAzimuthAngle}" }} }}'
          - '$.azimuthAngle'
        illuminationZenithAngle:
          - '{{"search":{{"zenithAngle":"{illuminationZenithAngle}" }} }}'
          - '$.zenithAngle'
        # Custom parameters (not defined in the base document referenced above)
        originalTileMeta: '{$.originalTileMeta#replace_str("sentinel-s2-l1c.s3.amazonaws.com","roda.sentinel-hub.com/sentinel-s2-l2a")}'
        thumbnail: '{$.thumbnail#replace_str("sentinel-s2-l1c.s3.amazonaws.com","roda.sentinel-hub.com/sentinel-s2-l1c")}'
        quicklook: '{thumbnail}'
        downloadLink: 's3://sentinel-s2-l2a/{awsPathL2A}'
        awsPath: '$.null'
        productMetaLink: '$.null'
        productPath: '$.null'
        productInfo: 'https://roda.sentinel-hub.com/sentinel-s2-l2a/{awsPathL2A}/productInfo.json'
        id:
          - '{id#s2msil2a_title_to_aws_productinfo}'
          - '{title}'
        processedL2A:
          - '{{"search":{{"processedL2A":"{processedL2A}" }} }}'
          - '$.processedL2A'
        awsPathL2A:
          - '{{"search":{{"awsPathL2A":"{awsPathL2A}" }} }}'
          - '$.awsPathL2A'

  download: !plugin
    type: AwsDownload
    requester_pays: True
    products:
      CBERS4_MUX_L2:
        default_bucket: 'cbers-pds'
        complementary_url_key:
          - metaDownloadLink
      CBERS4_AWFI_L2:
        complementary_url_key:
          - metaDownloadLink
      CBERS4_PAN5M_L2:
        complementary_url_key:
          - metaDownloadLink
      CBERS4_PAN10M_L2:
        complementary_url_key:
          - metaDownloadLink
      S1_SAR_GRD:
        default_bucket: 'sentinel-s1-l1c'
        build_safe: true
      S2_MSI_L1C:
        default_bucket: 'sentinel-s2-l1c'
        build_safe: true
        complementary_url_key:
          - productPath
      S2_MSI_L2A:
        default_bucket: 'sentinel-s2-l2a'
        build_safe: true
        fetch_metadata:
          fetch_url: '{productInfo}'
          fetch_format: json
          update_metadata:
            title: '$.name'
            id: '{title}'
            productPath: '$.path'
        complementary_url_key:
          - productPath
  auth: !plugin
    type: AwsAuth

---
!provider
  name: theia
  priority: 0
  description: French National Space Agency (CNES) catalog for Sentinel 2 products, Pleiades and Landsat products
  roles:
    - host
  url: https://theia.cnes.fr/atdistrib/rocket/
  search: !plugin
    type: QueryStringSearch
    api_endpoint: 'https://theia.cnes.fr/atdistrib/resto2/api/collections/{collection}/search.json'
    need_auth: false
    pagination:
      next_page_url_tpl: '{url}?{search}&maxRecords={items_per_page}&page={page}'
      total_items_nb_key_path: '$.properties.totalResults'
      # 2021/03/19: Returns a 400 error code if greater than 500.
      max_items_per_page: 500
    discover_metadata:
      auto_discovery: true
      metadata_pattern: '^(?!collection)[a-zA-Z0-9_]+$'
      search_param: '{metadata}={{{metadata}}}'
      metadata_path: '$.properties.*'
    metadata_mapping:
      # Opensearch resource identifier within the search engine context (in our case
      # within the context of the data provider)
      uid: '$.id'
      # OpenSearch Parameters for Collection Search (Table 3)
      productType:
        - productType
        - '$.properties.productType'
      platform:
        - platform
        - '$.properties.platform'
      platformSerialIdentifier:
        - 'platform={platformSerialIdentifier}'
        - '$.properties.platform'
      instrument:
        - instrument
        - '$.properties.instrument'
      processingLevel:
        - processingLevel
        - '$.properties.processingLevel'

      # INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
      title: '$.properties.title'
      keyword: '$.properties.keywords'
      resolution: '$.properties.resolution'
      organisationName: '$.properties.organisationName'
      publicationDate: '$.properties.published'
      accessConstraint: '$.properties.license'

      # OpenSearch Parameters for Product Search (Table 5)
      parentIdentifier: '$.properties.parentIdentifier'
      orbitNumber: '$.properties.orbitNumber'
      cloudCover:
        - 'cloudCover=[0,{cloudCover}]'
        - '$.properties.cloudCover'
      snowCover: '$.properties.snowCover'
      productVersion: '$.properties.version'
      creationDate: '$.properties.productionDate'
      modificationDate: '$.properties.modificationDate'
      processingDate: '$.properties.processingDate'
      sensorMode: '$.properties.sensorMode'
      archivingCenter: '$.properties.archivingCenter'
      processingMode: '$.properties.processingMode'

      # OpenSearch Parameters for Acquistion Parameters Search (Table 6)
      availabilityTime: '$.properties.availabilityTime'
      acquisitionStation: '$.properties.acquisitionStation'
      acquisitionSubType: '$.properties.acquisitionSubType'
      startTimeFromAscendingNode:
        - 'startDate={startTimeFromAscendingNode#to_iso_date}'
        - '$.properties.startDate'
      completionTimeFromAscendingNode:
        - 'completionDate={completionTimeFromAscendingNode#to_iso_date}'
        - '$.properties.completionDate'
      illuminationAzimuthAngle: '$.properties.illuminationAzimuthAngle'
      illuminationZenithAngle: '$.properties.illuminationZenithAngle'
      illuminationElevationAngle: '$.properties.illuminationElevationAngle'
      polarizationMode: '$.properties.polarizationMode'
      polarizationChannels: '$.properties.polarisationChannels'
      antennaLookDirection: '$.properties.antennaLookDirection'
      minimumIncidenceAngle: '$.properties.minimumIncidenceAngle'
      maximumIncidenceAngle: '$.properties.maximumIncidenceAngle'
      dopplerFrequency: '$.properties.dopplerFrequency'
      incidenceAngleVariation: '$.properties.incidenceAngleVariation'

      # Custom parameters (not defined in the base document referenced above)
      id: '$.properties.productIdentifier'
      tileIdentifier:
        - 'location=T{tileIdentifier#replace_str(r"^T(.*)$",r"\1")}'
        - '{$.properties.location#replace_str(r"^T(.*)$",r"\1")}'
      # The geographic extent of the product
      geometry:
        - 'geometry={geometry#to_rounded_wkt}'
        - '$.geometry'
      # The url of the quicklook
      quicklook: '$.properties.quicklook'
      # The url to download the product "as is" (literal or as a template to be completed either after the search result
      # is obtained from the provider or during the eodag download phase)
      downloadLink: '$.properties.services.download.url'

      # Additional metadata provided by the providers but that don't appear in the reference spec
      # Or has a different signification for the provider
      thumbnail: '$.properties.thumbnail'

      # storageStatus set to ONLINE for consistency between providers
      storageStatus: '{$.null#replace_str("Not Available","ONLINE")}'

  products:
    S2_MSI_L2A_MAJA:
      processingLevel: LEVEL2A
      collection: SENTINEL2
    S2_MSI_L2B_MAJA_SNOW:
      processingLevel: L2B-SNOW
      collection: Snow
    S2_MSI_L2B_MAJA_WATER:
      processingLevel: L2B-WATER
      collection: WaterQual
    S2_MSI_L3A_WASP:
      processingLevel: LEVEL3A
      collection: SENTINEL2
    SPOT_SWH:
      collection: SPOTWORLDHERITAGE
      processingLevel: LEVEL1C
    SPOT_SWH_OLD:
      collection: SWH1
      processingLevel: LEVEL1C
    SPOT5_SPIRIT:
      collection: Spirit
      processingLevel: 1A
    VENUS_L1C:
      collection: VENUS
      processingLevel: LEVEL1C
    VENUS_L2A_MAJA:
      collection: VENUS
      processingLevel: LEVEL2A
    VENUS_L3A_MAJA:
      collection: VENUS
      processingLevel: LEVEL3A
    L8_REFLECTANCE:
      collection: LANDSAT
      productType: REFLECTANCE
    L57_REFLECTANCE:
      collection: Landsat57
      productType: REFLECTANCE
    PLD_PAN:
      collection: Pleiades
      productType: Panchromatique+(Pan)
    PLD_XS:
      collection: Pleiades
      productType: Multispectral+(XS)
    PLD_BUNDLE:
      collection: Pleiades
      productType: Bundle+(Pan,+XS)
    PLD_PANSHARPENED:
      collection: Pleiades
      productType: Pansharpened+(Pan%2BXS)
    OSO:
      collection: OSO
    GENERIC_PRODUCT_TYPE:
      productType: '{productType}'
      collection: '{collection}'
      processingLevel: '{processingLevel}'
  download: !plugin
    type: HTTPDownload
    base_uri: 'https://theia.cnes.fr/atdistrib/resto2'
    extract: true
    order_enabled: true
    auth_error_code: 403
    dl_url_params:
      issuerId: theia
  auth: !plugin
    type: TokenAuth
    auth_uri: 'https://theia.cnes.fr/atdistrib/services/authenticate/'

---
!provider
  name: peps
  priority: 1
  description: |
    The PEPS platform, the French "mirror site", redistributes the products of Sentinel satellites,
    S1A, S1B, S2A and S2B, S3A and S3B from COPERNICUS, the European system for the Earth monitoring.
  roles:
    - host
  url: https://peps.cnes.fr
  search: !plugin
    type: QueryStringSearch
    api_endpoint: 'https://peps.cnes.fr/resto/api/collections/{collection}/search.json'
    need_auth: false
    pagination:
      next_page_url_tpl: '{url}?{search}&maxRecords={items_per_page}&page={page}'
      total_items_nb_key_path: '$.properties.totalResults'
      # 2021/03/19: 500 is the max, no error if greater
      max_items_per_page: 500
    discover_metadata:
      auto_discovery: true
      metadata_pattern: '^(?!collection)[a-zA-Z0-9_]+$'
      search_param: '{metadata}={{{metadata}}}'
      metadata_path: '$.properties.*'
    metadata_mapping:
      # Opensearch resource identifier within the search engine context (in our case
      # within the context of the data provider)
      uid: '$.id'
      # OpenSearch Parameters for Collection Search (Table 3)
      productType:
        - productType
        - '$.properties.productType'
      platform: '$.properties.collection'
      platformSerialIdentifier:
        - platform
        - '$.properties.platform'
      instrument:
        - instrument
        - '$.properties.instrument'
      processingLevel:
        - processingLevel
        - '$.properties.processingLevel'

      # INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
      title: '$.properties.title'
      topicCategory: '$.properties.topicCategory'
      keyword: '$.properties.keywords'
      abstract: '$.properties.description'
      resolution:
        - 'resolution'
        - '$.properties.resolution'
      organisationName:
        - 'organisationName'
        - '$.properties.organisationName'
      publicationDate: '$.properties.published'
      accessConstraint: '$.properties.license'

      # OpenSearch Parameters for Product Search (Table 5)
      parentIdentifier:
        - 'parentIdentifier'
        - '$.properties.parentIdentifier'
      acquisitionType: '$.properties.acquisitionType'
      orbitNumber:
        - 'orbitNumber'
        - '$.properties.orbitNumber'
      orbitDirection:
        - 'orbitDirection'
        - '$.properties.orbitDirection'
      swathIdentifier:
        - 'swath'
        - '$.properties.swath'
      cloudCover:
        - 'cloudCover=[0,{cloudCover}]'
        - '$.properties.cloudCover'
      snowCover:
        - 'snowCover=[0,{snowCover}]'
        - '$.properties.snowCover'
      productVersion: '$.properties.version'
      productQualityStatus: '$.properties.onlineQualityCheck'
      processorName: '$.properties.processingName'
      processingCenter: '$.properties.processingCenter'
      creationDate: '$.properties.dhusIngestDate'
      modificationDate: '$.properties.updated'
      sensorMode:
        - 'sensorMode'
        - '$.properties.sensorMode'

      # OpenSearch Parameters for Acquistion Parameters Search (Table 6)
      startTimeFromAscendingNode:
        - startDate
        - '$.properties.startDate'
      completionTimeFromAscendingNode:
        - completionDate
        - '$.properties.completionDate'
      polarizationMode:
        - 'polarisation'
        - '$.properties.polarisation'

      # Custom parameters (not defined in the base document referenced above)
      id:
        - 'productIdentifier={id#remove_extension}'
        - '$.properties.productIdentifier'
      tileIdentifier:
        - 'tileid'
        - '$.properties.mgrs'
      # The geographic extent of the product
      geometry:
        - 'geometry={geometry#to_rounded_wkt}'
        - '$.geometry'
      # The url of the quicklook
      quicklook: '$.properties.quicklook'
      # The url to download the product "as is" (literal or as a template to be completed either after the search result
      # is obtained from the provider or during the eodag download phase)
      downloadLink: '$.properties.services.download.url'
      # storageStatus: must be one of ONLINE, STAGING, OFFLINE
      storageStatus: '{$.properties.storage.mode#get_group_name((?P<ONLINE>disk|tier2)|(?P<STAGING>staging)|(?P<OFFLINE>unknown|tape|tier3))}'

      # Additional metadata provided by the providers but that don't appear in the reference spec
      thumbnail: '$.properties.thumbnail'

  products:
    S1_SAR_OCN:
      productType: OCN
      collection: S1
      metadata_mapping:
        cloudCover: '$.null'
    S1_SAR_GRD:
      productType: GRD
      collection: S1
      metadata_mapping:
        cloudCover: '$.null'
    S1_SAR_SLC:
      productType: SLC
      collection: S1
      metadata_mapping:
        cloudCover: '$.null'
    S2_MSI_L1C:
      collection: S2ST
      productType: S2MSI1C
    S2_MSI_L2A:
      collection: S2ST
      productType: S2MSI2A
    GENERIC_PRODUCT_TYPE:
      productType: '{productType}'
      collection: '{collection}'
  download: !plugin
    type: HTTPDownload
    base_uri: 'https://peps.cnes.fr/resto'
    extract: true
    archive_depth: 2
    order_enabled: true
    auth_error_code: 401
    dl_url_params:
      issuerId: peps
  auth: !plugin
    type: GenericAuth
    auth_uri: 'https://peps.cnes.fr/resto/api/users/connect'
---
!provider
  name: creodias
  priority: 0
  description: CloudFerro DIAS
  roles:
    - host
  url: https://creodias.eu/
  search: !plugin
    type: ODataV4Search
    api_endpoint: 'https://datahub.creodias.eu/odata/v1/Products'
    need_auth: false
    dont_quote:
      - '['
      - ']'
      - '$'
      - '='
      - '&'
      - ':'
    pagination:
      next_page_url_tpl: '{url}?{search}&$top={items_per_page}&$skip={skip}&$expand=Attributes&$count=True'
      total_items_nb_key_path: '$."@odata.count"'
      max_items_per_page: 1_000
    results_entry: 'value'
    free_text_search_operations:
      $filter:
        union: ' or '
        wrapper: '{}'
        operations:
          and:
            - "Collection/Name eq '{collection}'"
            - "OData.CSC.Intersects(area=geography'{geometry#to_ewkt}')"
            - "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'productType' and att/OData.CSC.StringAttribute/Value eq '{productType}')"
            - "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'platformShortName' and att/OData.CSC.StringAttribute/Value eq '{platform}')"
            - "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'platformSerialIdentifier' and att/OData.CSC.StringAttribute/Value eq '{platformSerialIdentifier}')"
            - "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'spatialResolution' and att/OData.CSC.StringAttribute/Value eq '{resolution}')"
            - "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'authority' and att/OData.CSC.StringAttribute/Value eq '{organisationName}')"
            - "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'orbitNumber' and att/OData.CSC.StringAttribute/Value eq '{orbitNumber}')"
            - "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'orbitDirection' and att/OData.CSC.StringAttribute/Value eq '{orbitDirection}')"
            - "Attributes/OData.CSC.DoubleAttribute/any(att:att/Name eq 'cloudCover' and att/OData.CSC.DoubleAttribute/Value le {cloudCover})"
            - "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'operationalMode' and att/OData.CSC.StringAttribute/Value eq '{sensorMode}')"
            - "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'polarisationChannels' and att/OData.CSC.StringAttribute/Value eq '{polarizationChannels}')"
            - "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'tileId' and att/OData.CSC.StringAttribute/Value eq '{tileIdentifier}')"
            - 'ContentDate/Start gt {startTimeFromAscendingNode#to_iso_utc_datetime}'
            - 'ContentDate/End lt {completionTimeFromAscendingNode#to_iso_utc_datetime}'
            - contains(Name,'{id}')
    discover_metadata:
      auto_discovery: true
      metadata_pattern: '^(?!collection)[a-zA-Z0-9]+$'
      search_param:
        free_text_search_operations:
          $filter:
            operations:
              and:
                -  "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq '{metadata}' and att/OData.CSC.StringAttribute/Value eq '{{{metadata}}}')"
      metadata_path: '$.Attributes.*'
    discover_product_types:
      fetch_url: 'https://datahub.creodias.eu/stac/collections'
      result_type: json
      results_entry: '$.collections[*]'
      generic_product_type_id: '$.id'
      generic_product_type_parsable_properties:
        collection: '$.id'
      generic_product_type_parsable_metadata:
        abstract: '$.description'
        instrument: '{$.summaries.instruments#csv_list}'
        platform: '{$.summaries.constellation#csv_list}'
        platformSerialIdentifier: '{$.summaries.platform#csv_list}'
        processingLevel: '$.summaries."processing:level"'
        keywords: '{$.keywords#csv_list}'
        license: '$.license'
        title: '$.title'
        missionStartDate: '$.extent.temporal.interval[0][0]'
    per_product_metadata_query: false
    metadata_pre_mapping:
      metadata_path: '$.Attributes'
      metadata_path_id: 'Name'
      metadata_path_value: 'Value'
    metadata_mapping:
      # Opensearch resource identifier within the search engine context (in our case
      # within the context of the data provider)
      # Queryable parameters are set with null as 1st configuration list value to mark them as queryable,
      #   but `free_text_search_operations.$filter.operations.and` entries are then used instead.
      uid: '$.Id'
      # OpenSearch Parameters for Collection Search (Table 3)
      productType:
        - null
        - '$.Attributes.productType'
      platform:
        - null
        - '$.Attributes.platformShortName'
      platformSerialIdentifier:
        - null
        - '$.Attributes.platformSerialIdentifier'
      instrument:
        - null
        - '$.Attributes.instrumentShortName'
      processingLevel:
        - null
        - '$.Attributes.processingLevel'
      # INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
      title: '$.Name'
      resolution:
        - null
        - '$.Attributes.spatialResolution'
      organisationName:
        - null
        - '$.Attributes.authority'
      publicationDate: '$.PublicationDate'
      # OpenSearch Parameters for Product Search (Table 5)
      orbitNumber:
        - null
        - '$.Attributes.orbitNumber'
      orbitDirection:
        - null
        - '$.Attributes.orbitDirection'
      cloudCover:
        - null
        - '$.Attributes.cloudCover'
      modificationDate: '$.ModificationDate'
      sensorMode:
        - null
        - '$.Attributes.operationalMode'
      # OpenSearch Parameters for Acquistion Parameters Search (Table 6)
      startTimeFromAscendingNode:
        - null
        - '$.ContentDate.Start'
      completionTimeFromAscendingNode:
        - null
        - '$.ContentDate.End'
      polarizationChannels:
        - null
        - '$.Attributes.polarisationChannels'
      # Custom parameters (not defined in the base document referenced above)
      id:
        - null
        - '{$.Name#remove_extension}'
      tileIdentifier:
        - null
        - '$.Attributes.tileId'
      # The geographic extent of the product
      geometry:
        - null
        - '{$.Footprint.`sub(/.*(SRID=[0-9]+;.*)/, \\1)`#from_ewkt}'
      # The url to download the product "as is" (literal or as a template to be completed either after the search result
      # is obtained from the provider or during the eodag download phase)
      downloadLink: 'https://datahub.creodias.eu/odata/v1/Products({uid})/$value'
      # storageStatus: must be one of ONLINE, STAGING, OFFLINE
      storageStatus: '{$.Online#get_group_name((?P<ONLINE>True)|(?P<OFFLINE>False))}'
      collection:
        - null
        - $.null
      # Additional metadata provided by the providers but that don't appear in the reference spec
      productIdentifier: '$.S3Path'
  download: !plugin
    type: HTTPDownload
    base_uri: 'https://zipper.creodias.eu/download/'
    extract: true
    order_enabled: false
    archive_depth: 2
  auth: !plugin
    type: KeycloakOIDCPasswordAuth
    auth_base_uri: 'https://identity.cloudferro.com/auth'
    realm: 'dias'
    client_id: 'CLOUDFERRO_PUBLIC'
    client_secret: 'dc0aca03-2dc6-4798-a5de-fc5aeb6c8ee1'
    token_provision: qs
    token_qs_key: 'token'
    auth_error_code: 401
  products:
    # S2
    S2_MSI_L1C:
      collection: SENTINEL-2
      productType: S2MSI1C
    S2_MSI_L2A:
      collection: SENTINEL-2
      productType: S2MSI2A
    # S1
    S1_SAR_RAW:
      productType: RAW
      collection: SENTINEL-1
      metadata_mapping:
        cloudCover: '$.null'
    S1_SAR_GRD:
      productType: GRD
      collection: SENTINEL-1
      metadata_mapping:
        cloudCover: '$.null'
    S1_SAR_SLC:
      productType: SLC
      collection: SENTINEL-1
      metadata_mapping:
        cloudCover: '$.null'
    S1_SAR_OCN:
      productType: OCN
      collection: SENTINEL-1
      metadata_mapping:
        cloudCover: '$.null'
    # S3 SRAL
    S3_SRA:
      productType: SR_1_SRA___
      collection: SENTINEL-3
      metadata_mapping:
        cloudCover: '$.null'
    S3_SRA_A:
      productType: SR_1_SRA_A_
      collection: SENTINEL-3
      metadata_mapping:
        cloudCover: '$.null'
    S3_SRA_BS:
      productType: SR_1_SRA_BS
      collection: SENTINEL-3
      metadata_mapping:
        cloudCover: '$.null'
    S3_LAN:
      productType: SR_2_LAN___
      collection: SENTINEL-3
      metadata_mapping:
        cloudCover: '$.null'
    S3_WAT:
      productType: SR_2_WAT___
      collection: SENTINEL-3
      metadata_mapping:
        cloudCover: '$.null'
    # S3 OLCI
    S3_EFR:
      productType: OL_1_EFR___
      collection: SENTINEL-3
    S3_ERR:
      productType: OL_1_ERR___
      collection: SENTINEL-3
    S3_OLCI_L2LRR:
      productType: OL_2_LRR___
      collection: SENTINEL-3
    S3_OLCI_L2LFR:
      productType: OL_2_LFR___
      collection: SENTINEL-3
    S3_OLCI_L2WRR:
      productType: OL_2_WRR___
      collection: SENTINEL-3
    S3_OLCI_L2WFR:
      productType: OL_2_WFR___
      collection: SENTINEL-3
    # S3 SLSTR
    S3_SLSTR_L1RBT:
      productType: SL_1_RBT___
      collection: SENTINEL-3
    S3_SLSTR_L2LST:
      productType: SL_2_LST___
      collection: SENTINEL-3
    S3_SLSTR_L2WST:
      productType: SL_2_WST___
      collection: SENTINEL-3
    S3_SLSTR_L2AOD:
      productType: SL_2_AOD___
      collection: SENTINEL-3
    S3_SLSTR_L2FRP:
      productType: SL_2_FRP___
      collection: SENTINEL-3
    # S3 SY
    S3_SY_AOD:
      productType: SY_2_AOD___
      collection: SENTINEL-3
    S3_SY_SYN:
      productType: SY_2_SYN___
      collection: SENTINEL-3
    S3_SY_V10:
      productType: SY_2_V10___
      collection: SENTINEL-3
    S3_SY_VG1:
      productType: SY_2_VG1___
      collection: SENTINEL-3
    S3_SY_VGP:
      productType: SY_2_VGP___
      collection: SENTINEL-3
    # S5P L1
    S5P_L1B_IR_SIR:
      productType: L1B_IR_SIR
      collection: SENTINEL-5P
    S5P_L1B_IR_UVN:
      productType: L1B_IR_UVN
      collection: SENTINEL-5P
    S5P_L1B_RA_BD1:
      productType: L1B_RA_BD1
      collection: SENTINEL-5P
    S5P_L1B_RA_BD2:
      productType: L1B_RA_BD2
      collection: SENTINEL-5P
    S5P_L1B_RA_BD3:
      productType: L1B_RA_BD3
      collection: SENTINEL-5P
    S5P_L1B_RA_BD4:
      productType: L1B_RA_BD4
      collection: SENTINEL-5P
    S5P_L1B_RA_BD5:
      productType: L1B_RA_BD5
      collection: SENTINEL-5P
    S5P_L1B_RA_BD6:
      productType: L1B_RA_BD6
      collection: SENTINEL-5P
    S5P_L1B_RA_BD7:
      productType: L1B_RA_BD7
      collection: SENTINEL-5P
    S5P_L1B_RA_BD8:
      productType: L1B_RA_BD8
      collection: SENTINEL-5P
    # S5P L2
    S5P_L2_NO2:
      productType: L2__NO2___
      collection: SENTINEL-5P
    S5P_L2_CLOUD:
      productType: L2__CLOUD_
      collection: SENTINEL-5P
    S5P_L2_O3:
      productType: L2__O3____
      collection: SENTINEL-5P
    S5P_L2_CO:
      productType: L2__CO____
      collection: SENTINEL-5P
    S5P_L2_AER_AI:
      productType: L2__AER_AI
      collection: SENTINEL-5P
    S5P_L2_O3_PR:
      productType: L2__O3__PR
      collection: SENTINEL-5P
    S5P_L2_O3_TCL:
      productType: L2__O3_TCL
      collection: SENTINEL-5P
    S5P_L2_AER_LH:
      productType: L2__AER_LH
      collection: SENTINEL-5P
    S5P_L2_HCHO:
      productType: L2__HCHO__
      collection: SENTINEL-5P
    S5P_L2_CH4:
      productType: L2__CH4___
      collection: SENTINEL-5P
    S5P_L2_NP_BD3:
      productType: L2__NP_BD3
      collection: SENTINEL-5P
    S5P_L2_NP_BD6:
      productType: L2__NP_BD6
      collection: SENTINEL-5P
    S5P_L2_NP_BD7:
      productType: L2__NP_BD7
      collection: SENTINEL-5P
    S5P_L2_SO2:
      productType: L2__SO2___
      collection: SENTINEL-5P
    GENERIC_PRODUCT_TYPE:
      productType: '{productType}'
      collection: '{collection}'
---
!provider
  name: mundi
  priority: 0
  description: Atos DIAS
  roles:
    - host
  url: https://mundiwebservices.com/
  search: !plugin
    type: QueryStringSearch
    api_endpoint: 'https://{collection}.browse.catalog.mundiwebservices.com/opensearch'
    need_auth: false
    result_type: 'xml'
    results_entry: '//ns:entry'
    literal_search_params:
      format: atom
      relation: intersects
    pagination:
      next_page_url_tpl: '{url}?{search}&maxRecords={items_per_page}&startIndex={skip_base_1}'
      total_items_nb_key_path: '//os:totalResults/text()'
      # 2021/03/19: 50 is the max, no error if greater
      max_items_per_page: 50
    discover_metadata:
      auto_discovery: true
      metadata_pattern: '^(?!collection)[a-zA-Z0-9]+$'
      search_param: '{metadata}={{{metadata}}}'
      metadata_path: '*'
    metadata_mapping:
      # Opensearch resource identifier within the search engine context (in our case
      # within the context of the data provider)
      uid: 'ns:id/text()'
      # OpenSearch Parameters for Collection Search (Table 3)
      productType:
        - 'productType'
        - 'eo:productType/text()'
      doi: 'eo:doi/text()'
      platform: 'eo:platform/text()'
      platformSerialIdentifier: 'eo:platformSerialIdentifier/text()'
      instrument:
        - 'instrument'
        - 'eo:instrument/text()'
      sensorType: 'eo:sensorType/text()'
      compositeType: 'eo:compositeType/text()'
      processingLevel:
        - 'processingLevel'
        - 'eo:processingLevel/text()'
      orbitType: 'eo:orbitType/text()'
      spectralRange: 'eo:spectralRange/text()'
      wavelengths: 'eo:wavelengths/text()'
      hasSecurityConstraints: 'eo:hasSecurityConstraints/text()'
      dissemination: 'eo:dissemination/text()'

      # INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
      title: 'ns:title/text()'
      topicCategory: 'ns:category/text()'
      keyword: 'eo:keyword/text()'
      abstract: 'ns:summary/text()'
      resolution: 'eo:resolution/text()'
      organisationName: 'dc:creator/text()'
      organisationRole: 'eo:organisationRole/text()'
      publicationDate: 'ns:published/text()'
      lineage: 'eo:lineage/text()'
      useLimitation: 'eo:useLimitation/text()'
      accessConstraint: 'eo:accessConstraint/text()'
      otherConstraint: 'dc:rights/text()'
      classification: 'DIAS:productConfidentialityLevel/text()'
      language: 'eo:language/text()'
      specification: 'eo:specification/text()'

      # OpenSearch Parameters for Product Search (Table 5)
      parentIdentifier: 'eo:parentIdentifier/text()'
      acquisitionType: 'eo:acquisitionType/text()'
      orbitNumber: 'eo:orbitNumber/text()'
      orbitDirection: 'eo:orbitDirection/text()'
      track: 'eo:track/text()'
      frame: 'eo:frame/text()'
      swathIdentifier: 'eo:swathIdentifier/text()'
      cloudCover:
        - 'cloudCover=[0,{cloudCover}]'
        - 'eo:cloudCover/text()'
      snowCover: 'eo:snowCover/text()'
      lowestLocation: 'eo:lowestLocation/text()'
      highestLocation: 'eo:highestLocation/text()'
      productVersion: 'DIAS:processingBaseline/text()'
      productQualityStatus: 'eo:productQualityStatus/text()'
      productQualityDegradationTag: 'eo:productQualityDegradationTag/text()'
      processorName: 'eo:processorName/text()'
      processingCenter: 'eo:processingCenter/text()'
      creationDate: 'DIAS:onlineDate/text()'
      modificationDate: 'ns:updated/text()'
      processingDate: 'eo:processingDate/text()'
      sensorMode: 'eo:sensorMode/text()'
      archivingCenter: 'eo:archivingCenter/text()'
      processingMode: 'eo:processingMode/text()'

      # OpenSearch Parameters for Acquistion Parameters Search (Table 6)
      availabilityTime: 'DIAS:onlineDate/text()'
      acquisitionStation: 'eo:acquisitionStation/text()'
      acquisitionSubType: 'eo:acquisitionSubType/text()'
      startTimeFromAscendingNode:
        - 'timeStart={startTimeFromAscendingNode#to_iso_utc_datetime}'
        - 'DIAS:sensingStartDate/text()'
      completionTimeFromAscendingNode:
        - 'timeEnd={completionTimeFromAscendingNode#to_iso_utc_datetime}'
        - 'DIAS:sensingStopDate/text()'
      illuminationAzimuthAngle: 'eo:illuminationAzimuthAngle/text()'
      illuminationZenithAngle: 'eo:illuminationZenithAngle/text()'
      illuminationElevationAngle: 'eo:illuminationElevationAngle/text()'
      polarizationMode: 'eo:polarizationMode/text()'
      polarizationChannels: '{eo:polarisationChannels/text()#replace_str("/"," ")}'
      antennaLookDirection: 'eo:antennaLookDirection/text()'
      minimumIncidenceAngle: 'eo:minimumIncidenceAngle/text()'
      maximumIncidenceAngle: 'eo:maximumIncidenceAngle/text()'
      dopplerFrequency: 'eo:dopplerFrequency/text()'
      incidenceAngleVariation: 'eo:incidenceAngleVariation/text()'

      # Custom parameters (not defined in the base document referenced above)
      id:
        - 'uid={id#remove_extension}'
        - 'dc:identifier/text()'
      tileIdentifier:
        - 'tileIdentifier'
        - 'DIAS:tileIdentifier/text()'
      # The geographic extent of the product
      geometry:
        - 'geometry={geometry#to_rounded_wkt}'
        - '{georss:polygon|georss:where#from_georss}'
      defaultGeometry: '-180 -90 180 90'
      # The url of the quicklook
      quicklook: 'media:group/media:content[media:category="QUICKLOOK"]/@url'
      # The url to download the product "as is" (literal or as a template to be completed either after the search result
      # is obtained from the provider or during the eodag download phase)
      downloadLink: 'ns:link[@rel="enclosure"]/@href'
      # storageStatus: must be one of ONLINE, STAGING, OFFLINE
      storageStatus: 'DIAS:onlineStatus/text()'
      # order link formated for POST request usage
      orderLink: 'https://apis.mundiwebservices.com/odrapi/0.1/request?{{"productId":"{id}","collectionId":"{platform}"}}'
      searchLink: 'https://{platform}.browse.catalog.mundiwebservices.com/opensearch?uid={id}'
      # Additional metadata provided by the providers but that don't appear in the reference spec
      thumbnail: 'media:group/media:content[media:category="THUMBNAIL"]/@url'
  download: !plugin
    type: S3RestDownload
    base_uri: 'https://mundiwebservices.com/dp'
    extract: true
    auth_error_code: 401
    bucket_path_level: 0
    # order mechanism
    order_enabled: true
    order_method: 'POST'
    order_headers:
      accept: application/json
      Content-Type: application/json
    order_on_response:
      metadata_mapping:
        order_id: '{$.requestId#replace_str("Not Available","")}'
        reorder_id: '{$.message.`sub(/.*requestId: ([a-z0-9]+)/, \\1)`#replace_str("Not Available","")}'
        orderStatusLink: 'https://apis.mundiwebservices.com/odrapi/0.1/request/{order_id}{reorder_id}'
    order_status_method: 'GET'
    order_status_percent: status
    order_status_success:
      status: Success
    order_status_on_success:
      need_search: true
      result_type: 'xml'
      results_entry: '//ns:entry'
      metadata_mapping:
        downloadLink: 'ns:link[@rel="enclosure"]/@href'
        storageStatus: 'DIAS:onlineStatus/text()'
  auth: !plugin
    # Mixed HTTPHeaderAuth and TokenAuth
    type: TokenAuth
    auth_uri: 'https://apis.mundiwebservices.com/token'
    token_type: json
    token_key: access_token
    credentials:
      grant_type: password
    headers:
      Authorization: Basic cXJic1pTVjBUWTdkZzlNRGQ0Q3ZQVXkyQV9NYToxMU5oNGtFQjFva2NvZlBlRkRYemhudWxGR3dh
      Cookie: "seeedtoken={apikey}"
  products:
    # S1
    S1_SAR_GRD:
      productType: GRD
      collection: Sentinel1
      metadata_mapping:
        cloudCover: 'null/text()'
    S1_SAR_SLC:
      productType: SLC
      collection: Sentinel1
      metadata_mapping:
        cloudCover: 'null/text()'
    S1_SAR_OCN:
      productType: OCN
      collection: Sentinel1
      metadata_mapping:
        cloudCover: 'null/text()'
    S1_SAR_RAW:
      productType: RAW
      collection: Sentinel1
      metadata_mapping:
        cloudCover: 'null/text()'
    # S2
    S2_MSI_L1C:
      productType: IMAGE
      processingLevel: L1C
      collection: Sentinel2
    S2_MSI_L2A:
      productType: IMAGE
      processingLevel: L2A
      collection: Sentinel2
    # S3 SRAL
    S3_SRA:
      productType: SR_1_SRA___
      collection: Sentinel3
      metadata_mapping:
        cloudCover: 'null/text()'
    S3_SRA_A:
      productType: SR_1_SRA_A_
      collection: Sentinel3
      metadata_mapping:
        cloudCover: 'null/text()'
    S3_SRA_BS:
      productType: SR_1_SRA_BS
      collection: Sentinel3
      metadata_mapping:
        cloudCover: 'null/text()'
    S3_LAN:
      productType: SR_2_LAN___
      collection: Sentinel3
      metadata_mapping:
        cloudCover: 'null/text()'
    # S3 OLCI
    S3_EFR:
      productType: OL_1_EFR___
      collection: Sentinel3
    S3_ERR:
      productType: OL_1_ERR___
      collection: Sentinel3
    S3_OLCI_L2LRR:
      productType: OL_2_LRR___
      collection: Sentinel3
    S3_OLCI_L2LFR:
      productType: OL_2_LFR___
      collection: Sentinel3
    # S3 SLSTR
    S3_SLSTR_L1RBT:
      productType: SL_1_RBT___
      collection: Sentinel3
    S3_SLSTR_L2LST:
      productType: SL_2_LST___
      collection: Sentinel3
    S3_SLSTR_L2FRP:
      productType: SL_2_FRP___
      collection: Sentinel3
    # S3 SY
    S3_SY_SYN:
      productType: SY_2_SYN___
      collection: Sentinel3
    S3_SY_V10:
      productType: SY_2_V10___
      collection: Sentinel3
    S3_SY_VG1:
      productType: SY_2_VG1___
      collection: Sentinel3
    S3_SY_VGP:
      productType: SY_2_VGP___
      collection: Sentinel3
    # S5P L1
    S5P_L1B_IR_SIR:
      productType: L1B_IR_SIR
      collection: Sentinel5p
    S5P_L1B_IR_UVN:
      productType: L1B_IR_UVN
      collection: Sentinel5p
    S5P_L1B_RA_BD1:
      productType: L1B_RA_BD1
      collection: Sentinel5p
    S5P_L1B_RA_BD2:
      productType: L1B_RA_BD2
      collection: Sentinel5p
    S5P_L1B_RA_BD3:
      productType: L1B_RA_BD3
      collection: Sentinel5p
    S5P_L1B_RA_BD4:
      productType: L1B_RA_BD4
      collection: Sentinel5p
    S5P_L1B_RA_BD5:
      productType: L1B_RA_BD5
      collection: Sentinel5p
    S5P_L1B_RA_BD6:
      productType: L1B_RA_BD6
      collection: Sentinel5p
    S5P_L1B_RA_BD7:
      productType: L1B_RA_BD7
      collection: Sentinel5p
    S5P_L1B_RA_BD8:
      productType: L1B_RA_BD8
      collection: Sentinel5p
    # S5P L2
    S5P_L2_NO2:
      productType: L2__NO2___
      collection: Sentinel5p
    S5P_L2_CLOUD:
      productType: L2__CLOUD_
      collection: Sentinel5p
    S5P_L2_O3:
      productType: L2__O3____
      collection: Sentinel5p
    S5P_L2_CO:
      productType: L2__CO____
      collection: Sentinel5p
    S5P_L2_AER_AI:
      productType: L2__AER_AI
      collection: Sentinel5p
    S5P_L2_O3_PR:
      productType: L2__O3__PR
      collection: Sentinel5p
    S5P_L2_O3_TCL:
      productType: L2__O3_TCL
      collection: Sentinel5p
    S5P_L2_AER_LH:
      productType: L2__AER_LH
      collection: Sentinel5p
    S5P_L2_HCHO:
      productType: L2__HCHO__
      collection: Sentinel5p
    S5P_L2_CH4:
      productType: L2__CH4___
      collection: Sentinel5p
    S5P_L2_NP_BD3:
      productType: L2__NP_BD3
      collection: Sentinel5p
    S5P_L2_NP_BD6:
      productType: L2__NP_BD6
      collection: Sentinel5p
    S5P_L2_NP_BD7:
      productType: L2__NP_BD7
      collection: Sentinel5p
    S5P_L2_SO2:
      productType: L2__SO2___
      collection: Sentinel5p
    GENERIC_PRODUCT_TYPE:
      productType: '{productType}'
      collection: '{collection}'
      instrument: '{instrument}'
      processingLevel: '{processingLevel}'
---
!provider
  name: onda
  priority: 0
  description: Serco DIAS
  roles:
    - host
  url: https://www.onda-dias.eu/cms/
  search: !plugin
    type: ODataV4Search
    api_endpoint: 'https://catalogue.onda-dias.eu/dias-catalogue/Products'
    need_auth: false
    dont_quote:
      - '['
      - ']'
      - '$'
      - '='
      - '&'
      - ':'
    pagination:
      count_endpoint: 'https://catalogue.onda-dias.eu/dias-catalogue/Products/$count'
      next_page_url_tpl: '{url}?{search}&$top={items_per_page}&$skip={skip}&$expand=Metadata'
      # 2021/03/19: 2000 is the max, if greater 200 response but contains an error message
      max_items_per_page: 2_000
    results_entry: 'value'
    literal_search_params:
      $format: json
    free_text_search_operations:
      $search:
        union: ' OR '
        wrapper: '"{}"'
        operations:
          AND:
            - 'footprint:"Intersects({geometry#to_rounded_wkt})"'
            - 'productType:{productType}'
            - 'platformName:{platform}'
            - 'cloudCoverPercentage:[0 TO {cloudCover}]'
            - 'beginPosition:[{startTimeFromAscendingNode#to_iso_utc_datetime} TO *]'
            - 'endPosition:[* TO {completionTimeFromAscendingNode#to_iso_utc_datetime}]'
            - '{id#remove_extension}'
            - 'platformSerialIdentifier:{platformSerialIdentifier}'
            - 'instrumentShortName:{instrument}'
            - 'processingLevel:{processingLevel}'
            - 'sensorType:{sensorType}'
            - 'topicCategory:{topicCategory}'
            - 'lineage:{lineage}'
            - 'orbitNumber:{orbitNumber}'
            - 'orbitDirection:{orbitDirection}'
            - 'processingBaseline:{productVersion}'
            - 'generalQualityFlag:{productQualityStatus}'
            - 'sensorOperationalMode:{sensorMode}'
            - 'tileIdentifier:{tileIdentifier}'
    discover_metadata:
      auto_discovery: true
      metadata_pattern: '^[a-zA-Z0-9]+$'
      search_param:
        free_text_search_operations:
          $search:
            operations:
              AND:
                - '{metadata}:{{{metadata}}}'
      metadata_path: '$.Metadata.*'
    per_product_metadata_query: false
    metadata_pre_mapping:
      metadata_path: '$.Metadata'
      metadata_path_id: 'id'
      metadata_path_value: 'value'
    metadata_mapping:
      # Opensearch resource identifier within the search engine context (in our case
      # within the context of the data provider)
      # Queryable parameters are set with null as 1st configuration list value to mark them as queryable,
      #   but `free_text_search_operations.$search.operations.AND` entries are then used instead.
      uid: '$.id'
      # OpenSearch Parameters for Collection Search (Table 3)
      productType:
        - null
        - '$.Metadata.productType'
      platform:
        - null
        - '$.Metadata.platformName'
      platformSerialIdentifier:
        - null
        - '$.Metadata.platformSerialIdentifier'
      instrument:
        - null
        - '$.Metadata.instrumentShortName'
      processingLevel:
        - null
        - '$.Metadata.processingLevel'
      sensorType:
        - null
        - '$.Metadata.sensorType'

      # INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
      title: '$.Metadata.filename'
      topicCategory:
        - null
        - '$.Metadata.topicCategory'
      lineage:
        - null
        - '$.Metadata.lineage'

      # OpenSearch Parameters for Product Search (Table 5)
      orbitNumber:
        - null
        - '$.Metadata.orbitNumber'
      orbitDirection:
        - null
        - '$.Metadata.orbitDirection'
      cloudCover:
        - null
        - '$.Metadata.cloudCoverPercentage'
      productVersion:
        - null
        - '$.Metadata.processingBaseline'
      productQualityStatus:
        - null
        - '$.Metadata.generalQualityFlag'
      creationDate: '$.Metadata.creationDate'
      processingDate: '$.Metadata.processingDate'
      sensorMode:
        - null
        - '$.Metadata.sensorOperationalMode'

      # OpenSearch Parameters for Acquistion Parameters Search (Table 6)
      startTimeFromAscendingNode:
        - null
        - '$.beginPosition'
      completionTimeFromAscendingNode:
        - null
        - '$.endPosition'
      polarizationChannels: '{$.Metadata.polarisationChannels#replace_str(","," ")}'

      # Custom parameters (not defined in the base document referenced above)
      id:
        - null
        - '{$.Metadata.filename#remove_extension}'
      tileIdentifier:
        - null
        - '$.Metadata.tileIdentifier'
      # The geographic extent of the product
      geometry:
        - null
        - '$.footprint'
      # The url of the quicklook
      quicklook: '$.quicklook'
      # The url to download the product "as is" (literal or as a template to be completed either after the search result
      # is obtained from the provider or during the eodag download phase)
      downloadLink: '%(base_uri)s({uid})/$value'
      # storageStatus: must be one of ONLINE, STAGING, OFFLINE
      storageStatus: '{$.offline#get_group_name((?P<ONLINE>False)|(?P<OFFLINE>True))}'
      # Url used for ordering product if it is offline/archived
      orderLink: 'https://catalogue.onda-dias.eu/dias-catalogue/Products({uid})/Ens.Order'
  products:
    # S1
    S1_SAR_OCN:
      productType: '*OCN*'
      metadata_mapping:
        cloudCover: '$.null'
    S1_SAR_GRD:
      productType: '*GRD*'
      metadata_mapping:
        cloudCover: '$.null'
    S1_SAR_SLC:
      productType: '*SLC*'
      metadata_mapping:
        cloudCover: '$.null'
    S1_SAR_RAW:
      productType: '*RAW*'
      metadata_mapping:
        cloudCover: '$.null'
    # S2
    S2_MSI_L1C:
      productType: S2MSI1C
    S2_MSI_L2A:
      productType: S2MSI2A
    # S3 SRAL
    S3_SRA:
      productType: SR_1_SRA___
      metadata_mapping:
        cloudCover: '$.null'
    S3_SRA_A:
      productType: SR_1_SRA_A_
      metadata_mapping:
        cloudCover: '$.null'
    S3_SRA_BS:
      productType: SR_1_SRA_BS
      metadata_mapping:
        cloudCover: '$.null'
    S3_LAN:
      productType: SR_2_LAN___
      metadata_mapping:
        cloudCover: '$.null'
    S3_WAT:
      productType: SR_2_WAT___
      metadata_mapping:
        cloudCover: '$.null'
    # S3 OLCI
    S3_EFR:
      productType: OL_1_EFR___
    S3_ERR:
      productType: OL_1_ERR___
    S3_OLCI_L2LFR:
      productType: OL_2_LFR___
    S3_OLCI_L2LRR:
      productType: OL_2_LRR___
    S3_OLCI_L2WRR:
      productType: OL_2_WRR___
    S3_OLCI_L2WFR:
      productType: OL_2_WFR___
    # S3 SLSTR
    S3_SLSTR_L1RBT:
      productType: SL_1_RBT___
    S3_SLSTR_L2LST:
      productType: SL_2_LST___
    S3_SLSTR_L2WST:
      productType: SL_2_WST___
    S3_SLSTR_L2FRP:
      productType: SL_2_FRP___
    # S3 SY
    S3_SY_AOD:
      productType: SY_2_AOD___
    S3_SY_SYN:
      productType: SY_2_SYN___
    S3_SY_V10:
      productType: SY_2_V10___
    S3_SY_VG1:
      productType: SY_2_VG1___
    S3_SY_VGP:
      productType: SY_2_VGP___
    # S5P L1
    S5P_L1B_RA_BD1:
      productType: L1B_RA_BD1
    S5P_L1B_RA_BD2:
      productType: L1B_RA_BD2
    S5P_L1B_RA_BD3:
      productType: L1B_RA_BD3
    S5P_L1B_RA_BD4:
      productType: L1B_RA_BD4
    S5P_L1B_RA_BD5:
      productType: L1B_RA_BD5
    S5P_L1B_RA_BD6:
      productType: L1B_RA_BD6
    S5P_L1B_RA_BD7:
      productType: L1B_RA_BD7
    S5P_L1B_RA_BD8:
      productType: L1B_RA_BD8
    # S5P L2
    S5P_L2_NO2:
      productType: L2__NO2___
    S5P_L2_CLOUD:
      productType: L2__CLOUD_
    S5P_L2_O3:
      productType: L2__O3____
    S5P_L2_CO:
      productType: L2__CO____
    S5P_L2_AER_AI:
      productType: L2__AER_AI
    S5P_L2_O3_PR:
      productType: L2__O3__PR
    S5P_L2_AER_LH:
      productType: L2__AER_LH
    S5P_L2_HCHO:
      productType: L2__HCHO__
    S5P_L2_CH4:
      productType: L2__CH4___
    S5P_L2_NP_BD3:
      productType: L2__NP_BD3
    S5P_L2_NP_BD6:
      productType: L2__NP_BD6
    S5P_L2_NP_BD7:
      productType: L2__NP_BD7
    S5P_L2_SO2:
      productType: L2__SO2___
    # Landsat
    L8_OLI_TIRS_C1L1:
      platform: 'Landsat-*'
    GENERIC_PRODUCT_TYPE:
      productType: '{productType}'
      platform: '{platform}'
  download: !plugin
    type: HTTPDownload
    base_uri: 'https://catalogue.onda-dias.eu/dias-catalogue/Products'
    extract: true
    auth_error_code: 401
    order_enabled: true
    order_method: 'POST'
    order_headers:
      Content-Type: application/json
  auth: !plugin
    type: GenericAuth

---
!provider
  name: astraea_eod
  priority: 0
  roles:
    - host
  description: Astraea Earth OnDemand
  url: https://earthondemand.astraea.earth
  search: !plugin
    type: StacSearch
    api_endpoint: https://eod-catalog-svc-prod.astraea.earth/search
    need_auth: false
    pagination:
      # 2021/03/19: The docs (https://eod-catalog-svc-prod.astraea.earth/api.html#operation/getSearchSTAC)
      # say the max is 10_000. In practice 1_000 products are returned if more are asked (even greater
      # than 10_000), without any error.
      # This provider doesn't implement any pagination, let's just try to get the maximum number of
      # products available at once then, so we stick to 10_000.
      max_items_per_page: 10_000
    metadata_mapping:
      # redefine the following mapppings as the provider does not support advanced queries/filtering,
      # these parameters will not be queryable
      doi: '$.properties."sci:doi"'
      processingLevel: '$.properties."processing:level"'
      platform: '$.properties.constellation'
      platformSerialIdentifier: '$.properties.platform'
      instrument: '$.properties.instruments'
      # INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
      title: '$.id'
      abstract: '$.properties.description'
      resolution: '$.properties.gsd'
      publicationDate: '$.properties.published'
      # OpenSearch Parameters for Product Search (Table 5)
      orbitNumber: '$.properties."sat:relative_orbit"'
      orbitDirection: '$.properties."sat:orbit_state"'
      cloudCover: '$.properties."eo:cloud_cover"'
      sensorMode: '$.properties."sar:instrument_mode"'
      creationDate: '$.properties.created'
      modificationDate: '$.properties.updated'
      productVersion: '$.properties.version'
      # OpenSearch Parameters for Acquistion Parameters Search (Table 6)
      availabilityTime: '$.properties.availabilityTime'
      acquisitionStation: '$.properties.acquisitionStation'
      acquisitionSubType: '$.properties.acquisitionSubType'
      illuminationAzimuthAngle: '$.properties."view:sun_azimuth"'
      illuminationElevationAngle: '$.properties."view:sun_elevation"'
      polarizationChannels: '$.properties."sar:polarizations"'
      dopplerFrequency: '$.properties."sar:frequency_band"'
  products:
    S1_SAR_GRD:
      productType: sentinel1_l1c_grd
      metadata_mapping:
        cloudCover: '$.null'
        platformSerialIdentifier: '$.id.`split(_, 0, -1)`'
        polarizationMode: '$.id.`sub(/.{14}([A-Z]{2}).*/, \\1)`'
        awsPath: '$.assets.productInfo.href.`sub(/(.*)\/productInfo\.json/, \\1)`'
        assets: |
          {$.assets#dict_update([["manifest.safe",[["title","manifest.safe"],["href","{awsPath}/manifest.safe"],["roles",["metadata"]],["type","application/xml"]]]])}
    S2_MSI_L1C:
      productType: sentinel2_l1c
      metadata_mapping:
        platformSerialIdentifier: '$.id.`split(_, 0, -1)`'
        tileInfo: '{$.assets.tileInfo.href#replace_str(r"s3(.*)sentinel-s2-l1c/",r"https\1roda.sentinel-hub.com/sentinel-s2-l1c/")}'
        tilePath: |
          $.assets.tileInfo.href.`sub(/.*\/sentinel-s2-l1c\/(tiles\/.*)\/tileInfo\.json/, \\1)`
    S2_MSI_L2A:
      productType: sentinel2_l2a
      metadata_mapping:
        platformSerialIdentifier: '$.id.`split(_, 0, -1)`'
        tileInfo: '{$.assets.tileInfo.href#replace_str(r"s3(.*)sentinel-s2-l2a/",r"https\1roda.sentinel-hub.com/sentinel-s2-l2a/")}'
        tilePath: |
          $.assets.tileInfo.href.`sub(/.*\/sentinel-s2-l2a\/(tiles\/.*)\/tileInfo\.json/, \\1)`
    LANDSAT_C2L1:
      productType: landsat8_c2l1t1
    MODIS_MCD43A4:
      productType: mcd43a4
    NAIP:
      productType: naip
    GENERIC_PRODUCT_TYPE:
      productType: '{productType}'
  download: !plugin
    type: AwsDownload
    requester_pays: True
    flatten_top_dirs: True
    products:
      S1_SAR_GRD:
        default_bucket: 'sentinel-s1-l1c'
        build_safe: true
        complementary_url_key:
          - awsPath
      S2_MSI_L1C:
        default_bucket: 'sentinel-s2-l1c'
        build_safe: true
        fetch_metadata:
          fetch_url: '{tileInfo}'
          fetch_format: json
          update_metadata:
            title: '$.productName'
            productPath: '$.productPath'
        complementary_url_key:
          - productPath
          - tilePath
      S2_MSI_L2A:
        default_bucket: 'sentinel-s2-l2a'
        build_safe: true
        fetch_metadata:
          fetch_url: '{tileInfo}'
          fetch_format: json
          update_metadata:
            title: '$.productName'
            productPath: '$.productPath'
        complementary_url_key:
          - productPath
          - tilePath
  auth: !plugin
    type: AwsAuth

---
!provider
  name: usgs_satapi_aws
  priority: 0
  roles:
    - host
  description: USGS Landsatlook SAT API
  url: https://landsatlook.usgs.gov/stac-server
  search: !plugin
    type: StacSearch
    api_endpoint: https://landsatlook.usgs.gov/stac-server/search
    need_auth: false
    pagination:
      # 2021/03/19: no more than 10_000 (if greater, returns a 500 error code)
      # but in practive if an Internal Server Error is returned for more than
      # about 500 products.
      max_items_per_page: 500
    metadata_mapping:
      assets: '{$.assets#recursive_sub_str(r"https?(.*)landsatlook.usgs.gov/data/",r"s3\1usgs-landsat/")}'
      awsProductId: '{$.assets.thumbnail.href#replace_str(r".+/([A-Z0-9_]+)/[\w.]+$",r"\1")}'
  products:
    LANDSAT_C2L1:
      productType: landsat-c2l1
    LANDSAT_C2L2_SR:
      productType: landsat-c2l2-sr
    LANDSAT_C2L2_ST:
      productType: landsat-c2l2-st
    LANDSAT_C2L2ALB_BT:
      productType: landsat-c2l2alb-bt
    LANDSAT_C2L2ALB_SR:
      productType: landsat-c2l2alb-sr
    LANDSAT_C2L2ALB_ST:
      productType: landsat-c2l2alb-st
    LANDSAT_C2L2ALB_TA:
      productType: landsat-c2l2alb-ta
    GENERIC_PRODUCT_TYPE:
      productType: '{productType}'
  download: !plugin
    type: AwsDownload
    requester_pays: True
    flatten_top_dirs: True
  auth: !plugin
    type: AwsAuth

---
!provider
  name: earth_search
  priority: 0
  roles:
    - host
  description: Earth Search
  url: https://www.element84.com/earth-search/
  search: !plugin
    type: StacSearch
    api_endpoint: https://earth-search.aws.element84.com/v0/search
    need_auth: false
    discover_product_types:
      results_entry: '$.collections[?id!="sentinel-s2-l2a-cogs"]'
    pagination:
      # Override the default next page url key path of StacSearch because the next link returned
      # by Earth Search is invalid (as of 2021/04/29). Entry set to null (None) to avoid using the
      # next page retrieval mechanism, `next_page_url_tpl` will be used instead (inherited from StacSearch)
      # Remove that entry if Earth Search updates that and returns a valid link.
      next_page_url_key_path: null
      # 2021/04/28: Earth-Search relies on Sat-API whose docs (http://sat-utils.github.io/sat-api/#search-stac-items-by-simple-filtering-)
      # say the max is 10_000. In practice a too high number (e.g. 5_000) returns a 502 error ({"message": "Internal server error"}).
      # Let's set it to a more robust number: 500
      max_items_per_page: 500
    metadata_mapping:
      utmZone:
        - '{{"query":{{"sentinel:utm_zone":{{"eq":"{utmZone}"}}}}}}'
        - '$.properties."sentinel:utm_zone"'
      latitudeBand:
        - '{{"query":{{"sentinel:latitude_band":{{"eq":"{latitudeBand}"}}}}}}'
        - '$.properties."sentinel:latitude_band"'
      gridSquare:
        - '{{"query":{{"sentinel:grid_square":{{"eq":"{gridSquare}"}}}}}}'
        - '$.properties."sentinel:grid_square"'
      tileIdentifier:
        - '{{"query":{{"sentinel:utm_zone":{{"eq":"{tileIdentifier#slice_str(0,2,1)}"}},"sentinel:latitude_band":{{"eq":"{tileIdentifier#slice_str(2,3,1)}"}},"sentinel:grid_square":{{"eq":"{tileIdentifier#slice_str(3,5,1)}"}}}}}}'
        - '{utmZone}{latitudeBand}{gridSquare}'
  products:
    S2_MSI_L1C:
      productType: sentinel-s2-l1c
      metadata_mapping:
        title: '$.properties."sentinel:product_id"'
        platformSerialIdentifier: '$.id.`split(_, 0, -1)`'
        polarizationMode: '$.id.`sub(/.{14}([A-Z]{2}).*/, \\1)`'
        productPath: |
          $.properties."sentinel:product_id".`sub(/([S2AB]{3})_MSIL1C_([0-9]{4})([0-9]{2})([0-9]{2})(T.*)/, products!\\2!\\3!\\4!\\1_MSIL1C_\\2\\3\\4\\5)`.`sub(/!0*/, /)`
        tilePath: |
          $.assets.info.href.`sub(/.*/sentinel-s2-l1c\/(tiles\/.*)\/tileInfo\.json/, \\1)`
        utmZone:
          - '{{"query":{{"mgrs:utm_zone":{{"eq":"{utmZone}"}}}}}}'
          - '$.properties."mgrs:utm_zone"'
        latitudeBand:
          - '{{"query":{{"mgrs:latitude_band":{{"eq":"{latitudeBand}"}}}}}}'
          - '$.properties."mgrs:latitude_band"'
        gridSquare:
          - '{{"query":{{"mgrs:grid_square":{{"eq":"{gridSquare}"}}}}}}'
          - '$.properties."mgrs:grid_square"'
        tileIdentifier:
          - '{{"query":{{"mgrs:utm_zone":{{"eq":"{tileIdentifier#slice_str(0,2,1)}"}},"mgrs:latitude_band":{{"eq":"{tileIdentifier#slice_str(2,3,1)}"}},"mgrs:grid_square":{{"eq":"{tileIdentifier#slice_str(3,5,1)}"}}}}}}'
          - '{utmZone}{latitudeBand}{gridSquare}'
    S2_MSI_L2A:
      productType: sentinel-s2-l2a
      metadata_mapping:
        title: '$.properties."sentinel:product_id"'
        platformSerialIdentifier: '$.id.`split(_, 0, -1)`'
        polarizationMode: '$.id.`sub(/.{14}([A-Z]{2}).*/, \\1)`'
        productPath: |
          $.properties."sentinel:product_id".`sub(/([S2AB]{3})_MSIL2A_([0-9]{4})([0-9]{2})([0-9]{2})(T.*)/, products!\\2!\\3!\\4!\\1_MSIL2A_\\2\\3\\4\\5)`.`sub(/!0*/, /)`
        tilePath: |
          $.assets.info.href.`sub(/.*/sentinel-s2-l2a\/(tiles\/.*)\/tileInfo\.json/, \\1)`
    L8_OLI_TIRS_C1L1:
      productType: landsat-8-l1-c1
    GENERIC_PRODUCT_TYPE:
      productType: '{productType}'
  download: !plugin
    type: AwsDownload
    requester_pays: True
    flatten_top_dirs: True
    products:
      S2_MSI_L1C:
        default_bucket: 'sentinel-s2-l1c'
        build_safe: true
        complementary_url_key:
          - productPath
          - tilePath
      S2_MSI_L2A:
        default_bucket: 'sentinel-s2-l2a'
        build_safe: true
        complementary_url_key:
          - productPath
          - tilePath
  auth: !plugin
    type: AwsAuth

---
!provider
  name: earth_search_cog
  priority: 0
  roles:
    - host
  description: Earth Search with Cloud Optimized GeoTIFF (COG) formatted assets
  url: https://www.element84.com/earth-search/
  search: !plugin
    type: StacSearch
    api_endpoint: https://earth-search.aws.element84.com/v0/search
    need_auth: false
    discover_product_types:
      fetch_url: null
    pagination:
      # Override the default next page url key path of StacSearch because the next link returned
      # by Earth Search is invalid (as of 2021/04/29). Entry set to null (None) to avoid using the
      # next page retrieval mechanism, `next_page_url_tpl` will be used instead (inherited from StacSearch)
      # Remove that entry if Earth Search updates that and returns a valid link.
      next_page_url_key_path: null
      # 2021/04/28: Earth-Search relies on Sat-API whose docs (http://sat-utils.github.io/sat-api/#search-stac-items-by-simple-filtering-)
      # say the max is 10_000. In practice a too high number (e.g. 5_000) returns a 502 error ({"message": "Internal server error"}).
      # Let's set it to a more robust number: 500
      max_items_per_page: 500
    metadata_mapping:
      platformSerialIdentifier: '$.id.`split(_, 0, -1)`'
      polarizationMode: '$.id.`sub(/.{14}([A-Z]{2}).*/, \\1)`'
  products:
    S2_MSI_L2A_COG:
      productType: sentinel-s2-l2a-cogs
    GENERIC_PRODUCT_TYPE:
      productType: '{productType}'
  download: !plugin
    type: HTTPDownload
    base_uri: 'https://sentinel-cogs.s3.us-west-2.amazonaws.com'
---
!provider
  name: earth_search_gcs
  priority: 0
  roles:
    - host
  description: Google Cloud Storage through Earth Search
  url: https://www.element84.com/earth-search/
  search: !plugin
    type: StacSearch
    api_endpoint: https://earth-search.aws.element84.com/v0/search
    need_auth: false
    discover_product_types:
      fetch_url: null
    pagination:
      # Override the default next page url key path of StacSearch because the next link returned
      # by Earth Search is invalid (as of 2021/04/29). Entry set to null (None) to avoid using the
      # next page retrieval mechanism, `next_page_url_tpl` will be used instead (inherited from StacSearch)
      # Remove that entry if Earth Search updates that and returns a valid link.
      next_page_url_key_path: null
      # 2021/04/28: Earth-Search relies on Sat-API whose docs (http://sat-utils.github.io/sat-api/#search-stac-items-by-simple-filtering-)
      # say the max is 10_000. In practice a too high number (e.g. 5_000) returns a 502 error ({"message": "Internal server error"}).
      # Let's set it to a more robust number: 500
      max_items_per_page: 500
  products:
    S2_MSI_L1C:
      productType: sentinel-s2-l1c
      metadata_mapping:
        title: '$.properties."sentinel:product_id"'
        platformSerialIdentifier: '$.id.`split(_, 0, -1)`'
        polarizationMode: '$.id.`sub(/.{14}([A-Z]{2}).*/, \\1)`'
        utmZone: '$.properties."sentinel:utm_zone"'
        latitudeBand: '$.properties."sentinel:latitude_band"'
        gridSquare: '$.properties."sentinel:grid_square"'
        downloadLink: 's3://gcp-public-data-sentinel-2/tiles/{utmZone}/{latitudeBand}/{gridSquare}/{title}.SAFE'
    L8_OLI_TIRS_C1L1:
      productType: landsat-8-l1-c1
      metadata_mapping:
        wrsPath: '$.properties."landsat:wrs_path"'
        wrsRow: '$.properties."landsat:wrs_row"'
        downloadLink: 's3://gcp-public-data-landsat/LC08/01/{wrsPath:03d}/{wrsRow:03d}/{title}'
    GENERIC_PRODUCT_TYPE:
      productType: '{productType}'
  download: !plugin
    type: AwsDownload
    base_uri: https://storage.googleapis.com
    ignore_assets: True
    products:
      S2_MSI_L1C:
        default_bucket: 'gcp-public-data-sentinel-2'
  auth: !plugin
    type: AwsAuth
---
!provider
  name: ecmwf
  priority: 0
  description: ECMWF archive products
  roles:
    - host
  url: https://www.ecmwf.int
  api: !plugin
    type: EcmwfApi
    api_endpoint: https://api.ecmwf.int/v1
    extract: false
    metadata_mapping:
      productType: '$.productType'
      title: '$.id'
      startTimeFromAscendingNode: '{$.startTimeFromAscendingNode#to_iso_date}'
      completionTimeFromAscendingNode:
        - 'date={startTimeFromAscendingNode#to_iso_date}/to/{completionTimeFromAscendingNode#to_iso_date(-1,)}'
        - '{$.completionTimeFromAscendingNode#to_iso_date}'
      id: '$.id'
      # The geographic extent of the product
      geometry:
        - 'area={geometry#to_nwse_bounds_str(/)}'
        - '$.geometry'
      # storageStatus set to ONLINE for consistency between providers
      storageStatus: '{$.null#replace_str("Not Available","ONLINE")}'
      downloadLink: 'https://apps.ecmwf.int/datasets/data/{dataset}'
      # ECMWF/MARS specific parameters
      # Full list here https://confluence.ecmwf.int/display/UDOC/Keywords+in+MARS+and+Dissemination+requests
      accuracy:
        - accuracy
        - '$.accuracy'
      anoffset:
        - anoffset
        - '$.anoffset'
      bitmap:
        - bitmap
        - '$.bitmap'
      block:
        - block
        - '$.block'
      channel:
        - channel
        - '$.channel'
      class:
        - class
        - '$.class'
      database:
        - database
        - '$.database'
      dataset:
        - dataset
        - '$.dataset'
      diagnostic:
        - diagnostic
        - '$.diagnostic'
      direction:
        - direction
        - '$.direction'
      domain:
        - domain
        - '$.domain'
      duplicates:
        - duplicates
        - '$.duplicates'
      expect:
        - expect
        - '$.expect'
      expver:
        - expver
        - '$.expver'
      fcmonth:
        - fcmonth
        - '$.fcmonth'
      fcperiod:
        - fcperiod
        - '$.fcperiod'
      fieldset:
        - fieldset
        - '$.fieldset'
      filter:
        - filter
        - '$.filter'
      format:
        - format
        - '$.format'
      frame:
        - frame
        - '$.frame'
      frequency:
        - frequency
        - '$.frequency'
      grid:
        - grid
        - '$.grid'
      hdate:
        - hdate
        - '$.hdate'
      ident:
        - ident
        - '$.ident'
      interpolation:
        - interpolation
        - '$.interpolation'
      intgrid:
        - intgrid
        - '$.intgrid'
      iteration:
        - iteration
        - '$.iteration'
      latitude:
        - latitude
        - '$.latitude'
      levelist:
        - levelist
        - '$.levelist'
      levtype:
        - levtype
        - '$.levtype'
      longitude:
        - longitude
        - '$.longitude'
      lsm:
        - lsm
        - '$.lsm'
      method:
        - method
        - '$.method'
      number:
        - number
        - '$.number'
      obsgroup:
        - obsgroup
        - '$.obsgroup'
      obstype:
        - obstype
        - '$.obstype'
      origin:
        - origin
        - '$.origin'
      packing:
        - packing
        - '$.packing'
      padding:
        - padding
        - '$.padding'
      param:
        - param
        - '$.param'
      priority:
        - priority
        - '$.priority'
      product:
        - product
        - '$.product'
      range:
        - range
        - '$.range'
      refdate:
        - refdate
        - '$.refdate'
      reference:
        - reference
        - '$.reference'
      reportype:
        - reportype
        - '$.reportype'
      repres:
        - repres
        - '$.repres'
      resol:
        - resol
        - '$.resol'
      rotation:
        - rotation
        - '$.rotation'
      section:
        - section
        - '$.section'
      source:
        - source
        - '$.source'
      step:
        - step
        - '$.step'
      stream:
        - stream
        - '$.stream'
      system:
        - system
        - '$.system'
      target:
        - target
        - '$.target'
      time:
        - time
        - '$.time'
      truncation:
        - truncation
        - '$.truncation'
      type:
        - type
        - '$.type'
      use:
        - use
        - '$.use'
  products:
    # See Archive Catalog in https://apps.ecmwf.int/archive-catalogue/
    # See available Public Datasets in https://apps.ecmwf.int/datasets/
    TIGGE_CF_SFC:
      class: ti
      dataset: tigge
      expver: prod
      type: cf
      levtype: sfc
      origin: ecmf
      grid: 0.5/0.5
      param: 59/134/136/146/147/151/165/166/167/168/172/176/177/179/189/235/228001/228002/228039/228139/228141/228144/228164/228228
      step: 0
      time: 00:00
    GENERIC_PRODUCT_TYPE:
      dataset: '{productType}'
---
!provider
  name: cop_ads
  priority: 0
  description: Copernicus Atmosphere Data Store
  roles:
    - host
  url: https://ads.atmosphere.copernicus.eu/
  api: !plugin
    type: CdsApi
    api_endpoint: https://ads.atmosphere.copernicus.eu/api/v2
    extract: false
    metadata_mapping:
      productType: '$.productType'
      title: '$.id'
      startTimeFromAscendingNode: '{$.startTimeFromAscendingNode#to_iso_date}'
      completionTimeFromAscendingNode:
        - 'date={startTimeFromAscendingNode#to_iso_date}/{completionTimeFromAscendingNode#to_iso_date(-1,)}'
        - '{$.completionTimeFromAscendingNode#to_iso_date}'
      id: '$.id'
      # The geographic extent of the product
      geometry:
        - 'area={geometry#to_nwse_bounds_str(/)}'
        - '$.geometry'
      # storageStatus set to ONLINE for consistency between providers
      storageStatus: '{$.null#replace_str("Not Available","ONLINE")}'
      downloadLink: 'https://ads.atmosphere.copernicus.eu/cdsapp#!/dataset/{dataset}'
      # Copernicus ADS specific parameters
      accuracy:
        - accuracy
        - '$.accuracy'
      anoffset:
        - anoffset
        - '$.anoffset'
      api_product_type:
        - product_type
        - '$.api_product_type'
      bitmap:
        - bitmap
        - '$.bitmap'
      block:
        - block
        - '$.block'
      channel:
        - channel
        - '$.channel'
      class:
        - class
        - '$.class'
      database:
        - database
        - '$.database'
      dataset:
        - dataset
        - '$.dataset'
      date_range:
        - date_range
        - '$.date_range'
      day:
        - day
        - '$.day'
      diagnostic:
        - diagnostic
        - '$.diagnostic'
      direction:
        - direction
        - '$.direction'
      domain:
        - domain
        - '$.domain'
      duplicates:
        - duplicates
        - '$.duplicates'
      expect:
        - expect
        - '$.expect'
      expver:
        - expver
        - '$.expver'
      fcmonth:
        - fcmonth
        - '$.fcmonth'
      fcperiod:
        - fcperiod
        - '$.fcperiod'
      fieldset:
        - fieldset
        - '$.fieldset'
      filter:
        - filter
        - '$.filter'
      format:
        - format
        - '$.format'
      frame:
        - frame
        - '$.frame'
      frequency:
        - frequency
        - '$.frequency'
      grid:
        - grid
        - '$.grid'
      hdate:
        - hdate
        - '$.hdate'
      ident:
        - ident
        - '$.ident'
      interpolation:
        - interpolation
        - '$.interpolation'
      intgrid:
        - intgrid
        - '$.intgrid'
      iteration:
        - iteration
        - '$.iteration'
      latitude:
        - latitude
        - '$.latitude'
      levelist:
        - levelist
        - '$.levelist'
      leadtime_hour:
        - leadtime_hour
        - '$.leadtime_hour'
      levtype:
        - levtype
        - '$.levtype'
      longitude:
        - longitude
        - '$.longitude'
      lsm:
        - lsm
        - '$.lsm'
      method:
        - method
        - '$.method'
      model_level:
        - model_level
        - '$.model_level'
      month:
        - month
        - '$.month'
      number:
        - number
        - '$.number'
      obsgroup:
        - obsgroup
        - '$.obsgroup'
      obstype:
        - obstype
        - '$.obstype'
      origin:
        - origin
        - '$.origin'
      packing:
        - packing
        - '$.packing'
      padding:
        - padding
        - '$.padding'
      param:
        - param
        - '$.param'
      pressure_level:
        - pressure_level
        - '$.pressure_level'
      priority:
        - priority
        - '$.priority'
      product:
        - product
        - '$.product'
      range:
        - range
        - '$.range'
      refdate:
        - refdate
        - '$.refdate'
      reference:
        - reference
        - '$.reference'
      reportype:
        - reportype
        - '$.reportype'
      repres:
        - repres
        - '$.repres'
      resol:
        - resol
        - '$.resol'
      rotation:
        - rotation
        - '$.rotation'
      section:
        - section
        - '$.section'
      source:
        - source
        - '$.source'
      step:
        - step
        - '$.step'
      stream:
        - stream
        - '$.stream'
      system:
        - system
        - '$.system'
      target:
        - target
        - '$.target'
      time:
        - time
        - '$.time'
      truncation:
        - truncation
        - '$.truncation'
      type:
        - type
        - '$.type'
      use:
        - use
        - '$.use'
      variable:
        - variable
        - '$.variable'
      year:
        - year
        - '$.year'
  products:
    # See available Public Datasets in https://ads.atmosphere.copernicus.eu/cdsapp#!/search?type=dataset
    CAMS_GACF_AOT:
      dataset: cams-global-atmospheric-composition-forecasts
      stream: oper
      class: mc
      expver: '0001'
      step: 0
      levtype: SFC
      variable:
        - sea_salt_aerosol_optical_depth_550nm
        - dust_aerosol_optical_depth_550nm
        - organic_matter_aerosol_optical_depth_550nm
        - black_carbon_aerosol_optical_depth_550nm
        - sulphate_aerosol_optical_depth_550nm
      time: '00:00'
      leadtime_hour: 0
      type: forecast
      format: netcdf
    CAMS_GACF_RH:
      dataset: cams-global-atmospheric-composition-forecasts
      stream: oper
      class: mc
      expver: '0001'
      step: 0
      variable: 'relative_humidity'
      pressure_level: ['1', '2', '3', '5', '7', '10', '20', '30', '50', '70', '100', '150', '200', '250', '300', '400', '500', '600', '700', '850', '925', '1000']
      time: '00:00'
      leadtime_hour: 0
      type: forecast
      format: netcdf
    CAMS_GACF_MR:
      dataset: cams-global-atmospheric-composition-forecasts
      stream: oper
      class: mc
      expver: '0001'
      step: 0
      variable:
        - sea_salt_aerosol_0.03-0.5um_mixing_ratio
        - sea_salt_aerosol_0.5-5um_mixing_ratio
        - sea_salt_aerosol_5-20um_mixing_ratio
        - dust_aerosol_0.03-0.55um_mixing_ratio
        - dust_aerosol_0.55-0.9um_mixing_ratio
        - dust_aerosol_0.9-20um_mixing_ratio
        - hydrophilic_organic_matter_aerosol_mixing_ratio
        - hydrophobic_organic_matter_aerosol_mixing_ratio
        - hydrophilic_black_carbon_aerosol_mixing_ratio
        - hydrophobic_black_carbon_aerosol_mixing_ratio
        - sulphate_aerosol_mixing_ratio
      model_level: ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32', '33', '34', '35', '36', '37', '38', '39', '40', '41', '42', '43', '44', '45', '46', '47', '48', '49', '50', '51', '52', '53', '54', '55', '56', '57', '58', '59', '60', '61']
      time: '00:00'
      leadtime_hour: 0
      type: forecast
      format: netcdf
    CAMS_EAC4:
      dataset: cams-global-reanalysis-eac4
      stream: oper
      class: mc
      expver: '0001'
      step: 0
      variable:
        - dust_aerosol_0.03-0.55um_mixing_ratio
        - dust_aerosol_0.55-0.9um_mixing_ratio
        - dust_aerosol_0.9-20um_mixing_ratio
        - dust_aerosol_optical_depth_550nm
        - hydrophilic_black_carbon_aerosol_mixing_ratio
        - hydrophilic_organic_matter_aerosol_mixing_ratio
        - hydrophobic_black_carbon_aerosol_mixing_ratio
        - hydrophobic_organic_matter_aerosol_mixing_ratio
        - sea_salt_aerosol_0.03-0.5um_mixing_ratio
        - sea_salt_aerosol_0.5-5um_mixing_ratio
        - sea_salt_aerosol_5-20um_mixing_ratio
        - sea_salt_aerosol_optical_depth_550nm
        - sulphate_aerosol_optical_depth_550nm
      model_level: ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32', '33', '34', '35', '36', '37', '38', '39', '40', '41', '42', '43', '44', '45', '46', '47', '48', '49', '50', '51', '52', '53', '54', '55', '56', '57', '58', '59', '60']
      time: '00:00'
      format: netcdf
    GENERIC_PRODUCT_TYPE:
      dataset: '{productType}'
      stream: oper
      class: mc
      expver: '0001'
      step: 0
      time: '00:00'
      format: netcdf
---
!provider
  name: cop_cds
  priority: 0
  description: Copernicus Climate Data Store
  roles:
    - host
  url: https://cds.climate.copernicus.eu/
  api: !plugin
    type: CdsApi
    api_endpoint: https://cds.climate.copernicus.eu/api/v2
    extract: false
    metadata_mapping:
      productType: '$.productType'
      title: '$.id'
      startTimeFromAscendingNode: '{$.startTimeFromAscendingNode#to_iso_date}'
      completionTimeFromAscendingNode:
        - 'date={startTimeFromAscendingNode#to_iso_date}/{completionTimeFromAscendingNode#to_iso_date(-1,)}'
        - '{$.completionTimeFromAscendingNode#to_iso_date}'
      id: '$.id'
      # The geographic extent of the product
      geometry:
        - 'area={geometry#to_nwse_bounds_str(/)}'
        - '$.geometry'
      # storageStatus set to ONLINE for consistency between providers
      storageStatus: '{$.null#replace_str("Not Available","ONLINE")}'
      downloadLink: 'https://cds.climate.copernicus.eu/cdsapp#!/dataset/{dataset}'
      # Copernicus CDS specific parameters
      accuracy:
        - accuracy
        - '$.accuracy'
      anoffset:
        - anoffset
        - '$.anoffset'
      api_product_type:
        - product_type
        - '$.api_product_type'
      bitmap:
        - bitmap
        - '$.bitmap'
      block:
        - block
        - '$.block'
      channel:
        - channel
        - '$.channel'
      class:
        - class
        - '$.class'
      database:
        - database
        - '$.database'
      dataset:
        - dataset
        - '$.dataset'
      date_range:
        - date_range
        - '$.date_range'
      day:
        - day
        - '$.day'
      diagnostic:
        - diagnostic
        - '$.diagnostic'
      direction:
        - direction
        - '$.direction'
      domain:
        - domain
        - '$.domain'
      duplicates:
        - duplicates
        - '$.duplicates'
      expect:
        - expect
        - '$.expect'
      expver:
        - expver
        - '$.expver'
      fcmonth:
        - fcmonth
        - '$.fcmonth'
      fcperiod:
        - fcperiod
        - '$.fcperiod'
      fieldset:
        - fieldset
        - '$.fieldset'
      filter:
        - filter
        - '$.filter'
      format:
        - format
        - '$.format'
      frame:
        - frame
        - '$.frame'
      frequency:
        - frequency
        - '$.frequency'
      grid:
        - grid
        - '$.grid'
      hdate:
        - hdate
        - '$.hdate'
      ident:
        - ident
        - '$.ident'
      interpolation:
        - interpolation
        - '$.interpolation'
      intgrid:
        - intgrid
        - '$.intgrid'
      iteration:
        - iteration
        - '$.iteration'
      latitude:
        - latitude
        - '$.latitude'
      levelist:
        - levelist
        - '$.levelist'
      leadtime_hour:
        - leadtime_hour
        - '$.leadtime_hour'
      levtype:
        - levtype
        - '$.levtype'
      longitude:
        - longitude
        - '$.longitude'
      lsm:
        - lsm
        - '$.lsm'
      method:
        - method
        - '$.method'
      model_level:
        - model_level
        - '$.model_level'
      month:
        - month
        - '$.month'
      number:
        - number
        - '$.number'
      obsgroup:
        - obsgroup
        - '$.obsgroup'
      obstype:
        - obstype
        - '$.obstype'
      origin:
        - origin
        - '$.origin'
      packing:
        - packing
        - '$.packing'
      padding:
        - padding
        - '$.padding'
      param:
        - param
        - '$.param'
      pressure_level:
        - pressure_level
        - '$.pressure_level'
      priority:
        - priority
        - '$.priority'
      product:
        - product
        - '$.product'
      range:
        - range
        - '$.range'
      refdate:
        - refdate
        - '$.refdate'
      reference:
        - reference
        - '$.reference'
      reportype:
        - reportype
        - '$.reportype'
      repres:
        - repres
        - '$.repres'
      resol:
        - resol
        - '$.resol'
      rotation:
        - rotation
        - '$.rotation'
      section:
        - section
        - '$.section'
      source:
        - source
        - '$.source'
      step:
        - step
        - '$.step'
      stream:
        - stream
        - '$.stream'
      system:
        - system
        - '$.system'
      target:
        - target
        - '$.target'
      time:
        - time
        - '$.time'
      truncation:
        - truncation
        - '$.truncation'
      type:
        - type
        - '$.type'
      use:
        - use
        - '$.use'
      variable:
        - variable
        - '$.variable'
      year:
        - year
        - '$.year'
  products:
    # See available Public Datasets in https://cds.climate.copernicus.eu/cdsapp#!/search?type=dataset
    ERA5_SL:
      dataset: reanalysis-era5-single-levels
      api_product_type: reanalysis
      time: '00:00'
      format: netcdf
    GENERIC_PRODUCT_TYPE:
      dataset: '{productType}'
      time: '00:00'
      format: netcdf
---
!provider
  name: sara
  priority: 0
  description: Sentinel Australasia Regional Access
  search: !plugin
    type: QueryStringSearch
    # The endpoint is based off of the collection. There is a generic endpoint,
    # but can be very slow if not enough metdata is provided.
    api_endpoint: 'https://copernicus.nci.org.au/sara.server/1.0/api/collections/{collection}/search.json'
    need_auth: false
    pagination:
      next_page_url_tpl: '{url}?{search}&maxRecords={items_per_page}&page={page}'
      total_items_nb_key_path: '$.properties.totalResults'
      # 2021/03/19: 500 is the max, no error if greater
      max_items_per_page: 500
    discover_metadata:
      auto_discovery: true
      metadata_pattern: '^(?!collection)[a-zA-Z0-9_]+$'
      search_param: '{metadata}={{{metadata}}}'
      metadata_path: '$.properties.*'
    metadata_mapping:
      # Opensearch resource identifier within the search engine context (in our case
      # within the context of the data provider)
      uid: '$.id'
      # OpenSearch Parameters for Collection Search (Table 3)
      productType:
        - productType
        - '$.properties.productType'
      platform: '$.properties.collection'
      platformSerialIdentifier:
        - platform
        - '$.properties.platform'
      instrument:
        - instrument
        - '$.properties.instrument'
      processingLevel:
        - processingLevel
        - '$.properties.processingLevel'
      # INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
      title: '$.properties.title'
      topicCategory: '$.properties.topicCategory'
      keyword: '$.properties.keywords'
      abstract: '$.properties.description'
      resolution:
        - 'resolution'
        - '$.properties.resolution'
      organisationName:
        - 'organisationName'
        - '$.properties.organisationName'
      publicationDate: '$.properties.published'
      accessConstraint: '$.properties.license'
      # OpenSearch Parameters for Product Search (Table 5)
      parentIdentifier:
        - 'parentIdentifier'
        - '$.properties.parentIdentifier'
      acquisitionType: '$.properties.acquisitionType'
      orbitNumber:
        - 'orbitNumber'
        - '$.properties.orbitNumber'
      orbitDirection:
        - 'orbitDirection'
        - '$.properties.orbitDirection'
      swathIdentifier:
        - 'swath'
        - '$.properties.swath'
      cloudCover:
        - 'cloudCover=[0,{cloudCover}]'
        - '$.properties.cloudCover'
      snowCover:
        - 'snowCover=[0,{snowCover}]'
        - '$.properties.snowCover'
      productVersion: '$.properties.version'
      productQualityStatus: '$.properties.onlineQualityCheck'
      processorName: '$.properties.processingName'
      processingCenter: '$.properties.processingCenter'
      creationDate: '$.properties.dhusIngestDate'
      modificationDate: '$.properties.updated'
      sensorMode:
        - 'sensorMode'
        - '$.properties.sensorMode'
      # OpenSearch Parameters for Acquistion Parameters Search (Table 6)
      startTimeFromAscendingNode:
        - startDate
        - '$.properties.startDate'
      completionTimeFromAscendingNode:
        - completionDate
        - '$.properties.completionDate'
      polarizationMode:
        - 'polarisation'
        - '$.properties.polarisation'
      # Custom parameters (not defined in the base document referenced above)
      id:
        - 'productIdentifier={id#remove_extension}'
        - '$.properties.productIdentifier'
      # The geographic extent of the product
      geometry:
        - 'geometry={geometry#to_rounded_wkt}'
        - '$.geometry'
      # The url of the quicklook
      quicklook: '$.properties.quicklook'
      # The url to download the product "as is" (literal or as a template to be completed either after the search result
      # is obtained from the provider or during the eodag download phase)
      downloadLink: '$.properties.services.download.url'
      # storageStatus set to ONLINE for consistency between providers
      storageStatus: '{$.null#replace_str("Not Available","ONLINE")}'
      # Additional metadata provided by the providers but that don't appear in the reference spec
      thumbnail: '$.properties.thumbnail'
  products:
    # Sentinel 1
    S1_SAR_OCN:
      productType: OCN
      collection: S1
      instrument: C-SAR
    S1_SAR_GRD:
      productType: GRD
      collection: S1
      instrument: C-SAR
    S1_SAR_SLC:
      productType: SLC
      collection: S1
      instrument: C-SAR
    # Sentinel 2
    S2_MSI_L1C:
      collection: S2
      productType: S2MSIL1C
      instrument: MSI
      processingLevel: L1C
    S2_MSI_L2A:
      collection: S2
      productType: S2MSIL2A
      instrument: MSI
      processingLevel: L2A
    # OLCI products
    # L1
    S3_EFR:
      productType: OL_1_EFR___
      collection: S3
      instrument: OLCI
      processingLevel: LEVEL-1
    S3_ERR:
      productType: OL_1_ERR___
      collection: S3
      instrument: OLCI
      processingLevel: LEVEL-1
    # output during radiometric calibration mode
    S3_RAC:
      productType: OL_1_RAC___
      collection: S3
      instrument: OLCI
      processingLevel: LEVEL-1
    # L2
    S3_OLCI_L2LRR:
      productType: OL_2_LRR___
      collection: S3
      instrument: OLCI
      processingLevel: LEVEL-2
    S3_OLCI_L2LFR:
      productType: OL_2_LFR___
      collection: S3
      instrument: OLCI
      processingLevel: LEVEL-2
    S3_OLCI_L2WRR:
      productType: OL_2_WRR___
      collection: S3
      instrument: OLCI
      processingLevel: LEVEL-2
    S3_OLCI_L2WFR:
      productType: OL_2_WFR___
      collection: S3
      instrument: OLCI
      processingLevel: LEVEL-2
    # SLSTR products
    S3_SLSTR_L1RBT:
      productType: SL_1_RBT___
      collection: S3
      instrument: SLSTR
      processingLevel: LEVEL-1
    S3_SLSTR_L2LST:
      productType: SL_2_LST___
      collection: S3
      instrument: SLSTR
      processingLevel: LEVEL-2
    S3_SLSTR_L2WST:
      productType: SL_2_WST___
      collection: S3
      instrument: SLSTR
      processingLevel: LEVEL-2
    S3_SLSTR_L2AOD:
      productType: SL_2_AOD___
      collection: S3
      instrument: SLSTR
      processingLevel: LEVEL-2
    S3_SLSTR_L2FRP:
      productType: SL_2_FRP___
      collection: S3
      instrument: SLSTR
      processingLevel: LEVEL-2
    # SRAL
    # L1
    S3_SRA:
      productType: SR_1_SRA___
      collection: S3
      instrument: SRAL
      processingLevel: LEVEL-1
    S3_SRA_A:
      productType: SR_1_SRA_A_
      collection: S3
      instrument: SRAL
      processingLevel: LEVEL-1
    S3_SRA_BS:
      productType: SR_1_SRA_BS
      collection: S3
      instrument: SRAL
      processingLevel: LEVEL-1
    # L2
    S3_LAN:
      productType: SR_2_LAN___
      collection: S3
      instrument: SRAL
      processingLevel: LEVEL-2
    S3_WAT:
      productType: SR_2_WAT___
      collection: S3
      instrument: SRAL
      processingLevel: LEVEL-2
    # Synergy products
    S3_SY_AOD:
      productType: SY_2_AOD___
      collection: S3
      instrument: SYNERGY
      processingLevel: LEVEL-2
    S3_SY_SYN:
      productType: SY_2_SYN___
      collection: S3
      instrument: SYNERGY
      processingLevel: LEVEL-2
    S3_SY_V10:
      productType: SY_2_V10___
      collection: S3
      instrument: SYNERGY
      processingLevel: LEVEL-2W
    S3_SY_VG1:
      productType: SY_2_VG1___
      collection: S3
      instrument: SYNERGY
      processingLevel: LEVEL-2
    S3_SY_VGP:
      productType: SY_2_VGP___
      collection: S3
      instrument: SYNERGY
      processingLevel: LEVEL-2
    GENERIC_PRODUCT_TYPE:
      productType: '{productType}'
      collection: '{collection}'
  download: !plugin
    type: HTTPDownload
    base_uri: 'https://copernicus.nci.org.au/'
    extract: true
    archive_depth: 2
    order_enabled: true
    auth_error_code: 403
  auth: !plugin
    type: GenericAuth
    method: basic
---
!provider
  name: meteoblue
  priority: 0
  roles:
    - host
  description: Meteoblue
  url: https://www.meteoblue.com
  search: !plugin
    type: BuildPostSearchResult
    api_endpoint: 'https://my.meteoblue.com/dataset/query'
    need_auth: true
    pagination:
      next_page_query_obj: '{{"checkOnly":true}}'
    discover_metadata:
      auto_discovery: true
      metadata_pattern: '^[a-zA-Z0-9_]+$'
      search_param: '{{{{"{metadata}":{{{metadata}#to_geojson}} }}}}'
      metadata_path: '$.*'
    metadata_mapping:
      startTimeFromAscendingNode: '{$.startTimeFromAscendingNode#to_iso_utc_datetime}'
      completionTimeFromAscendingNode:
        - '{{"timeIntervals": [ "{startTimeFromAscendingNode#to_iso_date}/{completionTimeFromAscendingNode#to_iso_date(-1,)}" ] }}'
        - '{$.completionTimeFromAscendingNode#to_iso_utc_datetime}'
      geometry:
        - '{{"geometry": {geometry#to_geojson} }}'
        - '$.geometry'
      productType: '$.queries[0].domain'
      storageStatus: '{$.requiresJobQueue#get_group_name((?P<ONLINE>False)|(?P<OFFLINE>True))}'
      downloadLink: 'https://my.meteoblue.com/dataset/query'
      # Meteoblue specific parameters
      datapoints: '$.datapoints'
      requiresJobQueue: '$.requiresJobQueue'
      requiresComplexJobQueue: '$.requiresComplexJobQueue'
      units:
        - '{{"units": {units#to_geojson} }}'
        - '$.units'
      queries:
        - '{{"queries": {queries#to_geojson} }}'
        - '$.queries'
      format:
        - '{{"format": {format#to_geojson} }}'
        - '$.format'
      timeIntervalsAlignment:
        - '{{"timeIntervalsAlignment": {timeIntervalsAlignment#to_geojson} }}'
        - '$.timeIntervalsAlignment'
      orderLink: '{downloadLink#replace_str(r"^(.*)(\")(queries\")(.)",r"\1\2runOnJobQueue\2\4 true, \2\3\4")}'
  products:
    NEMSGLOBAL_TCDC:
      queries: [{'domain':'NEMSGLOBAL','gapFillDomain':null,'timeResolution':'daily','codes':[{'code':71,'level':'sfc','aggregation':'mean'}]}]
      format: netCDF
      units: {'temperature':'C','velocity':'km/h','length':'metric','energy':'watts'}
      timeIntervalsAlignment:
      geometry: {"type": "Polygon", "coordinates": [[[180, -90], [180, 90], [-180, 90], [-180, -90], [180, -90]]]}
    NEMSAUTO_TCDC:
      queries: [{'domain':'NEMSAUTO','gapFillDomain':null,'timeResolution':'daily','codes':[{'code':71,'level':'sfc','aggregation':'mean'}]}]
      format: netCDF
      units: {'temperature':'C','velocity':'km/h','length':'metric','energy':'watts'}
      timeIntervalsAlignment:
      geometry: {"type": "Polygon", "coordinates": [[[180, -90], [180, 90], [-180, 90], [-180, -90], [180, -90]]]}
    GENERIC_PRODUCT_TYPE:
      productType: '{productType}'
      geometry: {"type": "Polygon", "coordinates": [[[180, -90], [180, 90], [-180, 90], [-180, -90], [180, -90]]]}
  download: !plugin
    type: HTTPDownload
    base_uri: 'https://my.meteoblue.com/dataset/query'
    method: POST
    extract: False
    order_enabled: true
    order_method: 'POST'
    order_on_response:
      metadata_mapping:
        order_id: '$.id'
        downloadLink: 'http://queueresults.meteoblue.com/{order_id}'
        downloadMethod: '{$.null#replace_str("Not Available","GET")}'
        orderStatusLink: 'http://my.meteoblue.com/queue/status/{order_id}'
    order_status_method: 'GET'
    order_status_percent: percentCompleted
    order_status_error:
      status: error
    outputs_extension: .nc
  auth: !plugin
    type: HttpQueryStringAuth
    auth_uri: 'http://my.meteoblue.com/dataset/meta?dataset=NEMSAUTO'
---
!provider
  name: cop_dataspace
  priority: 0
  description: Copernicus Data Space Ecosystem
  roles:
    - host
  url: https://dataspace.copernicus.eu/
  search: !plugin
    type: ODataV4Search
    api_endpoint: 'https://catalogue.dataspace.copernicus.eu/odata/v1/Products'
    need_auth: false
    dont_quote:
      - '['
      - ']'
      - '$'
      - '='
      - '&'
      - ':'
    pagination:
      next_page_url_tpl: '{url}?{search}&$top={items_per_page}&$skip={skip}&$expand=Attributes&$count=True'
      total_items_nb_key_path: '$."@odata.count"'
      max_items_per_page: 1_000
    results_entry: 'value'
    free_text_search_operations:
      $filter:
        union: ' or '
        wrapper: '{}'
        operations:
          and:
            - "Collection/Name eq '{collection}'"
            - "OData.CSC.Intersects(area=geography'{geometry#to_ewkt}')"
            - "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'productType' and att/OData.CSC.StringAttribute/Value eq '{productType}')"
            - "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'platformShortName' and att/OData.CSC.StringAttribute/Value eq '{platform}')"
            - "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'platformSerialIdentifier' and att/OData.CSC.StringAttribute/Value eq '{platformSerialIdentifier}')"
            - "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'spatialResolution' and att/OData.CSC.StringAttribute/Value eq '{resolution}')"
            - "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'authority' and att/OData.CSC.StringAttribute/Value eq '{organisationName}')"
            - "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'orbitNumber' and att/OData.CSC.StringAttribute/Value eq '{orbitNumber}')"
            - "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'orbitDirection' and att/OData.CSC.StringAttribute/Value eq '{orbitDirection}')"
            - "Attributes/OData.CSC.DoubleAttribute/any(att:att/Name eq 'cloudCover' and att/OData.CSC.DoubleAttribute/Value le {cloudCover})"
            - "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'operationalMode' and att/OData.CSC.StringAttribute/Value eq '{sensorMode}')"
            - "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'polarisationChannels' and att/OData.CSC.StringAttribute/Value eq '{polarizationChannels}')"
            - "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'tileId' and att/OData.CSC.StringAttribute/Value eq '{tileIdentifier}')"
            - 'ContentDate/Start gt {startTimeFromAscendingNode#to_iso_utc_datetime}'
            - 'ContentDate/End lt {completionTimeFromAscendingNode#to_iso_utc_datetime}'
            - contains(Name,'{id}')
    discover_metadata:
      auto_discovery: true
      metadata_pattern: '^(?!collection)[a-zA-Z0-9]+$'
      search_param:
        free_text_search_operations:
          $filter:
            operations:
              and:
                -  "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq '{metadata}' and att/OData.CSC.StringAttribute/Value eq '{{{metadata}}}')"
      metadata_path: '$.Attributes.*'
    per_product_metadata_query: false
    metadata_pre_mapping:
      metadata_path: '$.Attributes'
      metadata_path_id: 'Name'
      metadata_path_value: 'Value'
    metadata_mapping:
      # Opensearch resource identifier within the search engine context (in our case
      # within the context of the data provider)
      # Queryable parameters are set with null as 1st configuration list value to mark them as queryable,
      #   but `free_text_search_operations.$filter.operations.and` entries are then used instead.
      uid: '$.Id'
      # OpenSearch Parameters for Collection Search (Table 3)
      productType:
        - null
        - '$.Attributes.productType'
      platform:
        - null
        - '$.Attributes.platformShortName'
      platformSerialIdentifier:
        - null
        - '$.Attributes.platformSerialIdentifier'
      instrument:
        - null
        - '$.Attributes.instrumentShortName'
      processingLevel:
        - null
        - '$.Attributes.processingLevel'
      # INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
      title: '$.Name'
      resolution:
        - null
        - '$.Attributes.spatialResolution'
      organisationName:
        - null
        - '$.Attributes.authority'
      publicationDate: '$.PublicationDate'
      # OpenSearch Parameters for Product Search (Table 5)
      orbitNumber:
        - null
        - '$.Attributes.orbitNumber'
      orbitDirection:
        - null
        - '$.Attributes.orbitDirection'
      cloudCover:
        - null
        - '$.Attributes.cloudCover'
      modificationDate: '$.ModificationDate'
      sensorMode:
        - null
        - '$.Attributes.operationalMode'
      # OpenSearch Parameters for Acquistion Parameters Search (Table 6)
      startTimeFromAscendingNode:
        - null
        - '$.ContentDate.Start'
      completionTimeFromAscendingNode:
        - null
        - '$.ContentDate.End'
      polarizationChannels:
        - null
        - '$.Attributes.polarisationChannels'
      # Custom parameters (not defined in the base document referenced above)
      id:
        - null
        - '{$.Name#remove_extension}'
      tileIdentifier:
        - null
        - '$.Attributes.tileId'
      # The geographic extent of the product
      geometry:
        - null
        - '{$.Footprint.`sub(/.*(SRID=[0-9]+;.*)/, \\1)`#from_ewkt}'
      # The url to download the product "as is" (literal or as a template to be completed either after the search result
      # is obtained from the provider or during the eodag download phase)
      downloadLink: 'https://catalogue.dataspace.copernicus.eu/odata/v1/Products({uid})/$value'
      # storageStatus: must be one of ONLINE, STAGING, OFFLINE
      storageStatus: '{$.Online#get_group_name((?P<ONLINE>True)|(?P<OFFLINE>False))}'
      collection:
        - null
        - $.null
      # Additional metadata provided by the providers but that don't appear in the reference spec
      productIdentifier: '$.S3Path'
  download: !plugin
    type: HTTPDownload
    base_uri: 'https://catalogue.dataspace.copernicus.eu/odata/v1/Products'
    extract: true
    order_enabled: false
    archive_depth: 2
  auth: !plugin
    type: KeycloakOIDCPasswordAuth
    auth_base_uri: 'https://identity.dataspace.copernicus.eu/auth'
    realm: 'CDSE'
    client_id: 'cdse-public'
    client_secret: null
    token_provision: qs
    token_qs_key: 'token'
    auth_error_code: 401
  products:
    # S2
    S2_MSI_L1C:
      collection: SENTINEL-2
      productType: S2MSI1C
    S2_MSI_L2A:
      collection: SENTINEL-2
      productType: S2MSI2A
    # S1
    S1_SAR_RAW:
      productType: RAW
      collection: SENTINEL-1
      metadata_mapping:
        cloudCover: '$.null'
    S1_SAR_GRD:
      productType: GRD
      collection: SENTINEL-1
      metadata_mapping:
        cloudCover: '$.null'
    S1_SAR_SLC:
      productType: SLC
      collection: SENTINEL-1
      metadata_mapping:
        cloudCover: '$.null'
    S1_SAR_OCN:
      productType: OCN
      collection: SENTINEL-1
      metadata_mapping:
        cloudCover: '$.null'
    # S3 SRAL
    S3_SRA:
      productType: SR_1_SRA___
      collection: SENTINEL-3
      metadata_mapping:
        cloudCover: '$.null'
    S3_SRA_A:
      productType: SR_1_SRA_A_
      collection: SENTINEL-3
      metadata_mapping:
        cloudCover: '$.null'
    S3_SRA_BS:
      productType: SR_1_SRA_BS
      collection: SENTINEL-3
      metadata_mapping:
        cloudCover: '$.null'
    S3_LAN:
      productType: SR_2_LAN___
      collection: SENTINEL-3
      metadata_mapping:
        cloudCover: '$.null'
    S3_WAT:
      productType: SR_2_WAT___
      collection: SENTINEL-3
      metadata_mapping:
        cloudCover: '$.null'
    # S3 OLCI
    S3_EFR:
      productType: OL_1_EFR___
      collection: SENTINEL-3
    S3_ERR:
      productType: OL_1_ERR___
      collection: SENTINEL-3
    S3_OLCI_L2LRR:
      productType: OL_2_LRR___
      collection: SENTINEL-3
    S3_OLCI_L2LFR:
      productType: OL_2_LFR___
      collection: SENTINEL-3
    S3_OLCI_L2WRR:
      productType: OL_2_WRR___
      collection: SENTINEL-3
    S3_OLCI_L2WFR:
      productType: OL_2_WFR___
      collection: SENTINEL-3
    # S3 SLSTR
    S3_SLSTR_L1RBT:
      productType: SL_1_RBT___
      collection: SENTINEL-3
    S3_SLSTR_L2LST:
      productType: SL_2_LST___
      collection: SENTINEL-3
    S3_SLSTR_L2WST:
      productType: SL_2_WST___
      collection: SENTINEL-3
    S3_SLSTR_L2AOD:
      productType: SL_2_AOD___
      collection: SENTINEL-3
    S3_SLSTR_L2FRP:
      productType: SL_2_FRP___
      collection: SENTINEL-3
    # S3 SY
    S3_SY_AOD:
      productType: SY_2_AOD___
      collection: SENTINEL-3
    S3_SY_SYN:
      productType: SY_2_SYN___
      collection: SENTINEL-3
    S3_SY_V10:
      productType: SY_2_V10___
      collection: SENTINEL-3
    S3_SY_VG1:
      productType: SY_2_VG1___
      collection: SENTINEL-3
    S3_SY_VGP:
      productType: SY_2_VGP___
      collection: SENTINEL-3
    # S5P L1
    S5P_L1B_RA_BD1:
      productType: L1B_RA_BD1
      collection: SENTINEL-5P
    S5P_L1B_RA_BD2:
      productType: L1B_RA_BD2
      collection: SENTINEL-5P
    S5P_L1B_RA_BD3:
      productType: L1B_RA_BD3
      collection: SENTINEL-5P
    S5P_L1B_RA_BD4:
      productType: L1B_RA_BD4
      collection: SENTINEL-5P
    S5P_L1B_RA_BD5:
      productType: L1B_RA_BD5
      collection: SENTINEL-5P
    S5P_L1B_RA_BD6:
      productType: L1B_RA_BD6
      collection: SENTINEL-5P
    S5P_L1B_RA_BD7:
      productType: L1B_RA_BD7
      collection: SENTINEL-5P
    S5P_L1B_RA_BD8:
      productType: L1B_RA_BD8
      collection: SENTINEL-5P
    # S5P L2
    S5P_L2_NO2:
      productType: L2__NO2___
      collection: SENTINEL-5P
    S5P_L2_CLOUD:
      productType: L2__CLOUD_
      collection: SENTINEL-5P
    S5P_L2_O3:
      productType: L2__O3____
      collection: SENTINEL-5P
    S5P_L2_CO:
      productType: L2__CO____
      collection: SENTINEL-5P
    S5P_L2_AER_AI:
      productType: L2__AER_AI
      collection: SENTINEL-5P
    S5P_L2_O3_PR:
      productType: L2__O3__PR
      collection: SENTINEL-5P
    S5P_L2_AER_LH:
      productType: L2__AER_LH
      collection: SENTINEL-5P
    S5P_L2_HCHO:
      productType: L2__HCHO__
      collection: SENTINEL-5P
    S5P_L2_CH4:
      productType: L2__CH4___
      collection: SENTINEL-5P
    S5P_L2_NP_BD3:
      productType: L2__NP_BD3
      collection: SENTINEL-5P
    S5P_L2_NP_BD6:
      productType: L2__NP_BD6
      collection: SENTINEL-5P
    S5P_L2_NP_BD7:
      productType: L2__NP_BD7
      collection: SENTINEL-5P
    S5P_L2_SO2:
      productType: L2__SO2___
      collection: SENTINEL-5P
    GENERIC_PRODUCT_TYPE:
      productType: '{productType}'
      collection: '{collection}'
---
!provider
  name: planetary_computer
  priority: 0
  roles:
    - host
  description: Microsoft Planetary Computer
  url: https://planetarycomputer.microsoft.com
  search: !plugin
    type: StacSearch
    api_endpoint: https://planetarycomputer.microsoft.com/api/stac/v1/search
    need_auth: false
    pagination:
      max_items_per_page: 1000
    metadata_mapping:
      tileIdentifier:
        - '{{"query":{{"s2:mgrs_tile":{{"eq":"{tileIdentifier}"}}}}}}'
        - '$.properties."s2:mgrs_tile"'
  products:
    S1_SAR_GRD:
      productType: sentinel-1-grd
    S2_MSI_L2A:
      productType: sentinel-2-l2a
    LANDSAT_C2L1:
      productType: landsat-c2-l1
    LANDSAT_C2L2:
      productType: landsat-c2-l2
    MODIS_MCD43A4:
      productType: modis-43A4-061
    NAIP:
      productType: naip
    GENERIC_PRODUCT_TYPE:
      productType: '{productType}'
  download: !plugin
    type: HTTPDownload
    base_uri: 'https://planetarycomputer.microsoft.com/api/stac/download'
    flatten_top_dirs: True
    auth_error_code: 403
  auth: !plugin
    type: SASAuth
    auth_uri: 'https://planetarycomputer.microsoft.com/api/sas/v1/sign?href={url}'
    signed_url_key: href
    headers:
      Ocp-Apim-Subscription-Key: "{apikey}"
---
!provider
  name: hydroweb_next
  priority: 0
  roles:
    - host
  description: hydroweb.next thematic hub for hydrology data access
  url: https://hydroweb.next.theia-land.fr
  search: !plugin
    type: StacSearch
    api_endpoint: https://hydroweb.next.theia-land.fr/api/v1/rs-catalog/stac/search
    need_auth: true
    auth_error_code: 401
    pagination:
      max_items_per_page: 10_000
    metadata_mapping:
      startTimeFromAscendingNode:
        - '{{"query":{{"end_datetime":{{"gte":"{startTimeFromAscendingNode#to_iso_utc_datetime}"}}}}}}'
        - '$.properties.start_datetime'
      completionTimeFromAscendingNode:
        - '{{"query":{{"start_datetime":{{"lte":"{completionTimeFromAscendingNode#to_iso_utc_datetime}"}}}}}}'
        - '$.properties.end_datetime'
  products:
    GENERIC_PRODUCT_TYPE:
      productType: '{productType}'
  download: !plugin
    type: HTTPDownload
    base_uri: https://hydroweb.next.theia-land.fr
    flatten_top_dirs: true
    auth_error_code: 401
  auth: !plugin
    type: HTTPHeaderAuth
    headers:
      X-API-Key: "{apikey}"

Parameters mapping#

EODAG maps each provider’s specific metadata parameters to a common model using OGC OpenSearch Extension for Earth Observation. Extra parameters having no equivalent in this model are mapped as is.

Depending on the provider, some parameters are queryable or not. This is configured in providers.yml:

  • If a parameter metadata-mapping is a list, the first element will help constructing the query (using format()), and the 2nd will help extracting its values from the query result (using jsonpath)

  • If a parameter metadata-mapping is a string, it will not be queryable and this string will help extracting its values from the query result (using jsonpath).

some_provider:
   search:
      metadata_mapping:
         queryableParameter:
            - 'this_is_query_string={queryableParameter}'
            - '$.jsonpath.in.result.to.parameter'
         nonQueryableParameter: '$.jsonpath.in.result.to.another_parameter'

The following tables list the parameters supported by providers, and if they are queryable or not.

OpenSearch parameters (CSV)#

parameter

astraea_eod

cop_ads

cop_cds

cop_dataspace

creodias

earth_search

earth_search_cog

earth_search_gcs

ecmwf

mundi

onda

peps

planetary_computer

sara

theia

usgs_satapi_aws

abstract

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

accessConstraint

metadata only

metadata only

metadata only

metadata only

acquisitionStation

metadata only

queryable metadata

queryable metadata

queryable metadata

metadata only

queryable metadata

metadata only

queryable metadata

acquisitionSubType

metadata only

queryable metadata

queryable metadata

queryable metadata

metadata only

queryable metadata

metadata only

queryable metadata

acquisitionType

metadata only

metadata only

metadata only

antennaLookDirection

metadata only

metadata only

archivingCenter

metadata only

metadata only

availabilityTime

metadata only

queryable metadata

queryable metadata

queryable metadata

metadata only

queryable metadata

metadata only

queryable metadata

classification

metadata only

cloudCover

metadata only

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

completionTimeFromAscendingNode

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

compositeType

metadata only

creationDate

metadata only

queryable metadata

queryable metadata

queryable metadata

metadata only

metadata only

metadata only

queryable metadata

metadata only

metadata only

queryable metadata

dissemination

metadata only

doi

metadata only

queryable metadata

queryable metadata

queryable metadata

metadata only

queryable metadata

queryable metadata

dopplerFrequency

metadata only

queryable metadata

queryable metadata

queryable metadata

metadata only

queryable metadata

metadata only

queryable metadata

frame

metadata only

hasSecurityConstraints

metadata only

highestLocation

metadata only

illuminationAzimuthAngle

metadata only

queryable metadata

queryable metadata

queryable metadata

metadata only

queryable metadata

metadata only

queryable metadata

illuminationElevationAngle

metadata only

queryable metadata

queryable metadata

queryable metadata

metadata only

queryable metadata

metadata only

queryable metadata

illuminationZenithAngle

metadata only

metadata only

incidenceAngleVariation

metadata only

metadata only

instrument

metadata only

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

keyword

metadata only

metadata only

metadata only

metadata only

language

metadata only

lineage

metadata only

queryable metadata

lowestLocation

metadata only

maximumIncidenceAngle

metadata only

metadata only

minimumIncidenceAngle

metadata only

metadata only

modificationDate

metadata only

metadata only

metadata only

queryable metadata

queryable metadata

queryable metadata

metadata only

metadata only

queryable metadata

metadata only

metadata only

queryable metadata

orbitDirection

metadata only

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

metadata only

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

orbitNumber

metadata only

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

metadata only

queryable metadata

queryable metadata

queryable metadata

queryable metadata

metadata only

queryable metadata

orbitType

metadata only

organisationName

queryable metadata

queryable metadata

metadata only

queryable metadata

queryable metadata

metadata only

organisationRole

metadata only

otherConstraint

metadata only

parentIdentifier

metadata only

queryable metadata

queryable metadata

metadata only

platform

metadata only

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

metadata only

queryable metadata

metadata only

queryable metadata

metadata only

queryable metadata

queryable metadata

platformSerialIdentifier

metadata only

queryable metadata

queryable metadata

queryable metadata

metadata only

queryable metadata

metadata only

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

processingCenter

metadata only

metadata only

metadata only

processingDate

metadata only

metadata only

metadata only

processingLevel

metadata only

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

processingMode

metadata only

metadata only

processorName

metadata only

metadata only

metadata only

productQualityDegradationTag

metadata only

productQualityStatus

metadata only

queryable metadata

metadata only

metadata only

productType

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

productVersion

metadata only

queryable metadata

queryable metadata

queryable metadata

metadata only

queryable metadata

metadata only

queryable metadata

metadata only

metadata only

queryable metadata

publicationDate

metadata only

metadata only

metadata only

queryable metadata

queryable metadata

queryable metadata

metadata only

metadata only

queryable metadata

metadata only

metadata only

queryable metadata

resolution

metadata only

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

metadata only

queryable metadata

queryable metadata

queryable metadata

metadata only

queryable metadata

sensorMode

metadata only

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

metadata only

queryable metadata

queryable metadata

queryable metadata

queryable metadata

metadata only

queryable metadata

sensorType

metadata only

queryable metadata

snowCover

metadata only

queryable metadata

queryable metadata

metadata only

spectralRange

metadata only

startTimeFromAscendingNode

metadata only

queryable metadata

queryable metadata

metadata only

metadata only

metadata only

queryable metadata

queryable metadata

queryable metadata

metadata only

queryable metadata

queryable metadata

metadata only

swathIdentifier

metadata only

queryable metadata

queryable metadata

title

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

topicCategory

metadata only

queryable metadata

metadata only

metadata only

track

metadata only

useLimitation

metadata only

wavelengths

metadata only

Provider/eodag specific parameters (CSV)#

parameter

astraea_eod

cop_ads

cop_cds

cop_dataspace

creodias

earth_search

earth_search_cog

earth_search_gcs

ecmwf

mundi

onda

peps

planetary_computer

sara

theia

usgs_satapi_aws

assets

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

awsProductId

metadata only

collection

queryable metadata

queryable metadata

defaultGeometry

metadata only

downloadLink

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

geometry

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

gridSquare

queryable metadata

id

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

metadata only

queryable metadata

latitudeBand

queryable metadata

orderLink

metadata only

metadata only

polarizationChannels

metadata only

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

metadata only

metadata only

queryable metadata

metadata only

queryable metadata

polarizationMode

metadata only

metadata only

queryable metadata

queryable metadata

metadata only

productIdentifier

metadata only

metadata only

quicklook

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

searchLink

metadata only

specification

metadata only

storageStatus

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

thumbnail

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

tileIdentifier

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

queryable metadata

uid

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

metadata only

utmZone

queryable metadata