Hint

You can run this notebook in a live session with .

ECMWF API plugin for EODAG

This tutorial will show you how use eodag to download data from ECWMF using EcmwfApi eodag plugin. The API plugin and this tutorial have been developed in the context of DOMINO-X.

from eodag import EODataAccessGateway, setup_logging

setup_logging(1)  # 0: nothing, 1: only progress bars, 2: INFO, 3: DEBUG
dag = EODataAccessGateway()
dag.set_preferred_provider("ecmwf")

Search (build download request) from an existing product type:

In this example, we will request data from TIGGE public dataset, using a product type already configured in eodag (TIGGE_CF_SFC) or using a custom request to access the same data.

Retrieval request can be customized from ecmwf web ui on https://apps.ecmwf.int/datasets/data/tigge.

For performance purpose, we add a param (total cloud cover, tcc) to the request as TIGGE_CF_SFC is configured to ask for all available param values. See ecmwf availables parameters. You can add more MARS keywords, that will overwrite default values configured for TIGGE_CF_SFC product type.

# Request for all parameters
products_from_product_type, total_count = dag.search(
    geom=[-60, 30, -130, 70],
    start="2021-11-01",
    end="2021-11-02",
    productType="TIGGE_CF_SFC",
)
print(
    "%s product built %s, \n having param=%s\n"
    % (
        total_count,
        products_from_product_type[0],
        products_from_product_type[0].properties["param"],
    )
)
# Request for total cloud cover
products_from_product_type, total_count = dag.search(
    geom=[-60, 30, -130, 70],
    start="2021-11-01",
    end="2021-11-02",
    productType="TIGGE_CF_SFC",
    param="tcc",
)
print(
    "%s product built %s, \n having param=%s"
    % (
        total_count,
        products_from_product_type[0],
        products_from_product_type[0].properties["param"],
    )
)

1 product built EOProduct(id=TIGGE_CF_SFC_20211101_31613e05afb70ccea479a4e2ce057227b15de05b, provider=ecmwf),
 having param=59/134/136/146/147/151/165/166/167/168/172/176/177/179/189/235/228001/228002/228039/228139/228141/228144/228164/228228

1 product built EOProduct(id=TIGGE_CF_SFC_20211101_28b0b1a0da723aba1682661621bd7b21fe12f1ea, provider=ecmwf),
 having param=tcc

Send product retrieval request and download when available

• download performed using ECMWF credentials set in ~/.config/eodag/eodag.yml as for other EO providers:

ecmwf:
    priority:
    api:
        outputs_prefix: /data/eodag_data
        credentials:
            username: john.doe@csgroup.eu
            password: my-ecmwf-api-access-key

• you can check your request status from https://apps.ecmwf.int/webmars/joblist/

• See ECMWF guidelines if request is queued for a long time

product_path = dag.download(products_from_product_type[0])
product_path

'/data/eodag_data/TIGGE_CF_SFC_20211101_28b0b1a0da723aba1682661621bd7b21fe12f1ea.grib'

Open dataset with xarray and cfgrib, then plot using matplotlib

import xarray as xr
import matplotlib.pyplot as plt

ds = xr.load_dataset(product_path, engine="cfgrib")
ds

<xarray.Dataset>
Dimensions:           (latitude: 81, longitude: 141)
Coordinates:
    number            int64 0
    time              datetime64[ns] 2021-11-01
    step              timedelta64[ns] 00:00:00
    entireAtmosphere  float64 0.0
  * latitude          (latitude) float64 70.0 69.5 69.0 68.5 ... 31.0 30.5 30.0
  * longitude         (longitude) float64 230.0 230.5 231.0 ... 299.5 300.0
    valid_time        datetime64[ns] 2021-11-01
Data variables:
    tcc               (latitude, longitude) float32 100.0 100.0 ... 8.406 10.62
Attributes:
    GRIB_edition:            2
    GRIB_centre:             ecmf
    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             European Centre for Medium-Range Weather Forecasts
    history:                 2022-05-11T09:27 GRIB to CDM+CF via cfgrib-0.9.1...

xarray.Dataset

• Dimensions:
  • latitude: 81
  • longitude: 141
• Coordinates: (7)
  • number
    ()
    int64
    0

    long_name :

    ensemble member numerical id

    units :

    1

    standard_name :

    realization

    array(0)

  • time
    ()
    datetime64[ns]
    2021-11-01

    long_name :

    initial time of forecast

    standard_name :

    forecast_reference_time

    array('2021-11-01T00:00:00.000000000', dtype='datetime64[ns]')

  • step
    ()
    timedelta64[ns]
    00:00:00

    long_name :

    time since forecast_reference_time

    standard_name :

    forecast_period

    array(0, dtype='timedelta64[ns]')

  • entireAtmosphere
    ()
    float64
    0.0

    long_name :

    original GRIB coordinate for key: level(entireAtmosphere)

    units :

    1

    array(0.)

  • latitude
    (latitude)
    float64
    70.0 69.5 69.0 ... 31.0 30.5 30.0

    units :

    degrees_north

    standard_name :

    latitude

    long_name :

    latitude

    stored_direction :

    decreasing

    array([70. , 69.5, 69. , 68.5, 68. , 67.5, 67. , 66.5, 66. , 65.5, 65. , 64.5,
           64. , 63.5, 63. , 62.5, 62. , 61.5, 61. , 60.5, 60. , 59.5, 59. , 58.5,
           58. , 57.5, 57. , 56.5, 56. , 55.5, 55. , 54.5, 54. , 53.5, 53. , 52.5,
           52. , 51.5, 51. , 50.5, 50. , 49.5, 49. , 48.5, 48. , 47.5, 47. , 46.5,
           46. , 45.5, 45. , 44.5, 44. , 43.5, 43. , 42.5, 42. , 41.5, 41. , 40.5,
           40. , 39.5, 39. , 38.5, 38. , 37.5, 37. , 36.5, 36. , 35.5, 35. , 34.5,
           34. , 33.5, 33. , 32.5, 32. , 31.5, 31. , 30.5, 30. ])

  • longitude
    (longitude)
    float64
    230.0 230.5 231.0 ... 299.5 300.0

    units :

    degrees_east

    standard_name :

    longitude

    long_name :

    longitude

    array([230. , 230.5, 231. , 231.5, 232. , 232.5, 233. , 233.5, 234. , 234.5,
           235. , 235.5, 236. , 236.5, 237. , 237.5, 238. , 238.5, 239. , 239.5,
           240. , 240.5, 241. , 241.5, 242. , 242.5, 243. , 243.5, 244. , 244.5,
           245. , 245.5, 246. , 246.5, 247. , 247.5, 248. , 248.5, 249. , 249.5,
           250. , 250.5, 251. , 251.5, 252. , 252.5, 253. , 253.5, 254. , 254.5,
           255. , 255.5, 256. , 256.5, 257. , 257.5, 258. , 258.5, 259. , 259.5,
           260. , 260.5, 261. , 261.5, 262. , 262.5, 263. , 263.5, 264. , 264.5,
           265. , 265.5, 266. , 266.5, 267. , 267.5, 268. , 268.5, 269. , 269.5,
           270. , 270.5, 271. , 271.5, 272. , 272.5, 273. , 273.5, 274. , 274.5,
           275. , 275.5, 276. , 276.5, 277. , 277.5, 278. , 278.5, 279. , 279.5,
           280. , 280.5, 281. , 281.5, 282. , 282.5, 283. , 283.5, 284. , 284.5,
           285. , 285.5, 286. , 286.5, 287. , 287.5, 288. , 288.5, 289. , 289.5,
           290. , 290.5, 291. , 291.5, 292. , 292.5, 293. , 293.5, 294. , 294.5,
           295. , 295.5, 296. , 296.5, 297. , 297.5, 298. , 298.5, 299. , 299.5,
           300. ])

  • valid_time
    ()
    datetime64[ns]
    2021-11-01

    standard_name :

    time

    long_name :

    time

    array('2021-11-01T00:00:00.000000000', dtype='datetime64[ns]')

• Data variables: (1)
  • tcc
    (latitude, longitude)
    float32
    100.0 100.0 100.0 ... 8.406 10.62

    GRIB_paramId :

    228164

    GRIB_dataType :

    cf

    GRIB_numberOfPoints :

    11421

    GRIB_typeOfLevel :

    entireAtmosphere

    GRIB_stepUnits :

    1

    GRIB_stepType :

    instant

    GRIB_gridType :

    regular_ll

    GRIB_NV :

    0

    GRIB_Nx :

    141

    GRIB_Ny :

    81

    GRIB_cfName :

    unknown

    GRIB_cfVarName :

    tcc

    GRIB_gridDefinitionDescription :

    Latitude/longitude

    GRIB_iDirectionIncrementInDegrees :

    0.5

    GRIB_iScansNegatively :

    0

    GRIB_jDirectionIncrementInDegrees :

    0.5

    GRIB_jPointsAreConsecutive :

    0

    GRIB_jScansPositively :

    0

    GRIB_latitudeOfFirstGridPointInDegrees :

    70.0

    GRIB_latitudeOfLastGridPointInDegrees :

    30.0

    GRIB_longitudeOfFirstGridPointInDegrees :

    230.0

    GRIB_longitudeOfLastGridPointInDegrees :

    300.0

    GRIB_missingValue :

    9999

    GRIB_name :

    Total Cloud Cover

    GRIB_shortName :

    tcc

    GRIB_totalNumber :

    51

    GRIB_units :

    %

    long_name :

    Total Cloud Cover

    units :

    %

    standard_name :

    unknown

    array([[100.      , 100.      , 100.      , ..., 100.      , 100.      ,
            100.      ],
           [ 98.72461 ,  99.77539 ,  99.947266, ..., 100.      , 100.      ,
            100.      ],
           [ 99.796875,  99.84375 ,  99.8125  , ..., 100.      ,  99.99805 ,
             99.99805 ],
           ...,
           [ 99.890625,  99.83789 ,  88.103516, ...,  99.02539 ,  99.44922 ,
            100.      ],
           [ 99.98047 ,  93.453125,  49.73242 , ...,  98.03711 ,  95.6875  ,
             49.289062],
           [ 98.5332  ,  65.98242 ,  33.322266, ...,  64.30664 ,   8.40625 ,
             10.619141]], dtype=float32)

• Attributes: (7)

  GRIB_edition :

  2

  GRIB_centre :

  ecmf

  GRIB_centreDescription :

  European Centre for Medium-Range Weather Forecasts

  GRIB_subCentre :

  0

  Conventions :

  CF-1.7

  institution :

  European Centre for Medium-Range Weather Forecasts

  history :

  2022-05-11T09:27 GRIB to CDM+CF via cfgrib-0.9.10.1/ecCodes-2.25.0 with {"source": "../../../../../../../data/eodag_data/TIGGE_CF_SFC_20211101_28b0b1a0da723aba1682661621bd7b21fe12f1ea.grib", "filter_by_keys": {}, "encode_cf": ["parameter", "time", "geography", "vertical"]}

ds.tcc.plot()

<matplotlib.collections.QuadMesh at 0x7f3d39772670>

To go further, search using a custom MARS request:

Here we use a set of custom parameters corresponding to TIGGE_CF_SFC, which should result to the same request sent to ecmwf.

If you want to eodag from a MARS request obtained from https://apps.ecmwf.int/datasets/data/tigge, you can put in this ecmwf_req_params the content of your MARS request without area and date settings, which must be passed to eodag as geom, start and end parameters.

ecmwf_req_params = {
    "class": "ti",
    "dataset": "tigge",
    "expver": "prod",
    "type": "cf",
    "levtype": "sfc",
    "origin": "ecmf",
    "grid": "0.5/0.5",
    "step": 0,
    "time": "00:00",
    "param": "tcc",
}

products_from_mars_req, total_count = dag.search(
    geom={
        "latmax": 70,
        "lonmax": -130,
        "latmin": 30,
        "lonmin": -60,
    },
    start="2021-11-01",
    end="2021-11-02",
    **ecmwf_req_params
)
# title and downloadLink properties must be the same with the two request methods,
# as they are built from ECMWF request arguments
if (
    products_from_mars_req[0].properties["title"]
    == products_from_product_type[0].properties["title"]
    and products_from_mars_req[0].properties["downloadLink"]
    == products_from_product_type[0].properties["downloadLink"]
):
    print(
        "Request using productType or directly MARS parameters result to the \n",
        "same product title %s \n and downloadLink %s"
        % (
            products_from_mars_req[0].properties["title"],
            products_from_mars_req[0].properties["downloadLink"],
        )
    )

Request using productType or directly MARS parameters result to the
 same product title TIGGE_CF_SFC_20211101_28b0b1a0da723aba1682661621bd7b21fe12f1ea
 and downloadLink https://apps.ecmwf.int/datasets/data/tigge?date=2021-11-01/to/2021-11-01&area=70.0/-130.0/30.0/-60.0&class=ti&dataset=tigge&expver=prod&grid=0.5/0.5&levtype=sfc&origin=ecmf&param=tcc&step=0&time=00:00&type=cf