{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Get forecast data from meteoblue using EODAG\n", "\n", "In this tutorial we will show you how to use eodag to download forecast data from [meteoblue](https://content.meteoblue.com/en/business-solutions/weather-apis/dataset-api). The provider configuration and this tutorial have been developed in the context of DOMINO-X." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import datetime\n", "from eodag import EODataAccessGateway, setup_logging\n", "\n", "setup_logging(1) # 0: nothing, 1: only progress bars, 2: INFO, 3: DEBUG\n", "dag = EODataAccessGateway()\n", "dag.set_preferred_provider(\"meteoblue\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Search (get data availability and build download request)\n", "\n", "There are two use-cases, a search for a product already configured in EODAG, or a search for a dataset not already configured, where you will have a little more to do.\n", "\n", "### 1.a. Search from an existing product type:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1 product built EOProduct(id=NEMSGLOBAL_TCDC_20230107_df9666f4d5ad8a5bfad24e6eca78475b2359139e, provider=meteoblue),\n", " using queries=[{'domain': 'NEMSGLOBAL', 'gapFillDomain': None, 'timeResolution': 'daily', 'codes': [{'code': 71, 'level': 'sfc', 'aggregation': 'mean'}]}]\n", "\n" ] } ], "source": [ "tomorrow = (datetime.date.today() + datetime.timedelta(days=1)).isoformat()\n", "after_tomorrow = (datetime.date.today() + datetime.timedelta(days=2)).isoformat()\n", "aoi_bbox = [-5, 40, 10, 45]\n", "\n", "products_from_product_type, total_count = dag.search(\n", " start=tomorrow,\n", " end=after_tomorrow,\n", " geom=aoi_bbox,\n", " productType=\"NEMSGLOBAL_TCDC\",\n", ")\n", "print(\n", " \"%s product built %s,\\n using queries=%s\\n\"\n", " % (\n", " total_count,\n", " products_from_product_type[0],\n", " products_from_product_type[0].properties[\"queries\"],\n", " )\n", ")" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "('OFFLINE', 11544)" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "products_from_product_type[0].properties[\"storageStatus\"], products_from_product_type[0].properties[\"datapoints\"]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that the product is `OFFLINE`, which means that it is not directly avalaible for download and will need to be ordered first.\n", "\n", "We also displayed the `datapoints` property (credits), needed for this download (search does not consume datapoints)." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### 1.b. Search using a custom request:\n", "\n", "Here we use a set of custom parameters corresponding to `NEMSGLOBAL_TCDC`, which should result to the same request sent to `meteoblue`.\n", "\n", "You can compose your own request using meteoblue documentation: [dataset API](https://docs.meteoblue.com/en/weather-apis/dataset-api/dataset-api), [available datasets/models](https://docs.meteoblue.com/en/meteo/data-sources/datasets), [available variables](https://docs-staging.meteoblue.com/en/meteo/variables/weather-variables), or also meteoblue [dataset API configurator](https://www.meteoblue.com/en/weather-api/dataset-api/index)." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Request using productType or directly meteoblue query result to the\n", " same downloadLink https://my.meteoblue.com/dataset/query?{\"format\": \"netCDF\", \"geometry\": {\"coordinates\": [[[-5.0, 40.0], [-5.0, 45.0], [10.0, 45.0], [10.0, 40.0], [-5.0, 40.0]]], \"type\": \"Polygon\"}, \"queries\": [{\"codes\": [{\"aggregation\": \"mean\", \"code\": 71, \"level\": \"sfc\"}], \"domain\": \"NEMSGLOBAL\", \"gapFillDomain\": null, \"timeResolution\": \"daily\"}], \"timeIntervals\": [\"2023-01-07/2023-01-07\"], \"units\": {\"energy\": \"watts\", \"length\": \"metric\", \"temperature\": \"C\", \"velocity\": \"km/h\"}}\n" ] } ], "source": [ "meteoblue_req_params = {\n", " \"queries\":[\n", " {\n", " \"domain\":\"NEMSGLOBAL\",\"gapFillDomain\":None,\"timeResolution\":\"daily\",\n", " \"codes\":[{\"code\":71,\"level\":\"sfc\",\"aggregation\":\"mean\"}],\n", " }\n", " ],\n", " \"format\": \"netCDF\",\n", " \"units\":{\"temperature\":\"C\",\"velocity\":\"km/h\",\"length\":\"metric\",\"energy\":\"watts\"},\n", " \"timeIntervalsAlignment\": None,\n", "}\n", "\n", "products_from_custom_req, total_count = dag.search(\n", " start=tomorrow,\n", " end=after_tomorrow,\n", " geom=aoi_bbox,\n", " **meteoblue_req_params,\n", ")\n", "# downloadLink property must be the same with the two request methods,\n", "# as they are built from the same custom request arguments\n", "if (\n", " products_from_custom_req[0].properties[\"downloadLink\"]\n", " == products_from_product_type[0].properties[\"downloadLink\"]\n", "):\n", " print(\n", " \"Request using productType or directly meteoblue query result to the\\n\",\n", " \"same downloadLink %s\"\n", " % (\n", " products_from_custom_req[0].properties[\"downloadLink\"],\n", " )\n", " )" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Download product\n", "\n", "- download performed using meteoblue credentials set in `~/.config/eodag/eodag.yml` as for other EO providers:\n", "```yaml\n", " meteoblue:\n", " priority:\n", " search:\n", " auth:\n", " credentials:\n", " apikey: my-meteoblue-api-key\n", " download:\n", " outputs_prefix: /data/eodag_data\n", "```\n", "- as the product is OFFLINE, download will be performed in 2 times:\n", " - first order\n", " - then loop to try downloading as soon as order is available" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "bff4cd737daa4e85b22b40afc9146a99", "version_major": 2, "version_minor": 0 }, "text/plain": [ "0.00B [00:00, ?B/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "[Retry #1] Waiting 1s until next download try for ordered product (retry every 0.016666666666666666' for 10')" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "'/data/eodag_data/NEMSGLOBAL_TCDC_20230107_df9666f4d5ad8a5bfad24e6eca78475b2359139e.nc'" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "product_path = dag.download(products_from_product_type[0], wait=1/60, timeout=10)\n", "product_path" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Open dataset with `xarray` and plot over a map using `folium`" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import xarray as xr\n", "import folium\n", "from folium import plugins" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:            (x: 480, y: 1, time: 1)\n",
       "Coordinates:\n",
       "  * time               (time) float64 1.673e+09\n",
       "Dimensions without coordinates: x, y\n",
       "Data variables:\n",
       "    lat                (x, y) float32 40.33 40.33 40.33 40.33 ... 45.0 45.0 45.0\n",
       "    lon                (x, y) float32 -4.688 -4.219 -3.75 ... 8.906 9.375 9.844\n",
       "    asl                (x, y) float32 920.5 702.9 619.4 ... 363.9 332.3 167.0\n",
       "    Cloud Cover Total  (x, y, time) float32 78.71 71.71 70.96 ... 100.0 100.0\n",
       "Attributes:\n",
       "    domain:   NEMSGLOBAL
" ], "text/plain": [ "\n", "Dimensions: (x: 480, y: 1, time: 1)\n", "Coordinates:\n", " * time (time) float64 1.673e+09\n", "Dimensions without coordinates: x, y\n", "Data variables:\n", " lat (x, y) float32 40.33 40.33 40.33 40.33 ... 45.0 45.0 45.0\n", " lon (x, y) float32 -4.688 -4.219 -3.75 ... 8.906 9.375 9.844\n", " asl (x, y) float32 920.5 702.9 619.4 ... 363.9 332.3 167.0\n", " Cloud Cover Total (x, y, time) float32 78.71 71.71 70.96 ... 100.0 100.0\n", "Attributes:\n", " domain: NEMSGLOBAL" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds = xr.load_dataset(product_path)\n", "ds" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "reshaped_data = np.dstack([ds.lat[:,0], ds.lon[:,0], ds[\"Cloud Cover Total\"][:,0,0]]).reshape(len(ds.lon),3)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Make this Notebook Trusted to load map: File -> Trust Notebook
" ], "text/plain": [ "" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "fmap = folium.Map(\n", " [43.5, 5], \n", " zoom_start=6, \n", " tiles=\"stamentoner\"\n", ")\n", "\n", "fmap.add_child(plugins.HeatMap(\n", " reshaped_data.tolist(),\n", " radius=19, \n", " blur=22, \n", "))\n", "folium.LayerControl().add_to(fmap)\n", "\n", "fmap " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "eodag-dev", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10 (default, Nov 14 2022, 12:59:47) \n[GCC 9.4.0]" }, "vscode": { "interpreter": { "hash": "946c0635d97db016b4c650c0f14014bdbd70e0aaae6a8f4b82f75907772dbc16" } }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": { "26c8d99f516a41ea93ae2e6f906e1fa0": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "" } }, "3730d134a52344a38ebb2a08c47e7337": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_8f002dbbe9114b20938e09bdc2c55e9b", "style": "IPY_MODEL_532af338e28949aa98f067a61a2e0c4c", "value": " 1/1 [00:00<00:00, 149.31file/s]" } }, "532af338e28949aa98f067a61a2e0c4c": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "74ad68b8f0a84eefa62c04bf3b52a58c": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "display": "inline-flex", "flex_flow": "row wrap", "width": "100%" } }, "8f002dbbe9114b20938e09bdc2c55e9b": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "a17c710092214a258ca738b0480ea6a4": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_b715be9c03fe4026a52ee6888a74b32f", "style": "IPY_MODEL_d0de144210fc43918cff436c31068473", "value": "NEMSGLOBAL_TCDC_20230107_df9666f4d5ad8a5bfad24e6eca78475b2359139e: 100%" } }, "ac7ad84ab08d431c935f798bcf71c717": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "layout": "IPY_MODEL_b672bebdee084a2eb13e195168026927", "max": 1, "style": "IPY_MODEL_26c8d99f516a41ea93ae2e6f906e1fa0", "value": 1 } }, "b672bebdee084a2eb13e195168026927": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "flex": "2" } }, "b715be9c03fe4026a52ee6888a74b32f": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "bff4cd737daa4e85b22b40afc9146a99": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_a17c710092214a258ca738b0480ea6a4", "IPY_MODEL_ac7ad84ab08d431c935f798bcf71c717", "IPY_MODEL_3730d134a52344a38ebb2a08c47e7337" ], "layout": "IPY_MODEL_74ad68b8f0a84eefa62c04bf3b52a58c" } }, "d0de144210fc43918cff436c31068473": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } } }, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 4 }