Plugins
Contents
Plugins#
EODAG uses a two-level plugin system. Plugin topics are abstract interfaces for a specific functionality of EODAG like Search or Download. EODAG providers are implementations of at least one plugin topic. The more plugin topics are implemented by a provider, the more functionality of eodag are available for this provider.
Plugin Management#
The plugin machinery is managed by one instance of PluginManager
. The instance manager knows
how to discover plugins at runtime, using setuptools entry points mechanism.
Plugins Available#
EODAG currently advertises 5 types of plugins: Search, Download, Crunch, Authentication and Api.
The providers are implemented with a triplet of Search/Authentication/Download plugins or with an Api plugin:
Provider |
Search |
Authentication |
Download |
---|---|---|---|
aws_eos |
PostJsonSearch |
AwsAuth |
AwsDownload |
theia |
QueryStringSearch |
TokenAuth |
HTTPDownload |
peps |
QueryStringSearch |
GenericAuth |
HTTPDownload |
creodias |
ODataV4Search |
KeycloakOIDCPasswordAuth |
HTTPDownload |
mundi |
QueryStringSearch |
HTTPHeaderAuth |
S3RestDownload |
onda |
ODataV4Search |
GenericAuth |
HTTPDownload |
astraea_eod |
StacSearch |
AwsAuth |
AwsDownload |
usgs_satapi_aws |
StacSearch |
AwsAuth |
AwsDownload |
earth_search |
StacSearch |
AwsAuth |
AwsDownload |
earth_search_cog |
StacSearch |
None |
HTTPDownload |
earth_search_gcs |
StacSearch |
AwsAuth |
AwsDownload |
usgs |
UsgsApi |
UsgsApi |
UsgsApi |
ecmwf |
EcmwfApi |
EcmwfApi |
EcmwfApi |
cop_ads |
CdsApi |
CdsApi |
CdsApi |
cop_cds |
CdsApi |
CdsApi |
CdsApi |
meteoblue |
BuildPostSearchResult |
HttpQueryStringAuth |
HTTPDownload |
cop_dataspace |
ODataV4Search |
KeycloakOIDCPasswordAuth |
HTTPDownload |
planetary_computer |
StacSearch |
SASAuth |
HTTPDownload |
hydroweb_next |
StacSearch |
HTTPHeaderAuth |
HTTPDownload |
Creating Plugins#
EODAG plugins can be separated from the core eodag package. They live in your Python site-packages as standalone Python
packages or modules, with eodag’s specific entry points that will be looked at by the plugin manager to load plugins at
runtime. These entry points are: eodag.plugins.{topic}
, with possible values for topic
being api
, search
,
crunch
, auth
and download
.
All you have to do to create eodag plugins is start a new Python project, add eodag as a dependancy to your project,
and then create the plugin type you want in Python module(s) by importing its base class from eodag.plugins.{topic}.base
and subclassing it. Here is an example for an Api
plugin class located in a module
named your_package.your_module
:
from eodag.plugins.apis.base import Api
class SampleApiPlugin(Api):
def query(self):
pass
def download(self):
pass
Then, in the setup.py of your Python project, add this:
setup(
...
entry_points={
...
'eodag.plugins.api': [
'SampleApiPlugin = your_package.your_module:SampleApiPlugin'
],
...
},
...
)
See what the PyPa explains to better understand this concept. In EODAG, the name you give to your plugin in the setup.py script’s entry point doesn’t matter, but we prefer it to be the same as the class name of the plugin. What matters is that the entry point must be a class deriving from one of the 5 plugin topics supported. Be particularly careful with consistency between the entry point name and the super class of you plugin class. Here is a list of entry point names and the plugin topic to which they map:
‘eodag.plugins.api’ :
Api
‘eodag.plugins.auth’ :
Authentication
‘eodag.plugins.crunch’ :
Crunch
‘eodag.plugins.download’ :
Download
‘eodag.plugins.search’ :
Search
As an example of external plugin, you can have a look to eodag-sentinelsat.
Provider configuration#
The plugin structure is reflected in the internal providers configuration file. Here is a sample:
provider_name:
priority: 1
products:
# List of supported product types
# This is a mapping containing all the information required by the search plugin class to perform its job.
# The mapping is available in the config attribute of the search plugin as config['products']
S2_MSI_L1C:
a-config-key-needed-by-search-plugin-to-search-this-product-type: value
another-config-key: another-value
# Whether this product type is partially supported by this provider (the provider does not contain all the
# products of this type)
partial: True
...
search:
plugin: CustomSearchPluginClass
api_endpoint: https://mandatory.config.key/
a-key-conf-used-by-the-plugin-class-init-method: value
another-random-key: random-value
# A mapping between the search result of the provider and the eodag way of describing EO products (the keys are
# the same as in the OpenSearch specification)
metadata_mapping:
...
...
download:
plugin: CustomDownloadPlugin
# Same as with search for random config keys as needed by the plugin class
...
auth:
plugin: CustomAuthPlugin
# Same as with search for random config keys as needed by the plugin class
...
Note however, that for a provider which already has a Python library for accessing its products, the configuration varies a little bit. It does not have the ‘search’ and ‘download’ keys. Instead, there is a single ‘api’ key like this:
provider_name:
...
api:
plugin: ApiPluginClassName
...
An EOProduct
has a properties
attribute
which is built based on how its metadata are set in the provider configuration. The following converters can be used to
transform the values collected from the provider:
- metadata_mapping.format_metadata(*args, **kwargs)#
Format a string of form {<field_name>#<conversion_function>}
- The currently understood converters are:
datetime_to_timestamp_milliseconds
: converts a utc date string to a timestamp in millisecondsto_rounded_wkt
: simplify the WKT of a geometryto_bounds_lists
: convert to list(s) of boundsto_nwse_bounds
: convert to North,West,South,East boundsto_nwse_bounds_str
: convert to North,West,South,East bounds string with given separatorto_geojson
: convert to a GeoJSON (via __geo_interface__ if exists)from_ewkt
: convert EWKT to shapely geometry / WKT in DEFAULT_PROJto_ewkt
: convert to EWKT (Extended Well-Known text)from_georss
: convert GeoRSS to shapely geometry / WKT in DEFAULT_PROJcsv_list
: convert to a comma separated listto_iso_utc_datetime_from_milliseconds
: convert a utc timestamp in given milliseconds to a utc iso datetimeto_iso_utc_datetime
: convert a UTC datetime string to ISO UTC datetime stringto_iso_date
: remove the time part of a iso datetime stringremove_extension
: on a string that contains dots, only take the first part of the list obtained by splitting the string on dotsget_group_name
: get the matching regex group namereplace_str
: execute “string”.replace(old, new)recursive_sub_str
: recursively substitue in the structure (e.g. dict) values matching a regexslice_str
: slice a string (equivalent to s[start, end, step])fake_l2a_title_from_l1c
: used to generate SAFE format metadata for data from AWSs2msil2a_title_to_aws_productinfo
: used to generate SAFE format metadata for data from AWS
For example:
search:
...
metadata_mapping:
publicationDate: '{$.data.timestamp#to_iso_utc_datetime_from_milliseconds}'
...