Software Development Kit (SDK)

Overview

The Data Manager Software Development Kit (DM SDK) allows users to access the Data Manager programmatically, bypassing the standard web interface. This opens up possibilities which would prove tedious or time-consuming to conduct on a web form. The user can automate the composition and submission of the query form and the retrieval of request status and result output.

The DM SDK comes in the form of a pip package. It runs with Python 3.8 and later and depends only on the Python Standard Library.

The workflow will involve the Session, QueryForm and Query classes. Session mediates access to the Data Manager ReST API and it allows the user to retrieve request history, details and results. QueryForm manages the composition, validation and submission of Data Manager queries. After form submission the user can access the Query object to read the request ID used to check progress or retrieve results.

Session

The Session class manages authentication and communication with the Data Manager API. It also provides methods for checking request status and retrieving results.

If you do not provide credentials when creating a Session object, you will be prompted for your username and password. Your password will not echo to the console. A token is cached at .cache/Technosylva/dmsdk in your home directory and reused while it remains valid.

Methods

Example

python

>>> from dmsdk import Session
>>> session = Session()
Data Manager username: theUserName
Data Manager password:
>>> session.get_request_state('225a9239-91ce-4a7d-9687-2fbd6ab4be06')
'FINISHED'

QueryForm

The QueryForm class handles query composition, validation, and submission. It requires a Session object to initialize. Because some fields depend on others, you should fill out the form in the same order as the web interface: select the dataset first, then variables, then spatial and temporal fields, and so on. The class will raise an exception if you violate a field dependency.

Parameters that accept multiple values (shown as plural nouns in the method names, such as aggs) expect a list, even when passing a single value: ['mean'] not 'mean'.

Methods and Fields

Dataset

Variables

Region

Statistics and Aggregations

Date and Time

Output

Submission

Valid Values

Aggregations: 'max', 'min', 'median', 'mean', 'sum', 'p.01', 'p.1', 'p1', 'p2', 'p3', 'p4', 'p5', 'p10', 'p25', 'p75', 'p90', 'p95', 'p96', 'p97', 'p98', 'p99', 'p99.9', 'p99.99'

Temporal resolutions: 'hourly', 'daily', 'monthly', 'annual', 'period'

Timezones: 'UTC', 'PST', 'MST', 'CST', 'EST'

Example

python

from dmsdk import Session, QueryForm

session = Session()
form = QueryForm(session)

form.select_dataset('dataset1')
form.get_variables()

varlist = [('NDVI', None), ('temperature 2m', 'Kelvin')]
form.select_variables(varlist)

form.get_shapefiles()
form.select_shapefile('Counties')
form.get_shape_names()
form.select_shapes(['Los Angeles', 'San Bernardino'])

form.select_spatial_aggregations(['min', 'mean'])
form.select_temporal_resolution('monthly')
form.select_temporal_aggregations(['mean'])
form.select_from_datetime('2021-01-01 00:00:00')
form.select_to_datetime('2021-03-01 00:00:00')
form.select_timezone('UTC')
form.select_output_format('csv')
form.select_description('test query')

form.submit()
query = form.get_query()

Note that the examples here and in the examples directory have sample names for the various entities (datasets, variables, shapefiles, etc) which don't correspond to objects saved in the Data Manager. The user should ensure that scripts request objects appertaining to the account used by consulting the appropriate property in the QueryForm object (eg form.variables).

Query

The Query object is returned by QueryForm.get_query(). Once a form has been submitted, Query will have a request_id field that you can pass to Session.get_request_data() or Session.get_request_state(). The Query.json() method returns the JSON representation of the query parameters.

Example Script: Batch Queries Across Shapes and Time Periods

The following script demonstrates how to run the same query across multiple shapes and time periods, wait for each result, and save it with a descriptive filename. This pattern is useful when querying large datasets that need to be broken into smaller time windows due to resource constraints.

python

import os
from datetime import datetime
from time import sleep

from dmsdk import Session, QueryForm


def compose_query_form(form, shape, start_date, end_date):
    """Compose a query form based on provided parameters."""
    form.select_dataset('dataset1')
    form.get_variables()
    form.select_variables([('temperature 2m', 'Celsius')])
    form.get_shapefiles()
    form.select_shapefile('Counties')
    form.get_shape_names()
    form.select_shapes([shape])
    form.select_temporal_resolution('daily')
    form.select_temporal_aggregations(['mean'])
    form.select_from_datetime(start_date)
    form.select_to_datetime(end_date)
    form.select_timezone('PST')
    form.select_output_format('netcdf')
    form.select_description(
        f'temp2m_{shape}_daily-mean_{start_date.year}0101-{end_date.year}0101'
    )
    return form


def submit_requests():
    """Submit queries for multiple shapes and five-year time periods."""
    if not os.path.exists('results'):
        os.mkdir('results')

    session = Session()

    for shape in ['shape1', 'shape2', 'shape3']:
        start_date = datetime(2002, 1, 1)
        while start_date < datetime(2020, 1, 1):
            end_date = datetime(start_date.year + 5, 1, 1)
            form = QueryForm(session)
            compose_query_form(form, shape, start_date, end_date)
            query = form.get_query()
            form.submit()

            while True:
                print('Waiting one minute...')
                sleep(60)
                request_state = session.get_request_state(query.request_id)
                print('Request state:', request_state)
                if request_state == 'FINISHED':
                    save_path = session.save_result(query.request_id, 'results')
                    os.rename(
                        save_path,
                        os.path.join('results', query.description + '.nc')
                    )
                    break

            start_date = end_date


if __name__ == '__main__':
    submit_requests()

Note that you do not need to wait for one query to finish before submitting the next: Data Manager queues requests automatically. The polling loop in this example is included to demonstrate how to check status and rename result files as they complete.