sensortoolkit.testing_attrib_objs._referencemonitor.ReferenceMonitor

class ReferenceMonitor(data_source=None, site_name=None, site_id=None)[source]

Bases: object

Object for storing and accessing reference monitor (FRM/FEM) data as well as instrument and monitoring site attributes.

Parameters
  • data_source (str, optional) –

    The name of the data service that reference measurements were acquired from. Defaults to None.

    Options:

    • airnow: Reference data from the AirNow API

    • aqs: Reference data from U.S. EPA’s Air Quality System

    • airnowtech: Reference data from AirNowTech, files downloaded locally.

    • local: A catch-all category for datasets stored locally on the user’s system.

  • site_name (str, optional) – The name of the air monitoring site. Defaults to None.

  • site_id (str, optional) – The unique identification number for the air monitoring site. Typically the AQS Site ID if applicable. Defaults to None.

Raises

ValueError – If both the site name and site ID are not None but the data source is None, raise ValueError as the source must be specified to indicate where reference data are located. Also, a ValueError will be raised if the passed data source, site name, and site id do not point to a valid reference data subdirectory.

Methods

get_method_name

Return the name of the reference method (instrument name) for the passed parameter.

load_data

Load reference data from locally acquired files.

query_airnow

Send a data query to the AirNow API.

query_aqs

Send a data query to the AQS API.

reference_setup

Interactive method for configuring reference data ingestion.

Attributes

data_source

default_proj_path

project_path

get_method_name(param)[source]

Return the name of the reference method (instrument name) for the passed parameter.

Parameters

param (str) – The parameter for which the reference monitor name will be retrieved.

Returns

Name of the reference monitor used to collect measurements for the passed parameter.

Return type

method_name (str)

load_data(bdate, edate, param_list, path=None, met_data=True)[source]

Load reference data from locally acquired files.

Parameters
  • bdate (str) – The beginning date (“YYYY-MM-DD” format) for the sensor testing period.

  • edate (str) – The ending date (“YYYY-MM-DD” format) for the sensor testing period.

  • param_list (list) – A list of SDFS parameters measured by reference monitors at the monitoring site for which data will be loaded.

  • path (str, optional) –

    Full path to the data directory where processed reference datasets are located. If None (default), will attempt to construct path from an existing setup configuration if the sensortoolkit.ReferenceMonitor.reference_setup() method was run prior to calling load_data().

    Note

    If path is not null (i.e., a valid directory path is passed) and a reference monitor configuration was created with sensortoolkit.ReferenceMonitor.reference_setup() prior to running load_data(), the path will point to datasets located at the indicated path instead of datasets pointed to by the path configured by the reference_setup() method.

  • met_data (bool, optional) – If true, meteorological data will be loaded in addition to datasets corresponding to parameters passed to param_list. Defaults to true (U.S. EPA’s documents for recommended performance testing protocols, metrics, and target values encourage users to report meteorological conditions for sensor performance evaluations and reports).

Returns

None.

query_airnow(key, param_list, bdate, edate, bbox=None, bbox_size=0.01)[source]

Send a data query to the AirNow API.

Parameters
  • key (str) – The API authentication key code.

  • param_list (list) – A list of SDFS parameters measured by reference monitors at the monitoring site for which data will be loaded.

  • bdate (str) – The beginning date (“YYYY-MM-DD” format) for the sensor testing period.

  • edate (str) – The ending date (“YYYY-MM-DD” format) for the sensor testing period.

  • bbox (dict, optional) –

    A bounding box of coordinates within which data will be queried. Defaults to None. bbox accepts four key-value pair entries, where the following keys should be specified:

    • 'minLat': The northern-most latitude for the bounding box.

    • 'maxLat': The southern-most latitude for the bounding box.

    • 'minLong': The western-most longitude for the bounding box.

    • 'maxLong': The eastern-most longitude for the bounding box.

    Values for each key should be type float.

  • bbox_size (float, optional) – DESCRIPTION. Defaults to 0.01.

Returns

None.

query_aqs(username, key, param_list, bdate, edate, site_id=None, query_met_data=True)[source]

Send a data query to the AQS API.

Parameters
  • username (str) – The email account registered with the API service.

  • key (str) – The API authentication key code.

  • param_list (list) – A list of SDFS parameters measured by reference monitors at the monitoring site for which data will be loaded.

  • bdate (str) – The beginning date (“YYYY-MM-DD” format) for the sensor testing period.

  • edate (str) – The ending date (“YYYY-MM-DD” format) for the sensor testing period.

  • site_id (dict, optional) –

    The AQS site ID for the air monitoring site from which reference measurements will be returned by the API. The site ID is passed as a dictionary with three key-value pairs:

    • 'state': The two digit FIPS code for the state (leading zero included).

    • 'county': The three digit FIPS code for the county (leading zeros included) located within the state.

    • 'site': The four digit AQS site number within the county (leading zeros included).

    Note

    If site_id is not null, the site ID passed will be used instead of any previously configured reference monitor configuration via the sensortoolkit.ReferenceMonitor.reference_setup() method.

  • query_met_data (bool, optional) – If true, meteorological data for temperature and relative humidity measurements will be queried in addition to parameters passed to param_list. Defaults to true (U.S. EPA’s documents for recommended performance testing protocols, metrics, and target values encourage users to report meteorological conditions for sensor performance evaluations and reports).

Returns

None.

reference_setup()[source]

Interactive method for configuring reference data ingestion.

Setup for Local Datasets:

  1. Selecting the Reference Data Service or Source:

    Choose the service or source from which reference data were acquired. Choose from the following options:

    • 'local': Data files aqcuired locally (e.g., local transfer from agency overseeing reference instrumentation at air monitoring site).

    • 'airnowtech': User has downloaded files from the AirNowTech system and has saved files locally to the user’s system.

    • 'aqs': User will query EPA’s Air Quality System (AQS) API for reference data.

    • 'airnow': User will query the AirNow API for reference data.

  2. Adding Monitoring Site Information:

    Users input information about the ambient monitoring site at which the collocation study was conducted.

  3. Selecting File Data Type:

    Choose the corresponding data file type for recorded reference datasets from '.csv', '.txt', '.xlsx'.

  4. Selecting Data Files:

    Choose the selection scheme for pointing to recorded data files from the following options:

    • 'directory', which will locate and copy all of the data files in the specified directory for the indicated data type

    • 'recursive directory', which will locate and copy all data files within the specified directory and any subdirectories contained within the indicated folder path

    • 'files' which copies over files that the user manually selects within a directory.

  5. Copying Data Files:

    Recorded reference datasets are copied from the selected file or folder location to the ../data/reference_data/[reference_source]/[site_name]_[site_id]/raw directory path.

  6. Selecting the Column Header Index:

    Users indicate the integer index position for the row of header data in reference datasets.

  7. Parsing Reference Datasets:

    The first few rows of recorded reference datasets located in the ../data/reference_data/[reference_source]/[site_name]_[site_id]/raw directory path are imported and the names of column headers are located based on the indicated head index. A list of unique column headers is stored for subsequent reassignment of column header names.

  8. Specifying Timestamp Columns:

    Users indicate the column(s) containing date/timestamp information.

  9. Specifying the Parameter Renaming Scheme and Monitor Information:

    Users indicate the SDFS parameters corresponding to column names discovered in step 5. This creates a parameter renaming dictionary for reassigning the names of header labels. Details about the reference monitors used to measure the selected SDFS parameters are also indicated at this step.

  10. Configuring Timestamp Column Formatting:

    Users indicate the date/time formatting for date/time column(s) indicated in step 6. Formatting should correspond to the strftime formatting keywords located at https://strftime.org/

  11. Specifying the DateTime Index Time Zone:

    Users indicate the time zone associated with the date/time column(s). Timezones should be valid timezone names recognized by the pytz library.

  12. Saving the Setup Configuration to setup.json:

    The setup configuration specified by the user is saved to a setup.json file for subsequent use by the ingestion module for importing recorded reference datasets and conversion to SDFS datasets.

  13. Reference Data Ingestion and Saving Processed Datasets:

    Recorded reference datasets are ingested via the configuration specified and processed version of these datasets that have been converted to SDFS format are saved as comma-separated value files to the /data/reference_data/[source]/processed/[site_name]_[site_id].

Setup for AirNowTech Datsets:

Setup process is the same as steps #1-5 for the local data file setup process. Subsequently, these steps are followed by:

  1. Pre-processing AirNowTech Datasets:

    Local AirNowTech files that the user selected in the previous step are imported and processed versions of these datasets are ingested into the SDFS format via the sensortoolkit.reference.preprocess_airnowtech() method. Processed datasets are subsequently saved as comma-separated value files to /data/reference_data/airnowtech/processed/[site_name]_[site_id]

  2. Saving the Setup Configuration to setup.json:

    The setup configuration specified by the user is saved to a setup.json file.

Setup for Querying the AQS API

Setup process is the same as steps 1 and 2 for the local data file setup process.

Important

Users must provide a site AQS ID in step 2 to successfully query the API service.

The setup configuration is subsequently saved to a reference_setup.json. Users should then query the AQS API via the ReferenceMonitor.query_aqs() method.

Setup for Querying the AirNow API

Setup process is the same as steps 1 and 2 for the local data file setup process.

Important

In step 2, users must provide the latitude and longitude coordinates for the monitoring site where air sensors were collocated to successfully query the API service.

The setup configuration is subsequently saved to a reference_setup.json. Users should then query the AirNow API via the ReferenceMonitor.query_airnow() method.

Returns

None.