CPD3 Ingest for CR1000 logged GMD met data

This describes the conversion and configuration used to ingest the met data logged with the new CR1000 system.

In this context time formats are any that are accepted using the standard time conversion. An empty field specifies an infinite bound (either the the very start or very end of data, depending on the field). This means that both standard ISO8601 times (e.x. “2016-09-01T19:18:01Z”) and that “Excel” style (e.x. “2016-09-01 19:18:01”) are accepted. Unless otherwise specified in the times, the time zone is assumed to be UTC.

All files are comma separated values, with standard quoting and escaping. This means that fields with embedded commas can have quotes placed around them to treat them as a single field. Quotes can be escaped by tripling them (e.x. “A value with ”““quotes””“ goes here”). This applies to both the data files and the configuration files.

In the context of the configuration files, lines starting with a “#” are ignored (comments). Currently the system ignores any trailing fields after the significant ones. This allows those fields to be used as comments or notes.

Configuration files are stored in the met file store, under the directory “config/cr1000”. So the visible path on NFS file shares is “/met/config/cr1000”, all further locations below are relative to this prefix.

The configuration system used consists of a series of overlays with time bounds, generated for each line. Each overlay is applied to the existing configuration for the parts it affects and the times it is active for. By analogy, think of a stack of transparencies on an overhead projector, each one having part of the final picture produced by light shining through them. In this analogy, the first (earlier in the file order) lines are the lower transiencies, with subsequent ones able to “overwrite” what they set.

When loading a station configuration the first file loaded is “global.cfg”. This file is loaded for all stations and contains non-station specific settings (e.x. the location of the logger serial number).

After loading the global configuration the station specific configuration is then loaded, for example “sum.cfg”. This sets station specific settings (e.x. the column locations). It should also set any disambiguation description metadata (e.x. the sensor heights for multiple wind birds).

At this point the ingest begins processing the input data. When reading the input data sensor serial numbers are determined from the station configuration and the current contents of the file. This serial number is combined with sensor type to load the sensor configuration.

When loading the sensor configuration a similar path to the station configuration is used. First the file “sensors/$TYPE/default.cfg” (where “$TYPE” is the sensor type specified in the station configuration) is loaded, then a file with the serial number is loaded at “sensors/$TYPE/$SERIAL.cfg” (where “$SERIAL” is the serial number). In general “default.cfg” should contain the “standard” parameters, while the serial number specific one contains the actual measured calibration for that specific sensor.

Lines in the station and global configuration take the form of:

Start,End,Parameter,Type[,fields...]

The start and end are the time bounds that the line is effective for. The parameter mostly follows CPD3 naming conventions but is one of:

• LOGGER - Specifies the logger serial number.
• Tx - Declares the logger panel temperature (C).
• Vx - Declares the logger panel voltage (V).
• WS - Declares a wind speed (m/s).
• WD - Declares a wind direction (degrees true).
• WSx - Declares a wind gust speed (m/s).
• P - Declares a pressure (hPa).
• T - Declares a temperature (C).
• TD - Declares a dewpoint temperature (C).
• U - Declares a relative humidity (%).
• WI - Declares a precipitation rate (mm/h).

All the variable declarations can also accept an instance number (e.x. “WS1”). Winds, temperatures, dewpoints, and humidities are assumed to have one by default, with the “1” instance being the “primary” one (for winds this is 10m one, for temperatures this is the one nearest to the humidity/dewpoint sensor).

If the parameter ends with “:metadata” the line sets metadata about the variable instead of its input configuration. Currently the only type of metadata accepted it “description”, which sets a string appended to the end of the description used to disambiguate multiple duplicate measurements. It is not required for cases where there is only a single measurement of a parameter.

The field contents and the serial number lookup behaviour depend on the type specified.

Sensor configuration files are similar to the station ones, with each line having the form:

Start,End,[Parameters...]

The interpretation of the parameters depends on the type specified.

If the type does not reference on of the below ones, it is assumed to be a general sensor type specification. The type sets the sensor type as described above (after conversion to lower case).

The first field after the type is the serial number specification, if this starts with “@” then it specifies the serial number literally. This can be used to override an erroneous logger setting temporarily. For example setting this to “@1234” sets the serial number to “1234”. Otherwise this field specifies the field index (starting from one) to look up the serial number in.

The next field specifies the field number (starting from one) to look up data values from.

So the final line for a default typed parameter setting looks like:

Start,End,Parameter,TYPE,SERIAL,VALUE

In the sensor configuration files, only a single parameter is interpreted, with the first one being the calibration polynomial to apply. This polynomial is specified in ascending power order (e.x. the first field is the offset, the second is the slope, etc). The coefficients are separated by “:”, “;”, or “,” (note that if using a “,” the whole field must be quoted). So to specify the polynomial 1+2x+3x^2, the field would be “1:2:3”. Empty or omitted calibrations are considered to be identity.

So the final line for a the sensor configuration looks like:

Start,End,CALIBRATION

This is the type used for “ventus” or “rmy” wind sensors. The serial number functions as described by the default type. These sensors accept two calibration fields, the first being calibration applied to the speed and the second the one applied to the direction. Each calibration is specified as described above.

So the final line for a the sensor configuration looks like:

Start,End,SPEEDCALIBRATION,DIRECTIONCALIBRATION

This specifies an rtd temperature sensor. The serial number functions as described by the default type. The first field in the sensor configuration is the calibration applied to the converted temperature (in degrees C), as described above. That is, this calibration is applied AFTER the Callendar-Van Dusen calculation. The second field is the parameters to the Callendar-Van Dusen equation, in the order: a, d, e, g, h, i, j. Like the calibration these constants are a single field separated by “:”, “;”, or “,” (note that if using a “,” the whole field must be quoted).

So the final line for a rtd sensor configuration looks like:

Start,End,CALIBRATION,CVDPARAMETERS

This specifies a literal calibration without looking up a serial number. This can be used to copy values directly from the file without needing a sensor type. The first field after the type is the field index (starting from one) of the value and the second is the calibration applied to it (as described above). The calibration can be omitted to apply identity.

So the final line for a calibration typed parameter setting looks like:

Start,End,Parameter,calibration,VALUE,COEFFICIENTS

This specifies a literal string copy without conversion to a number. This is generally only used in specialized cases (e.x. the logger serial number). The first field after the type is the field index (starting from one) of the value.

So the final line for a string typed parameter setting looks like:

Start,End,Parameter,string,VALUE