The graphical system provides a flexible system to create exported data products.

This document provides some basic examples of how to use the data export interface. Changes made in the export dialog are preserved as long as CPD3 is not closed. So if you need a complex export product made repeatedly between sessions, the scripting system is likely a better way to handle that.

Basic Data Export

The most common case for data export is to simply create a file with comma separated values for a set of selected variables. To do this, start by opening the "Export Data" dialog from the "File" menu. This will open the main export data dialog with times copied from the currently visible data in the main window. Unless otherwise specified (see below) the source for data is based on the current data display mode when the dialog was openned. So, to export raw data, open the dialog from raw mode. To export hourly averaged data, first switch to averaged display mode from the "Display" menu, then open the dialog. The dialog will initially open in the input selection component.

Export data input dialog

  1. Use the "Input" button at the top is used to change the source of data to be processed. This will initially be selected when the dialog is first opened.

  2. Within the input selection, the drop down below it selects the abstract "data source" that is exported. Initially, the archive source will be selected. The default archive exported is determined by the display mode when the dialog is opened.

  3. Enter the time range to export. This will default to the currently visible data when the dialog was opened.

  4. Select the variables to export. Clicking on a variable toggles its selection, so you can simply click on the desired variables.

  5. After you have selected data, use the "Output" button near the bottom to display output parameters.

If you do not change the output mode, the data will simply be shown in another dialog. You can re-open the export dialog and it will remember the previously selected parameters, as long as the main CPD3 window has not been closed.

Export data output dialog

  1. Now the dialog should be changed so that the "Output" section is selected.

  2. Immediately below the output section, the first drop down allows you to select how the data are exported. Select "Convert or process data" so that data will be converted to CSV values, instead of CPD3’s native data format.

  3. After changing the data processing mode, the drop down below it will change to the type of processing applied. "Export" should already be selected. This component is what is used to convert data into various CSV formats. Options for conversion appear below the selected conversion type.

  4. The first option for data export is the output file. For this example, check the box next to the file name entry. If you do not check the box, then the output will be displayed in another window which you can then save.

  5. Use the "disk" button to the right of the file name entry to select the file to write to. Here you should give the file a ".csv" extension to make sure it is recognized as the correct file type.

  6. Finally press the "Ok" button at the bottom to perform the CSV data export.

After accepting the export dialog, a processing progress window may appear. Once the export is complete, the dialog will close and you can then open the file in another program.

Simple Data Averaging

Sometimes data need to be averaged before exporting. For example, exporting not-yet passed data without contamination removal (i.e. automatically flagged data kept in). To do this, the process is similar to the basic export, but involves a processing stage to export data.

Start by opening the export dialog as above. For most averaging, you will also want to include the associated flags variable for an instrument. This is required because this is how the system tracks contamination flagging. So, when exporting BsG_S11, you should also include F1_S11 to ensure proper averaging.

Average data input dialog

  1. Select the associated flags variable.

  2. Instead of selecting "Output" next, use the "Add Processing Stage" button to add a data transformation (averaging in this case).

Some variables require additiona "supporting" variables to be averaged. For example averaging raw BaG_A11 (absorption) requires L_A11 and IrG_A11. In this case, the additional variables are required because they are needed to perform difference measurements. When starting from hourly averages, these additional variables are not required: the variable has already been "flattended" by the first averaging step.

Another common case is wind speed and direction, where both are required to average either one (for vector calculation).

If a required supporting variable is missing during averaging, the "primary" one will be set to missing values.
Average data processing select

  1. Select the "avg" component to average data.

  2. Accept the added component.

Average data processing configuration

  1. The processing stage that was just added will be selected in the dialog now. You can use the buttons to the right to change the ordering of processing or to remove the processing stage.

  2. Press the "contamination" button. This will cause the checkbox to the right to be selected. The checkbox means that this option is "set" (i.e. the defaults are being overridden). The button will stay depressed and an area below the option will appear that controls the effect of the option.

  3. Enter the word "none" as described in the help text above the box. This setting disables all contamination removal from averaging, so the contamination flagged data are included.

  4. Press the "Output" button to proceed to normal data output as described above.

Since the default averaging is one hour, no further changes are required. However, you can select the averaging specifically with the "interval" option near the bottom.

There may be two scroll bars that appear on the dialog due to graphical display limitations. The inner one scrolls the options being configured (e.g. the processing stage settings). The outer one scrolls the list of stages and the input/output selection.

Complex Data Selection

Another possible data export is to include some QA/QC’ed data (clean data) but add some variables that are not part of normal passing. To do this, you can use the "Advanced" data selection.

Advanced input selection

  1. Change to the "Advanced" tab of the input selection.

  2. Press the "Add" button in the table to add two rows.

  3. Enter clean as the first archive. You may have to type it directly, since it may not appear in the list.

  4. Enter .*_S11 in the variable field. This is a regular expression indicating all data ending with _S11 (i.e. all nephelometer data).

  5. Enter raw in the second archive.

  6. Select a raw only variable, here a status temperature sensor is used. This status temperature does not normally occur in the clean data, but it can be exported along side the clean variables above. As with the clean archive, you may have to type the variable directly. Variables are case sensitive, while archives are not.

  7. Accept the export to proceed as described above.

Edited Data Export

To export edited data that have not been passed yet, you need to use an alternate "input" source for the export processing. This has a complexity in that the way edited data are generated by the system means that all "clean" and "avgh" variables are generated at once. It’s usually necessary to filter the list of outputs after generation, so that resulting file is manageable.

Edited input dialog

  1. Change the input source to "Convert or process data".

  2. From the list that appears below the input type selection, choose "generate_edited". This will appear near the bottom of the list, past all the acquisition data converters.

  3. Enter the time range to be generated.

  4. If needed, enter the station to export.

  5. If you still have the averaging stage from before, use the remove button to remove it.

  6. Add a new processing stage.

Edited add select componet

  1. Choose "select" from the component list.

  2. Accept the dialog to add the select component.

Edited select configure

  1. Use the (inner, if there are two) scroll bar to find the "include" option.

  2. Press the "include" button to activate the data include filtering.

  3. Change the the "Advanced" tab of the selection that appears below the option.

  4. Add a selection.

  5. Enter avgh in the archive field to select only hourly averages from the edited data being generated.

  6. Enter BsG_S11 to select only a single variable from all the edited data.

  7. Accept the export as above.