Individual values in CPD3 can represent both traditional time-series type data as well as much more complex ones used in configuring the system. Thus the format for expressing these values needs to encompass all this potential complexity. So, CPD3 uses a more advanced syntax for representing the data than what is normally generated through simple exporting.

Data values in CPD3 can be either simple single datum or complex tree-like structures. That is, a "value" in the context of CPD3 consists of a either a map from zero or more "names" to further values or a final single datum. For example a final datum might be a simple real number or a string of text. A map might be a "hash" consisting of a mapping from textual names to further values (either datum or further type of maps).

So the final structure of data looks like a tree of maps ending in leafs of datum. There are no balance requirements and in the simplest case the only part of the tree is just the only datum. This means that data can be described in a hierarchical fashion, with values further up in the hierarchy being maps to the ones in lower parts of it.

The supported types of single datum are:

Real numbers

A single number with possible fractional components. These can also be undefined, the meaning of which varies by context.

Integers

A single integer. These can also be undefined, the meaning of which varies by context.

Strings

A string of text. In some contexts this can also have locale specific values (meaning it may appear differently depending on the system language setting). Strings can contain any unicode character, but some contexts are not able to display all of them (e.g. a text only console).

Boolean

A value that can be either true or false.

Flags

Zero or more strings of text usually representing flags. Each string may only be present once in the flags. A string present indicates that the flag is "set".

Binary data

Zero or more bytes of unprocessed data.

Invalid

A special type of value indicating that the value is explicitly undefined and typeless. The most common occurrence of invalid values is when accessing those that are not defined at all in the hierarchy. Invalid values can also be used to undefined the entire hierarchy below them, when used as part of the transparency stack.

The second type of values are those that map from a one type to zero or more further values. The supported type of mapping values are:

Arrays

A list of contagiously indexed values. That is, an array consists of zero or more values in an ordered sequence and the map is from the position in the list to the value. In most contexts arrays are considered to be index from zero: the first element in the list is zero, the second is one, etc.

Matrix

Similar to an array but with two or more dimensions. For example, a two dimensional matrix can be thought of as an array of arrays (of equal size). Matrices do not support ragged elements. That is, all rows, columns, or further dimensions are the same size.

Hash

This is the name used by the system to represent a mapping from a string of text to values. This can also be called an associative array. A hash can also contain a default value (an empty string) that is used when a lookup is performed and does not match any explicitly defined ones. Each string of text maps to exactly one value (no duplicate keys allowed).

Keyframe

This is the name used to representing a mapping from a (defined) real number to values. Keyframes are used when the order of the numbers matters. Each real number maps to exactly one value (no duplicate keys allowed).

There is also a special type of value called an "overlay". An overlay is similar to a datum (in that it cannot itself have values under it), but instead of representing a final value, it instructs the system to insert an entire hierarchic where it is as a replacement. The overlay itself is a path name (described below) that causes the existing contents at that path to replace the overlay before evaluation. This allows for parts of the hierarchy to be reused without duplicating them in definitions.

Finally there are special types of values to represent metadata about other values. These are used primarily in the metadata archive to describe normal values. However, they can also appear in normal configuration that creates other data. For example, the acquisition configuration describing the raw values might include metadata to be written out with them. For all types of metadata there is a mapping from text to metadata values in the same way a hash maps text to values. Hash and keyframe metadata may also include value specific mappings. For example, hash metadata may have metadata values about specific text in the data hashes.

Paths and Values

In order to describe all this in a concise fashion the hierarchy of values can be described a way similar to file paths on an operating system. In this analogy the final datum values are the "files" while the series of maps leading up to them are the "directories". Continuing this analogy, a syntax similar to Unix style path names can be used. This means separating the path components with / and escaping forbidden characters (with \) as required. Using this, a single datum is described by a pair of the path (its "location") and the actual value. The syntax the system common uses has this pair separated by a comma.

Example 1. Simple path definition
/Profiles/aerosol/DisableDefaultComponents,TRUE

Datum Value Types

The simple datum types above can all be defined using strings of characters. As mentioned above, this is most commonly used in the flat text configuration syntax.

Real Numbers

Values that are real numbers are simply interpreted from a normal text representation of the number.

Example 2. Simple real number

2.3

Values support both traditional definitions as well as scientific notation (using "e" notation).

Example 3. Scientific notation

2.5E-3

In order to disambiguate real numbers from integers, the fractional part can always be used (even if it is zero).

Finally real numbers can also be set to undefined by setting them to the special value NaN or a metadata defined missing value code, if available.

Integers

Integer values are simply a single integer number interpreted from a text representation.

Example 4. Simple integer

42

Additionally values may be prefixed with (case insensitive) strings to indicate the base which is used to interpret the number.

0x

Base 16 (hexadecimal).

0o

Base 8 (octal).

0i

Base 10 (decimal).

0b

Base 2 (binary).

Example 5. Hexadecimal integer

0xAB12

In order to disambiguate integer from real numbers, the prefix 0i can be used to explicitly specify base ten.

Finally integers can also be set to undefined by setting them to the special value iNaN or a metadata defined missing value code, if available.

Text Strings

In the simplest form a text string is just zero or more unicode (UTF-8, usually) characters enclosed in double quote characters (").

Example 6. Simple string

"A string of text"

In order to allow strings that have quotes or other special characters (for example newlines), the input allows for \ C-like escaping of characters. For example \" produces a double quote character without ending the string. Similarly \n translates to a newline character.

Example 7. String with double quotes

"A string of \"quoted\" text"

A string can also have locale specific values that are substituted base on the system language setting. These are specified by prefixing the locale specific values with the locale name, after the initial default locale-less value.

Example 8. Locacle specific string

"The default value" en_US"US english only value"

Boolean

Boolean values can be either true or false. Values are case-insensitive.

TRUE, T, ON, or YES

Set the value to true.

FALSE, F, OFF, or NO

Set the value to false.

A non-zero integer

If this is not ambiguous with an integer, then set the value to true.

A zero valued integer

If this is not ambiguous with an integer, then set the value to false.

Example 9. True value

TRUE

Example 10. False value

FALSE

Flags

Flags consist of zero or more unique strings that are combined to generate a list of "set" flags. Each individual flag is separated from the others by a | character. The entire list of flags also begins with another | character.

Example 11. Set of flags

|Flag1|Flag2

Example 12. Empty flags

|

Binary Data

Binary data are represented with base-64 encoded values enclosed in a set of { and }.

Example 13. Binary data value

{QmluYXJ5IGRhdGE=}

Invalid Values

Explicitly invalid values can be designated with a single _ (an underscore).

Example 14. Explicit invalid value

_

Overlays

Overlays are described by a path (see below) prefixed with a ~ (a tilda character). Everything after the tilda is the path that is overlaid in place of the value.

Example 15. Overlay specification

~/Templates/Profiles/Default

Explicit Type

A final form of datum is an operation that simply sets the type of the value. This will set normal datum values to their empty or undefined state if they did not already exist or where a different type. The more common case is using it to set the type of a mapping value that has nothing under it. This is required, since such empty values would not otherwise appear in a flat representation of the hierarchy.

Explicit types are set by a value starting with = (an equals) followed by a case-insensitive name of the type. The types recognized are:

real, number, double, or d

A real number.

integer or i

An integer.

boolean, bool, or b

A boolean value.

text, string, or s

A boolean value.

binary or n

Binary data.

array or a

An array.

matrix or t

A matrix.

keyframe or t

A keyframe.

hash or h

A hash map.

flags or f

Flags.

*r, mr, metanumber, or metareal

Real number metadata.

*i , mi, metainteger, or metaint

Integer metadata.

*b, mb, metaboolean, or metabool

Boolean metadata.

*s, ms, metastring, metatext

Text string metadata.

*n, mn, or metabytes

Raw bytes data metadata.

*a, ma or metaarray

Array metadata.

*e, me, or metakeyframe

Keyframe metadata.

*h, mh, *c or metahash

Hash map metadata.

*f, mf, *l or metaflags

Flags metadata.

invalid, undefined, or undef

An explicitly undefined value. The same as a simple _ as described above.

Example 16. An empty hash value

=HASH

Note that in most cases the above accept the same prefix type values as the path syntax described below.

Hierarchy Paths

In order to designate a particular datum within the hierarchy a filesystem-like path is used as described above. The path itself consists of components separated by / characters. Each component determines the specific leaf within that element of the path. A special "component" of .. (two periods) indicates to move the path up one level. That is, the path will walk up one level when a simple .. is encountered instead of going down one level. Components of the path obey \ escaping. For example, \* is interpreted as just *. Path of just an escaped underscore (\_) is interpreted as the default (empty) hash key. This can be used to allow paths that start with reserved prefixes (see below).

In general the path begins with a / indicate the very top of the hierarchy. It is possible to use relative paths (i.e. ones that do not start with a /) in which case the path is evaluated from the parent in question. This type of syntax generally only makes sense in overlays beginning with ...

General Path

If the component of the path does not match any of the other definitions it is interpreted as a hash map key. This is the most common case and allows most paths to just be simple specifications of keys separated by / characters.

Example 17. Hash path

/A/Path/With/Hashes

Array

Array path components are integers prefixed with # designating an index (from zero) into an array.

Example 18. Array path

/#0/#1/#2

Matrix

If the path consists of one or more integers separated by :, :, or , (with escaping if required) and surrounded by [ and ] then it is a matrix path, which the indices in ascending order each starting from zero.

Example 19. Matrix path

/[0:1]/[0:1:2]

Keyframe

If the path is a real number prefixed with @ then it is a point in a keyframe map.

Example 20. Keyframe path

/@0.1/@0/@-3.5

Metadata

Metadata paths are similar to general paths, except that they contain a prefix starting with * and a single character indicating the metadata type.

The supported prefix types are:

*r

Real number metadata.

*i

Integer metadata.

*b

Boolean metadata.

*s

String metadata.

*n

Binary value metadata.

*a

Array metadata.

*t

Matrix metadata.

*e

Keyframe metadata.

*h

Hash map metadata.

*f

Flags metadata.

*c

Specific hash value metadata.

*l

Specific flag metadata.

Example 21. Metadata path

/*hHashMetadata/*cHashValueMetadata/*rFormat

Additionally a single ^ can be used to specify "any" type of metadata. This is not normally used in the internal syntax, but can be used for user specification when the incoming type of metadata is not known ahead of time.

Example 22. Generic metadata

/SomePrefix/^AnyMetadataType