Individual values in CPD3 can represent both traditional time-series type data as well as much more complex ones used in configuring the system. Thus the format for expressing these values needs to encompass all this potential complexity. So, CPD3 uses a more advanced syntax for representing the data than what is normally generated through simple exporting.
Data values in CPD3 can be either simple single datum or complex tree-like structures. That is, a "value" in the context of CPD3 consists of a either a map from zero or more "names" to further values or a final single datum. For example a final datum might be a simple real number or a string of text. A map might be a "hash" consisting of a mapping from textual names to further values (either datum or further type of maps).
So the final structure of data looks like a tree of maps ending in leafs of datum. There are no balance requirements and in the simplest case the only part of the tree is just the only datum. This means that data can be described in a hierarchical fashion, with values further up in the hierarchy being maps to the ones in lower parts of it.
The supported types of single datum are:
- Real numbers
-
A single number with possible fractional components. These can also be undefined, the meaning of which varies by context.
- Integers
-
A single integer. These can also be undefined, the meaning of which varies by context.
- Strings
-
A string of text. In some contexts this can also have locale specific values (meaning it may appear differently depending on the system language setting). Strings can contain any unicode character, but some contexts are not able to display all of them (e.g. a text only console).
- Boolean
-
A value that can be either true or false.
- Flags
-
Zero or more strings of text usually representing flags. Each string may only be present once in the flags. A string present indicates that the flag is "set".
- Binary data
-
Zero or more bytes of unprocessed data.
- Invalid
-
A special type of value indicating that the value is explicitly undefined and typeless. The most common occurrence of invalid values is when accessing those that are not defined at all in the hierarchy. Invalid values can also be used to undefined the entire hierarchy below them, when used as part of the transparency stack.
The second type of values are those that map from a one type to zero or more further values. The supported type of mapping values are:
- Arrays
-
A list of contagiously indexed values. That is, an array consists of zero or more values in an ordered sequence and the map is from the position in the list to the value. In most contexts arrays are considered to be index from zero: the first element in the list is zero, the second is one, etc.
- Matrix
-
Similar to an array but with two or more dimensions. For example, a two dimensional matrix can be thought of as an array of arrays (of equal size). Matrices do not support ragged elements. That is, all rows, columns, or further dimensions are the same size.
- Hash
-
This is the name used by the system to represent a mapping from a string of text to values. This can also be called an associative array. A hash can also contain a default value (an empty string) that is used when a lookup is performed and does not match any explicitly defined ones. Each string of text maps to exactly one value (no duplicate keys allowed).
- Keyframe
-
This is the name used to representing a mapping from a (defined) real number to values. Keyframes are used when the order of the numbers matters. Each real number maps to exactly one value (no duplicate keys allowed).
There is also a special type of value called an "overlay". An overlay is similar to a datum (in that it cannot itself have values under it), but instead of representing a final value, it instructs the system to insert an entire hierarchic where it is as a replacement. The overlay itself is a path name (described below) that causes the existing contents at that path to replace the overlay before evaluation. This allows for parts of the hierarchy to be reused without duplicating them in definitions.
Finally there are special types of values to represent metadata about other values. These are used primarily in the metadata archive to describe normal values. However, they can also appear in normal configuration that creates other data. For example, the acquisition configuration describing the raw values might include metadata to be written out with them. For all types of metadata there is a mapping from text to metadata values in the same way a hash maps text to values. Hash and keyframe metadata may also include value specific mappings. For example, hash metadata may have metadata values about specific text in the data hashes.
Paths and Values
In order to describe all this in a concise fashion the hierarchy of values can be described a way similar to file paths on an operating system.
In this analogy the final datum values are the "files" while the series of maps leading up to them are the "directories".
Continuing this analogy, a syntax similar to Unix style path names can be used.
This means separating the path components with /
and escaping forbidden characters (with \
) as required.
Using this, a single datum is described by a pair of the path (its "location") and the actual value.
The syntax the system common uses has this pair separated by a comma.
/Profiles/aerosol/DisableDefaultComponents,TRUE
Datum Value Types
The simple datum types above can all be defined using strings of characters. As mentioned above, this is most commonly used in the flat text configuration syntax.
Real Numbers
Values that are real numbers are simply interpreted from a normal text representation of the number.
2.3
Values support both traditional definitions as well as scientific notation (using "e" notation).
2.5E-3
In order to disambiguate real numbers from integers, the fractional part can always be used (even if it is zero).
Finally real numbers can also be set to undefined by setting them to the special value NaN
or a metadata defined missing value code, if available.
Integers
Integer values are simply a single integer number interpreted from a text representation.
42
Additionally values may be prefixed with (case insensitive) strings to indicate the base which is used to interpret the number.
0x
-
Base 16 (hexadecimal).
0o
-
Base 8 (octal).
0i
-
Base 10 (decimal).
0b
-
Base 2 (binary).
0xAB12
In order to disambiguate integer from real numbers, the prefix 0i
can be used to explicitly specify base ten.
Finally integers can also be set to undefined by setting them to the special value iNaN
or a metadata defined missing value code, if available.
Text Strings
In the simplest form a text string is just zero or more unicode (UTF-8, usually) characters enclosed in double quote characters ("
).
"A string of text"
In order to allow strings that have quotes or other special characters (for example newlines), the input allows for \
C-like escaping of characters.
For example \"
produces a double quote character without ending the string.
Similarly \n
translates to a newline character.
"A string of \"quoted\" text"
A string can also have locale specific values that are substituted base on the system language setting. These are specified by prefixing the locale specific values with the locale name, after the initial default locale-less value.
"The default value" en_US"US english only value"
Boolean
Boolean values can be either true or false. Values are case-insensitive.
TRUE
,T
,ON
, orYES
-
Set the value to true.
FALSE
,F
,OFF
, orNO
-
Set the value to false.
- A non-zero integer
-
If this is not ambiguous with an integer, then set the value to true.
- A zero valued integer
-
If this is not ambiguous with an integer, then set the value to false.
TRUE
FALSE
Flags
Flags consist of zero or more unique strings that are combined to generate a list of "set" flags.
Each individual flag is separated from the others by a |
character.
The entire list of flags also begins with another |
character.
|Flag1|Flag2
|
Binary Data
Binary data are represented with base-64 encoded values enclosed in a set of {
and }
.
{QmluYXJ5IGRhdGE=}
Invalid Values
Explicitly invalid values can be designated with a single _
(an underscore).
_
Overlays
Overlays are described by a path (see below) prefixed with a ~
(a tilda character).
Everything after the tilda is the path that is overlaid in place of the value.
~/Templates/Profiles/Default
Explicit Type
A final form of datum is an operation that simply sets the type of the value. This will set normal datum values to their empty or undefined state if they did not already exist or where a different type. The more common case is using it to set the type of a mapping value that has nothing under it. This is required, since such empty values would not otherwise appear in a flat representation of the hierarchy.
Explicit types are set by a value starting with =
(an equals) followed by a case-insensitive name of the type.
The types recognized are:
real
,number
,double
, ord
-
A real number.
integer
ori
-
An integer.
boolean
,bool
, orb
-
A boolean value.
text
,string
, ors
-
A boolean value.
binary
orn
-
Binary data.
array
ora
-
An array.
matrix
ort
-
A matrix.
keyframe
ort
-
A keyframe.
hash
orh
-
A hash map.
flags
orf
-
Flags.
*r
,mr
,metanumber
, ormetareal
-
Real number metadata.
*i
,mi
,metainteger
, ormetaint
-
Integer metadata.
*b
,mb
,metaboolean
, ormetabool
-
Boolean metadata.
*s
,ms
,metastring
,metatext
-
Text string metadata.
*n
,mn
, ormetabytes
-
Raw bytes data metadata.
*a
,ma
ormetaarray
-
Array metadata.
*e
,me
, ormetakeyframe
-
Keyframe metadata.
*h
,mh
,*c
ormetahash
-
Hash map metadata.
*f
,mf
,*l
ormetaflags
-
Flags metadata.
invalid
,undefined
, orundef
-
An explicitly undefined value. The same as a simple
_
as described above.
=HASH
Note that in most cases the above accept the same prefix type values as the path syntax described below.
Hierarchy Paths
In order to designate a particular datum within the hierarchy a filesystem-like path is used as described above.
The path itself consists of components separated by /
characters.
Each component determines the specific leaf within that element of the path.
A special "component" of ..
(two periods) indicates to move the path up one level.
That is, the path will walk up one level when a simple ..
is encountered instead of going down one level.
Components of the path obey \
escaping.
For example, \*
is interpreted as just *
.
Path of just an escaped underscore (\_
) is interpreted as the default (empty) hash key.
This can be used to allow paths that start with reserved prefixes (see below).
In general the path begins with a /
indicate the very top of the hierarchy.
It is possible to use relative paths (i.e. ones that do not start with a /
) in which case the path is evaluated from the parent in question.
This type of syntax generally only makes sense in overlays beginning with ..
.
General Path
If the component of the path does not match any of the other definitions it is interpreted as a hash map key.
This is the most common case and allows most paths to just be simple specifications of keys separated by /
characters.
/A/Path/With/Hashes
Array
Array path components are integers prefixed with #
designating an index (from zero) into an array.
/#0/#1/#2
Matrix
If the path consists of one or more integers separated by :
, :
, or ,
(with escaping if required) and surrounded by [
and ]
then it is a matrix path, which the indices in ascending order each starting from zero.
/[0:1]/[0:1:2]
Keyframe
If the path is a real number prefixed with @
then it is a point in a keyframe map.
/@0.1/@0/@-3.5
Metadata
Metadata paths are similar to general paths, except that they contain a prefix starting with *
and a single character indicating the metadata type.
The supported prefix types are:
*r
-
Real number metadata.
*i
-
Integer metadata.
*b
-
Boolean metadata.
*s
-
String metadata.
*n
-
Binary value metadata.
*a
-
Array metadata.
*t
-
Matrix metadata.
*e
-
Keyframe metadata.
*h
-
Hash map metadata.
*f
-
Flags metadata.
*c
-
Specific hash value metadata.
*l
-
Specific flag metadata.
/*hHashMetadata/*cHashValueMetadata/*rFormat
Additionally a single ^
can be used to specify "any" type of metadata.
This is not normally used in the internal syntax, but can be used for user specification when the incoming type of metadata is not known ahead of time.
/SomePrefix/^AnyMetadataType