JSON Archive Proxy Reference Manual

Version 3.0.0-SNAPSHOT

Sebastian Marsching

aquenos GmbH

Table of Contents
I. Overview
II. What’s new in JSON Archive Proxy 3.x
1. JSON Archive Proxy 3.0
III. Installation
1. JSON Archive Proxy reader plug-in
2. JSON Archive Proxy server
A. JSON archive access protocol 1.0
1. Requesting the list of available archives
2. Searching for channels
3. Retrieving samples for a channel

Chapter I. Overview

The JSON Archive Proxy for CSS is intended as a simple tool for integrating archive data-sources into Control System Studio without having to install lots of dependencies (e.g. libraries for accessing database systems). Furthermore, the JSON Archive Proxy can be used for easily integrating third-party applications that can act either as data sources, providing archived PV data to CSS, or as clients that can read archived PV data from CSS data-sources.

The JSON Archive Proxy consists of two components that can be used in combination (simply acting as a proxy for an archive reader plug-in) or separately for integrating a third-party application with CSS. The first component is the JSON Archive Proxy reader plug-in that can be installed into CSS in order to access an archive source exposed through the JSON/HTTP protocol. The second component is the JSON Archive Proxy server that can expose the data sources provided by any CSS archive reader plug-in through the JSON/HTTP protocol.

These two components are completely independent and communicate with each other using JSON/HTTP only. This is why it is easy to replace either component with a different implementation thus facilitating the integration of third-party applications with CSS. One example of such a third-party application is the Cassandra PV Archiver which exposes its archive data through the JSON/HTTP protocol so that it can be accessed from CSS using the JSON Archive Proxy reader plug-in.

Chapter II. What’s new in JSON Archive Proxy 3.x

Table of Contents
1. JSON Archive Proxy 3.0

1. JSON Archive Proxy 3.0

In comparison to older versions, the JSON Archive Proxy 3.0 is now compatible with newer versions of Control System Studio (in particular versions 4.3 and 4.4). The JSON/HTTP protocol has stayed the same, so that the server and reader components can simply be mixed with older versions of the JSON Archive Proxy.

With version 3.0, the distribution format of the JSON Archive Proxy has changed. It is no longer distributed as a downloadable archive. Instead, the software is distributed through a P2 repository (Eclipse update-site). Installing the reader plug-in from the update site is both simpler and more reliable than manually fiddling with the Eclipse / CSS installation. However, it also means that in order to use the server component, one now has to build a product which adds the necessary archive reader plug-ins.

Chapter III. Installation

The installation process is different for the reader and the server component. While the reader component can simply be installed into CSS, the server component has to be used for building a custom Eclipse product.

1. JSON Archive Proxy reader plug-in

Installing the JSON Archive Proxy reader plug-in into CSS is very simple. One only has to select Install New Software... from the Help menu in CSS and point it to the URL of the update site for the JSON Archive Proxy release that is supposed to be installed. The URL of the update site can be found on the JSON Archive Proxy web-site. Next, the “JSON Archive Proxy Reader” feature has to be selected. The “JSON Archive Proxy Reader Developer Resources” feature only contains source files and does not have to be installed. It is only useful when creating a new Eclipse target platform for plug-in development. The “JSON Archive Proxy Server” feature is only intended for creating a new server product and must never be installed into CSS. This feature will not work in a regular CSS installation and might interfere with other CSS features.

The “JSON Archive Proxy Reader” feature depends on a few plug-ins that are not provided by the same repository. In particular, it depends on the CSS archive APIs and the Jackson JSON library. These plug-ins are provided by the CSS and Eclipse repositories, so the update-sites for the respective CSS and Eclipse releases should also be present in the CSS software configuration. Typically, these update sites are already present in most CSS products and do not have to be added explicitly.

After installing the plug-in, one can add a data-source for the CSS data browser. The URLs recognized by the JSON Archive Proxy reader have the form json:http://server.example.com:8080/ where in this example server.example.com is the server’s hostname and 8080 is the TCP port on which the server is listening for incoming connections. The path might differ from the path given in the example if a server does not expose the archive-access API at its root.

2. JSON Archive Proxy server

The JSON Archive Proxy server is the component that can expose any CSS archive reader plug-in through the JSON/HTTP protocol used by the JSON Archive Proxy reader plug-in. In order to be useful, the JSON Archive Proxy server plug-in has to be bundled with the archive reader plug-in that is used for accessing the underlying archive. This can be achieved by creating a custom Eclipse product that depends on both the JSON Archive Proxy server feature and the feature that provides the relevant archive reader plug-in.

Creating a custom Eclipse product is outside the scope of this document, but the source tree of the JSON Archive Proxy provides an example project using Tycho for building such a product in the com.aquenos.csstudio.json.server.product directory.

In general, for building such a product, one has to ensure that the update-sites of the Eclipse RCP release, the CSS release, and the JSON Archive Proxy release are present in the target platform used for building the product. The product definition should specify the com.aquenos.csstudio.archive.json.server.engine product and the com.aquenos.csstudio.archive.json.server.application application. In addition to the archive-reader specific features, it has to contain the com.aquenos.csstudio.archive.json.server.feature, org.eclipse.equinox.server.jetty, and the org.eclipse.equinox.core.feature features.

When using the com.aquenos.csstudio.archive.json.server.application application as the base of the product, the launcher will support two command-line parameters:

The optional -port argument takes the ports number on which the server will listen for incoming requests as its parameter. If not specified, the server will listen on port 8080 by default.

The mandatory -archive-url argument takes the URL of the underlying archive data-source as its parameter. The format of this URL depends on the archive reader plug-in being used.

Appendix A. JSON archive access protocol 1.0

The JSON-based archive access protocol is the protocol that is used for communication between the JSON Archive Proxy reader plug-in and the JSON Archive Proxy server.

The base URL used for all requests concerning the JSON-based archive-access protocol 1.0 depends on the server implementation and the deployment configuration. The JSON Archive Proxy server provides the API right at the root, so that the base URL for accessing a the JSON Archive Proxy server running on myserver.example.com and listening on port 8080 is http://myserver.example.com:8080/. This base URL has to be prepended to all URLs that are mentioned in this protocol specification. Other server implementations might choose a different path or default port. Please refer to the documentation of the respective server implementation for details.

All requests are made by specifying query parameters in the URL. The request body is always empty. The response is always sent in the JSON format (MIME type application/json) unless there is an error (which is identified by a corresponding HTTP status code). All requests are sent as GET requests.

The JSON Archive Proxy server and the JSON Archive Proxy reader plug-in support deflate and gzip compression of the response body. Other client and server implementations should also support these compression methods when possible. For JSON data, compression can dramatically reduce the amount of data that has to be transferred, so clients should support compression when possible. In any, case clients and servers supporting compression must also be compatible with clients or servers that do not support compression, falling back to uncompressed data transfer.

1. Requesting the list of available archives

Request

The request URL for retrieving the list of available archives has the following form:

/archive/[?prettyPrint]

If the optional prettyPrint parameter is present, the output is formatted nicely, which can be useful for debugging. Usually, this parameter should be omitted because this will result in a more compact representation, saving bandwidth.

Response

The response is a JSON array, each element being one available archive (JSON object). Each of these JSON objects has the following fields:

Field nameInternal data typeJSON data typeDescription
keyintnumber (must be in integer format) numeric key identifying the archive (unique)
namestringstring name of the archive (might not be unique)
descriptionstringstring description of the archive

Example

Request:

GET /archive-access/api/1.0/archive/?prettyPrint HTTP/1.0

Response:

[ {
  "key" : 1,
  "name" : "Cassandra",
  "description" : "Cassandra PV Archive"
} ]

2. Searching for channels

Request

The request URL used in order to search for channels takes one of the following two forms:

/archive/<archive key>/channels-by-pattern/<glob pattern expression>
  ↪[?prettyPrint]
/archive/<archive key>/channels-by-regexp/<regular expression>;[?prettyPrint] 

The archive key is the numeric key of the archive as specified in the list of archives (typically 1).

The search for a channel name can be done with a glob pattern or a regular expression. In either case, the pattern needs to be URL encoded so that all special characters (in particular those that have a special meaning in a URL, like the question mark) are encoded with %xx where xx is the hexadecimal character code. This includes the special wildcard characters that are part of the pattern. When the expression contains non-ASCII characters, those characters are expected to be specified in UTF-8 encoding.

When using a glob pattern, the channels-by-pattern URL has to be used. In the glob pattern expression, the ? and * characters have a special meaning. The question mark acts as wildcard that matches exactly one character. The asterisk acts as a wildcard that matches an arbitrary number of characters (including zero characters).

When using a regular expression, the channels-by-regexp URL has to be used. The regular expression must be specified in a form that is understood by the java.util.regex.Pattern.compile(java.lang.String) method.

If the optional prettyPrint parameter is present, the output is formatted nicely, which can be useful for debugging. Usually, this parameter should be omitted because this will result in a more compact representation, saving bandwidth.

Response

The response is a JSON array, containing JSON strings, where each string is a channel name. When no matching channel is found, an empty array is returned.

Example

Request:

GET /archive-access/api/1.0/archive/1/channels-by-pattern/my%2AExample?
  ↪prettyPrint HTTP/1.0

Response:

[ "myTest1Example", "myTest2Example" ]

3. Retrieving samples for a channel

Request

The request URL for retrieving samples for a specific channel has the following form:

/archive/<archive key>/samples/<channel name>?start=<start time-stamp>&
  ↪end=<end time-stamp>[&count=<desired number of samples>][&prettyPrint]

The archive key is the numeric key of the archive as specified in the list of archives (typically 1).

The channel name is the name of the channel for which samples are requested. The channel name must be URL encoded so that all special characters (in particular those that have a special meaning in a URL, like the question mark) are encoded with %xx where xx is the hexadecimal character code. When the channel name contains non-ASCII characters, those characters are expected to be specified in UTF-8 encoding.

The start time-stamp specifies the start of the interval for which samples are requested. The time stamp is specified as the number of nanoseconds since epoch (January 1st, 1970, 00:00:00 UTC).

The end time-stamp specifies the end of the interval for which samples are requested. The time stamp is specified as the number of nanoseconds since epoch (January 1st, 1970, 00:00:00 UTC).

The count parameter is optional. If specified, the desired number of samples is a strictly positive number that specifies the number of samples that should be returned. The number of samples returned will usually not match this number exactly. However, if samples with various densities are available, the density which will result in the number of samples closest to the requested number is chosen. If this parameter is not specified, raw samples are used.

If the optional prettyPrint parameter is present, the output is formatted nicely, which can be useful for debugging. Usually, this parameter should be omitted because this will result in a more compact representation, saving bandwidth.

Response

The response is a JSON array, each element being one sample (JSON object). In addition to the samples between the start and the end time-stamp, one sample at or before the start time-stamp and one sample at or after the end time-stamp is returned (if such samples exist at all). This way, the returned data is sufficient for creating a plot covering the whole interval, even if the specified time stamps do not exactly match the time stamps of samples.

Each of the sample objects can have the following fields:

Field nameInternal data typeJSON data typeDescription
timebig integernumber (must be in integer format) time-stamp in nanoseconds since epoch (January 1st, 1970, 00:00:00 UTC)
severitysee belowobject alarm severity
statusstringstring alarm status (might contain additional information about the severity)
qualitystringstring sample quality - one of “Original” or “Interpolated” (not case-sensitive)
metaDatasee belowobject meta-data of the sample
typestringstring sample type - must be one of “double”, “enum”, “long”, “minMaxDouble”, or “string” (not case-sensitive)
valuedepends on sample typearray array of values making up the sample
minimumdoublenumber or string minimum value – must be in number format unless it cannot be expressed as a JSON number (e.g. infinity)
maximumdoublenumber or string maximum value – must be in number format unless it cannot be expressed as a JSON number (e.g. infinity)

The type, time, severity, status, quality, and value fields are always present. The minimum and maximum fields are only present if the type is minMaxDouble. The type field must always come before the value field.

The quality field indicates whether the sample is a raw sample (“Original”) or a decimated sample (“Interpolated”).

The metaData field may be present for all types except the string type. The format of the meta-data depends on the type (see below).

At places where a number may also be expressed as a JSON string, the use of a string is reserved to cases where the number cannot be represented as a JSON number (infinity and not-a-number). Valid strings are inf, infinity, +inf, +infinity, -inf, -infinity, and nan (all not case-sensitive).

The value is always represented as a JSON array. The type of the array elements depends on the sample type:

Sample typeElement JSON typeRemarks
doublenumber or string must be in number format unless it cannot be expressed as a JSON number (e.g. infinity)
enumnumber must be in integer format, numbers outside the interval [-231, 231-1] may be truncated
longnumber must be in integer format, numbers outside the interval [-263, 263-1] may be truncated
minMaxDoublenumber or string must be in number format unless it cannot be expressed as a JSON number (e.g. infinity)
stringstring 

The minMaxDouble type is used for samples that have been aggregates from several raw samples and the minimum and maximum represent the least and the greatest value of any of the original samples. Sample of type minMaxDouble typically have a quality of “Interpolated” because they represent decimated samples.

The severity is a JSON object with the following fields (all mandatory):

Field nameInternal data typeJSON data typeDescription
levelstringstring sample severity - one of “OK”, “MINOR”, “MAJOR”, or “INVALID” (all not case-sensitive)
hasValuebooleanboolean tells whether the sample has a value (or just signals a condition with a certain severity)

The meta-data is a JSON object. The format depends on the sample type. Samples that are of the string type do not have meta data. Samples that are of the enum type can have meta data in the following format (all fields are mandatory):

Field nameInternal data typeJSON data typeDescription
typestringstring value is always “enum” (not case-sensitive)
statesarray of stringsarray of strings labels for the enum states

Samples that are of the double, long, or minMaxDouble type can have meta data in the following format (all fields are mandatory):

Field nameInternal data typeJSON data typeDescription
typestringstring value is always “numeric” (not case-sensitive)
precisionintegernumber number of fractional digits to be displayed, must be in integer format
unitstringstring engineering units of the value
displayLowdoublenumber or string lower display limit – must be in number format unless it cannot be expressed as a JSON number (e.g. infinity)
displayHighdoublenumber or string upper display limit – must be in number format unless it cannot be expressed as a JSON number (e.g. infinity)
warnLowdoublenumber or string lower warning limit – must be in number format unless it cannot be expressed as a JSON number (e.g. infinity)
warnHighdoublenumber or string upper warning limit – must be in number format unless it cannot be expressed as a JSON number (e.g. infinity)
alarmLowdoublenumber or string lower alarm limit – must be in number format unless it cannot be expressed as a JSON number (e.g. infinity)
alarmHighdoublenumber or string upper alarm limit – must be in number format unless it cannot be expressed as a JSON number (e.g. infinity)

Example

Request:

GET /archive-access/api/1.0/archive/1/samples/testCalc?start=0&
  ↪end=1500000000000000000&prettyPrint HTTP/1.0

Response:

[ {
  "time" : 1468429059824011000,
  "severity" : {
    "level" : "OK",
    "hasValue" : true
  },
  "status" : "NO_ALARM",
  "quality" : "Original",
  "metaData" : {
    "type" : "numeric",
    "precision" : 2,
    "units" : "V",
    "displayLow" : 0.0,
    "displayHigh" : 0.0,
    "warnLow" : "NaN",
    "warnHigh" : 12.0,
    "alarmLow" : "NaN",
    "alarmHigh" : 15.0
  },
  "type" : "double",
  "value" : [ 7.0 ]
}, {
  "time" : 1468429060825564000,
  "severity" : {
    "level" : "MINOR",
    "hasValue" : true
  },
  "status" : "HIGH",
  "quality" : "Original",
  "metaData" : {
    "type" : "numeric",
    "precision" : 2,
    "units" : "V",
    "displayLow" : 0.0,
    "displayHigh" : 0.0,
    "warnLow" : "NaN",
    "warnHigh" : 12.0,
    "alarmLow" : "NaN",
    "alarmHigh" : 15.0
  },
  "type" : "double",
  "value" : [ 12.0 ]
} ]