Cassandra Archiver Manual

Version 2.0 of the Cassandra Archiver changes the structure of the database used to store the samples. The new structure brings significant improvements for the performance of Cassandra, in particular for large databases.

Existing databases cannot be converted to the new format. Instead the new database has to be setup first and then the data needs to be transferred manually.

Other changes mainly affect the internal code structure which has been tidied up. In particular, major changes have been made to the sample compressor, which also profits from the new database structure.

Finally an option for changing the consistency levels used for reads and writes has been added. Expert users can use this setting to tweak the availability trade-off between read and write scenarios.

Version 2.1 of the Cassandra Archiver uses Apache Cassandra 1.2 instead of version 1.1 that was used for earlier releases. This way users of the Cassandra Archiver can benefit from the many useful features that are new in Cassandra 1.2 (e.g. virtual nodes).

Another change affects the format that channel values are stored in. This format has been optimized, so that integers typically need considerably less space than before. Cassandra Archiver 2.1 can read values in the old format, but values written by Cassandra Archiver 2.0 cannot be read by previous versions. This change does not affect the database structure, thus no special actions have to be taken when upgrading.

Version 2.2 of the Cassandra Archiver brings an improved version of the sample compressor algorithm. Instead of polling the database for new samples, new samples are now send to the compressor thread using a queue, thus significantly reducing the number of read requests needed. The retention mechanism has also been optimized, so that samples are now deleted once about every four hours instead of constantly. These retention runs and compression runs for channels without new samples are spread out over a period of about 4 hours, thus avoiding load peaks. In summary these changes reduce the system load for compressed samples significantly.

The database structure and format have not changed compared to version 2.1, thus older versions of the reader can be used in combination with version 2.2 of the archive engine.

Version 2.3 of the Cassandra Archiver improves the way how the retention period is applied. In earlier versions, individual samples would be deleted when they expired. This caused many tombstones to accumulate in the Cassandra database, affecting performance and possibly even making queries for the corresponding channel impossible due to limitations in Apache Cassandra.

The new algorithm only deletes complete “buckets” of samples, thus reducing the number of tombstones created significantly. In addition to that, the way how the archive reader reads samples has been improved slightly, so that it is less prone to be affected by tombstones due to the deletion of old samples.

The database structure and format have not changed compared to version 2.1 and 2.2, thus older versions of the reader can be used in combination with version 2.2 of the archive engine.

Apache Cassandra is a column-oriented database management system (CDBMS), which is optimized for storing large amounts (tera- or even petabytes) of data grouped in column-families. It is a a special form of a key-value store. Unlike a relational database management system (RDBMS) it is not optimized for storing relational data or modifying data in a transactional way. The main advantages of a CDBMS compared to a RDBMS are superior read-write performance, linear scalability and high availability at low operation costs.

In a CDBMS data is stored in column families. Each column family contains an arbitrary number of rows, which (for a multi-node setup) are distributed over all cluster nodes. Each row is identified by a unique row key. Each row contains one or more columns. Each column is identified by a column name, that must be unique for the respective row. Each column can but does not have to store a value. Row keys, column names and column values are stored as array of bytes. The meaning of the bytes depend on the application accessing the database. Therefore, data-types are completely transparent to the CDBMS.

In a multi-node setup, data is distributed across the nodes, so that the amount of data stored is not limited by the disk-space of a single computer. Typically, low-price servers which are not fault-tolerant are used and the same data is stored on multiple nodes (typically three). The database clients and servers have built-in facilities, that automatically switch to a different node, if the first node fails. Therefore, the database cluster is fault-tolerant and highly availabe, although cheap, unreliable computers are used.

This document does not provide a detailed introduction into column-oriented database management systems or Apache Cassandra. For understanding the concepts of a CDBMS, the original paper about Google Bigtable by Chang et al. is a good starting point.

If you want to setup a cluster of Cassandra servers or are interested in advanced configuration options and performance tuning, you should read the Apache Cassandra Documentation provided by DataStax. However, this manual describes the basic steps needed to setup a single-node Cassandra cluster for getting started with the Cassandra Archiver.

The Cassandra Archiver for Control System Studio is a set of plugins that extend the existing archive reader and writer architecture so that a database hosted by Apache Cassandra can be used instead of a traditional RDBMS like MySQL or Oracle.

By using a column-oriented database management system, huge amounts of channel samples can be archived. The Cassandra Archiver uses two column families for storing all samples of all channels for a certain compression level. Several samples of a channel are aggregated in a so called bucket and each bucket is stored in one row. Each row storeing a bucket is identified by a key that aggregates the channel name, the size of the bucket and the start time-stamp of the bucket (both in nanoseconds). Each column of a bucket-row stores one sample, using the column's name for the sample's time-stamp and the column's value for the sample's value and meta-data (e.g. alarm severity). As data stored in column-families is compressed before being written to disk by default, the space requirements of the database are reduced. Due to the way Cassandra stores data, the read and write perfomance of the database is not reduced by using compression. In fact, using compression can even slightly increase the data throughput.

The Cassandra Archiver can be regarded as a hybrid between the RDB Archiver and the Channel Archiver. Like the RDB Archiver, the Cassandra Archiver uses an existing, well-tested database management system for storing data. However, like the Channel Archiver, the Cassandra Archiver uses a storage format that is more optimized for storing channel samples and can provide high write and read rates.

The HyperArchiver uses a similar concept as the Channel Archiver. However, it uses Hypertable to store the samples and MySQL to store the configuration, while the Cassandra Archiver stores the configuration and the samples in the same database, simplifying installation and maintenance. For a HyperArchiver setup, where the Hypertable server is not running on the same node as the archive engine, the source code of the HyperArchiver has to be modified, because important configuration values are hard-coded. Unlike Apache Cassandra, which does not have a single-point of failure, Hypertable has a master server, which, when down, causes the whole cluster to fail. Besides, Cassandra is implemented as pure Java and thus 100 percent platform independent, while Hypertable needs to be compiled for each supported platform. In summary, the Cassandra Archiver is easier to setup and maintain and more reliable than the HyperArchiver, making it the better choice for most scenarios.

This section explains the various column families which are used to store the configuration and samples. If you are not interested in the details, you can simply skip this section and read on at the next chapter. The information in this section is not needed for setting up the Cassandra Archiver.

For row keys and column names, which have several parts, a composite type is used.

This section describes the steps needed for setting up the Apache Cassandra server for use with the Cassandra Archiver.

	Important
	Earlier versions of the Cassandra Archiver (before version 2.0.0) used a different database structure which required an order-preserving partitioner to be used. Newer version of the Cassandra archiver (starting with version 2.0.0) do not require this any longer and in fact should not be installed on a cluster with an order-preserving partitioner, because this will cause hot-spots.

5.1.2. Configuration

Apache Cassandra stores its configuration in /path/to/cassandra/conf. For a simple, single-node configuration, there are two relevant files: cassandra.yaml and log4j-server.properties.

Data Paths

Before starting Cassandra, you either have to change the paths where Cassandra stores its data, or you have to create the directories used by default and make sure the user, that is running Cassandra can write to these directories.

There are four directories Cassandra uses to store data. The first three are configured in cassandra.yaml. The option data_file_directories is set to /var/lib/cassandra/data by default and defines where the actual data from the various column families is saved. The option saved_caches_directory defaults to /var/lib/cassandra/saved_caches and is used to store cached data. The third option is the commitlog_directory, which defaults to /var/lib/cassandra/commitlog. This directory is used for storing the write-ahead log. If you aim for maximum performance, you might want to consider storing the commit-log on a different disk than the data directories. For simple test setups however, storing the commit-log on the same disk is fine.

The last directory is configured in log4j-server.properties and is used to store the server log. The option log4j.appender.R.File defaults to /var/log/cassandra/system.log. In contrast to the other options, this option specifies the file and not a directory.

Configuring the Network Interface

	Note
	If the Cassandra server, the Cassandra Archiver Engine and the Control System Studio client are all running on the same machine, you can skip this step.

There are five configuration regarding the network interface used by the cassandra server. The first three options (storage_port, ssl_storage_port and listen_address) are only relevant for a multi-node Cassandra cluster and thus outside the scope of this manual.

The other two options (rpc_address and rpc_port) are relevant if you want to run Control System Studio or the archive engine on different machines than the Cassandra server. By default rpc_address is configured to only listen on the loopback interface. You should change this to the IP address of the network interface your machine uses to connect to the rest of the network. If you are sure, your hostname and IP address configuration is correct (in particular /etc/hosts and /etc/hostname are configured correctly), you can also set a blank value, to make Cassandra deterine the right IP address by itself.

The rpc_port option needs only to be changed, if you run two or more Cassandra servers on the same host, or a different service uses the same port. By default TCP port 9160 is used for the Thrift service. If you change this port number, you also have to adjust the setting in the archive engine and archive reader configurations.

Configuring Authentication Options

	Note
	Configuring the authentication options is completely optional. By default, Cassandra grants full write-access to all connections without any authentication. If using Cassandra in a production environment, you might want to use authentication for better security however.

Cassandra's security system divides into two components: authentication and authorization. Authentication is the task of checking credentials provided by a client and assigning a principal. Authentication is the task of checking whether a specific principal may perform a certain operation.

By default Cassandra is distributed with an authenticator which accepts any credentials and an authority which grants any permission to any principal.

If you want to enable authentication and authorization starting with Cassandra 1.2 there is a built-in mechanism that stores the required data in the Cassandra database. Please refer to the Cassandra manual for configuring authentication and authorization.

You usually will want to configure two users for the Cassandra archiver: One with read-access, used by the archive reader, and one with write-access, used by the archive engine, configuration tool and clean-up tool.

You can create the two users using the following commands in cqlsh:

CREATE USER archive_write with PASSWORD 'somePassword' NOSUPERUSER;
CREATE USER archive_read with PASSWORD 'anotherPassword' NOSUPERUSER;

You can grant the privileges using the following commands:

GRANT ALTER PERMISSION on KEYSPACE "cssArchive" to archive_write;
GRANT CREATE PERMISSION on KEYSPACE "cssArchive" to archive_write;
GRANT DROP PERMISSION on KEYSPACE "cssArchive" to archive_write;
GRANT MODIFY PERMISSION on KEYSPACE "cssArchive" to archive_write;
GRANT SELECT PERMISSION on KEYSPACE "cssArchive" to archive_write;
GRANT SELECT PERMISSION on KEYSPACE "cssArchive" to archive_read;

Starting the Server

The Cassandra server can be started using the script /path/to/cassandra/bin/cassandra. You can use the -f flag to start Cassandra in foreground (recommended when testing Cassandra the first time).

Creating the Keyspace for the Cassandra Archiver

In order to use the Cassandra Archiver, you first have to create the keyspace and the column families used by the the archiver. You can do this by starting /path/to/cassandra/bin/cassandra-cli -h <hostname or IP address of your Cassandra server>. If you enabled authentication for your Cassandra server, you have to specify additional parameters. Call cassandra-cli -h for getting a list of all supported command-line parameters.

Once you successfully started the Cassandra CLI and it is connected to the Cassandra server, you can execute the following command to create the keyspace for the Cassandra Archiver.

CREATE KEYSPACE cssArchive;

For compatibility with older versions of the Cassandra Archiver, the default keyspace name used is cssArchive. However, the use of mixed-case keyspace names is discouraged in recent Apache Cassandra versions and can cause problems when using CQL to access the keyspace. Therefore, you should prefer the keyspace name css_archive when deploying a new cluster. However, you will have to configure the keyspace name for the tools using the Cassandra server, if you do not use the keyspace name cssArchive. See Section 5.2.2, “Configuration” for instructions on how to configure the keyspace name by creating a plugin_customization.ini for the archive engine, archive clean-up tool and archive configuration tool.

Basically, the configuration format used by the Cassandra Archiver is the same that is used by the RDB Archiver. However, the syntax is extended by a new tag used to configure compression levels.

For explaining the syntax of the configuration file, we use a simple example:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<engineconfig>
  <group>
    <name>firstGroup</name>
    <channel>
      <name>firstChannel</name>
      <period>0.5</period>
      <monitor/>
      <compression-level retention-period="86400"/>
      <compression-level compression-period="30"/>
      <compression-level compression-period="300"/>
    </channel>
    <channel>
      <enable/>      
      <name>secondChannel</name>
      <period>1</period>
      <scan/>
    </channel>
  </group>
  <group>
    <name>anotherGroup</name>
    
    <channel>
      <name>someOtherChannel</name>
      <period>10</period>
      <scan/>
      <compression-level compression-period="30"/>
      <compression-level compression-period="300"/>
    </channel>
  </group>
  <group>
    <name>__disabled_channels</name>
    
    <channel>
      <name>someOldChannel</name>
      <period>5</period>
      <scan/>
      <compression-level compression-period="30"/>
    </channel>
  </group>
</engineconfig>

Every engine configuration is enclosed by the engineconfig tag. Within the engineconfig there must be at least one group. Each group must have a name. The group name must be unique within the engine configuration.

Within a group, there can be an arbitrary number of channel tags. Each channel must specify a name. The channel name must be unique across all engine configurations.

A channel must also specify a period and either the scan or monitor mode.

In scan mode, the period specifies the interval (as a floating point number in seconds) between the snapshots taken from the channel. If the channel has not changed since the last snapshot the new snapshot is discarded.

In monitor mode, every change received for the channel is saved. In this case, period specifies the expected change rate. This is used to allocate the queue, which stores new samples, before they are written to the database. If the specified period is too long and the actual change rate is higher, samples might be lost, because the queue fills up. If the specified period is much shorter than the actual change period, more memory than needed is allocated for the channel. The size of the buckets storing the samples is also determined based on the configured period. Choosing a period that is too far off the actual period will either result in too many buckets being created (hurting the read and write performance) or in too many samples being stored in a single bucket (which means that rows will grow very big). Therefore you should choose the period as what you expect for the average change rate. If you are woried about losing samples in periods of bursts, you can increase the org.csstudio.archive.engine/buffer_reserve property.

The compression-level tag is optional and its meaning is discussed in the next section.

The group name __disabled_channels is reserved for special use: You can use this group for channels, which you want to disable permanently (e.g. because the corresponding device has been removed), but which you still want to see in the data browser. Moving a channel to this group basically has the same effect as moving it to a group with an enabling channel that is always false, but it avoids the warning about the channel being disconnected.

Unlike the RDB Archiver, the Cassandra Archiver does not perform compression of samples for each read request, but stores the compressed samples instead. This has the advantage, that for queries requesting samples for a long period, less data has to be read and thus the query can be answered more quickly.

The compression levels are independently configured for each channel. Each compression-level (except the compression level that stores the raw samples and has an implicit compression period of zero) level must specify a compression-period attribute. This interval (an integer number of seconds) specifies the time between two compressed samples. If two consecutive samples have the same (average) value as well as the same minimum and maximum bounds, the seconds sample is not saved. All compressed samples are aligned to January 1st, 1970, 00:00:00 UTC. This way, the compressed samples from two different channels but using the same compression period are aligned with respect to each other.

The retention-period attribute is optional for the compression-level tag. If a positive retention period (in integer seconds) is defined, samples that are older than the newest sample minus the specified period are deleted. The retention-period attribute is also valid for the special raw compression level.

The raw compression-level is always defined, even if you do not specify a compression-level for it. By default its retention period is zero (meaning that samples are never deleted).

Important

When specifying a retention period, you have to make sure that all compressed samples have been calculated before the samples needed for this calculation are deleted. Compressed samples are usually calculated from the compression level with the next shorter compression period, that is an even integer fraction of the compression period of the level to be calculated. However, if no such compression level exists, the raw samples are used however. As a rule of thumb, the retention period for any compression level should be at least double the largest compression period for the same channel.

For loading or updating an engine configuration, you have to use the archive config tool, which is distributed in the archive-config-tool directory of the binary distribution. For importing an engine configuration file, you can call ./ArchiveConfigTool.sh -engine myEngineName -config myEngineConfig.xml -import. If you want to replace the configuration of an existing engine, you have to add the -replace_engine parameter. Replacing an engine configuration will first delete the existing configuration and than import the new configuration. Thus, it is equivalent to first using the -delete_config parameter and then importing the configuration with the -replace_engine parameter. Deleting an engine configuration will never delete the samples associated with the engine's channels. However, if a channel does not exist in the configuration, there is no way to retrieve the samples using the archive reader. Therefore, instead of completely deleting channels, you should move them to a disabled group, if you want to be able to retrieve historic data. If you finally want to delete samples for deleted channels, you have to use the clean-up tool.

If the default connection parameters (Cassandra host is localhost, port is 9160, keyspace name is cssArchive and no authentication is used) are not correct for your setup, you either have to specify the connection parameters as command-line parameters, or you have to specify a plug-in customization file. Call ./ArchiveConfigTool -help for a list of all supported command-line parameters.

Important

No channels should be modified using the archive configuration tool, while the engine for the respective channel is running (in particular no channel should be deleted). This means that you should not use the -import or -delete_config actions for an archive engine that is running and that you should not use the -steal_channels option, while any engine is running. You also should not modify the configuration while the archive clean-up tool is running.

If you want to delete the samples for non-existing channels or want to clean-up small inconsistencies, which can occur if a write operation is interrupted, you can use the clean-up tool.

The clean-up tool is distributed in the archive-cleanup-tool directory of the binary distribution. You can start it by invoking ./ArchiveCleanUpTool.sh. If you are using non-default connection parameters, the same considerations as for the archive config tool apply.

	Important
	The run of the clean-up tool can take a very long time. During this time you should not use the archive config tool, because new configurations added by the config tool and the respective samples might be deleted by the clean-up tool. However, the archive engine can run while the clean-up process is running.