Cassandra Archiver Manual

Apache Cassandra is a column-oriented database management system (CDBMS), which is optimized for storing large amounts (tera- or even petabytes) of data grouped in column-families. It is a a special form of a key-value store. Unlike a relational database management system (RDBMS) it is not optimized for storing relational data or modifying data in a transactional way. The main advantages of a CDBMS compared to a RDBMS are superior read-write performance, linear scalability and high availability at low operation costs.

In a CDBMS data is stored in column families. Each column family contains an arbitrary number of rows, which (for a multi-node setup) are distributed over all cluster nodes. Each row is identified by a unique row key. Each row contains one or more columns. Each column is identified by a column name, that must be unique for the respective row. Each column can but does not have to store a value. Row keys, column names and column values are stored as array of bytes. The meaning of the bytes depend on the application accessing the database. Therefore, data-types are completely transparent to the CDBMS.

In a multi-node setup, data is distributed across the nodes, so that the amount of data stored is not limited by the disk-space of a single computer. Typically, low-price servers which are not fault-tolerant are used and the same data is stored on multiple nodes (typically three). The database clients and servers have built-in facilities, that automatically switch to a different node, if the first node fails. Therefore, the database cluster is fault-tolerant and highly availabe, although cheap, unreliable computers are used.

This document does not provide a detailed introduction into column-oriented database management systems or Apache Cassandra. For understanding the concepts of a CDBMS, the original paper about Google Bigtable by Chang et al. is a good starting point.

If you want to setup a cluster of Cassandra servers or are interested in advanced configuration options and performance tuning, you should read the Apache Cassandra Documentation provided by DataStax. However, this manual describes the basic steps needed to setup a single-node Cassandra cluster for getting started with the Cassandra Archiver.

The Cassandra Archiver for Control System Studio is a set of plugins that extend the existing archive reader and writer architecture so that a database hosted by Apache Cassandra can be used instead of a traditional RDBMS like MySQL or Oracle.

By using a column-oriented database management system, huge amounts of channel samples can be archived. The Cassandra Archiver uses one column family for storing all channel samples. Each row stores one sample and is identified by a key that aggregates the channel name, the time-stamp of the sample and the compression-level name. The columns of each row store the sample's value and meta-data (e.g. alarm severity). As the data in the samples column-family is compressed before being written to disk, the space requirements of the database are reduced. Due to the way Cassandra stores data, the read and write perfomance of the database is not reduced by using compression. In fact, using compression can even slightly increase the data throughput.

The Cassandra Archiver can be regarded as a hybrid between the RDB Archiver and the Channel Archiver. Like the RDB Archiver, the Cassandra Archiver uses an existing, well-tested database management system for storing data. However, like the Channel Archiver, the Cassandra Archiver uses a storage format that is more optimized for storing channel samples and can provide high write and read rate.

The HyperArchiver uses a similar concept as the Channel Archiver. However, it uses Hypertable to store the samples and MySQL to store the configuration, while the Cassandra Archiver stores the configuration and the samples in the same database, simplifying installation and maintenance. For a HyperArchiver setup, where the Hypertable server is not running on the same node as the archive engine, the source code of the HyperArchiver has to be modified, because important configuration values are hard-coded. Unlike Apache Cassandra, which does not have a single-point of failure, Hypertable has a master server, which, when down, causes the whole cluster to fail. Besides, Cassandra is implemented as pure Java and thus 100 percent platform independent, while Hypertable needs to be compiled for each supported platform. In summary, the Cassandra Archiver is easier to setup and maintain and more reliable than the HyperArchiver, making it the better choice for most scenarios.

This section explains the various column families which are used to store the configuration and samples. If you are not interested in the details, you can simply skip this section and read on at the next chapter. The information in this section is not needed for setting up the Cassandra Archiver.

For row keys which have several parts, the various parts are seperated by a null byte. All row-keys are prepended by a (binary) MD5 hash followed by a null byte in order to make sure that they are evenly distributed across the cluster nodes. The MD5 hash is calculated by appending the constituent byte arrays of the key (without a separating null byte) and then caclulating the MD5 hash of the result byte array.

This section describes the steps needed for setting up the Apache Cassandra server for use with the Cassandra Archiver.

	Important
	This section contains important information about configuration options that must be set for the Cassandra Archiver to work correctly. Thus, you should carefully read this section (in particular the section called “Configuring the Partitioner”), even if you already have a running Cassandra server.

4.1.2. Configuration

Apache Cassandra stores its configuration in /path/to/cassandra/conf. For a simple, single-node configuration, there are two relevant files: cassandra.yaml and log4j-server.properties.

Data Paths

Before starting Cassandra, you either have to change the paths where Cassandra stores its data, or you have to create the directories used by default and make sure the user, that is running Cassandra can write to these directories.

There are four directories Cassandra uses to store data. The first three are configured in cassandra.yaml. The option data_file_directories is set to /var/lib/cassandra/data by default and defines where the actual data from the various column families is saved. The option saved_caches_directory defaults to /var/lib/cassandra/saved_caches and is used to store cached data. The third option is the commitlog_directory, which defaults to /var/lib/cassandra/commitlog. This directory is used for storing the write-ahead log. If you aim for maximum performance, you might want to consider storing the commit-log on a different disk than the data directories. For most setups however, storing the commit-log on the same disk is fine.

The last directory is configured in log4j-server.properties and is used to store the server log. The option log4j.appender.R.File defaults to /var/log/cassandra/system.log. In contrast to the other options, this option specifies the file and not a directory.

Configuring the Partitioner

	Important
	An order-preserving partitioner must be used for the Cassandra Archiver. The partioner cannot be changed after data has been stored in the database, therefore you have to change this option before starting Cassandra the first time.

In cassandra.yaml the partitioner option has to be changed to refer to org.apache.cassandra.dht.ByteOrderedPartitioner. The Cassandra Archiver uses key-range queries for retrieving samples in a specific time range, so that an ordered partioner must be used. If you have other applications, which do not use ranged queries, you should run them on a different Cassandra cluster using the random partioner. Using applications which are desgined for use with the random partionier on a cluster with an ordered partioner will lead to unequal data distribution across the cluster nodes and bad read and write performance.

Configuring the Network Interface

	Note
	If the Cassandra server, the Cassandra Archiver Engine and the Control System Studio client are all running on the same machine, you can skip this step.

There are five configuration regarding the network interface used by the cassandra server. The first three options (storage_port, ssl_storage_port and listen_address) are only relevant for a multi-node Cassandra cluster and thus outside the scope of this manual.

The other two options (rpc_address and rpc_port) are relevant if you want to run Control System Studio or the archive engine on different machines than the Cassandra server. By default rpc_address is configured to only listen on the loopback interface. You should change this to the IP address of the network interface your machine uses to connect to the rest of the network. If you are sure, your hostname and IP address configuration is correct (in particular /etc/hosts and /etc/hostname are configured correctly), you can also set a blank value, to make Cassandra deterine the right IP address by itself.

The rpc_port option needs only to be changed, if you run two or more Cassandra servers on the same host, or a different service uses the same port. By default TCP port 9160 is used for the Thrift service. If you change this port number, you also have to adjust the setting in the archive engine and archive reader configurations.

Configuring Authentication Options

	Note
	Configuring the authentication options is completely optional. By default, Cassandra grants full write-access to all connections without any authentication. If using Cassandra in a production environment, you might want to use authentication for better security however.

Cassandra's security system divides into two components: authentication and authorization. Authentication is the task of checking credentials provided by a client and assigning a principal. Authentication is the task of checking whether a specific principal may perform a certain operation.

By default Cassandra is distributed with an authenticator which accepts any credentials and an authority which grants any permission to any principal.

The SimpleAuthenticator and SimpleAuthority are part of the Cassandra source code but are not distributed with the binary distribution.

For your convenience, a JAR file with the compiled versions of the two classes is distributed with the Cassandra Archiver in the cassandra-simpleauth directory.

Copy this JAR to the lib directory of the Cassandra installation and add the following two lines to the end of the cassandra-env.sh configuration file:

JVM_OPTS="$JVM_OPTS -Dpasswd.properties=$CASSANDRA_CONF/passwd.properties"
JVM_OPTS="$JVM_OPTS -Daccess.properties=$CASSANDRA_CONF/access.properties"

Besides adding these system properties, you also have to adjust the authenticator and authority options in cassandra.yaml to refer to org.apache.cassandra.auth.SimpleAuthenticator and org.apache.cassandra.auth.SimpleAuthority respectively.

You also have to create the configuration files passwd.properties and access.properties in the conf directory of the Cassandra installation.

The passwd.properties file uses a simple syntax where the property name is the username and the property value is the clear-text password for the user. The following examples defines four users with different passwords:

admin=superSafePassword
archive-read=somePassword
archive-write=someDifferentPassword
archive-config=anotherPassword

The access.properties uses a syntax, where the property name represents a privilege and the property value is a comma-separated list of principals, which are granted that privilege. The following example assigns four levels of privileges: The user admin may perform any operation, the user archive-read may read data from the column-families in the cssArchive keyspace and the user archive-write may write data to the column-families samples, channelConfigurations and compressionLevelConfigurations in the cssArchive keyspace, and the user archive-config may write data to any column family in the cssArchive keyspace:

<modify-keyspaces>=admin
cssArchive.<ro>=archive-read,archive-write,archive-config
cssArchive.<rw>=admin
cssArchive.engineConfiguration.<ro>=archive-read,archive-write
cssArchive.engineConfiguration.<rw>=archive-config,admin
cssArchive.engineConfigurationToGroups.<ro>=archive-read,archive-write
cssArchive.engineConfigurationToGroups.<rw>=archive-config,admin
cssArchive.groupConfiguration.<ro>=archive-read,archive-write
cssArchive.groupConfiguration.<rw>=archive-config,admin
cssArchive.groupConfigurationToChannels.<ro>=archive-read,archive-write
cssArchive.groupConfigurationToChannels.<rw>=archive-config,admin
cssArchive.channelConfiguration.<ro>=archive-read
cssArchive.channelConfiguration.<rw>=archive-config,archive-write,admin
cssArchive.channelConfigurationToCompressionLevels.<ro>=archive-read,archive-write
cssArchive.channelConfigurationToCompressionLevels.<rw>=archive-config,admin
cssArchive.compressionLevelConfiguration.<ro>=archive-read
cssArchive.compressionLevelConfiguration.<rw>=archive-config,archive-write,admin
cssArchive.samples.<ro>=archive-read
cssArchive.samples.<rw>=archive-config,archive-write,admin

Starting the Server

The Cassandra server can be started using the script /path/to/cassandra/bin/cassandra. You can use the -f flag to start Cassandra in foreground (recommended when testing Cassandra the first time).

Creating the Keyspace for the Cassandra Archiver

In order to use the Cassandra Archiver, you first have to create the keyspace and the column families used by the the archiver. You can do this by starting /path/to/cassandra/bin/cassandra-cli -h <hostname or IP address of your Cassandra server>. If you enabled authentication for your Cassandra server, you have to specify additional parameters. Call cassandra-cli -h for getting a list of all supported command-line parameters.

Once you successfully started the Cassandra CLI and it is connected to the Cassandra server, you can execute the following commands to create the keyspace and the column families for the Cassandra Archiver.

CREATE KEYSPACE cssArchive;
USE cssArchive;
CREATE COLUMN FAMILY engineConfiguration;
CREATE COLUMN FAMILY engineConfigurationToGroups;
CREATE COLUMN FAMILY groupConfiguration;
CREATE COLUMN FAMILY groupConfigurationToChannels;
CREATE COLUMN FAMILY channelConfiguration;
CREATE COLUMN FAMILY channelConfigurationToCompressionLevels;
CREATE COLUMN FAMILY compressionLevelConfiguration;
CREATE COLUMN FAMILY samples WITH
  compression_options = {
    sstable_compression: DeflateCompressor,
    chunk_length_kb: 256
  };

Instead of cssArchive you can use a different name for the keyspace. However, you will have to configure the keyspace name for the tools using the Cassandra server, if you do not use the default keyspace name. The column-family names are fixed and cannot be changed.

You can change the chunk_length_kb option for the samples column family. Choosing the right chunk length is a trade-off between the optimal compression ratio and the best performance for random reads. Using a value of 256 kilobytes should be okay for most environments, because on one hand random reads of samples are rare, so there is no significant benefit from using a smaller chunk size. On the other hand, for the kind of data typically stored in the samples column family, increasing the chunk size will not improve the compression ratio significantly.

Basically, the configuration format used by the Cassandra Archiver is the same that is used by the RDB Archiver. However, the syntax is extended by a new tag used to configure compression levels.

For explaining the syntax of the configuration file, we use a simple example:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<engineconfig>
  <group>
    <name>firstGroup</name>
    <channel>
      <name>firstChannel</name>
      <period>0.5</period>
      <monitor/>
      <compression-level name="raw" retention-period="86400"/>
      <compression-level name="30s" compression-period="30"/>
      <compression-level name="5m" compression-period="300"/>
    </channel>
    <channel>
      <enable/>      
      <name>secondChannel</name>
      <period>1</period>
      <scan/>
    </channel>
  </group>
  <group>
    <name>anotherGroup</name>
    
    <channel>
      <name>someOtherChannel</name>
      <period>10</period>
      <scan/>
      <compression-level name="30s" compression-period="30"/>
      <compression-level name="5m" compression-period="300"/>
    </channel>
  </group>
</engineconfig>

Every engine configuration is enclosed by the engineconfig tag. Within the engineconfig there must be at least one group. Each group must have a name. The group name must be unique within the engine configuration.

Within a group, there can be an arbitrary number of channel tags. Each channel must specify a name. The channel name must be unique across all engine configurations.

A channel must also specify a period and either the scan or monitor mode.

In scan mode, the period specifies the interval (as a floating point number in seconds) between the snapshots taken from the channel. If the channel has not changed since the last snapshot the new snapshot is discarded.

In monitor mode, every change received for the channel is saved. In this case, period specifies the expected change rate. This is used to allocate the queue, which stores new samples, before they are written to the database. If the specified period is too long and the actual change rate is higher, samples might be lost, because the queue fills up. If the specified period is much shorter than the actual change period, more memory than needed is allocated for the channel. As computer memory is rather cheap today, you should rather choose this value too small than too big.

The compression-level tag is optional and its meaning is discussed in the next section.

Unlike the RDB Archiver, the Cassandra Archiver does not perform compression of samples for each read request, but stores the compressed samples instead. This has the advantage, that for queries requesting samples for a long period, less data has to be read and thus the query can be answered more quickly.

The compression levels are independently configured for each channel. If no compression levels are configured, only raw samples are saved and they are never deleted. Each compression-level tag must have a name attribute. The compression-level name must be unique within the channel configuration. The special name raw is reserved for the raw samples, which are not calculated but represent the samples received from the channel.

Each compression-level except the raw level must specify a compression-period attribute. This interval (an integer number of seconds) specifies the time between two compressed samples. If two consecutive samples have the same (average) value as well as the same minimum and maximum bounds, the seconds sample is not saved. All compressed samples are aligned to January 1st, 1970, 00:00:00 UTC. This way, the compressed samples from two different channels but using the same compression period are aligned with respect to each other. The compression-period attribute is not valid for the special raw compression level.

The retention-period attribute is optional for the compression-level tag. If a positive retention period (in integer seconds) is defined, samples that are older than the newest sample minus the specified period are deleted. The retention-period attribute is also valid for the special raw compression level.

Important

When specifying a retention period, you have to make sure that all compressed samples have been calculated before the samples needed for this calculation are deleted. Compressed samples are usually calculated from the compression level with the next shorter compression period, that is an even integer fraction of the compression period of the level to be calculated. However, if no such compression level exists, the raw samples are used however. As a rule of thumb, the retention period for any compression level should be at least double the largest compression period for the same channel.

For loading or updating an engine configuration, you have to use the archive config tool, which is distributed in the archive-config-tool directory of the binary distribution. For importing an engine configuration file, you can call ./ArchiveConfigTool.sh -engine myEngineName -config myEngineConfig.xml -import. If you want to replace the configuration of an existing engine, you have to add the -replace_engine parameter. Replacing an engine configuration will first delete the existing configuration and than import the new configuration. Thus, it is equivalent to first using the -delete_config parameter and then importing the configuration with the -replace_engine parameter. Deleting an engine configuration will never delete the samples associated with the engine's channels. However, if a channel does not exist in the configuration, there is no way to retrieve the samples using the archive reader. Therefore, instead of completely deleting channels, you should move them to a disabled group, if you want to be able to retrieve historic data. If you finally want to delete samples for deleted channels, you have to use the clean-up tool.

If the default connection parameters (Cassandra host is localhost, port is 9160, keyspace name is cssArchive and no authentication is used) are not correct for your setup, you either have to specify the connection parameters as command-line parameters, or you have to specify a plug-in customization file. Call ./ArchiveConfigTool -help for a list of all supported command-line parameters.

If you want to delete the samples for non-existing channels or want to clean-up small inconsistencies, which can occur if a write operation is interrupted, you can use the clean-up tool.

The clean-up tool is distributed in the archive-cleanup-tool directory of the binary distribution. You can start it by invoking ./ArchiveCleanUpTool.sh. If you are using non-default connection parameters, the same considerations as for the archive config tool apply.

	Important
	The run of the clean-up tool can take a very long time. During this time you should not use the archive config tool, because new configurations added by the config tool and the respective samples might be deleted by the clean-up tool. However, the archive engine can run while the clean-up process is running.