The configuration options used by the Cassandra PV Archiver
server are
controlled through a configuration file in the
YAML
format.
The configuration file is located in the
conf
directory of the binary distribution or in the
/etc/cassandra-pv-archiver
directory when using the
Debian package.
In either case, the
configuration file is called
cassandra-pv-archiver.yaml
.
It is not an error if the configuration file does not exists at
the
expected location.
In this case the server starts using
default values for all
configuration options.
The path to the configuration file can be overridden by
specifying the
--config-file
command line option to the
cassandra-pv-archiver-server
script.
When this configuration option is specified, the default
location is not
used.
Unlike the configuration file in the default
location, a configuration
file specified with
--config-file
option must exist
and the server does not start if it is missing.
The configuration options are organized in a hierarchy. For the rest of this document, the first level of this hierarchy is called the section. The hierarchical path to a configuration option can either be specified inline or through indentation. For example, specifying
level1a:
option1: value1
level2:
option1: value2
level1b:
option1: value3
is equivalent to specifying
level1a.option1: value1 level1a.level2.option1: value2 level1b:option1: value3
The default values specified in this document are the default values that are used when a configuration option is not specified at all, not the value of the option that is specified in the configuration file distributed as part of the binary distribution or Debian package.
This section only describes the part of the configuration that is stored in the per-server configuration file, not the configuration that is stored in the database. Regarding the latter one, please refer to Section 4, “Administrative user interface” .
The
cassandra
section configures the server’s
connection to the Cassandra
cluster.
The
cassandra.hosts
option specifies the list
of hosts which are used for
initially establishing the connection
with the Cassandra
cluster.
This list does not have to contain all Cassandra
hosts because all
hosts in the cluster are detected
automatatically once the
connection to at least one host has
been established.
However, it is still a good idea to specify
more than one host here
because this will ensure that the
connection can be established
even if one of the hosts is
down when the Cassandra PV Archiver
server is started.
By default, the list only contains
localhost
.
The list of hosts has to be specified as a YAML list, using
the
regular or the inline list syntax. For example, a list
specifying
three hosts might look like this:
cassandra:
hosts:
- server1.example.com
- server2.example.com
- server3.example.com
The
cassandra.port
option specifies the port
number on which the Cassandra hosts
are listening for incoming
connections (for Cassandra’s
native protocol).
The default value is 9042, which is also
the default value used by
Cassandra.
The
cassandra.keyspace
option specifies the name
of the keyspace in which the
Cassandra PV Archiver stores its data.
The default value is
pv_archive
.
While strictly speaking mixed-case names are allowed, the
use of
such names is discouraged because many tools have
problem with them
and they typically require quoting.
For this
reason, the keyspace name should be all lower-case when
possible.
The
cassandra.username
option specifies the
username that is specified when
authenticating with the Cassandra
cluster.
When empty, the
connection to the Cassandra cluster is established
without
trying to authenticate the client.
The default value is the
empty string (no authentication).
The
cassandra.password
option specifies the
password that is specified when
authenticating with the Cassandra
cluster.
The password is
only used when the username is not empty.
The default value
is the empty string.
The
cassandra.fetchSize
option specifies the
default fetch size that is used when
reading data from the Cassandra
database.
The fetch size
specifies how many rows are read from the database in
a
single page.
Specifying a larger value typically improves
performance when
processing a query that returns many rows,
but results in more
memory usage in both the database server
and the client because the
full page of rows has to be kept
in memory.
The default value is zero, which causes the default fetch size of the Cassandra driver to be used. As of version 3.1.4 of the Cassandra driver, that default fetch size is 5000 rows. If specified, this option has to be set to an integer between 0 and 2147483647.
The fetch size specified here is only used for queries that do not explicitly specify a fetch size.
The
cassandra.useLocalConsistencyLevel
option
specifies the consistency level that is used for all
database
operations.
The default value is
false
.
This option only has an effect when the Cassandra cluster
is
distributed across multiple data centers.
By setting this
option to
true
, the
LOCAL_QUORUM
consistency level is used where
usually the
QUORUM
consistency level would be
used.
In the same way, the
LOCAL_SERIAL
consistency
level is used instead of the
SERIAL
consistency
level.
This option must only be enabled if only a single data center makes modifications to the data and all other data centers only use the database for read access. In this case, enabling this option can reduce the latency of operations because the client only has to wait for nodes local to the data center. The most likely scenario is a situation where all nodes running the Cassandra PV Archiver servers are in a single data center, but there is a second data center to which all data is replicated for disaster recovery.
![[Important]](docbook/images/important.png) | Important |
|---|---|
Never enable this option when there is more than one data center that is used for write access to the database. In this case, enabling this option will lead to data corruption because operations that are expected to result in a consistent state might actually leave inconsistencies.
This option merely provides a performance optimization, so
in case
of doubt, leave it at its default value of
|
The
server
section configures the archiving server
(for example the ID
assigned to each server instance and on which
address and ports
the archiving server listens).
While the address and port
settings can usually be left at their
defaults the server’s ID
has to be set.
Each server in the cluster is identified by a unique ID
(UUID).
As this UUID has to be unique for each server, there
is no
reasonable default value, but it has to be specified
explicitly.
The server’s UUID can be specified using the
server.uuid
option.
Alternatively, it can be specified by passing the
--server-uuid
parameter to the server’s start
script.
| Important |
|---|---|
Starting two server instances with the same UUID results in data corruption, regardless of whether these instances are started on the same host or different hosts. For this reason, care should be taken to ensure that each UUID is only used for exactly one process. |
As an alternative to specifying the server’s UUID in the
configuration file or on the command line, it is possible to
have a
separate file that specifies the UUID.
The path to this
file can be specified with the
server.uuidFile
option.
If this file exists, it is expected to contain a
single line with
the UUID that is then used as the server’s
UUID.
If this file does not exist, the server tries to create
it on
startup, using a randomly generated UUID.
By default
this option is not set so that the server expects an
explicitly specified UUID.
This option is particularly useful
in an environment where servers
are deployed automatically
and should thus automatically generate a
UUID the first time
they are started.
The
server.listenAddress
option specifies the IP
address (or the hostname resolving to
the IP address) on which the
server listens for incoming
connections.
If it is empty (the default), the server listens
on the first
non-loopback address that is found.
This means
that typically, this option only has to be set for
servers
that have more than one (non-loopback) interface.
The specified address is used for the administrative user-interface, the archive-access interface, and the inter-node communication interface. In addition to the specified address, the administrative user-interface and the archive-access interface are also made available on the loopback address.
This option should never be set to
localhost
,
127.0.0.1
,
::1
, or any other
loopback address because other servers will
try to contact the
server on the specified address and
obviously this will lead to
unexpected results when the
address is a loopback address.
The
server.adminPort
option specifies the TCP
port number on which the
administrative user-interface is made
available.
The default
is port 4812.
The
server.archiveAccessPort
option specifies the
TCP port number on which the
archive-access interface is made
available.
The default is
port 9812.
The archive-access interface is the web-interface
through which
clients access the data stored in the archive.
The
server.interNodeCommunicationPort
option
specifies the TCP port number on which the inter-node
communication
interface is made available.
The default is port
9813.
Like the name suggests, the inter-node communication
interface is
used for internal communication between
Cassandra PV Archiver
servers that is needed in order to
coordinate the cluster operation
(for example in case of
configuration changes).
The
server.interNodeCommunicationRequestTimeout
option specifies the timeout used for the communication
between
nodes.
The timeout is specified in milliseconds.
If
chosen too low, complex requests (e.g. a request to modify
the
configuration of many channels when importing a
configuration file)
may time out.
If chosen too high, requests
will take a very long time before
timing out in case of a
sudden server crash or network disruption.
The default value is 900000 milliseconds (15 minutes). Valid values are integer numbers between 1 and 2147483647.
The
throttling
section contains options for
throttling database operations.
The Cassandra PV Archiver server tries to run database
operations in
parallel in order to reduce the effective latency
of complex
operations (e.g. operations involing many channels).
However, depending on the exact configuration of the Cassandra
cluster
(for example the size of the cluster, network bandwidth
and latency,
hardware used for the cluster, load caused by
other applications), the
number of operations that can safely
be run in parallel might differ.
When running too many operations in parallel, this results in some of the operations timing out. This can be avoided by reducing the number of operations allowed to run in parallel. On the other hand, when operations never time out, one might try to increase the limits in order to improve the performance.
The limits can be controlled separately for read and write
operations
and for operations touching the channels’ meta-data
(for example the
configuration and information about sample
buckets) and the actual
samples.
Operations modifying channel
meta-data are typically carried out using
the
SERIAL
consistency level, so in this case write
operations typically
are more expensive than read operations.
Thus the limit for
write operations should be lower than the limit for
read
operations.
In the case of operations dealing with actual
samples, read operations
typically are more expensive than
write operation (due to how
Cassandra works internally), so the
limit for read operations shold be
lower than the limit for
write operations.
![[Note]](docbook/images/note.png) | Note |
|---|---|
When trying to optimize the throttling settings, it can be helpful to connect to the Cassandra PV Archiver server via JMX (for example using JConsole from the JDK). The current number of operations that are running and waiting is exposed via MBeans, so that it is possible to monitor how changing the throttling parameters affects the operation. |
The
throttling.maxConcurrentChannelMetaDataReadStatements
configuration option controls how many read operations for
channel
meta-data should be allowed to run in parallel.
Usually, these are statements reading from the
channels
,
channels_by_server
,
and
pending_channel_operations_by_server
tables.
Typically, this limit should be greater than the
limit set by the
throttling.maxConcurrentChannelMetaDataWriteStatements
option.
The default value is 64.
The
throttling.maxConcurrentChannelMetaDataWriteStatements
configuration option controls how many write operations for
channel
meta-data should be allowed to run in parallel.
Usually, these are statements writing to the
channels
,
channels_by_server
,
and
pending_channel_operations_by_server
tables.
Typically, such operations are light-weight
transactions and thus
this limit should be less than the
limit set by the
throttling.maxConcurrentChannelMetaDataReadStatements
option.
The default value is 16.
The
throttling.maxConcurrentControlSystemSupportReadStatements
configuration option controls how many read operations the
control-system supports (all of them combined) are allowed
to run in
parallel.
Usually, these are statements that read
actual samples and thus read
from the tables used by the
control-system support(s).
Typically, this limit should be
less than the limit set by the
throttling.maxConcurrentControlSystemSupportWriteStatements
option, but significantly greater than the limit set by the
throttling.maxConcurrentChannelMetaDataReadStatements
option.
The default value is 128.
The
throttling.maxConcurrentControlSystemSupportWriteStatements
configuration option controls how many write operations the
control-system supports (all of them combined) are allowed
to run in
parallel.
Usually, these are statements that write
actual samples (for each
sample that is written, an
INSERT
statement is
triggered) and that thus write to the tables
used by the
control-system support(s).
Typically, this limit
should be greater than the limit set by the
throttling.maxConcurrentControlSystemSupportReadStatements
option and significantly greater than the limits set by the
throttling.maxConcurrentChannelMetaDataReadStatements
and
throttling.maxConcurrentChannelMetaDataWriteStatements
options.
The default value is 512.
The
throttling.sampleDecimation.maxFetchedSamplesInMemory
configuration option controls how many samples may be
fetched into
memory when generating decimated samples.
The sample decimation process might consume a lot of memory when generating decimated samples from already existing source samples for a lot of channels. The amount of samples that may be fetched into memory is directly connected to memory usage. Each fetched sample occupies about 1 KB of memory (for scalar Channel Access samples), so one million samples are roughly equivalent to 1 GB of memory.
As the exact number of samples returned by a fetch operation
cannot
be known in advance, this threshold might actually be
exceeded
slightly.
The
maxRunningFetchOperations
option can be used to control by how much the threshold may
be exceeded.
The default value for this option is 1000000 samples.
The
throttling.sampleDecimation.maxRunningFetchOperations
configuration option controls how many fetch operations may
run in
parallel when generating decimated samples.
As the exact number of samples returned by a fetch operation
cannot
be known in advance, the threshold set by the
maxFetchedSamplesInMemory
option might actually be exceeded slightly.
This
configuration option can be used to control by how much the
threshold may be exceeded.
The max. number of running fetch
operations multiplied by the
fetch size
is the max. number of samples by which the limit might be
exceeded.
The default value for this option is 20.
The
controlSystemSupport
section contains the
configuration options for the various
control-system supports.
For each available control-system
support, this section has a
corresponding sub-section.
The
configuration options in these sub-sections are not handled by
the Cassandra PV Archiver server itself but passed as-is to
the
respective control-system support.
For this reason, the
names of the available options entirely depend
on the
respective control-system support.
Please refer to the
documentation of the respective control-system
support for
details.
For example, the documentation for the Channel Access
control-system
support is available in
Appendix D, Channel Access control-system support
.
The Cassandra PV Archiver server is based on the Spring Boot framework. For this reason, the options supported for configuring logging are actually the same ones that are supported by Spring Boot. These options are documented in the Spring Boot Reference Guide . The Cassanra PV Archiver server uses Logback as its logging backend, so the specifics of how to configure Logback for Spring Boot might also be interesting.
In order to get started more easily, this section contains a few pointers on how the logging configuration can be modified.
The log level can be set both globally and for specific subtrees of the class hierarchy. When specifying different log levels for different parts of the hierarchy, more specific definitions (the ones covering a smaller sub-tree of the hierarchy) take precedence over more general definitions.
The available log levels are
ERROR
,
WARN
,
INFO
,
DEBUG
, and
TRACE
.
Each log level contains the preceding log levels (for
example
the log level
INFO
also contains
ERROR
and
WARN
).
The log level for the root of the hierarchy (that is used
for all
loggers that do not have a more specific definition)
is set through
the
logging.root.level
option.
By default, this log level is set to
INFO
.
This results in a lot of diagnostic messages being logged,
so you
might want to consider reducing it to
WARN
.
The log level for individual parts of the hierarchy can be
set by
using a configuration option containing the path to
the respective
hierarchy level.
For example, in order to
enable DEBUG messages for all classes in
the
com.aquenos.cassandra.pvarchiver
package (and
its sub-packages), one could set
logging.com.aquenos.cassandra.pvarchiver.level
to
DEBUG
.
The path to the log file can be specified using the
logging.file
option.
If no log file is specified (the default),
log
messages are only written to the standard output.
In order to
log to more than one log file (for example depending
on the
log level or the class writing the log message) or in order
to disable logging to the standard output, one has to
specify a
custom logback configuration file (see the next
section).
When the configuration options directly available through
the
Cassandra PV Archiver server configuration-file are not
sufficient,
one can specify a custom Logback configuration
file.
The path to this file is specified using the
logging.config
option.
The
information
available in the
Spring Boot Reference Guide
might be useful when using this option.
In addition to the configuration options that can be specified
in the
server’s configuration file, there are two environment
variables that
can be passed to the server’s startup script.
When using the Debian package, these environment variables
should be
set in the file
/etc/default/cassandra-pv-archiver-server
.
The first environment variable is
JAVA_HOME
.
It specifies the path to the JRE.
When starting the Java
process, the server’s startup scripts uses the
$JAVA_HOME/bin/java
executable
(
%JAVA_HOME%/bin/java.exe
on Windows).
When
JAVA_HOME
is not set, the startup script uses the
java
executable that is in the search
PATH
of the shell executing the startup script.
The second environment variable is
JAVA_OPTS
.
When set, the value of this environment variable is added to
the
parameters passed to the
java
executable.
It can be used to configure JVM options like the
maximum heap size.