NOAA/seiscomp-training
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[installation] Init with inital config for global

Browse Source
This commit is contained in:
Dirk Rössler
2025-10-30 15:08:17 +01:00
commit 7640b452ed
3678 changed files with 2200095 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

904
share/doc/seiscomp/html/_sources/base/concepts/configuration.rst.txt Normal file
View File

@ -0,0 +1,904 @@
.. _concepts_configuration:
#############
Configuration
#############
Scope
=====
This chapter describes the configuration of a processing system and how all
the pieces gear into each other. It will cover module specific
configuration, inventory (station metadata) and module station bindings.
Overview
========
The |scname| framework defines a common schema to read configuration parameters.
This schema is used by all :ref:`SeisComP modules <concepts_modules>` with names
starting with **sc**, e.g. `scautopick`. Other modules might be available
as well such as third party modules which use other naming conventions.
A way to discover a module that uses the |scname| configuration schema
is to call it with ``--help``. The first lines of a typical output look like
this:
.. code-block:: sh
Generic:
-h [ --help ] Produce help message
-V [ --version ] Show version information
--config-file arg Use alternative configuration file
The reason for that is that there are also other modules which do not
use the |scname| core and client libraries such Seedlink plugins, Seedlink
itself and some others which are not part of the trunk source package. Those
modules need translators to generate their native configuration when
the configuration is updated (``seiscomp update-config``).
Again, the two indicators that a module uses the following configuration
schema are:
* The module name is prefixed with **sc**, e.g. *scautopick*
* The output of ``--help`` looks like the text fragment above
A typical configuration requires the following steps:
#. Generation and import of an inventory (station meta data): Read the concept
section :ref:`concepts_inventory`,
#. :ref:`Configuration of modules <global_modules_config>`,
#. :ref:`Configuration of bindings <global_bindings_config>`.
.. note::
:ref:`Standalone modules <concepts_modules>` can also run without inventory.
.. _concepts_configuration-configs:
Configuration of Modules
========================
|scname| modules provide 2 types of configurations:
* :ref:`Module configurations <global_modules_config>` define module control
parameters which are equally applied to all operations on waveforms, event
parameters, graphical user interfaces, etc.
**All** :ref:`daemon modules <concepts_modules_daemon>` and **some**
:ref:`command-line tools <concepts_modules_commandline>` provide module
configurations. *Module configurations are overridden by bindings
configurations.*
* :ref:`Binding configurations <global_bindings_config>` define control
parameters per station and even stream. They are used for station- and even
stream-specific definitions, e.g. the data acquisition from a particular
station or server using :ref:`seedlink` or the phase detection by
:ref:`scautopick`.
In contrast to module configuration, **only some**
:ref:`daemon modules <concepts_modules>` and a very few
:term:`GUI modules <GUI>` provide bindings. *Binding configurations override
module configurations.*
Whether or not a module considers bindings configurations can be read in the
module configuration panel of :ref:`scconfig`.
.. raw:: html
<div class="two column layout">
.. figure:: ../media/scconfig_no_bindings.png
:alt: scconfig: no bindings configurations
scconfig modules panel indicating that no bindings can be configured.
.. figure:: ../media/scconfig_has_bindings.png
:alt: scconfig: no bindings configurations
scconfig modules panel indicating that bindings can be configured.
.. raw:: html
</div>
.. _global_modules_config:
Module configuration
--------------------
Configurations for modules are saved as :file:`*.cfg` files. There are three
directories involved where configuration files can be stored:
#. :file:`$SEISCOMP_ROOT/etc/defaults/`: This directory ships with the distribution
of |scname| and should never be touched. All contained files might be
overwritten with the next software update.
#. :file:`$SEISCOMP_ROOT/etc/`: This directory will never be populated by a software
update and it is save to store global module configuration files there.
Depending on the system setup this directory might be read-only to users.
It is called the system configuration directory.
#. :file:`$HOME/.seiscomp/`: This directory is in the user's home directory and
it is meant to provide configurations which override default and system
configurations.
It is called the user configuration directory.
Furthermore there are two file names involved in each directory:
:file:`global.cfg` and :file:`[module].cfg`. The file :file:`global.cfg`
will be loaded by all modules and it is a good place to store common
parameters such as messaging connections or logging configurations.
The three directories and two files result in 6 locations to read all of a
module's configuration parameters:
#. :file:`$SEISCOMP_ROOT/etc/defaults/global.cfg`
#. :file:`$SEISCOMP_ROOT/etc/defaults/[module].cfg`
#. :file:`$SEISCOMP_ROOT/etc/global.cfg`
#. :file:`$SEISCOMP_ROOT/etc/[module].cfg`
#. :file:`$HOME/.seiscomp/global.cfg`
#. :file:`$HOME/.seiscomp/[module].cfg`
The order of the configuration files above also represents the order of loading.
Parameters can be available in any of these files. The last occurrence of a
parameter takes priority such as configurations in :file:`$HOME/.seiscomp/`.
Adjusting
~~~~~~~~~
:ref:`scconfig` provides a user-friendly graphical interface to adjust the
global and the module configuration parameters.
.. figure:: ../media/scconfig_config_modules.png
:alt: scconfig: module configuration
:align: center
:width: 18cm
scconfig modules module configuration panel.
Alternatively, you may adjust the module configuration files corresponding to
a module. An easy way of getting started is to copy the default configuration
file in :file:`$SEISCOMP_ROOT/etc/defaults/` to :file:`$SEISCOMP_ROOT/etc/` and
adjust it there.
.. warning::
Do not adjust any parameter in the default configuration files located in
:file:`$SEISCOMP_ROOT/etc/defaults/` as they will be overwritten by future
software updates.
Application
~~~~~~~~~~~
Module configurations are usually read when starting a module and cannot be
changed during runtime. For applying any change on module configuration
#. Save the configuration,
#. Restart or execute all affected modules.
In addition to a permanent configuration you may temporally override any
configuration parameter when executing a module. Provide the full structure of
the configuration parameter on the command line along with "=" between parameter
and value. This will identify the parameter as a configuration parameter
overridden on the command line. E.g. refer to the codes parameter of
:ref:`scrttv` in the streams section as `codes.streams`:
.. code-block:: sh
$ scrttv --streams.codes="GR.*.*.HH?"
.. _global_bindings_config:
Bindings configuration
----------------------
Bindings configure parameters specific to stations and for a certain module.
A station might require a custom set of parameters for e.g. data acquisition from
a remote data logger or server, for processing or
displaying. |scname| design is that bindings will be stored
in the database. All modules requiring this information read them from the
database. In this way consistent inventory and its bindings will be distributed
to all modules running locally or on remote computers.
.. hint::
Bindings can be conveniently configured in the Bindings panel of :ref:`scconfig`.
Read the section :ref:`scconfig-bindings` for instructions.
.. _config-fig-binding:
.. figure:: ../media/binding.*
:align: center
Binding
A binding holds the configuration how a station is used in a module.
Parameters defined in bindings override parameters in module configurations.
Bindings configurations are saved as stations bindings or as bindings profiles with
given names:
#. **Station binding parameters:** :file:`$SEISCOMP_ROOT/etc/key/[module]/station_NET_STA`
is used only be one station NET.STA.
#. **Binding profile parameters:** :file:`$SEISCOMP_ROOT/etc/key/[module]/profile_[name]`
can be used by many stations. Then any update of this file applies to all stations
bound to this profile.
Like in this list, the actual binding parameters for each module are
expected in either :file:`etc/key/$MODULE/station_$NET_$STA` or
:file:`etc/key/$MODULE/profile_$PROFILE`. That depends if a profile should
be used or not. If the station key file registers a module with a colon
following a name then this name is the profile name. If just a module is given
then the station binding file will be read.
The bindings themselves are links from the module to the station. They are written
into key files, e.g.
* :file:`$SEISCOMP_ROOT/etc/key/station_NET_STA`
Example of bindings profiles:
.. code-block:: properties
# Binding references
global:__HH
scautopick:default
seedlink:geofon
Example of bindings profiles and a station binding for slarchive:
.. code-block:: properties
# Binding references
global:__HH
scautopick:default
seedlink:geofon
slarchive
The key files are named like :file:`station_$NET_$STA`, e.g. :file:`station_GE_MORC`.
The names within the key files define for which modules the station registers. In the
above example it is the **global**, **scautopick**, **seedlink** and slarchive.
To list all modules a particular station is configured for is very simple by printing the content
of the station key file:
.. code-block:: sh
$ cat etc/key/station_GE_MORC
seedlink:geofon
global:BH
scautopick
Another way is a bit more complicated but at least all information is
there. To show all stations configured for SeedLink could be done this way:
.. code-block:: sh
$ for i in `find etc/key -type f -maxdepth 1 -name "station_*_*"`; do
> egrep -q '^seedlink(:.*){0,1}$' $i && echo $i;
> done
etc/key/station_GE_MORC
etc/key/station_GE_UGM
Modules usually read the bindings configuration from the database but not from files.
Therefore, the bindings are stored in the database when executing
.. code-block:: sh
$ seiscomp update-config
The database tables involved are ``ConfigModule``, ``ConfigStation``, ``Setup``,
``ParameterSet`` and ``Parameter``.
.. _config-fig-configdb-schema:
.. figure:: ../media/configdb-schema.png
:align: center
Configuration database schema
To illustrate the contents of the objects, the XML representation
is used below.
.. code-block:: xml
<Config>
<module publicID="Config/trunk" name="trunk" enabled="true">
...
</module>
</Config>
A ConfigModule with publicID *Config/trunk* is created with name *trunk*. This
ConfigModule is managed by the global initialization script (:file:`etc/init/trunk.py`)
and will be synchronized with configured bindings of all trunk modules. The
ConfigModule trunk is the one that is actually used by all configurations unless
configured otherwise with:
.. code-block:: sh
$ scapp --config-module test
Here :program:`scapp` would read ConfigModule *test*. Because a ConfigModule *test*
is not managed by :program:`seiscomp update-config` it is up to the user to create
it.
For each station that has at least one binding, a ConfigStation object is
attached to the ConfigModule:
.. code-block:: xml
<Config>
<module publicID="Config/trunk" name="trunk" enabled="true">
<station publicID="Config/trunk/GE/UGM"
networkCode="GE" stationCode="UGM" enabled="true">
...
</station>
</module>
</Config>
and finally one Setup per module:
.. code-block:: xml
<Config>
<module publicID="Config/trunk" name="trunk" enabled="true">
<station publicID="Config/trunk/GE/UGM"
networkCode="GE" stationCode="UGM" enabled="true">
<setup name="default" enabled="true">
<parameterSetID>
ParameterSet/trunk/Station/GE/UGM/default
</parameterSetID>
</setup>
<setup name="scautopick" enabled="true">
<parameterSetID>
ParameterSet/trunk/Station/GE/UGM/scautopick
</parameterSetID>
</setup>
</station>
</module>
</Config>
Here two setups have been created: *default* (which is a special case for
module *global* to be backwards compatible) and *scautopick* where each
refers to a ParameterSet by its publicID. The next XML fragment shows
the ParameterSet referred by the scautopick setup of station GE.UGM:
.. code-block:: xml
<Config>
<parameterSet publicID="ParameterSet/trunk/Station/GE/UGM/scautopick"
created="...">
<baseID>ParameterSet/trunk/Station/GE/UGM/default</baseID>
<moduleID>Config/trunk</moduleID>
<parameter publicID="...">
<name>timeCorr</name>
<value>-0.8</value>
</parameter>
<parameter publicID="...">
<name>detecFilter</name>
<value>
RMHP(10)&gt;&gt;ITAPER(30)&gt;&gt;BW(4,0.7,2)&gt;&gt;STALTA(2,80)
</value>
</parameter>
<parameter publicID="...">
<name>trigOff</name>
<value>1.5</value>
</parameter>
<parameter publicID="...">
<name>trigOn</name>
<value>3</value>
</parameter>
</parameterSet>
</Config>
The mapping to the binding configuration files is 1:1. Each parameter in
the configuration file is exactly one parameter in the database and their
names are matching exactly.
The concept of global bindings which are specialized for each module is
reflected by the *baseID* of the ParameterSet which points to setup *default*
of station GE.UGM:
.. code-block:: xml
<Config>
<parameterSet publicID="ParameterSet/trunk/Station/GE/UGM/default"
created="...">
<moduleID>Config/trunk</moduleID>
<parameter publicID="...">
<name>detecStream</name>
<value>BH</value>
</parameter>
</parameterSet>
</Config>
This ends up with a final configuration for scautopick and station GE.UGM:
=========== ==================
Name Value
=========== ==================
detecStream BH
timeCorr -0.8
detecFilter RMHP(10)>>ITAPER(30)>>BW(4,0.7,2)>>STALTA(2,80)
trigOff 1.5
trigOn 3
=========== ==================
which is the concatenation of the two files :file:`etc/key/global/station_GE_UGM`
and :file:`etc/key/scautopick/station_GE_UGM`. The filter grammar is explained in
:ref:`filter grammar section <filter-grammar>`.
The :file:`etc/key` directory is only used to ease the configuration of bindings.
They are finally transferred to the database or converted to other
representations a module might require to access them. The directory :file:`etc/key` is meant
for backup and used for copying bindings from one computer to another.
Adjusting
~~~~~~~~~
:ref:`scconfig` provides a graphical interface to adjust the global and the module
bindings parameters and to create the bindings. It populates the :file:`etc/key`
directory and it takes care that they synchronize with the database when processing
*Update configuration* in the *System* panel.
.. figure:: ../media/scconfig_config_bindings.png
:alt: scconfig: bindings configuration
:align: center
:width: 18cm
scconfig modules bindings configuration panel.
Alternatively, you may manually generate the binding parameter file
:file:`etc/key/global/station_GE_UGM` and
:file:`etc/key/scautopick/station_GE_UGM` and add the binding to the key file
corresponding to the station, e.g.,
:file:`$SEISCOMP_ROOT/etc/key/station_GE_UGM`.´
SeisComP shell
~~~~~~~~~~~~~~
Instead of creating bindings in :ref:`scconfig` or by adjusting key files
manually you may assign binding profiles to stations or networks or remove them
using the |scname| shell which is a specific shell:
#. Start the |scname| shell:
.. code-block:: sh
seiscomp shell
#. Issue a shell command. Examples:
* Read the help of the new shell:
.. code-block:: sh
================================================================================
SeisComP shell
================================================================================
Welcome to the SeisComP interactive shell. You can get help about
available commands with 'help'. 'exit' leaves the shell.
$ help
* add the existing scautopick binding profile, *default*, to the station
*GE.UGM*:
.. code-block:: sh
$ set profile scautopick default GE.UGM
or to all stations of the network *GE*:
.. code-block:: sh
$ set profile scautopick default GE.UGM
* remove the scautopick binding profile, *default*, from the station *GE.UGM*:
.. code-block:: sh
$ remove profile scautopick default GE.UGM
* exit the shell and get back to the Linux shell:
.. code-block:: sh
$ exit
Example: global bindings
~~~~~~~~~~~~~~~~~~~~~~~~
A binding is a set of name value pairs just like a configuration file but for
a station. In the file system a binding looks like this:
.. code-block:: properties
detecLocid = ""
detecStream = HH
Simple. These are actually two parameters from the global binding and it solves
a common issue elegantly: a station might provide a couple of channels, often
data in various sampling rates, e.g. LH, BH, SH and HH. Co-located stations
with velocity and acceleration sensors also provide at least two channel groups,
e.g. HL and HH. Those groups are also provided with different location code,
e.g. 00 and 10. To process a station a module needs to know which channel
it should process. To display a representative data channel a viewer needs to
know which channel to show. Global bindings solve that issue by defining the
"preferred" location code and channel code with the two parameters `detecLocid`
and `detecStream`. The TraceView, :ref:`scrttv` can read all available station
bindings, compose a fully qualified channel name from network code (part of the
binding), station code, detecLocid and detecStream. It will then lookup the
inventory information for that channel and display it.
The meta data of a binding are:
* Network code,
* Station code,
* Module name.
The actual binding data are the key value pairs.
Reading bindings
~~~~~~~~~~~~~~~~
Binding configuration parameters are read during a module start after the module
configuration parameters.
They override the module configuration.
Like module configurations, parameters defined in the global bindings can be
overridden in module bindings. The order of loading is:
#. :file:`$SEISCOMP_ROOT/etc/key/global/`: Global bindings configurations
#. :file:`$SEISCOMP_ROOT/etc/key/[module]`: Bindings configurations for a particular module.
If the module connects to a messaging server then it will receive the database parameters,
connect to it and read the bindings. Otherwise the user has to provide the
database address. A module never reads the key directory. It only
gets the bindings from the configuration tables.
As with inventory information there might are cases when a module should
not connect to a database and work offline, in particular when ``--ep`` is being
used.
In order to
read the bindings configuration from XML files (again in SCML format),
use :ref:`scxmldump` to dump the configuration XML file and let
the module use this XML file, ``--config-db`` must be used:
.. code-block:: sh
$ myapp --config-db config.xml
That will completely bypass the database (even if used for event information or
inventory) for reading station bindings.
To extract a configuration XML file from the database, :ref:`scxmldump` can be used:
.. code-block:: sh
$ scxmldump -fC -d localhost -o config.xml
One can also create a config XML file straight from the ``etc/key`` directory
with ``bindings2cfg``:
.. code-block:: sh
$ bindings2cfg -o config.xml
To create an XML file from a temporary key directory, an alternative directory
can be specified with ``--key-dir``:
.. code-block:: sh
$ bindings2cfg --key-dir /tmp/key -o config.xml
Application
~~~~~~~~~~~
Binding configurations are read when starting a module and cannot be
changed during runtime. For applying any change on binding configurations
#. Save the configuration,
#. Update the configuration
.. code-block:: sh
$ seiscomp update-config
#. Restart or execute all affected modules.
Summary
~~~~~~~
* Bindings are maintained as key files in :file:`etc/key`,
* Key files are another human readable representation of bindings,
* ``seiscomp update-config`` or ``seiscomp update-config trunk`` writes the
information from :file:`etc/key` to the database,
* A module never reads :file:`etc/key`,
* Bindings are being read from the database or an XML file.
Validation
----------
After changing configuration the affected modules should be tested, e.g. by
running with informative debug logging output. You may use :ref:`scdumpcfg` for
dumping the module an binding parameters into a summary file.
.. _concepts_configuration_parameters:
Format of Parameters
====================
The :term:`trunk` configuration files are simple text files where each line
is a name-value pair containing the parameter name and its value.
Parameter values can be provided as
* plain values or comma-separated lists on single or multiple lines
(read sections below) or as
* :ref:`variables <concepts_configuration_variables>`, allowing to refer to
previously defined parameters, e.g., in :file:`global.cfg` or to define relative
directories related to the |scname| installation or the Linux system.
.. warning::
In contrast to previous versions of |scname| the parameter names are now
case-sensitive. Use :ref`scchkcfg` to check configurations from previous
versions regarding case-sensitivity.
Basic
-----
Module and binding configuration files are simple text file where each line is a
name-value pair for one parameter. The parameter names are case-sensitive. The
format is a simple as:
.. code-block:: properties
agencyID = gempa
recordstream = slink://localhost:18000
Spaces in string parameters must be protected by quotes:
.. code-block:: properties
eventlist.filter.types.blacklist = "not existing"
Parameter groups are indicated by a separating dot ("."). The dot separates the
group from the parameter name or other groups. The item after the last dot is
the parameter.
Later assignments of parameters override previous ones so the order of lines in the
configuration file is important. The file is parsed top-down.
.. note::
Values are not type-checked. Type checking is part of the module
logic and will be handled there. The configuration file parser will not raise
an error if a string is assigned to a parameter that is expected to be an
integer.
Comments
--------
Everything following an unescaped **#** (hash) is a comment and is going to
be ignored. Blank lines and white spaces are ignored by the parser as well
unless quoted or escaped. Escaping is done by prepending a backslash (**\\\\**)
to the character that needs escaping. It backslash should be part of the
string, two backslashes should be used (**\\\\\\\\**).
.. code-block:: properties
agencyID = gempa # This is a comment
# The preceding empty line is ignored. Now the value of skyColor is replaced
# with 'blue'.
# The final value of a parameter is the result of the last assignment parsed
# from top to bottom.
agencyID = gempa
Lists
-----
Values can be either scalar values or lists. List items are separated by commas.
.. code-block:: python
# This is a list definition
rainbowColors = red, orange, yellow, green, blue, indigo, violet
If a value needs to include a comma, white space or any other special
character it can either be escaped with backslash ('\\') or quoted with double
quotes ("). Whitespaces are removed in unquoted and unescaped values.
.. code-block:: properties
# This is a comment
# The following list definitions have 2 items: 1,2 and 3,4
# quoted values
tuples = "1,2", "3,4"
# escaped values
tuples = 1\,2, 3\,4
The value of the parameter tuples is now `["1,2", "3,4"]`.
Multi-line
----------
Values can extend over multiple lines if a backslash is appended to each line
.. code-block:: properties
# Multi-line string
text = "Hello world. "\
"This text spawns 3 lines in the configuration file "\
"but only one line in the final value."
# Multiline list definition
rainbowColors = red,\
orange,\
yellow,\
green, blue,\
indigo, violet
Control characters
------------------
A limited set of control characters is allowed within strings.
.. csv-table::
:widths: 10 90
:align: left
:delim: ;
\\n; new line
\\t; tab
Example of a string consisting of two lines:
.. code-block:: properties
a = "123 456"\n"This is a new line with text"
The control characters must be outside of double quotes. Everything **within**
double quotes will **not** be decoded. Hence, "\n" will end up as the string
"\n" and not a new line character.
Namespaces
----------
A basic usage of variable names is to organize them in namespaces. A common
habit is to separate namespaces and variable names with a period character:
.. code-block:: properties
colors.sky = blue
colors.grass = green
Here a namespace called ``colors`` is used. The configuration file parser does
not care about namespaces at all. The final name (including the periods) is what
counts. But to avoid repeating namespaces again and again, declarations can
be wrapped in a namespace block. See the following example:
.. code-block:: properties
colors {
sky = blue
grass = green
}
Application code will still access ``colors.sky`` and ``colors.grass``.
Namespaces can be arbitrarily nested and even survive includes.
.. code-block:: properties
A {
B1 {
var1 = 123
}
B2 {
var1 = 456
}
}
The final list of parameter names is:
* A.B1.var1
* A.B2.var1
.. _concepts_configuration_variables:
Variables
=========
Environment or preceding configuration variables (configuration parameters) can
be used as values for :ref:`parameters <concepts_configuration_parameters>` in
the configuration of SeisComP modules with `${var}`, e.g.
.. code-block:: properties
plugins = ${plugins}, hypo71
homeDir = ${HOME}
myPath = ${homeDir}/test
myseiscomp = ${SEISCOMP_ROOT}
|scname| knows internal variables defining the it environment. The can be used
with `@var@`, e.g.
.. code-block:: properties
autoloc.stationConfig = @DATAGDIR@/autoloc/station.conf
Available internal |scname| variables are:
+------------------+-----------------------------+
| Variable | Value |
+==================+=============================+
| ROOTDIR | $SEISCOMP_ROOT |
+------------------+-----------------------------+
| DEFAUTLCONFIGDIR | $SEISCOMP_ROOT/etc/defaults |
+------------------+-----------------------------+
| SYSTEMCONFIGDIR | $SEISCOMP_ROOT/etc |
+------------------+-----------------------------+
| DATADIR | $SEISCOMP_ROOT/share |
+------------------+-----------------------------+
| CONFIGDIR | $HOME/.seiscomp |
+------------------+-----------------------------+
| LOGDIR | $HOME/.seiscomp/log |
+------------------+-----------------------------+
The list of internal |scname| variables can also be read in the information
panel of :ref:`scconfig`.
.. figure:: ../media/scconfig_information.png
:alt: scconfig: information panel
:align: center
:width: 18cm
scconfig information panel indicating the internal |scname| variables.
The internal |scname| variable CONFIGDIR can be re-defined by the SHELL
environment variable SEISCOMP_LOCAL_CONFIG. Setting SEISCOMP_LOCAL_CONFIG will
also effect LOGDIR which is automatically set to
:file:`$SEISCOMP_LOCAL_CONFIG/log`.
Example:
.. code-block:: sh
export SEISCOMP_LOCAL_CONFIG=/var/log/seiscomp
Related Tools
=============
* :ref:`bindings2cfg`
* :ref:`scchkcfg`
* :ref:`scconfig`
* :ref:`scdumpcfg`
* :ref:`scxmldump`

117
share/doc/seiscomp/html/_sources/base/concepts/database.rst.txt Normal file
View File

@ -0,0 +1,117 @@
.. _concepts_database:
********
Database
********
Scope
=====
This chapter provides an overview over databases supported by |scname|.
Overview
========
|scname| can store and read information from a relational database management
system (RDBMS). Supported are basically all existing RDBMS for which a plugin
can be written. Currently, :ref:`database plugins <concepts_plugins>` are
provided for
.. csv-table::
:widths: 1 1
:header: Database, Plugin Name
:align: left
MySQL / MariaDB, *dbmysql*
PostgreSQL, *dbpostgresql*
SQLite3, *dbsqlite3*
Database access
---------------
Typically, the database is accessed by the messaging (:ref:`scmaster`) for
reading and writing.
Most other modules can only read from the database but do not write into it.
Among the few exceptions which can also directly write to the database are
:ref:`scdb` and :ref:`scardac`.
The database connection provided by the messaging is configured by the
:ref:`scmaster module configuration <scmaster>`. :ref:`Modules <concepts_modules>`
connected to the messaging receive the read connection parameters through the
messaging connection. However, the default read connection by these and all
other modules may be set with :confval:`database` in
:ref:`global configuration <global-configuration>` or set on the command line
using :option:`--database` or simply :option:`-d`.
Read the sections :ref:`installation` and :ref:`getting-started` on the
installation and the configuration of the database backend and the initial setup
of the database itself, respectively.
The database connection may be used together with the *debug* option to print
the database commands along with debug log output. Example for using
:ref:`scolv` in offline mode with database debug output:
.. code-block:: sh
scolv -d localhost?debug --offline --debug
Database schema
---------------
The used database schema is well defined and respected by all modules which
access the database. It is similar to the SeisComML schema (:term:`SCML`,
a version of XML) and the C++ / Python class hierarchy of the datamodel
namespace / package.
Information of the following objects can be stored in the database as set out in
the :ref:`documentation of the data model <api-datamodel-python>`.
.. csv-table::
:widths: 25, 75
:header: Object, Description
:align: left
:delim: ;
:ref:`Config <concepts_configuration>`; station bindings
:ref:`DataAvailability <api-datamodel-python>`; information on continuous data records
:ref:`EventParameters <api-datamodel-python>`; derived objects like picks, amplitudes, magnitudes origins, events, etc.
:ref:`Inventory <concepts_inventory>`; station meta data
:ref:`Journaling <api-datamodel-python>`; information on commands and actions, e.g., by :ref:`scevent <scevent-journals>`
:ref:`QualityControl <api-datamodel-python>`; waveform quality control parameters
.. note::
The Config parameters just cover station bindings. Application/module specific
configurations (all .cfg files) are not stored in the database and only kept
in files.
The currently supported version of the database schema can be queried by any
module connecting to the data base using the option :option:`-V`. Example:
.. code-block:: sh
$ scm -V
scm
Framework: 6.0.0 Development
API version: 16.0.0
Data schema version: 0.12
GIT HEAD: 5e16580cc
Compiler: c++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Build system: Linux 6.2.0-26-generic
OS: Ubuntu 22.04.3 LTS / Linux
Related Modules
===============
* :ref:`scardac`
* :ref:`scdb`
* :ref:`scdbstrip`
* :ref:`scdispatch`
* :ref:`scquery`
* :ref:`scqueryqc`
* :ref:`scxmldump`

244
share/doc/seiscomp/html/_sources/base/concepts/inventory.rst.txt Normal file
View File

@ -0,0 +1,244 @@
.. _concepts_inventory:
*********
Inventory
*********
Scope
=====
This chapter describes the concept of inventories.
Overview
========
In |scname| the term :term:`inventory` refers to the meta data of data recording
stations.
Many |scname| :ref:`modules <concepts_modules>` require inventory information to
operate correctly and a correct inventory is therefore fundamental to |scname|.
The inventory will be read from the :ref:`database <concepts_database>`
by default. To insert or update the inventory in the database, this information
needs to be created or downloaded and converted. The following sections will
describe the process of populating the database and how applications get access
to it.
The format used for inventory is known as :term:`SCML`. Correct inventory in
:term:`SCML` format can be generated by :cite:t:`smp`. Some modules for
**converting from other formats** are listed in section
:ref:`concepts_inventory_format`. Tools for **processing inventory** are
listed in section :ref:`concepts_inventory_tools`.
The inventory shall contain all meta data describing the full recording system
and the
pre-processing of the raw data stream provided to the data processing modules
using the :ref:`RecordStream <concepts_recordstream>`. Inventories should be
always complete w.r.t the
processed data streams, correct and up-to-date. Beside network, station, location
and stream information they must include the full sensor and datalogger responses.
Older version used key files to configure available networks and stations.
Because the support of response meta-data was very limited, tools were build to
add this functionality. Since the release of Seattle the concept of key files
for station meta-data has been completely removed from the system. Now |scname|
only handles station meta-data in its own XML format called :term:`SCML`.
The task of supporting old key files, dataless SEED and other formats has been
out-sourced to external applications. The inventory synchronization is a
:ref:`two-stage process <config-fig-inventory-sync>`:
#. Convert :ref:`external formats <concepts_inventory_format>` into :term:`SCML`
#. Synchronize inventory pool with the database: merged all inventory files and
synchronize with the database using :program:`scinv snyc`.
.. _config-fig-inventory-sync:
.. figure:: ../media/inventory-sync.png
:align: center
:width: 12cm
Inventory synchronization as a two-stage process
All station meta-data are stored in :file:`etc/inventory`
and can be organized as needed. Either one file per network, a file containing the
complete inventory or one file for all instruments and one file per station.
The update script
.. code-block:: xml
$ scinv sync
loads the existing inventory from the database and merges each file in :file:`etc/inventory`.
Finally it removes all unreferenced objects and sends all updates to the database.
The |scname| :ref:`configuration <concepts_modules>` does not deal with station
meta-data anymore.
It only configures parameters for modules and module-station associations.
The management of the inventory can and should be handled by external tools
e.g. :cite:t:`smp`.
The |scname| documentation describes the
:ref:`data model including the inventory <api-datamodel-python>`.
.. _concepts_inventory_format:
Inventory Format
================
A typical inventory file in :term:`SCML` looks like this:
.. code-block:: xml
<?xml version="1.0" encoding="UTF-8"?>
<seiscomp xmlns="http://geofon.gfz-potsdam.de/ns/seiscomp-schema/0.11" version="0.11">
<Inventory>
<sensor>...
<datalogger>...
<network>...
</Inventory>
</seiscomp>
The version attribute of the ``seiscomp`` tag represents the schema version of
the |scname| data model which is consistent with the database schema version
and the version of all other representations.
.. note::
All geographic coordinates (latitudes, longitudes, elevation) are assumed in the
World Geodetic System 1984 (WGS84) reference system (National Imagery and
Mapping Agency 2000). Latitudes, longitudes are provided in degrees but
elevations are given in meters.
Inventories must be provided to |scname| in XML files in :term:`SCML` format. A
convenient way to generate clean and correct inventory files in :term:`SCML`
format is :cite:t:`smp`. Tools are provided to convert between other formats:
.. csv-table::
:widths: 1 1
:header: Module namespace, Conversion
:align: left
arclink2inv, Arclink XML to SeisComPML
:ref:`dlsv2inv`, dataless SEED to SeisComPML
:ref:`inv2dlsv`, SeisComPML to dataless SEED
:ref:`fdsnxml2inv`, FDSN StationXML to SeisComPML and back
Adding / Updating Inventory
===========================
To add inventory information to the |scname| database one either has to write directly
to the database with custom script (not recommended) or place :term:`SCML` files
in :file:`@SYSTEMCONFIGDIR@/inventory`. The service to import all data in that
directory can be called with
.. code-block:: sh
$ seiscomp update-config inventory
This command runs :ref:`scinv` finally. :ref:`scinv` merges all XML files and
synchronizes the merged local inventory tree with the database. That is a
rather complex process and for more information it is recommended to study
the source code of :ref:`scinv`. The bottom line is that inventory data is created
from :term:`SCML` files.
Because nobody will ever create such XML files by hand, tools are necessary.
A quite popular tools is the online station management portal (SMP)
:cite:p:`smp`. SMP will output XML in version of :term:`SCML` directly.
Another source is the FDSN station webservice (FDSNWS). If the |scname|
implementation of FDSNWS is used, the SeisComP XML format can be requested
directly as an extension. Otherwise FDSN StationXML will be provided. Inventory
in FDSN StationXML needs to be converted with either :ref:`fdsnxml2inv` or
:ref:`import_inv`. The latter is a more versatile application that also supports
several import formats.
.. code-block:: sh
$ import_inv fdsnxml download-1234.xml
This will automatically place the output XML file in
:file:`@SYSTEMCONFIGDIR@/inventory`. Afterwards call
.. code-block:: sh
$ seiscomp update-config inventory
to synchronize the changes with the database. If :program:`scconfig` is used,
then either the ``Sync`` button of the `Inventory` panel or the
``Update configuration`` button of the ``System`` panel must pressed.
**Summary**
In order to populate the database with inventory information, the following
steps have to be performed:
#. Convert existing station meta data formats to :term:`SCML`
#. Place all :term:`SCML` files at :file:`@SYSTEMCONFIGDIR@/inventory`
#. Synchronize the inventory files and write the meta data to the database. Run
.. code-block:: sh
$ scinv sync
or
.. code-block:: sh
$ seiscomp update-config inventory
Reading Inventory
=================
Applications usually connect to the database and read the necessary inventory
information. An application can decide whether it requires full response
information including sensor and data logger response functions or just channel
information without instrument descriptions. The latter performs faster and
some applications do not require full instrument information.
An application usually does not require special configuration to read inventory
information. A database connection is enough and it comes usually along with the
handshake message of the messaging server.
If the messaging is not involved, the database can be specified by the
command-line option ``-d``:
.. code-block:: sh
$ myapp -d [type]://[user]:[password]@[host]:[port]
There are cases when an application should be run without a database connection
but requires inventory information, particularly in combination with the
``--ep`` command line argument. To direct an application to an inventory XML file
(again in :term:`SCML` format), ``--inventory-db`` must be used:
.. code-block:: sh
$ myapp --inventory-db inventory.xml
The option ``--inventory-db`` will cause the module to completely bypass
the database for reading inventory information even if used for event
information. The file :file:`inventory.xml` can be created from the database
using :ref:`scxmldump`.
.. _concepts_inventory_tools:
Related Tools
=============
* arclink2inv
* :ref:`dlsv2inv`
* :ref:`fdsnxml2inv`
* :ref:`import_inv`
* :ref:`invextr`
* :ref:`inv2dlsv`
* :ref:`sccnv`
* :ref:`scinv`
* :ref:`scxmldump`
* :ref:`tab2inv`
* :ref:`tabinvmodifier`

82
share/doc/seiscomp/html/_sources/base/concepts/locators.rst.txt Normal file
View File

@ -0,0 +1,82 @@
.. _concepts_locators:
Locators
########
Locators receive :term:`phase picks <pick>` from modules such as :ref:`scautoloc`,
:ref:`screloc` or :ref:`scolv` for locating seismic or non-seismic sources. The
solutions may include source time and location with or without uncertainties.
They are used to form new :term:`origins <origin>` which can be treated
further.
|scname| ships with built-in locators:
* :ref:`FixedHypocenter <global_fixedhypocenter>` (FH)
* :ref:`Hypo71 <global_hypo71>`
* :ref:`iLoc <global_iloc>`
* :ref:`LOCSAT <global_locsat>`, the default locator in :ref:`scautoloc` and :ref:`scolv`
* :ref:`NonLinLoc <global_nonlinloc>`
* :ref:`StdLoc <global_stdloc>`
and a none built-in locator:
* :ref:`LocExt <global_locext>`.
While the built-in locators are well defined and documented, any other external
locator routine can be added to |scname| by configuration of the locator
:ref:`LocExt <global_locext>` and custom scripts.
LOCSAT and FixedHypocenter are native to |scname|. All other locators are
implemented as :term:`plugins <plugin>`. A :term:`plugin`
needs to be added to the list of loaded plugins by configuration of the global
parameter :confval:`plugins` for making the corresponding locator available to
|scname| applications.
A comparison of the locators is given in the table below.
.. note::
The specifications given below may depend on the configuration of the
respective locator. Please carefully read the documentation of the locators
for optimizing their performance.
.. csv-table::
:widths: 30 10 10 10 10 10 10 10
:header: , FH, Hypo71, iLoc, LocExt, LOCSAT, NonLinLoc, StdLoc
:align: center
**Applications**, ,,,,,,
phases considered by default, seismic / infrasound, seismic, seismic / infrasound / hydroacoustic, [3], seismic / infrasound, seismic, seismic
distance ranges of application, local / regional / teleseismic, local / regional, local / regional / teleseismic, [3], local / regional / teleseismic, local / regional / teleseismic, local / regional [4]
application with default configuration, regional / teleseismic, ❌, regional / global, [3], regional / teleseismic, ❌, local / regional [1]
**Algorithm**, ,,,,,,
inversion algorithm, linear, iterative, configurable, [3], grid search, probabilistic, configurable
automatic phase renaming, ❌, ❌, ✅, [3], ❌, ❌, ❌
considers network code, ✅, ❌, ✅, [3], ✅, ✅ [1], ✅
positive station elevation, ✅ [2/4], ✅, ✅, [3], ✅ [2], ✅, ✅
negative station elevation, ❌, ✅, ✅, [3], ❌, ✅, ✅
considers pick time, ✅, ✅, ✅, [3], ✅, ✅, ✅
considers pick slowness, ❌, ❌, ✅, [3], ✅, ❌, ❌
considers pick backazimuth, ❌, ❌, ✅, [3], ✅, ❌, ❌
speed, fast, fast, fast - intermediate, [3], fast, intermediate, fast - intermediate
**Velocity model**, ,,,,,,
velocity model, 1D [4], 1D, 1D, [3],1D, 1D / 3D, 1D / 3D [4]
independent Vp and Vs, ✅ [4], ❌, ✅, [3], ✅, ✅, ✅ [4]
default velocity model, iasp91 / tab, ❌, iasp91 / ak135, [3], iasp91 / tab, ❌, iasp91 / tab [1]
applies RSTT, ❌ , ❌, ✅, [3], ❌, ❌, ❌
**Hypocenter solution**, ,,,,,,
inverts for hypocenter location, ❌, ✅, ✅, [3], ✅, ✅, ✅
inverts for hypocenter time, ✅, ✅, ✅, [3], ✅, ✅, ✅
supports negative source depth, ❌, ✅, ❌, [3], ❌, ✅, ✅
**Configuration**, ,,,,,,
native or plugin to load, ✅, *hypo71*, *lociloc*, *locext*, ✅, *locnll*, *stdloc*
|scname| provides locator, ✅, ✅, ✅, ❌, ✅, ✅, ✅
operates without external files, ✅, ❌, ❌, ❌, ✅, ❌, ✅
operates without custom scripts, ✅, ✅, ✅, ❌, ✅, ✅, ✅
**Others**, ,,,,,,
remarks, intended for ground-truth tests / single-station location / any travel-time interface, ,operational at EMSC and ISC (earlier version), any external locator can be called by a custom script, currently the fastest locator in |scname| and the only one available to :ref:`scautoloc`, considers model uncertainties, uses travel-times from any travel-time interface
point of contact, :cite:t:`seiscomp-forum`, :cite:t:`seiscomp-forum`, `ibondar2014 @gmail.com <ibondar2014@gmail.com>`_, :cite:t:`seiscomp-forum`, :cite:t:`seiscomp-forum`, :cite:t:`seiscomp-forum`, :cite:t:`seiscomp-forum`
* [1]: requires initial or specific configuration
* [2]: requires correction file
* [3]: depends on selected locator
* [4]: depends on selected travel-time interface

175
share/doc/seiscomp/html/_sources/base/concepts/magnitudes.rst.txt Normal file
View File

@ -0,0 +1,175 @@
.. _concepts_magnitudes:
Magnitudes
##########
Magnitudes are computed based on amplitudes measured from waveforms. Different
types of amplitudes and magnitudes are available which are listed in
:ref:`scamp` and :ref:`scmag`.
Amplitudes
==========
Amplitudes can be measured automatically from waveforms
* During phase picking by :ref:`scautopick` with generally fixed time windows
due to the absence of knowledge about source parameters or by,
* :ref:`scamp` as soon as :term:`origins <origin>` are available. Depending
on the magnitude type, fixed or distance-dependent time windows apply.
and interactively using :ref:`scolv`.
Instrument simulation
---------------------
Amplitude measurements for some magnitude types require or allow the simulation
of instruments such as :py:func:`Wood-Anderson torsion seismometers <WA>`
(:cite:t:`richter-1935,uhrhammer-1990`), :py:func:`WWSSN_SP` or :py:func:`WWSSN_LP`.
The calibration parameters describing the Wood-Anderson seismometer are
configurable in global bindings or global module configuration:
:confval:`amplitudes.WoodAnderson.gain`, :confval:`amplitudes.WoodAnderson.T0`,
:confval:`amplitudes.WoodAnderson.h`. Specifically, the difference in magnitude
due to configuration using original values listed in
:cite:t:`richter-1935` and updated ones given in :cite:t:`uhrhammer-1990`
result in a constant offset of 0.13 in those magnitudes which apply
Wood-Anderson simulation, e.g. :term:`ML <magnitude, local (ML)>`,
:term:`MLv <magnitude, local vertical (MLv)>`, :term:`MLc <magnitude, local custom (MLc)>`.
Station Magnitudes
==================
Station magnitudes are computed automatically by :ref:`scmag` or interactively
by :ref:`scolv` from measured amplitudes based on distance-dependent
calibration curves which depend on magnitude type. When computing a set of
magnitudes in :ref:`scolv` which is different from the set configured in
:ref:`scmag`, then scmag may later add the missing magnitudes automatically.
Magnitude types for which the evaluation status is set to "rejected", e.g., in
scolv, will not be recomputed by scmag.
.. _concepts-magnitudes-correction:
Station corrections
-------------------
Linear station corrections applied to station magnitudes can be configured by
global :ref:`binding parameters <global_bindings_config>`:
#. Add a magnitude type profile where the name of the profile is the name of the
magnitude itself,
#. Configure the correction parameters.
When using binding profiles, all referencing stations will be affected equally
which is typically not intended. In contrast, applying station bindings requires
to set up many bindings which may not be intended either.
Therefore, you may add lines to the global module configuration in
:file:`global.cfg` where one line corresponds to one station with one magnitude
and the corresponding correction parameter. The groups and the name of the
parameters are identical to the global bindings parameters. All lines start with
"*module.trunk*". Example for an offset correction of
:term:`MLv <magnitude, local vertical (MLv)>` measured station GE.UGM:
.. code-block:: properties
module.trunk.GE.UGM.magnitudes.MLv.offset = 0.1
.. note::
The configuration of parameters starting with *module.trunk.* is not
supported by :ref:`scconfig`. All corresponding configurations must be done
by direclty editing the configuration file, e.g.,
:file:`seiscomp/etc/global.cfg`.
Network Magnitudes
==================
Network magnitudes are computed automatically by :ref:`scmag` or interactively
by :ref:`scolv` from station magnitudes based on averaging station magnitudes.
The averaging methods applied by :ref:`scmag` are configurable by
:confval:`magnitudes.average`. Available are (:cite:t:`rosenberger-1983`):
* *mean*: the mean value from all station magnitudes.
* *median*: the mean value from all station magnitudes.
* *trimmedMean(X)*: gnores outlier station magnitudes by first removing the
largest and the smallest *X* % of the observed values (percentiles). The mean is
formed from the remaining station magnitudes.
* *trimmedMedian(X)*: forms the median from all station magnitudes but returns
the uncertainty by ignoring the largest and the smallest *X* % station
magnitudes.
* *medianTrimmedMean(X)*: returns the mean magnitude from all station magnitudes
differing less than *X* magnitudes from the median.
Aliases
=======
New magnitude types (aliases) can be created based on existing magnitude and
amplitude types but configured specifically.
The setup procedure is outlined in the
:ref:`tutorial on magnitude aliases <tutorials_magnitude-aliases>`.
.. _concepts-magnitudes-regionalization:
Regionalization
===============
The computation of station magnitudes can be regionalized. This means that for
a specific region specific conditions apply when computing magnitudes. The
conditions include any parameter available for configuring a magnitude
including global binding parameters such as magnitude calibration, distance
and depth ranges, etc. As an example you may wish to apply different
attenuation curves for computing MLv magnitudes to earthquakes in Eastern and
in Western Canada.
Regionalization is achieved by adding magnitude-type profiles in the magnitudes
section of global module configuration parameters. Regionalization assumes
defaults from global bindings but overrides the values when configured. The
setup procedure including
:ref:`station corrections <concepts-magnitudes-correction>` is outlined in the
:ref:`tutorial on regionalization <tutorials_magnitude-region-aliases>`.
Moment Magnitudes
=================
Moment magnitudes can be derived from all other network magnitudes by mapping of
the original network magnitude, e.g., *Mx*, to a new moment magnitude *Mw(Mx)*.
The mapping function can be configured for all original magnitude types except
:term:`mB <magnitude, derived mB (Mw(mB))>` and
:term:`Mwp <magnitude, derived Mwp (Mw(Mwp))>` where the mapping is hardcoded.
Read the :ref:`tutorial on moment magnitudes <tutorials_mags_moment>` for the
configuration.
.. _concepts-magnitudes-summary:
Summary Magnitude
=================
In order to account for different phenomena related to magnitude computation
including magnitude saturation and application of different magnitude types at
specific distance and depth ranges of the sources a summary magnitude can be
computed from network magnitudes by :ref:`scmag`. The summary magnitude is
usually referred to as *M*. The name is configurable.
.. note::
Station, network and summary magnitudes are contained uniquely in one
:term:`origin`.
Preferred Magnitude
===================
From the list of computed network magnitudes and the summary magnitude,
:ref:`scevent` can automatically determine the preferred magnitude of the
:term:`event`. This may also be done interactively by operators in the
:ref:`Event tab of scolv <scolv-sec-event-tab>` or by
:ref:`custom commit buttons in scolv <sec-scolv-custom-commit>`.

242
share/doc/seiscomp/html/_sources/base/concepts/messaging.rst.txt Normal file
View File

@ -0,0 +1,242 @@
.. _concepts_messaging:
****************
Messaging system
****************
Scope
=====
This chapter describes the messaging system used for exchanging parameter messages between
different :ref:`SeisComP modules <concepts_modules>` during runtime.
Overview
========
A typical real-time |scname| system consists of data processing and other
modules (clients) and a messaging bus. The messaging bus connects all the
clients letting them exchange information by messages through this bus.
The messaging system is a fundamental concept of SeisComP following the
publish-subscribe pattern :cite:p:`wppubsub`. It is provided by the |scname|
module :ref:`scmaster`.
Clients can be producers (sending messages) and/or consumers (receiving
messages). The basic concept is really simple: A producer sends a message
(some event parameter, arbitrary content) and a consumer read that message and tries to
process it. That's it. That is a very generic approach which requires clients
to understand the message content of each other.
In |scname| this common messaging language consists of well defined message
types and contents. The vast majority of messages sent around are so called
**notifiers**.
Notifiers inform about changes on the data model. They carry just three
attributes:
* The unique identifier of the parent object
* The operation to apply (add, update or remove)
* The subject
Speaking XML, a notifier looks like this:
.. code-block:: xml
<?xml version="1.0" encoding="UTF-8"?>
<seiscomp xmlns="http://geofon.gfz-potsdam.de/ns/seiscomp-schema/0.11" version="0.11">
<Notifier parentID="EventParameters" operation="add">
<Pick publicID="Pick/....">
<!-- All pick attributes will be placed here -->
</Pick>
</Notifier>
</seiscomp>
Each client that receives such a notifier will take further actions if the
information meets its interest. A simple example is :ref:`scautoloc`. It
receives Pick objects in the form as written above. It stores the pick to its
internal data structures and when there are enough picks to form an Origin,
it does its processing and probably sends an Origin as a result (again as
notifier).
.. figure:: ../media/concepts/messaging/system.*
:alt: sketch of a distributed system
:align: center
Schematic view on a distributed SeisComP system.
.. _messaging-queue :
Queues
======
The messaging server (:ref:`scmaster`) manages a set of queues. Each queue is
independent and clients can only connect to *one* queue at a time within one
connection. Messages sent to a queue are sorted by their arrival time and
forwarded to clients in exactly that order. That is called a FIFO (first in
first out) queue.
For each queue a set of processors can be added where each message is
piped through. One example is the dbstore processor which applies notifiers to
a configured relational database management system (RDBMS).
One could configure scmaster with two queues, one for production which
populates a database and one for playbacks which does not populate a database.
A client can be directed to either the one or the other queue:
.. code-block:: sh
connection.server = localhost/production
or
.. code-block:: sh
scautopick -H localhost/production
.. _messaging-groups :
Groups
======
Groups are like topic filters for a message. A message is always being sent to
a specific group or a specific client (peer-to-peer). A client can subscribe
to one or many groups and it will only receive messages tagged with a group it
has subscribed to.
For example :ref:`scautopick` processes waveforms sending detections or
picks as pick notifiers to the PICK group. All clients subscribed to that group
would also receive this information. If the queue is configured with the
dbstore processor then it is guaranteed that the pick has been stored already
in the database at the time of message arrival at the clients.
A consumer of the PICK group messages is :ref:`scautoloc`. Whenever it receives
a new pick it will try to associate it or to locate a seismic event. After it
succeeded it will send an origin to the LOCATION group and so on.
.. _messaging-scheme :
Scheme
======
The messaging allows unencrypted or encrypted connections which are specified by
the scheme parameter of the connection:
* `scmp` (default): unencrypted,
* `scmps`: encrypted.
Scheme, host, port and :ref:`queue <messaging-scheme>` together form the
connection URL of the messaging system which is configurable in
:confval:`connection.server` or set by the command-line option ``-H``.
Examples:
* Connect to the production queue of the messaging on localhost with default port.
Not using encryption `scmp` can be assumed implicitly in global configuration ::
connection.server = localhost/production
On the command line use, e.g. ::
$ scolv -H localhost
* Connect to the production queue of the messaging on localhost with default port.
Using non-default secure encryption and port the scheme and the port must be
provided explicitly in global configuration ::
connection.server = scmps://localhost18181/production
On the command line use, e.g. ::
$ scolv -H scmps://localhost:18181/production
.. _messaging-db :
Database
========
The :ref:`database <concepts_database>` is populated by :ref:`scmaster` with all
kind of information which might depend on
each other. It is crucial for operations that write operations are well
synchronized. Part of the messaging contract is that clients get read-only
database access from :ref:`scmaster` and that they will never attempt to write
into the database. In a fully-fledged |scname| system the only instance that is
allowed to populate the database is the dbstore plugin of :ref:`scmaster`.
When connecting from a client to the messaging, the database connection parameters
are reported by :ref:`scmaster`. They can be overridden by the global configuration
or command-line option ``-d``.
Example: ::
scolv -H localhost -d mysql://sysop:sysop@localhost/seiscomp-test
.. _messaging-distribution :
Module distribution
===================
The messaging server accepts TCP/IP connections enabling a module actually
running anywhere in the network. It happens that this concept is not understood
and users run on each computer another :ref:`scmaster` instance to offload
processing to more computers. Unfortunately each scmaster instance is
configured with the same database connection which will most likely cause data
inconsistencies because each group of modules which is connected to an scmaster
instance does not know about the other scmaster clients and therefore ignore
the sent updates.
The correct approach is to run **one** scmaster on computer A which populates
database X. Modules running on computer A, use
.. code-block:: sh
connection.server = localhost/production
while modules running on computer B use
.. code-block:: sh
connection.server = computerA/production
The database connection which is used by :ref:`scmaster` will be reported to the
clients when they connect so no explicit database configuration is necessary.
The messaging connection can be explicitly provided on the command line using
the option ``-H`` considering the comments on the
:ref:`scheme <messaging-scheme>` and :ref:`queue <messaging-queue>`, e.g.
.. code-block:: sh
scolv -H computerA/production
Web Frontend
============
When running :ref:`scmaster` a web frontend is available which can be adjusted to
provide system information. The default port to access the web frontend is 18180:
.. code-block:: sh
firefox localhost:18180
.. figure:: ../media/concepts/messaging/scmaster_web.png
:alt: scmaster: web frontend
:align: center
:width: 10cm
scmaster: Web frontend
Related Modules
===============
* :ref:`scmaster`
* :ref:`scm`
* :ref:`scmm`

199
share/doc/seiscomp/html/_sources/base/concepts/modules.rst.txt Normal file
View File

@ -0,0 +1,199 @@
.. _concepts_modules:
****************
SeisComP modules
****************
Scope
=====
This document describes the difference between command-line and daemon modules.
Overview
========
SeisComP is not a single executable but it provides a set of :term:`trunk` and
:term:`standalone <standalone module>` :term:`modules <module>`. Trunk modules
can be
* :ref:`Command-line modules <concepts_modules_commandline>`,
* :ref:`Daemon modules <concepts_modules_daemon>`.
Modules are :ref:`configured by configuration files <concepts_modules_config>`
either to be used directly or to generate its native configuration. Modules that
need to convert the configuration or do not use the default configuration
options (see below) are called :term:`standalone modules <standalone module>`.
All other modules are called :term:`trunk` modules.
Examples for standalone modules are :ref:`seedlink`, :ref:`slarchive` and :ref:`slmon`.
.. note::
In order to start or execute modules without :command:`seiscomp exec`
(see below), the |scname| SHELL environment variables must be known to the
system. The variables and their values can be printed giving the full path to
the :ref:`seiscomp` tool. Example:
.. code-block:: sh
/home/sysop/seiscomp/bin/seiscomp print env
.. _concepts_modules_daemon:
Daemon Tools
============
Daemon tools are |scname| modules that can run automatically in the background
without user interaction, e.g., for automatic data acquisition or data
processing. The names of all daemon modules are listed when executing the
:ref:`seiscomp` tool with :command:`list modules`:
.. code-block:: sh
seiscomp list modules
Daemon modules can be started to run in the background:
.. code-block:: sh
seiscomp start scautopick
When starting a daemon module, all verbosity output is logged to
:ref:`log files <concepts_modules_logging>`. Daemon modules can also be executed
as :ref:`command-line tools <concepts_modules_commandline>`.
.. _concepts_modules_commandline:
Command-Line Tools
==================
All non-daemon modules are command-line modules. These command-line modules
and most :ref:`daemon modules <concepts_modules_daemon>` can be executed on
demand from the SHELL command-line. These modules can also be
utilities or :term:`graphical user interfaces (GUIs) <GUI>`.
For executing use the full path to the :ref:`seiscomp` tool or, when the
|scname| environment is known, just call the module name along with command-line
options, e.g.
.. code-block:: sh
seiscomp exec scolv [options]
scolv [options]
Command-line modules are found in :file:`@ROOTDIR@/bin/` but they are **NOT**
listed when executing
.. code-block:: sh
seiscomp list modules
Using options like :option:`-h` the list of available command-line options can
be learned.
.. code-block:: sh
scbulletin -h
In addition, any module configuration parameter can be specified on
the command line overriding the configured parameter. For indicating that a
module configuration parameter is set on the command line separate the value
from the parameter by '=' and provide the full set of sections separated by '.'.
Example:
.. code-block:: sh
scolv --picker.loadAllPicks=true
When executing a module, all verbosity output is logged to
:ref:`log files <concepts_modules_logging>`. The logging level can be controlled
by configuration or by the command-line option :option:`-v`.
.. code-block:: sh
scbulletin -vvvv
Detailed debugging information can also be printed on the command line during
runtime using the :option:`--debug` option:
.. code-block:: sh
scbulletin --debug
When starting a daemon module all verbosity output is stored in :file:`@LOGDIR@`
or :file:`$SEISCOMP_ROOT/var/log`. Daemon modules can also be executed as
:ref:`command-line tools <concepts_modules_commandline>`.
.. _concepts_modules_config:
Configuration
=============
Each :term:`standalone module` tries to read from three module configuration
files whereas :term:`trunk` modules try to read the six files. Note that
configuration parameters defined earlier are overwritten if defined in files
read in later:
+-----------------------------------------+------------+----------------+
| File | Standalone | Trunk |
+=========================================+============+================+
| $SEISCOMP_ROOT/etc/defaults/global.cfg | | X |
+-----------------------------------------+------------+----------------+
| $SEISCOMP_ROOT/etc/defaults/module.cfg | X | X |
+-----------------------------------------+------------+----------------+
| $SEISCOMP_ROOT/etc/global.cfg | | X |
+-----------------------------------------+------------+----------------+
| $SEISCOMP_ROOT/etc/module.cfg | X | X |
+-----------------------------------------+------------+----------------+
| ~/.seiscomp/global.cfg | | X |
+-----------------------------------------+------------+----------------+
| ~/.seiscomp/module.cfg | X | X |
+-----------------------------------------+------------+----------------+
In addition to the module configuration files some modules such as
:ref:`seedlink` or :ref:`scautopick` consider :term:`bindings <binding>`.
Bindings provide parameters specific to stations. They are configured as
per-station bindings or profiles used for multiple stations.
The :ref:`global configuration section <global-configuration>` describes all
available global configuration parameters for a trunk module. Modules typically
do not make use of all available global parameters because they may be disabled,
e.g., the messaging component. So the configuration of the messaging server is
disabled, too.
The concept section :ref:`Configuration <concepts_configuration>` provides more
details about configurations.
.. _concepts_modules_logging:
Logging
=======
Whenever operated, modules report the state of operation to log files. Trunk
modules report the module and the start up log to :file:`@LOGDIR@/[module].log` and
:file:`@ROOTDIR@/var/log/[module].log`, respectively. Standalone modules log to
:file:`@ROOTDIR@/var/log/[module].log` only. The log files are rotated and the
level of detail can be configured by :confval:`logging.level`. More parameters
in :confval:`logging.*` provide more control over logging, e.g., the log file
rotation.
.. _concepts_modules_aliaes:
Alias Modules
=============
Many :term:`trunk` and :term:`standalone modules <standalone module>` allow
generating aliases as symbolic links to another module.
These aliases are useful for running multiple instances of the same module with
different configuration.
Alias modules can be created or removed using the :ref:`seiscomp` tool by
providing the commands :command:`alias create` or :command:`alias remove`,
respectively. Read the documentation of :ref:`seiscomp <sec_seiscomp_aliases>`
for the details.

91
share/doc/seiscomp/html/_sources/base/concepts/plugins.rst.txt Normal file
View File

@ -0,0 +1,91 @@
.. _concepts_plugins:
****************
SeisComP plugins
****************
Scope
=====
This chapter describes the general use of plugins in SeisComP.
Overview
========
Plugins expand the functionality of :ref:`applications <concepts_modules>`.
They are C++ shared object libraries which are dynamically loaded at runtime
into an application.
Typical plugins provide access to:
* :ref:`Databases <concepts_database>`
* :ref:`Recordstream implementations <global_recordstream>`
* :ref:`Locator routines <sec_index_extensions>`
* :ref:`Amplitude and magnitude types <sec_index_extensions>`
* :ref:`Waveform quality <scqc>`.
The plugin files themselves are located in :file:`share/plugins`.
By just loading a plugin an application does not change
it's way to function magically. Common plugins just implement a certain
interface (see e.g. messaging or RecordStream) and exhibit that functionality
by adding a new entry to the internal interface factory. As an example an
application makes use of interface ``DatabaseInterface``. Technically it
creates a new object implementing a certain interface by calling the C++
method:
.. code-block:: sh
db = DatabaseInterface::Create("mysql");
Without having a plugin loaded the returned object will be NULL or to put it
in other words, there is not implementation for mysql available.
Once the plugin ``dbmysql`` is loaded into the application, an implementation
for type mysql is added and will be available to the application. It is
still required for the application to explicitly ask for a particular
interface. That is most likely left to the user by adding a corresponding
configuration option to the configuration file.
That means, if an application loads two plugins, e.g. ``dbmysql`` and
``dbpostgresql`` that does not mean that it will not read from two database
at a time. It means the user has now the option to either use a MySQL database
or a PostgreSQL database. He still needs to make his choice in the
configuration file.
Trunk plugins are only supported as shared object libraries and therefore are
required to be written in C++. Implementations for all available interfaces
can be added. An incomplete list of SeisComP C++ interfaces:
* :ref:`Messaging <concepts_messaging>`
* :ref:`Database <concepts_database>`
* :ref:`RecordStream <concepts_recordstream>`
* Record formats
* Map projections
* :ref:`Time domain filters <filter-grammar>`
* Importer
* Exporter
* **Amplitude processors**
* **Magnitude processors**
This is just a subset of available extensible interface factories. The
emphasized entries refer to the factories which are most commonly extended.
Location and Configuration
==========================
Plugins are located in :file:`$SEISCOMP_ROOT/share/pugins`. In order to make a
plugins available for a module it must be added to the configuration of
:confval:`plugins` of the global parameters of a module or in
:ref:`global_configuration`.
Configuring :confval:`plugins` with the name of a plugin will let the exclusively
use this plugin and no other ones, e.g. default plugins. Example: ::
plugins = evrc
In order to add a plugin to the default plugins or plugins loaded by before, e.g.
by the global configuration, load these

68
share/doc/seiscomp/html/_sources/base/concepts/recordstream.rst.txt Normal file
View File

@ -0,0 +1,68 @@
.. _concepts_RecordStream:
************
RecordStream
************
Scope
=====
This document describes the RecordStream interface for accessing waveform data.
Overview
========
A RecordStream refers to the interface which allows to retrieve records
(time series) from arbitrary sources. An implementation can either be real-time
and stream records continuously or time window based and just deliver what is
available at the time of requesting data.
A comprehensive list of RecordStream implementations is available in the
:ref:`technical documentation <global_recordstream>`.
RecordStream implementations have a name such as "slink", "fdsnws" or "file"
which is used as scheme in the configuration URL. The location part of the URL
is passed to the implementation. The scheme part is used to create the
implementation. As one might have noticed, that RecordStream
implementations can be added to existing applications with plugins.
What do they do actually?
Well, first of all they connect to or open the data source. If that fails, an
error is logged. Then they are configured with time windows and channel
identifieres. Once done, they are simply asked for new records in a loop. A
RecordStream implementation can run forever or finish after a short time.
The behavior depends on the implementation and configuration.
The application uses RecordStreams like that:
.. code-block:: python
# The RecordStream URL passed is slink://localhost:18000
scheme = URL.scheme() # scheme = 'slink'
location = URL.location() # location = 'localhost:18000'
rs = RecordStream.Create(scheme)
if not rs:
throw Error()
if not rs.setSource(location):
throw Error()
rs.setStartTime(Time(2019,1,1))
rs.addStream('GE', 'UGM', '', 'BH?')
rs.addStream('GE', 'MORC', '', 'BH?')
while ( rec = rs.next() )
do_something_with(rec)
In the example above the end time is not set, so actually an open time window
should be read. That works pretty well for the Seedlink implemtation but the
FDSNWS implementation would complain and issue an error because no end time
was set. So configuring a RecordStream for an application requires some
knowledge of the context and the supported features of the configured
implementation.
Although |scname| ships with the Seedlink server, the processing application
are not aware of the fact that they connect to Seedlink. All time series
retrieval is done with the RecordStream concept. There is no knowledge about
the underlying implementation. This leads to a high flexibility and
implementations can be added without the need to modify the base |scname|
sources.

67
share/doc/seiscomp/html/_sources/base/concepts/waveformarchives.rst.txt Normal file
View File

@ -0,0 +1,67 @@
.. _concepts_waveformarchives:
*****************
Waveform archives
*****************
Scope
=====
This chapter describes waveform archives for long-term storage of miniSEED data.
Overview
========
While real-time data sources provide data for a rather short amount of time,
long-term access to waveforms can be established through waveform archives.
The :ref:`RecordStream interface <concepts_RecordStream>` allows a combined access
to real-time data and data in :ref:`SDS <concepts_sds>` or other archives.
.. note::
It is assumed that instrument corrections applied to the recorded
waveform data result in data in units of the real observations and their unmodified value.
Therefore, it is recommended to store only unprocessed raw data in units of digital counts
in the waveform archives and to provide the complete :ref:`inventory <concepts_inventory>`
referring to input data given in counts.
.. _concepts_sds:
SDS archives
============
SeisComP uses the SeisComP Data Structure (SDS) for archiving miniSEED waveform data.
It has the structure:
.. code-block:: sh
archive
+ year
+ network code
+ station code
+ channel code
+ one file per day and location, e.g. NET.STA.LOC.CHAN.D.YEAR.DOY
SeisComP ships with :ref:`slarchive` to create SDS archives from a miniSEED waveform
buffer by :ref:`seedlink` in real time and with :ref:`scart` to intergrate miniSEED
records from files into an SDS archive. :ref:`scart` can also be used to retrieve
miniSEED records from an SDS archive.
Access to waveform archives
===========================
Access from SeisComP processing and GUI modules to waveform archives is realized by
:ref:`RecordStream implementations <concepts_RecordStream>`.
The continuity of SDS archives can be monitored by :ref:`scardac` and exposed by
the :ref:`fdsnws availability feature <fdsnws>`.
miniSEED waveforms in SDS archives, can interactively retrieved using :ref:`scart`.
Waveforms stored in SDS archives can be served to clients, e.g. using :ref:`fdsnws`.
Related modules
===============
* :ref:`caps_plugin`
* :ref:`fdsnws`
* :ref:`scardac`
* :ref:`scart`
* :ref:`slarchive`