seiscomp-training/share/doc/seiscomp/html/_sources/apps/ql2sc.rst.txt

.. highlight:: rst

.. _ql2sc:

#####
ql2sc
#####

**QuakeLink (gempa GmbH) to SeisComP event parameter exchange.**


Description
===========

ql2sc manages the import of SeisComP objects from one or several QuakeLink servers
into a SeisComP system in real time. Like :ref:`scimex` but contrary to
:ref:`scimport` the exchange of the SeisComP objects is event based. This means no
messages will be exchanged until the exporting system has produced an event.

The user may control at various levels which information to import. Whenever
possible server-side filters should be preferred to reduce both the network
bandwidth consumption as well as the CPU and memory utilization on the local
machine.

.. _ql2sc_event_filter:

Server-Side Event Filter
========================

QuakeLink provides a filter syntax similar to SQL-WHERE clauses which may be
used to filter interesting events on the server side:

.. code-block:: none

   clause    := condition[ AND|OR [(]clause[)]]
   condition := MAG|DEPTH|LAT|LON|PHASES|OTIME|UPDATED [op float|time]|[IS [NOT] NULL]
   op        := =|>|>=|<|<=|eq|gt|ge|lt|ge
   time      := %Y,%m,%d[,%H,%M,%S,%f]

E.g. the following filter string would select only those events with a minimum
magnitude of 6, detected by at least 10 stations and which are shallower than
100km:

.. code-block:: sql

   MAG >= 6.0 AND PHASES >= 10 AND DEPTH < 100


.. _ql2sc_object_filter:

Server-Side Object Filter
=========================

QuakeLink provides a coarse object filter for the most relevant SeisComP objects:

============ ==============================================================
Option       Impact
============ ==============================================================
picks        include picks
amplitudes   include amplitudes
arrivals     include origin arrivals
staMags      include origin station magnitudes
staMts       include moment tensor station contributions and phase settings
preferred    include only preferred origin and magnitude information
============ ==============================================================


.. _routing:

Local Object Filter and Routing
===============================

Subsequent to the server-side filters a routing table defines which objects to
import and to which message group to send them. Depending on the |scname| modules
listening to the specified message groups an object may be further processed.
Typically no modules (other than :ref:`scmaster`) is connected to the
``IMPORT_GROUP`` so that objects sent to this group are just stored to the
database. If an object should be discarded, the special group identifier ``NULL``
may be used.

The routing table is defined as a comma-separated list of
``object name:group name`` pairs. Also the routing rules are inherited
recursively within the SeisComP object tree. If no explicit rule exists for an
object, the routing of its parent is evaluated up to the ``EventParameters``
root node.

Examples
--------

.. code-block:: none

   EventParameters:IMPORT_GROUP

Imports everything

.. code-block:: none

   EventParameters:IMPORT_GROUP,Comment:NULL

Imports everything except comments

.. code-block:: none

   Origin:LOCATION,StationMagnitude:MAGNITUDE,Magnitude:MAGNITUDE

Sends origins and it's children arrival, origin uncertainty to the ``LOCATION``
group but the magnitude children to the ``MAGNITUDE`` group. Skips picks,
amplitudes, focal mechanisms and events.


Default routing table
---------------------

The default use case of ql2sc is to import earthquake solutions from other data
centers or in-house redundant SeisComP systems. The intention is not to
reprocess the solution but to add them to the local catalog.

By default we route:

* Picks and Amplitudes to the ``IMPORT_GROUP`` group to prevent processing by
  the local locator and amplitude processor
* Origins (including its StationMagnitude and Magnitude children) to the
  ``LOCATION`` to allow event association.
* FocalMechanisms to the ``FOCMECH`` group to trigger processing by specialized
  applications, e.g. graphical user interfaces for strong motion analysis or
  tsunami risk assessment.

We don't route events at all. With the help of :ref:`scevent` locations are
either associated to existing events or will create new events with local
settings.

We don't route StationMagnitudes and Magnitude to the ``MAGNITDUE`` group
because :ref:`scmag` subscribes to ``LOCATION`` and ``MAGNITUDE``. Separated
groups might lead to duplicated magnitude types in case a manual magnitude
solution is imported. In this case the foreign Origin with its Magnitudes would
be split into at least two messages, the first one containing the Origin, the
second one the Magnitude. The Origin message immediately triggers magnitude
calculation, potentially for a magnitude type which is received with the second
message.

The default routing table is set as given in :confval:`host.$name.routingTable`.


.. _agency_filter:

Agency List Filter
==================

In addition to the local object filter the user may choose to accept only those
objects originating from a set of trusted agencies. If at least one agency is
defined in the ``processing.whitelist.agencies`` or
``processing.blacklist.agencies`` configuration option, then the
``creationInfo.agencyID`` of amplitudes, arrivals, comments, events, focal
mechanisms, magnitudes, moment tensors, origins, picks and station magnitudes is
evaluated. Objects with unmatched or unset agency information are filtered out.
If objects with unset agency ID should match, then empty string ``""`` has to be
added to the white list.

The agency filter is applied on remote as well as local objects. In this way
remote objects may be excluded from import and local objects my be protected
from overriding or removing. Also the filter is applied recursively. If parent
object (e.g. an origin) is filtered out, all of its children (e.g. magnitudes)
are also skipped even if they carry a different agency ID.

.. note::

   The agency white list filter might be essential to avoid circular event
   updates between cross-connected SeisComP systems.


.. _publicID_filter:

PublicID Prefix Filter
======================

In addition to the :ref:`agency filter<agency_filter>` incoming or local objects
can be skipped by checking their publicID prefix. It behaves similar to the
:ref:`agency filter<agency_filter>` but checks the ``publicID`` attribute rather
than the ``creationInfo.agencyID`` attribute.
Prefixes can be configure as white- or blacklist with
``processing.whitelist.publicIDs = ...`` and
``processing.blacklist.publicIDs = ...``.


Workflow
========

Each event update received from a QuakeLink host is parsed and analyzed for
differences to the local database. The comparison starts at the level of the
top-level elements in the following order: picks, amplitudes, origins, focal
mechanisms, events.

For each top-level element the object tree is traversed in a depth-first search
order. Objects on the same level are processed in the order of their appearance.
The differences are collected as a list of notifier objects with the following
operation types:

====== ===========
Type   Description
====== ===========
ADD    The object does not exist locally
UPDATE The object does exist locally but differs from the remote one
REMOVE The object exist locally but not remotely
====== ===========

The ``ADD`` and ``REMOVE`` operation always generates notifies of the same type
for all children of the current object. ``ADD`` notifiers are collected top-down,
``REMOVE`` notifiers are collected bottom-up.

Because the order of child objects is arbitrary, e.g. the arrivals of an origin,
each object on the remote side has to be found in the set of local objects. For
public objects (e.g. origins, magnitudes, magnitudes..), the ``publicID`` property
is used for comparison. All other objects are compared by looking at their index
properties. For e.g. arrivals this is the ``pickID`` property, for comments the
``id`` property.

Ones all notifiers are collected they are send to the local messaging system.
For performance reasons and because of the processing logic of listening |scname|
modules ql2sc tries to batch as many notifiers as possible into one notifier
message. A separate notifier message is created if the target message group
changes between successive notifiers or if the configurable :confval:`batchSize`
limit is reached.

.. note::

   Care must be taken when configuring the ``batchSize`` limit. If the value
   is to big the overall message size limit (default: 1MB) may be exceeded
   resulting in an undeliverable message. On the other hand a much to small
   value will create unwanted results in the |scname| processing chain. If for
   instance picks are routed to the ``PICK`` group and the pick set is split
   into several notifier messages the local :ref:`scautoloc` might create
   locations based on an incomplete dataset.


Event Attributes
================

It might be desirable to synchronize event attributes set at the source with
the local system. In particular the event type, the type uncertainty, event
descriptions and comments might be of interest. Because it is not advisable
to route events and let :ref:`scevent` associate imported origins it can
happen that the imported event ID is different from the event ID of the local
system. The input host configuration parameter :confval:`syncEventAttributes`
controls that behaviour. It is set to true by default which means that imported
event attributes are going to be imported as well. ql2sc does not update
directly the attributes but commandates scevent in as many cases as possible
to do so. To find the matching local event it takes the first occurrence which
has associated the currently imported preferred origin.


Limitations
-----------

There are limitations to this process to avoid infinite loops when cross
connecting two systems. Prior to sending the commands to scevent to change a
particular attribute ql2sc checks if that attribute has been set already by
another module (via JournalEntry database table). If not, then ql2sc is allowed
to request an attribute change otherwise not. To illustrate the issue take the
following example:

scolv connected to system ``A`` changes the event type to 'earthquake'. ql2sc
of system ``B`` checks if the event type of the local event has been changed
already which is not the case and it requests that change. System ``A``
changes the event type again to 'unset'. ql2sc of system ``B`` notices that
someone has already changed the event type and it was ql2sc itself. It requests
again a change.

scolv connected to system ``B`` changes the event type to 'earthquake' again.
ql2sc of system ``A`` notices that ``scolv@A`` has already changed the
event type and ignores the request.

That simple case would not create an infinite loop even if ``ql2sc@A`` would
accept the last change. The situation changes immediately if two subsequent
attribute changes are being received by ``ql2sc@B`` while both of them are
already applied on system ``A``. ``ql2sc@B`` would "restore" the old state due
to the first received update and then apply the "final" state due to the
second update. Each update triggers again an update at system ``A`` and the
states start flapping. Without the described check there wouldn't be a well
defined exit condition.


Caveats
=======

Specific combinations of remote and local object filters may result in the loss
of data. If for instance origins are imported from system ``A`` to ``B`` and
additional magnitudes for the received origins are calculated on ``B``, care must
be taken. Without protection a new event update containing the same origin will
``REMOVE`` all newly calculated magnitudes on ``B`` since they are not included
in the magnitude set sent by ``A``.

To avoid losing these local magnitudes one may decide to block magnitudes from
import by routing them to ``NULL``. If magnitudes from ``A`` and from ``B``
should be available, an :ref:`agency filter<agency_filter>` or
:ref:`publicID filter<publicID_filter>` may be defined.

Make sure ``A`` and ``B`` use either distinct agency IDs or distinct publicID
patterns and add the agency ID of ``B`` to ``processing.blacklist.agencies`` or
the publicID prefix of ``B`` to ``processing.blacklist.publicIDs``.


.. _ql2sc_configuration:

Module Configuration
====================

| :file:`etc/defaults/global.cfg`
| :file:`etc/defaults/ql2sc.cfg`
| :file:`etc/global.cfg`
| :file:`etc/ql2sc.cfg`
| :file:`~/.seiscomp/global.cfg`
| :file:`~/.seiscomp/ql2sc.cfg`

ql2sc inherits :ref:`global options<global-configuration>`.



.. confval:: backLog

   Default: ``1800``

   Type: *int*

   Unit: *s*

   Number of seconds to fetch missed updates on start up.


.. confval:: cacheSize

   Default: ``5000``

   Type: *int*

   Number of public objects to cache.


.. confval:: batchSize

   Default: ``2000``

   Type: *int*

   Maximum number of notifiers to batch in one message. If set
   to 0 no size limit is enforced. Make sure to not hit the
   overall message size limited of 16MiB which is enforced by
   the messaging system.


.. confval:: eventAssociationTimeout

   Default: ``10``

   Type: *int*

   Unit: *s*

   If event synchronisation is enabled and an incoming origin
   is not yet associated with an event on the target machine,
   then this timeout defines the maximum number of seconds to
   wait for an association.


.. confval:: hosts

   Type: *list:string*

   Registration of the host profiles defining the connection
   parameters to the QuakeLink hosts.


.. note::
   **host.\***
   *Definition of host profiles. For each host profile a connection*
   *to one QuakeLink server can established. The profiles must be registered*
   *in 'hosts' to apply them.*



.. note::

   **host.$name.\***
   *Provide the connection parameters to one QuakeLink server.*
   $name is a placeholder for the name to be used and needs to be added to :confval:`hosts` to become active.

   .. code-block:: sh

      hosts = a,b
      host.a.value1 = ...
      host.b.value1 = ...
      # c is not active because it has not been added
      # to the list of hosts
      host.c.value1 = ...


.. confval:: host.$name.url

   Default: ``ql://localhost:18010``

   Type: *string*

   URL of the QuakeLink service, the scheme 'qls' enables SSL.
   
   Format: [ql[s]:\/\/][user:pwd\@][host][:port].
   
   If set to an empty string the application will run without any QuakeLink connection attempt.


.. confval:: host.$name.gzip

   Default: ``false``

   Type: *boolean*

   Enable\/disable GZip \(GNU zip\) compression.


.. confval:: host.$name.native

   Default: ``false``

   Type: *boolean*

   Request native data instead of XML format.
   Native data export may be disabled on some hosts.


.. confval:: host.$name.syncEventAttributes

   Default: ``true``

   Type: *boolean*

   Try to update the event attributes of the target event
   with the attributes of the source event which includes
   event type and event certainty. It will not import
   events but tries to find the associated event of the
   input preferred origin at the target system and will
   update the event attributes via journaling.


.. confval:: host.$name.keepAlive

   Default: ``false``

   Type: *boolean*

   Request server to send keep alive message every 30s to
   prevent connection reset by firewalls on long idle
   periods. If activated the client will reset the
   connection if no alive message is received within 60s.


.. confval:: host.$name.filter

   Type: *string*

   SQL like WHERE clause to filter the result set.
   
   clause    :\= condition[ AND\|OR [\(]clause[\)]] __
   condition :\= MAG\|DEPTH\|LAT\|LON\|PHASES\|OTIME\|UPDATED [op float\|time]\|[IS [NOT] NULL] __
   op        :\= \=\|>\|>\=\|<\|<\=\|eq\|gt\|ge\|lt\|ge __
   time      :\= %Y,%m,%d[,%H,%M,%S,%f]


.. confval:: host.$name.routingTable

   Default: ``Pick:IMPORT_GROUP,Amplitude:IMPORT_GROUP,FocalMechanism:EVENT,Origin:EVENT``

   Type: *list:string*

   Map datamodel class names to messaging groups. For unmapped objects
   the mapping of their parent objects is evaluated recursively. Objects
   may be excluded by mapping them to 'NULL'.


.. note::
   **host.$name.data.\***
   *Specify the XML components to fetch.*
   *Note: These options are not used if 'native' data is requested.*



.. confval:: host.$name.data.picks

   Default: ``true``

   Type: *boolean*

   Include picks


.. confval:: host.$name.data.amplitudes

   Default: ``true``

   Type: *boolean*

   Include amplitudes


.. confval:: host.$name.data.arrivals

   Default: ``true``

   Type: *boolean*

   Include origin arrivals


.. confval:: host.$name.data.staMags

   Default: ``true``

   Type: *boolean*

   Include origin station magnitudes


.. confval:: host.$name.data.staMts

   Default: ``true``

   Type: *boolean*

   Include moment tensor station contributions and phase settings


.. confval:: host.$name.data.preferred

   Default: ``true``

   Type: *boolean*

   Include only preferred origin and magnitude information


.. confval:: processing.blacklist.publicIDs

   Type: *list:string*

   Defines a whitelist of publicID prefixes that are
   allowed for processing. Separate items by comma.


.. confval:: processing.whitelist.publicIDs

   Type: *list:string*

   Defines a blacklist of publicID prefixes that are
   not allowed for processing. Separate items by comma.



Command-Line Options
====================

.. program:: ql2sc

:program:`ql2sc [options]`


Generic
-------

.. option:: -h, --help

   Show help message.

.. option:: -V, --version

   Show version information.

.. option:: --config-file arg

   Use alternative configuration file. When this option is
   used the loading of all stages is disabled. Only the
   given configuration file is parsed and used. To use
   another name for the configuration create a symbolic
   link of the application or copy it. Example:
   scautopick \-> scautopick2.

.. option:: --plugins arg

   Load given plugins.

.. option:: -D, --daemon

   Run as daemon. This means the application will fork itself
   and doesn't need to be started with \&.


Verbosity
---------

.. option:: --verbosity arg

   Verbosity level [0..4]. 0:quiet, 1:error, 2:warning, 3:info,
   4:debug.

.. option:: -v, --v

   Increase verbosity level \(may be repeated, eg. \-vv\).

.. option:: -q, --quiet

   Quiet mode: no logging output.

.. option:: --print-component arg

   For each log entry print the component right after the
   log level. By default the component output is enabled
   for file output but disabled for console output.

.. option:: --component arg

   Limit the logging to a certain component. This option can
   be given more than once.

.. option:: -s, --syslog

   Use syslog logging backend. The output usually goes to
   \/var\/lib\/messages.

.. option:: -l, --lockfile arg

   Path to lock file.

.. option:: --console arg

   Send log output to stdout.

.. option:: --debug

   Execute in debug mode.
   Equivalent to \-\-verbosity\=4 \-\-console\=1 .

.. option:: --trace

   Execute in trace mode.
   Equivalent to \-\-verbosity\=4 \-\-console\=1 \-\-print\-component\=1
   \-\-print\-context\=1 .

.. option:: --log-file arg

   Use alternative log file.


Messaging
---------

.. option:: -u, --user arg

   Overrides configuration parameter :confval:`connection.username`.

.. option:: -H, --host arg

   Overrides configuration parameter :confval:`connection.server`.

.. option:: -t, --timeout arg

   Overrides configuration parameter :confval:`connection.timeout`.

.. option:: -g, --primary-group arg

   Overrides configuration parameter :confval:`connection.primaryGroup`.

.. option:: -S, --subscribe-group arg

   A group to subscribe to.
   This option can be given more than once.

.. option:: --content-type arg

   Overrides configuration parameter :confval:`connection.contentType`.

.. option:: --start-stop-msg arg

   Set sending of a start and a stop message.


Database
--------

.. option:: --db-driver-list

   List all supported database drivers.

.. option:: -d, --database arg

   The database connection string, format:
   service:\/\/user:pwd\@host\/database.
   \"service\" is the name of the database driver which
   can be queried with \"\-\-db\-driver\-list\".