353 lines
13 KiB
ReStructuredText
353 lines
13 KiB
ReStructuredText
.. _sec-caps-retrieval:
|
|
|
|
Access Data on a CAPS Server
|
|
============================
|
|
|
|
A range of tools is available to access data and information on a CAPS server.
|
|
|
|
.. csv-table::
|
|
:header: "Name", "SW Package", "Description"
|
|
:widths: 15,15,70
|
|
|
|
":ref:`capstool <sec-caps-capstool>`","caps-tools","A command-line tool for retrieving data and meta information from a CAPS server"
|
|
":ref:`rifftool <sec-caps-file>`","caps-tools","A command-line tool for data inspection and extraction from individual CAPS data files (RIFF), e.g., in a CAPS archive"
|
|
":ref:`capssds <sec-caps-file>`","caps-tools","A virtual overlay file system presenting a CAPS archive directory as a read-only SDS archive with no extra disk space requirement."
|
|
":ref:`caps_plugin <sec-caps-seedlink>`","seiscomp","The plugin fetches miniSEED and :ref:`RAW <sec-pt-raw>` data from a CAPS server and provides the data to :program:`seedlink`"
|
|
":ref:`caps / capss<sec-caps-recstream>`","seiscomp","The RecordStream implementations for |appname|"
|
|
":ref:`cap2caps <sec-caps2caps>`","caps-plugins","Automatic or interactive synchronization of two CAPS servers"
|
|
":ref:`web interface <sec-caps-web>`","caps-server","The web interface provided by the CAPS server"
|
|
":ref:`FDSNWS <sec-caps-fdsnws>`","caps-server","FDSNWS dataselect interface provided by the CAPS server"
|
|
":ref:`WWS <sec-caps-wws>`","caps-server","Winston Waveform Server interface provided by the CAPS server"
|
|
":ref:`scardac <sec-caps-dataavailability>`","seiscomp","A command-line tool for generating availability information from CAPS archive"
|
|
|
|
|
|
.. _sec-caps-recstream:
|
|
|
|
RecordStream: caps/capss
|
|
------------------------
|
|
|
|
|scname| applications access waveform data through the
|
|
:term:`RecordStream` interface.
|
|
To fetch data from a CAPS server specific RecordStream implementations may be used:
|
|
|
|
* *caps*: regular RecordStream implementation to access the CAPS server,
|
|
* *capss*: RecordStream implementation to access the CAPS server secured by SSL,
|
|
user name and password. Similar to *https*, *capss* will establish a Secure Socket
|
|
Layer (SSL) communication.
|
|
|
|
|
|
.. _sec-caps-rs-config:
|
|
|
|
Configuration
|
|
~~~~~~~~~~~~~
|
|
|
|
In order to make use of the *caps* or the *capss* RecordStream configure the
|
|
RecordStream URL in :confval:`recordstream`. Let it point to the CAPS server
|
|
providing the data. Examples for *caps* and *capss*:
|
|
|
|
.. code-block:: properties
|
|
|
|
recordstream = caps://server:18002
|
|
recordstream = capss://foo:bar@server:18022
|
|
|
|
:ref:`Optional parameters <sec-caps-opt-params>` are available for
|
|
*caps*/*capss*.
|
|
|
|
.. note::
|
|
|
|
While the *caps*/*capss* :term:`RecordStream` provides data in real time
|
|
and from archive, some modules, e.g., :cite:t:`scart`, :cite:t:`fdsnws` or
|
|
:cite:t:`gis` should be strictly limited to reading from archive only by
|
|
the option ``arch``:
|
|
|
|
.. code-block:: properties
|
|
|
|
recordstream = caps://server:18002?arch
|
|
recordstream = capss://foo:bar@server:18022?arch
|
|
|
|
Otherwise requests attempting to fetch missing data may hang forever.
|
|
|
|
.. _sec-caps-opt-params:
|
|
|
|
Optional Parameters
|
|
~~~~~~~~~~~~~~~~~~~
|
|
|
|
Optional RecordStream parameters which can be combined:
|
|
|
|
- ``arch`` - read from CAPS archive only,
|
|
- ``ooo`` - out of order, data are fetched and provided in the order of their arrival in the CAPS server, useful for analysing if data have arrived out of order,
|
|
- ``pass`` - password if server requires authentication,
|
|
- ``request-file`` - file specifying the streams to be requested. One stream per line. Per line: net sta loc stream startTime endTime,
|
|
- ``timeout`` - timeout of acquisition in seconds. Data acquisition will be restarted,
|
|
- ``user`` - user name if server requires authentication.
|
|
|
|
|
|
.. csv-table::
|
|
:header: "URL", "Description"
|
|
|
|
"caps://server:18002?arch","Read data from CAPS archive. Stop data acquisition when all available waveforms are fetched."
|
|
"caps://server:18002?ooo","Fetch data in the original order of arrival."
|
|
"caps://server:18002?request-file=request.txt","Request only streams in time intervals given in request.txt"
|
|
"caps://server:18002?timeout=5","Apply a timeout of 5 seconds."
|
|
"capss://server:18022?user=foo&pass=bar", "Use secure protocol (SSL) with user
|
|
name and password. Read the section on :ref:`sec-conf-access-auth` for details
|
|
on the generation of user names and passwords."
|
|
|
|
|
|
Combination with other RecordStream interfaces
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The caps and the capss RecordStream may be combined with other
|
|
:term:`RecordStream` interfaces.
|
|
|
|
Examples:
|
|
|
|
#. Decimation
|
|
|
|
Use the decimation :term:`RecordStream` :cite:p:`recordstream`
|
|
interface to fetch data from a CAPS server running on localhost decimated to
|
|
1 sample per second.
|
|
|
|
*global configuration:*
|
|
|
|
.. code-block:: sh
|
|
|
|
recordstream = dec://caps/localhost:18002?rate=1
|
|
|
|
*command line parameter:*
|
|
|
|
.. code-block:: sh
|
|
|
|
-I dec://caps/localhost:18002?rate=1
|
|
|
|
#. Resample
|
|
|
|
Use the resample :term:`RecordStream` :cite:p:`recordstream`
|
|
interface to fetch data from a CAPS server running on localhost resampled to
|
|
16 samples per second.
|
|
|
|
*global configuration:*
|
|
|
|
.. code-block:: sh
|
|
|
|
recordstream = resample://caps/localhost:18002?rate=16
|
|
|
|
*command line parameter:*
|
|
|
|
.. code-block:: sh
|
|
|
|
-I resample://caps/localhost:18002?rate=16
|
|
|
|
|
|
.. _sec-caps-capstool:
|
|
|
|
CAPS command-line interface (CLI) client: capstool
|
|
--------------------------------------------------
|
|
|
|
:ref:`capstool` is a CAPS client application for retrieving data and listing
|
|
available streams from an operational CAPS server. The CAPS server may run
|
|
locally or remotely as all communication is performed over the network
|
|
(option:`-H`).
|
|
|
|
Data requests are based on time windows and stream IDs. The data is provided in
|
|
its origin format on stdout or, with :option:`-o` as a single file. In
|
|
particular capstool may be used to fetch miniSEED data and create miniSEED
|
|
files.
|
|
|
|
:ref:`capstool` can also be used for :ref:`testing the server
|
|
<sec-caps-server-testing>` as it provides information on available streams with
|
|
their time window (:option:`-Q`, :option:`-I`).
|
|
|
|
|
|
.. _sec-caps-file:
|
|
|
|
Data file access: rifftool/capssds
|
|
----------------------------------
|
|
|
|
The data files in the CAPS archive contain a small additional header describing
|
|
the data format and implementing an index for fast and in-order data retrieval.
|
|
Read the :ref:`format documentation <sec-packet-types>` for more details. In
|
|
contrast to miniSEED files in :term:`SDS` archives created, e.g., by
|
|
:cite:t:`slarchive` or :cite:t:`scart`, the original miniSEED files stored in
|
|
the CAPS archive cannot be directly read by common seismological applications.
|
|
|
|
You may access data files directly:
|
|
|
|
* Test and retrieve data files using :ref:`rifftool`. rifftool addresses
|
|
individual files directly and does not require the CAPS server to be running.
|
|
* Run :ref:`capssds` to create a virtual overlay file system presenting a CAPS
|
|
archive directory as a read-only :term:`SDS` archive with no extra disk space
|
|
requirement. The CAPS archive directory and file names are mapped. An
|
|
application reading from a file will only see :term:`miniSEED` records ordered
|
|
by record start time. You may connect to the virtual SDS archive using the
|
|
RecordStream SDS or directly read the single :term:`miniSEED` file. Other
|
|
seismological software such as ObsPy or Seisan may read directly from the SDS
|
|
archive of the files therein.
|
|
|
|
|
|
.. _sec-caps2caps:
|
|
|
|
Synchronize with another CAPS server: caps2caps
|
|
-----------------------------------------------
|
|
|
|
Use :ref:`caps2caps` to synchronize your CAPS server with another one. You may push
|
|
or pull data on either side. In contrast to the generation of regular :term:`SDS`
|
|
archives, e.g., by :program:`scart`, the CAPS server will not generate duplicate
|
|
data records if executing :ref:`caps2caps` multiple times. While synchronizing
|
|
observe the :ref:`web interface <sec-caps-web-interface>` for the statistics of
|
|
received, written or rejected data packages.
|
|
|
|
|
|
.. _sec-caps-seedlink:
|
|
|
|
Connect from SeedLink: caps_plugin
|
|
----------------------------------
|
|
|
|
The :ref:`caps_plugin` plugin fetches data from a CAPS server and provides the
|
|
data to :program:`seedlink`. The plugin can be configured and started as any
|
|
other plugin for :program:`seedlink` or executed on demand. Select *caps* when
|
|
choosing the plugin in the seedlink binding configuration.
|
|
|
|
**Examples:**
|
|
|
|
* Fetch data from a remote CAPS server and make them available in your :program:`seedlink` instance:
|
|
|
|
#. configure the :ref:`caps_plugin` plugin in the bindings configuration of your
|
|
:program:`seedlink` instance pointing to the remote CAPS server
|
|
|
|
#. enable and start :program:`seedlink`
|
|
|
|
.. code-block:: sh
|
|
|
|
seiscomp enable seedlink
|
|
seiscomp start seedlink
|
|
|
|
* Provide data from a CAPS server by seedlink on the same machine to external clients:
|
|
|
|
#. create another seedlink instance, e.g., :program:`seedlinkP`:
|
|
|
|
.. code-block:: sh
|
|
|
|
seiscomp alias create seedlinkP seedlink
|
|
|
|
#. configure the :ref:`caps_plugin` in the bindings configuration of :program:`seedlinkP`
|
|
pointing to the local CAPS server.
|
|
#. enable and start :program:`seedlinkP`:
|
|
|
|
.. code-block:: sh
|
|
|
|
seiscomp enable seedlinkP
|
|
seiscomp start seedlinkP
|
|
|
|
|
|
.. _sec-caps-web:
|
|
|
|
Web interface
|
|
-------------
|
|
|
|
The CAPS server ships with a :ref:`web interface <sec-caps-web-interface>`.
|
|
Beside allowing you to view server statistics, data stored stored on the server
|
|
can be downloaded for any time if the original format is :term:`miniSEED`.
|
|
|
|
For downloading miniSEED data
|
|
|
|
#. Select the stream(s) of interest.
|
|
#. Zoom in to the period of interest. Zooming in and out in time works by
|
|
right-mouse button actions just like in other |scname| GUI applications like
|
|
:cite:t:`scrttv`.
|
|
#. Click on the download button to download the miniSEED file. An error message
|
|
will be printed in case the original format is not miniSEED.
|
|
|
|
.. _fig-web-streams-download:
|
|
|
|
.. figure:: media/web_streams_download.png
|
|
:width: 18cm
|
|
|
|
Stream perspective of :term:`CAPS` Web interface allowing to download miniSEED
|
|
data for selected streams.
|
|
|
|
|
|
.. _sec-caps-fdsnws:
|
|
|
|
Built-in FDSNWS
|
|
---------------
|
|
|
|
|appname| speaks natively FDSN Web Services, FDSNWS :cite:p:`fdsn` providing
|
|
waveform data via dataselect. Information on events and stations are
|
|
not delivered. The waveform data will be delivered through the port configured in
|
|
:confval:`AS.http.port` or the port configured by your Apache server, if available.
|
|
Contact your system administrator for information on the Apache server.
|
|
Read the documentation of the :ref:`CAPS server <sec-caps-config>` for the configuration.
|
|
|
|
|
|
.. _sec-caps-wws:
|
|
|
|
Built-in Winston waveform server
|
|
--------------------------------
|
|
|
|
|appname| speeks natively Winston Waveform Server protocol, WWS, :cite:p:`wws`,
|
|
e.g., to :cite:t:`swarm` by USGS. Read the documentation of the
|
|
:ref:`CAPS server<sec-caps-config>` for the configuration.
|
|
|
|
|
|
.. _sec-caps-dataavailability:
|
|
|
|
Data availability information
|
|
-----------------------------
|
|
|
|
|
|
Web interface
|
|
~~~~~~~~~~~~~
|
|
|
|
The :ref:`Channels perspective of the CAPS web interface <sec-caps-web>`
|
|
indicates periods of availability and of gaps on level of network, station,
|
|
sensor location and channel. The resolution of colors and percentages is linked
|
|
to the granularity of the data detection which increases with shorter time
|
|
windows in order to optimize the speed of the calculation.
|
|
|
|
|
|
capstool
|
|
~~~~~~~~
|
|
|
|
The CAPS server stores information on received data segments
|
|
including their start and end times. Information on resulting gaps can be
|
|
retrieved by :ref:`capstool`. Example:
|
|
|
|
.. code-block:: sh
|
|
|
|
echo "2023,05,01,12,00,00 2023,05,03,00,00,00 NET * * *" | capstool -G --tolerance=0.5 -H localhost
|
|
|
|
|
|
scardac
|
|
~~~~~~~
|
|
|
|
The availability of data in the caps archive can be analyzed and written to the
|
|
|scname| database by the |scname| module :cite:t:`scardac`. For availability analysis
|
|
add the plugin *daccaps* to the list of :confval:`plugins` and URL of the caps
|
|
archive to the archive configuration of :cite:t:`scardac`. The *daccaps* plugin
|
|
ships with the gempa package *caps-server*.
|
|
|
|
Example configuration of :cite:t:`scardac` (:file:`scardac.cfg`):
|
|
|
|
.. code-block:: properties
|
|
|
|
plugins = ${plugins}, daccaps
|
|
archive = caps:///home/data/archive/caps/
|
|
|
|
Example call:
|
|
|
|
.. code-block:: sh
|
|
|
|
scardac --plugins="daccaps, dbmysql" -d localhost -a caps:///home/data/archive/caps/ --debug
|
|
|
|
.. note::
|
|
|
|
As of SeisComP in version 6, scardac has received significant optimization.
|
|
Instead of scanning the full archive, only files which have changed since the
|
|
last scan will be examined. This means that when scanning the entire archive,
|
|
the first run may be more time consuming than subsequent ones when executed
|
|
within reasonable intervals.
|
|
|
|
The data availability information can be retrieved from the database, e.g.,
|
|
using :cite:t:`fdsnws` or :cite:t:`scxmldump`.
|