[installation] Init with inital config for global
This commit is contained in:
352
share/doc/caps/html/_sources/base/data-retrieval.rst.txt
Normal file
352
share/doc/caps/html/_sources/base/data-retrieval.rst.txt
Normal file
@ -0,0 +1,352 @@
|
||||
.. _sec-caps-retrieval:
|
||||
|
||||
Access Data on a CAPS Server
|
||||
============================
|
||||
|
||||
A range of tools is available to access data and information on a CAPS server.
|
||||
|
||||
.. csv-table::
|
||||
:header: "Name", "SW Package", "Description"
|
||||
:widths: 15,15,70
|
||||
|
||||
":ref:`capstool <sec-caps-capstool>`","caps-tools","A command-line tool for retrieving data and meta information from a CAPS server"
|
||||
":ref:`rifftool <sec-caps-file>`","caps-tools","A command-line tool for data inspection and extraction from individual CAPS data files (RIFF), e.g., in a CAPS archive"
|
||||
":ref:`capssds <sec-caps-file>`","caps-tools","A virtual overlay file system presenting a CAPS archive directory as a read-only SDS archive with no extra disk space requirement."
|
||||
":ref:`caps_plugin <sec-caps-seedlink>`","seiscomp","The plugin fetches miniSEED and :ref:`RAW <sec-pt-raw>` data from a CAPS server and provides the data to :program:`seedlink`"
|
||||
":ref:`caps / capss<sec-caps-recstream>`","seiscomp","The RecordStream implementations for |appname|"
|
||||
":ref:`cap2caps <sec-caps2caps>`","caps-plugins","Automatic or interactive synchronization of two CAPS servers"
|
||||
":ref:`web interface <sec-caps-web>`","caps-server","The web interface provided by the CAPS server"
|
||||
":ref:`FDSNWS <sec-caps-fdsnws>`","caps-server","FDSNWS dataselect interface provided by the CAPS server"
|
||||
":ref:`WWS <sec-caps-wws>`","caps-server","Winston Waveform Server interface provided by the CAPS server"
|
||||
":ref:`scardac <sec-caps-dataavailability>`","seiscomp","A command-line tool for generating availability information from CAPS archive"
|
||||
|
||||
|
||||
.. _sec-caps-recstream:
|
||||
|
||||
RecordStream: caps/capss
|
||||
------------------------
|
||||
|
||||
|scname| applications access waveform data through the
|
||||
:term:`RecordStream` interface.
|
||||
To fetch data from a CAPS server specific RecordStream implementations may be used:
|
||||
|
||||
* *caps*: regular RecordStream implementation to access the CAPS server,
|
||||
* *capss*: RecordStream implementation to access the CAPS server secured by SSL,
|
||||
user name and password. Similar to *https*, *capss* will establish a Secure Socket
|
||||
Layer (SSL) communication.
|
||||
|
||||
|
||||
.. _sec-caps-rs-config:
|
||||
|
||||
Configuration
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
In order to make use of the *caps* or the *capss* RecordStream configure the
|
||||
RecordStream URL in :confval:`recordstream`. Let it point to the CAPS server
|
||||
providing the data. Examples for *caps* and *capss*:
|
||||
|
||||
.. code-block:: properties
|
||||
|
||||
recordstream = caps://server:18002
|
||||
recordstream = capss://foo:bar@server:18022
|
||||
|
||||
:ref:`Optional parameters <sec-caps-opt-params>` are available for
|
||||
*caps*/*capss*.
|
||||
|
||||
.. note::
|
||||
|
||||
While the *caps*/*capss* :term:`RecordStream` provides data in real time
|
||||
and from archive, some modules, e.g., :cite:t:`scart`, :cite:t:`fdsnws` or
|
||||
:cite:t:`gis` should be strictly limited to reading from archive only by
|
||||
the option ``arch``:
|
||||
|
||||
.. code-block:: properties
|
||||
|
||||
recordstream = caps://server:18002?arch
|
||||
recordstream = capss://foo:bar@server:18022?arch
|
||||
|
||||
Otherwise requests attempting to fetch missing data may hang forever.
|
||||
|
||||
.. _sec-caps-opt-params:
|
||||
|
||||
Optional Parameters
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Optional RecordStream parameters which can be combined:
|
||||
|
||||
- ``arch`` - read from CAPS archive only,
|
||||
- ``ooo`` - out of order, data are fetched and provided in the order of their arrival in the CAPS server, useful for analysing if data have arrived out of order,
|
||||
- ``pass`` - password if server requires authentication,
|
||||
- ``request-file`` - file specifying the streams to be requested. One stream per line. Per line: net sta loc stream startTime endTime,
|
||||
- ``timeout`` - timeout of acquisition in seconds. Data acquisition will be restarted,
|
||||
- ``user`` - user name if server requires authentication.
|
||||
|
||||
|
||||
.. csv-table::
|
||||
:header: "URL", "Description"
|
||||
|
||||
"caps://server:18002?arch","Read data from CAPS archive. Stop data acquisition when all available waveforms are fetched."
|
||||
"caps://server:18002?ooo","Fetch data in the original order of arrival."
|
||||
"caps://server:18002?request-file=request.txt","Request only streams in time intervals given in request.txt"
|
||||
"caps://server:18002?timeout=5","Apply a timeout of 5 seconds."
|
||||
"capss://server:18022?user=foo&pass=bar", "Use secure protocol (SSL) with user
|
||||
name and password. Read the section on :ref:`sec-conf-access-auth` for details
|
||||
on the generation of user names and passwords."
|
||||
|
||||
|
||||
Combination with other RecordStream interfaces
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The caps and the capss RecordStream may be combined with other
|
||||
:term:`RecordStream` interfaces.
|
||||
|
||||
Examples:
|
||||
|
||||
#. Decimation
|
||||
|
||||
Use the decimation :term:`RecordStream` :cite:p:`recordstream`
|
||||
interface to fetch data from a CAPS server running on localhost decimated to
|
||||
1 sample per second.
|
||||
|
||||
*global configuration:*
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
recordstream = dec://caps/localhost:18002?rate=1
|
||||
|
||||
*command line parameter:*
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
-I dec://caps/localhost:18002?rate=1
|
||||
|
||||
#. Resample
|
||||
|
||||
Use the resample :term:`RecordStream` :cite:p:`recordstream`
|
||||
interface to fetch data from a CAPS server running on localhost resampled to
|
||||
16 samples per second.
|
||||
|
||||
*global configuration:*
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
recordstream = resample://caps/localhost:18002?rate=16
|
||||
|
||||
*command line parameter:*
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
-I resample://caps/localhost:18002?rate=16
|
||||
|
||||
|
||||
.. _sec-caps-capstool:
|
||||
|
||||
CAPS command-line interface (CLI) client: capstool
|
||||
--------------------------------------------------
|
||||
|
||||
:ref:`capstool` is a CAPS client application for retrieving data and listing
|
||||
available streams from an operational CAPS server. The CAPS server may run
|
||||
locally or remotely as all communication is performed over the network
|
||||
(option:`-H`).
|
||||
|
||||
Data requests are based on time windows and stream IDs. The data is provided in
|
||||
its origin format on stdout or, with :option:`-o` as a single file. In
|
||||
particular capstool may be used to fetch miniSEED data and create miniSEED
|
||||
files.
|
||||
|
||||
:ref:`capstool` can also be used for :ref:`testing the server
|
||||
<sec-caps-server-testing>` as it provides information on available streams with
|
||||
their time window (:option:`-Q`, :option:`-I`).
|
||||
|
||||
|
||||
.. _sec-caps-file:
|
||||
|
||||
Data file access: rifftool/capssds
|
||||
----------------------------------
|
||||
|
||||
The data files in the CAPS archive contain a small additional header describing
|
||||
the data format and implementing an index for fast and in-order data retrieval.
|
||||
Read the :ref:`format documentation <sec-packet-types>` for more details. In
|
||||
contrast to miniSEED files in :term:`SDS` archives created, e.g., by
|
||||
:cite:t:`slarchive` or :cite:t:`scart`, the original miniSEED files stored in
|
||||
the CAPS archive cannot be directly read by common seismological applications.
|
||||
|
||||
You may access data files directly:
|
||||
|
||||
* Test and retrieve data files using :ref:`rifftool`. rifftool addresses
|
||||
individual files directly and does not require the CAPS server to be running.
|
||||
* Run :ref:`capssds` to create a virtual overlay file system presenting a CAPS
|
||||
archive directory as a read-only :term:`SDS` archive with no extra disk space
|
||||
requirement. The CAPS archive directory and file names are mapped. An
|
||||
application reading from a file will only see :term:`miniSEED` records ordered
|
||||
by record start time. You may connect to the virtual SDS archive using the
|
||||
RecordStream SDS or directly read the single :term:`miniSEED` file. Other
|
||||
seismological software such as ObsPy or Seisan may read directly from the SDS
|
||||
archive of the files therein.
|
||||
|
||||
|
||||
.. _sec-caps2caps:
|
||||
|
||||
Synchronize with another CAPS server: caps2caps
|
||||
-----------------------------------------------
|
||||
|
||||
Use :ref:`caps2caps` to synchronize your CAPS server with another one. You may push
|
||||
or pull data on either side. In contrast to the generation of regular :term:`SDS`
|
||||
archives, e.g., by :program:`scart`, the CAPS server will not generate duplicate
|
||||
data records if executing :ref:`caps2caps` multiple times. While synchronizing
|
||||
observe the :ref:`web interface <sec-caps-web-interface>` for the statistics of
|
||||
received, written or rejected data packages.
|
||||
|
||||
|
||||
.. _sec-caps-seedlink:
|
||||
|
||||
Connect from SeedLink: caps_plugin
|
||||
----------------------------------
|
||||
|
||||
The :ref:`caps_plugin` plugin fetches data from a CAPS server and provides the
|
||||
data to :program:`seedlink`. The plugin can be configured and started as any
|
||||
other plugin for :program:`seedlink` or executed on demand. Select *caps* when
|
||||
choosing the plugin in the seedlink binding configuration.
|
||||
|
||||
**Examples:**
|
||||
|
||||
* Fetch data from a remote CAPS server and make them available in your :program:`seedlink` instance:
|
||||
|
||||
#. configure the :ref:`caps_plugin` plugin in the bindings configuration of your
|
||||
:program:`seedlink` instance pointing to the remote CAPS server
|
||||
|
||||
#. enable and start :program:`seedlink`
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
seiscomp enable seedlink
|
||||
seiscomp start seedlink
|
||||
|
||||
* Provide data from a CAPS server by seedlink on the same machine to external clients:
|
||||
|
||||
#. create another seedlink instance, e.g., :program:`seedlinkP`:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
seiscomp alias create seedlinkP seedlink
|
||||
|
||||
#. configure the :ref:`caps_plugin` in the bindings configuration of :program:`seedlinkP`
|
||||
pointing to the local CAPS server.
|
||||
#. enable and start :program:`seedlinkP`:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
seiscomp enable seedlinkP
|
||||
seiscomp start seedlinkP
|
||||
|
||||
|
||||
.. _sec-caps-web:
|
||||
|
||||
Web interface
|
||||
-------------
|
||||
|
||||
The CAPS server ships with a :ref:`web interface <sec-caps-web-interface>`.
|
||||
Beside allowing you to view server statistics, data stored stored on the server
|
||||
can be downloaded for any time if the original format is :term:`miniSEED`.
|
||||
|
||||
For downloading miniSEED data
|
||||
|
||||
#. Select the stream(s) of interest.
|
||||
#. Zoom in to the period of interest. Zooming in and out in time works by
|
||||
right-mouse button actions just like in other |scname| GUI applications like
|
||||
:cite:t:`scrttv`.
|
||||
#. Click on the download button to download the miniSEED file. An error message
|
||||
will be printed in case the original format is not miniSEED.
|
||||
|
||||
.. _fig-web-streams-download:
|
||||
|
||||
.. figure:: media/web_streams_download.png
|
||||
:width: 18cm
|
||||
|
||||
Stream perspective of :term:`CAPS` Web interface allowing to download miniSEED
|
||||
data for selected streams.
|
||||
|
||||
|
||||
.. _sec-caps-fdsnws:
|
||||
|
||||
Built-in FDSNWS
|
||||
---------------
|
||||
|
||||
|appname| speaks natively FDSN Web Services, FDSNWS :cite:p:`fdsn` providing
|
||||
waveform data via dataselect. Information on events and stations are
|
||||
not delivered. The waveform data will be delivered through the port configured in
|
||||
:confval:`AS.http.port` or the port configured by your Apache server, if available.
|
||||
Contact your system administrator for information on the Apache server.
|
||||
Read the documentation of the :ref:`CAPS server <sec-caps-config>` for the configuration.
|
||||
|
||||
|
||||
.. _sec-caps-wws:
|
||||
|
||||
Built-in Winston waveform server
|
||||
--------------------------------
|
||||
|
||||
|appname| speeks natively Winston Waveform Server protocol, WWS, :cite:p:`wws`,
|
||||
e.g., to :cite:t:`swarm` by USGS. Read the documentation of the
|
||||
:ref:`CAPS server<sec-caps-config>` for the configuration.
|
||||
|
||||
|
||||
.. _sec-caps-dataavailability:
|
||||
|
||||
Data availability information
|
||||
-----------------------------
|
||||
|
||||
|
||||
Web interface
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
The :ref:`Channels perspective of the CAPS web interface <sec-caps-web>`
|
||||
indicates periods of availability and of gaps on level of network, station,
|
||||
sensor location and channel. The resolution of colors and percentages is linked
|
||||
to the granularity of the data detection which increases with shorter time
|
||||
windows in order to optimize the speed of the calculation.
|
||||
|
||||
|
||||
capstool
|
||||
~~~~~~~~
|
||||
|
||||
The CAPS server stores information on received data segments
|
||||
including their start and end times. Information on resulting gaps can be
|
||||
retrieved by :ref:`capstool`. Example:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
echo "2023,05,01,12,00,00 2023,05,03,00,00,00 NET * * *" | capstool -G --tolerance=0.5 -H localhost
|
||||
|
||||
|
||||
scardac
|
||||
~~~~~~~
|
||||
|
||||
The availability of data in the caps archive can be analyzed and written to the
|
||||
|scname| database by the |scname| module :cite:t:`scardac`. For availability analysis
|
||||
add the plugin *daccaps* to the list of :confval:`plugins` and URL of the caps
|
||||
archive to the archive configuration of :cite:t:`scardac`. The *daccaps* plugin
|
||||
ships with the gempa package *caps-server*.
|
||||
|
||||
Example configuration of :cite:t:`scardac` (:file:`scardac.cfg`):
|
||||
|
||||
.. code-block:: properties
|
||||
|
||||
plugins = ${plugins}, daccaps
|
||||
archive = caps:///home/data/archive/caps/
|
||||
|
||||
Example call:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
scardac --plugins="daccaps, dbmysql" -d localhost -a caps:///home/data/archive/caps/ --debug
|
||||
|
||||
.. note::
|
||||
|
||||
As of SeisComP in version 6, scardac has received significant optimization.
|
||||
Instead of scanning the full archive, only files which have changed since the
|
||||
last scan will be examined. This means that when scanning the entire archive,
|
||||
the first run may be more time consuming than subsequent ones when executed
|
||||
within reasonable intervals.
|
||||
|
||||
The data availability information can be retrieved from the database, e.g.,
|
||||
using :cite:t:`fdsnws` or :cite:t:`scxmldump`.
|
||||
Reference in New Issue
Block a user