365 lines
12 KiB
ReStructuredText
365 lines
12 KiB
ReStructuredText
.. highlight:: rst
|
|
|
|
.. _scart:
|
|
|
|
#####
|
|
scart
|
|
#####
|
|
|
|
**Import/export MiniSEED data to/from SDS archives.**
|
|
|
|
|
|
Description
|
|
===========
|
|
|
|
The archive tool scart reads and writes :term:`SDS` archives and files
|
|
in miniSEED format and checks miniSEED archives or prints stream information.
|
|
|
|
* **Dump mode:** Create miniSEED files (multiplexed), e.g. for playbacks, from
|
|
:term:`SDS` structured data (e.g. created by slarchive).
|
|
* **Dump mode:** Play back records directly out of an SDS structure.
|
|
* **Import mode:** Import multiplexed miniSEED files into a local SDS waveform
|
|
archive.
|
|
* **Import mode:** Import data using the :ref:`global_recordstream` interface
|
|
into a local SDS waveform archive.
|
|
* **Import mode:** Read data from any :ref:`global_recordstream` interface
|
|
and dump it to file.
|
|
* **Check mode:** Check an archive of miniSEED files for out-of-order records in
|
|
files.
|
|
|
|
.. warning::
|
|
|
|
* When creating :term:`SDS` archives, scart simply appends the new records to
|
|
existing ones. Multiple imports of the same data result in duplication.
|
|
* Out-of-order imports of waveforms into a SDS archive result in out-of-order
|
|
records which may not be processed. Clean your archive using :ref:`scmssort`.
|
|
* **Before importing miniSEED data** into an SDS archive they must be sorted
|
|
by time and duplicate records must be removed. Otherwise, the SDS archive
|
|
may not be correctly readable by other modules. Therefore, combine scart
|
|
with :ref:`scmssort` for multiplexing and removal of duplicates.
|
|
|
|
.. hint::
|
|
|
|
In dump and import mode output streams may be filtered by
|
|
|
|
* Time windows (:option:`-t`),
|
|
* Network-station-location-channel (NSLC) lists (:option:`--nslc`) created,
|
|
e.g., with
|
|
|
|
* :ref:`scinv` from inventories,
|
|
* :ref:`scdumpcfg` from bindings configuration,
|
|
* scart itself from other miniSEED files or achives.
|
|
|
|
In dump mode output streams may also be filtered by
|
|
|
|
* Time window - stream lists (:option:`--list`, dump mode) generated by
|
|
:ref:`scevtstreams` for particular events.
|
|
|
|
Time strings may be given in
|
|
* ISO time format, e.g., 2023-03-28T15:48:00
|
|
* or the 'old' SeisComP time format with empty character between date and time, e.g.,
|
|
'2023-03-28 15:48:00'.
|
|
|
|
When omitting seconds, minutes or hours values of zero are implied.
|
|
|
|
|
|
.. _scart-config:
|
|
|
|
Configuration
|
|
=============
|
|
|
|
scart can make use of :ref:`global_recordstream` implementations which are
|
|
provided by additional plugins. For loading additional plugins, e.g. the *xyz*
|
|
plugin create and configure :file:`scart.cfg`:
|
|
|
|
.. code-block:: sh
|
|
|
|
plugins = xyz
|
|
|
|
|
|
Examples
|
|
========
|
|
|
|
.. hint::
|
|
|
|
The usage of wildcards in place of network, station, location or channel code
|
|
is allowed in many options (:option:`-n`, :option:`-c`, :option:`-l`,
|
|
:option:`--list`, :option:`--nslc`) and follows these rules:
|
|
|
|
* Import mode: the wildcards are passed to the :ref:`global_recordstream` interface,
|
|
that interprets them. Normally both "*" and "?" are supported by RecordStreams.
|
|
* Dump mode: the wildcards are interpreted by scart command that supports "*" for
|
|
network, station, location codes and "*", "?", "(", ")", "|" for channel code.
|
|
|
|
#. Extract data from the default :term:`SDS` archive in :file:`$SEISCOMP_ROOT/var/lib/archive`
|
|
or from a local :term:`SDS` archive [SDS archive] into a miniSEED file :file:`file.mseed`
|
|
and sort by end time of the records:
|
|
|
|
.. code-block:: sh
|
|
|
|
scart -dsvE -t '[start-time]~[end-time]' [SDS archive] > [file.mseed]
|
|
scart -dsvE -t '[start-time]~[end-time]' > file.mseed
|
|
scart -dsvE -t '[start-time]~[end-time]' -n '[NET1],[NET2]' > file.mseed
|
|
scart -dsvE -t '[start-time]~[end-time]' -n '[NET]' -c '(E,H)H(1,2,3)' > file.mseed
|
|
scart -dsvE -t '[start-time]~[end-time]' -n '[N1.S1.L1.C1],[N2.S2.L2.C2]' > file.mseed
|
|
scart -dsvE -t '[start-time]~[end-time]' --nslc list.file > file.mseed
|
|
scart -dsvE -t --list list.file > file.mseed
|
|
|
|
|
|
It is possible to achieve the same result of the dump mode using a
|
|
combination of the input mode and the :ref:`scmssort` command, which allows
|
|
to read the input data from any supported :ref:`global_recordstream`,
|
|
not only an SDS archive:
|
|
|
|
.. code-block:: sh
|
|
|
|
scart -I [record-stream] --list list.file --stdout | scmssort -u -E -v > file.mseed
|
|
|
|
.. note::
|
|
|
|
Sorting data is computational expensive but required for waveform playbacks.
|
|
|
|
#. Push miniSEED data from file :file:`file.mseed` or standard input
|
|
(stdin) into a local :term:`SDS` archive or a file. Additionally, you may
|
|
check if the records of archived files are correctly ordered, filter by time
|
|
and/or with NSLC list and print the output streams:
|
|
|
|
.. code-block:: sh
|
|
|
|
scmssort -u -E [file.mseed] > sorted.mseed
|
|
scart -I sorted.mseed --with-filecheck [SDS archive]
|
|
|
|
scart -I [file.mseed] -t '[start-time]~[end-time]' --print-streams --nslc list.file -o [out.mseed]
|
|
|
|
cat sorted.mseed | scart -I - [SDS archive]
|
|
cat sorted.mseed | scart [SDS archive]
|
|
|
|
#. Collect data using the :ref:`global_recordstream` interface (e.g. FDSNWS server)
|
|
and write to a miniSEED file or import it into a local :term:`SDS` archive. The
|
|
data streams and the time spans can be defined in several ways. The data streams
|
|
(:option:`--list`) can be automatically generated, e.g., by :ref:`scevtstreams`.
|
|
|
|
.. code-block:: sh
|
|
|
|
scart -I fdsnws://[server]:80 --list list.file [SDS archive]
|
|
scart -I fdsnws://[server]:80 --list list.file -o file.mseed
|
|
scart -I fdsnws://[server]:80 -t '[start-time]~[end-time]' --nslc list.file [SDS archive]
|
|
scart -I fdsnws://[server]:80 -t '[start-time]~[end-time]' -n '[NET1],[NET2]' [SDS archive]
|
|
scart -I fdsnws://[server]:80 -t '[start-time]~[end-time]' -n '[NET]' -c 'EH?' [SDS archive]
|
|
scart -I fdsnws://[server]:80 -t '[start-time]~[end-time]' -n '[N1.S1.L1.C1],[N2.S2.L2.C2]' [SDS archive]
|
|
|
|
#. Check all files of an SDS archive or other directory structure for
|
|
miniSEED files with out-of-order records:
|
|
|
|
.. code-block:: sh
|
|
|
|
scart --check [archive]
|
|
|
|
#. Print stream information from miniSEED files in archives or from
|
|
:term:`RecordStream` without actually writing miniSEED data. In dump and
|
|
import mode use the :option:`--test` if miniSEED data shall be read but not
|
|
written.
|
|
|
|
.. code-block:: sh
|
|
|
|
scart --print-streams -I [miniSEED file] --test
|
|
scart --print-streams -d -t [time span] --test [SDS archive]
|
|
scart --print-streams --check [archive]
|
|
|
|
The output looks like this:
|
|
|
|
.. code-block:: sh
|
|
|
|
# streamID start end records samples samplingRate
|
|
GE.RGN..BHZ 2022-12-08T15:34:41.895Z 2022-12-08T15:52:19.145Z 58 21145 20.0
|
|
|
|
where the header and the text body are printed to stderr.
|
|
|
|
|
|
|
|
Command-Line Options
|
|
====================
|
|
|
|
.. program:: scart
|
|
|
|
:program:`scart [options] {archive-dir}`
|
|
|
|
The last option has to be the archive directory when dump mode is enabled.
|
|
When no archive directory is explicitly given,
|
|
\$SEISCOMP_ROOT\/var\/lib\/archive or the current directory
|
|
is used depending on whether \$SEISCOMP_ROOT has been set or not.
|
|
The default operation mode is import. That means that a multiplexed
|
|
MiniSEED file or another record source such as ArcLink is used to import
|
|
records into a SDS structure.
|
|
|
|
|
|
Verbosity
|
|
---------
|
|
|
|
.. option:: -v, --verbose
|
|
|
|
Verbose mode.
|
|
|
|
.. option:: -h, --help
|
|
|
|
Display a help message.
|
|
|
|
|
|
Mode
|
|
----
|
|
|
|
.. option:: --check
|
|
|
|
Check mode. Check all files
|
|
in the given directory for erroneous miniSEED records.
|
|
All sub\-directories are included. If no directory is given,
|
|
the default SDS archive is scanned. Checks are only complete
|
|
for files containing exactly one stream. More complete checks
|
|
are made with scmssort.
|
|
|
|
.. option:: -d, --dump
|
|
|
|
Set export \(dump\) mode. Records are retrieved from an archive and
|
|
written to standard output.
|
|
|
|
.. option:: -I
|
|
|
|
Import mode \(default\): Specify the recordstream URL to read
|
|
the data from for archiving. When using any other
|
|
recordstream than file, a stream list file is needed.
|
|
Specifying \- implies file:\/\/\- \(stdin\). If no mode is
|
|
explicitly specified, \-I file:\/\/\- is assumed.
|
|
|
|
|
|
Processing
|
|
----------
|
|
|
|
.. option:: -c channels
|
|
|
|
Channel filter to be applied to the data streams.
|
|
Default for Dump: \"\(B\|E\|H\|M\|S\)\(D\|H\|L\|N\)\(E\|F\|N\|Z\|1\|2\|3\)\"
|
|
Default for Import: \"\*\"
|
|
|
|
.. option:: -E
|
|
|
|
Dump mode: sort records according to their end time.
|
|
Default: start time.
|
|
|
|
.. option:: --files count
|
|
|
|
Dump mode: Specify the number of file handles to cache.
|
|
Default: 100.
|
|
|
|
.. option:: -i
|
|
|
|
Ignore records without data samples.
|
|
|
|
.. option:: -l, --list file
|
|
|
|
Import, dump mode: Use a stream list file with time windows instead
|
|
of defined networks and channels \(\-n, \-c and \-t are ignored\).
|
|
The list can be generated from events by scevtstreams. One
|
|
line per stream. Line format: starttime;endtime;streamID
|
|
The time format is the same as described in option '\-t'.
|
|
|
|
Example:
|
|
|
|
2019\-07\-17 02:00:00;2019\-07\-17 02:10:00;GR.CLL..BH?
|
|
|
|
.. option:: -m, --modify
|
|
|
|
Dump mode: Modify the record time for real time playback.
|
|
The first record time is NOW. The relative time of
|
|
successive records to the first one are kept.
|
|
|
|
.. option:: -n networks
|
|
|
|
Import, dump mode: Data stream selection as a comma separated list
|
|
\"stream1,stream2,streamX\" where each stream can be NET or NET.STA
|
|
or NET.STA.LOC or NET.STA.LOC.CHA.
|
|
If CHA is omitted, it defaults to the value of \-c option.
|
|
Default: \"\*\"
|
|
|
|
.. option:: --nslc file
|
|
|
|
Import, dump mode: Stream list file to be used instead of
|
|
defined networks and channels \(\-n and \-c are ignored\)
|
|
for filtering the data by the given streams. Dump mode:
|
|
Use in combination with \-t\! One line per stream, line
|
|
format: NET.STA.LOC.CHA
|
|
|
|
Example:
|
|
|
|
GR.CLL..BH?
|
|
|
|
.. option:: --rename rule
|
|
|
|
Import, dump mode: Rename stream data according to the provided
|
|
rule\(s\). A rule is \"[match\-stream:]rename\-stream\" and match\-stream
|
|
is optional. match\-stream and rename\-stream are in the
|
|
\"NET.STA.LOC.CHA\" format. match\-stream supports special
|
|
charactes \"?\" \"\*\" \"\|\" \"\(\" \"\)\". rename\-stream supports the
|
|
special character \"\-\" that can be used in place of NET, STA,
|
|
LOC, CHA codes with the meaning of not renaming those.
|
|
\"\-\" can also be used as the last character in CHA code.
|
|
Multiple rules can be provided as a comma separated list
|
|
or by providing multiple \-\-rename options.
|
|
|
|
.. option:: -s, --sort
|
|
|
|
Dump mode: Sort records by [start\-]time. To sort records by their
|
|
end time use \-E.
|
|
|
|
.. option:: --speed value
|
|
|
|
Dump mode: Specify the speed to dump the records. A value of 0 means
|
|
no delay otherwise speed is a multiplier of the real time difference
|
|
between the records. When feeding the records directly into the replay
|
|
pipe a value of 1 \(real time\) is recommended.
|
|
|
|
.. option:: -t, --time-window timeWindow
|
|
|
|
Import, dump mode: Specify the time window \(as one properly
|
|
quoted string\) to dump records for. Times are UTC and
|
|
separated by a tilde \"\~\". To dump one hour of
|
|
waveform data between 2008\/01\/01 00:00:00 and 2008\/01\/01
|
|
01:00:00 use
|
|
<\-t 2008\-01\-01T00:00:00\~2008\-01\-01T01:00:00>.
|
|
|
|
|
|
Output
|
|
------
|
|
|
|
.. option:: -o, --output
|
|
|
|
Dump, Import mode: Write data to given file instead of creating
|
|
a SDS archive. Deactivates \-\-stdout. Deactivated by \-\-test.
|
|
|
|
.. option:: --print-streams
|
|
|
|
Print stream information only and exit. Works in import, dump
|
|
and check mode. Output: NET.STA.LOC.CHA StartTime EndTime.
|
|
|
|
.. option:: --stdout
|
|
|
|
Import mode: Write to stdout instead of creating a SDS archive.
|
|
Deactivated by \-\-test and \-\-output.
|
|
|
|
.. option:: --test
|
|
|
|
Test input only, deactivate all miniSEED output. This switch is
|
|
useful for debugging and printing stream information with
|
|
\-\-print\-streams.
|
|
|
|
.. option:: --with-filecheck
|
|
|
|
Import mode: Check all accessed files. Unsorted or unreadable
|
|
files are reported to stderr. Checks are only complete
|
|
for files containing exactly one stream. More complete
|
|
checks are made with scmssort.
|
|
|
|
.. option:: --with-filename
|
|
|
|
Import mode: Print all accessed files to stderr after import.
|
|
|