686 lines
43 KiB
HTML
686 lines
43 KiB
HTML
<!DOCTYPE html>
|
||
|
||
<html lang="en">
|
||
<head>
|
||
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
|
||
|
||
<title>scardac — SeisComP Release documentation</title>
|
||
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
|
||
<link rel="stylesheet" href="../_static/seiscomp.css" type="text/css" />
|
||
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=72bcf2f2" />
|
||
<link rel="stylesheet" type="text/css" href="../_static/seiscomp.css?v=c6da7ce6" />
|
||
<link rel="stylesheet" type="text/css" href="../_static/graphviz.css?v=eafc0fe6" />
|
||
<script type="text/javascript" src="../_static/seiscomp.js"></script>
|
||
<script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
|
||
<script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js?v=823bb831"></script>
|
||
<script src="../_static/doctools.js?v=888ff710"></script>
|
||
<script src="../_static/sphinx_highlight.js?v=4825356b"></script>
|
||
<link rel="index" title="Index" href="../genindex.html" />
|
||
<link rel="search" title="Search" href="../search.html" />
|
||
<link rel="next" title="scwfas" href="scwfas.html" />
|
||
<link rel="prev" title="msrtsimul" href="msrtsimul.html" />
|
||
</head>
|
||
<body>
|
||
<div class="header">
|
||
<div class="container">
|
||
<div class="brand">
|
||
<img class="logo" src="../_static/brands/seiscomp/text/white.svg"/>
|
||
<!-- span class="title">SeisComP Release</span -->
|
||
<span class="version">6.9.0</span>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="nav">
|
||
<div class="container">
|
||
<div class="content"><a class="pull-right" id="sidebar-toggle">TOC</a>
|
||
|
||
<div class="related" role="navigation" aria-label="related navigation">
|
||
<ul>
|
||
<li class="right">
|
||
<a href="../genindex.html" title="General Index"
|
||
accesskey="I">
|
||
index
|
||
</a>
|
||
</li>
|
||
<li class="right">
|
||
<a href="scwfas.html" title="scwfas"
|
||
accesskey="N">
|
||
next
|
||
</a>
|
||
</li>
|
||
<li class="right">
|
||
<a href="msrtsimul.html" title="msrtsimul"
|
||
accesskey="P">
|
||
previous
|
||
</a>
|
||
</li>
|
||
<li class="nav-item nav-item-0">
|
||
<a href="../index.html">Home</a>
|
||
</li>
|
||
<li class="nav-item nav-item-1">
|
||
<a href="../modules.html" >Modules</a>
|
||
</li>
|
||
<li class="nav-item nav-item-2">
|
||
<a href="../toc/acquisition.html" accesskey="U">Acquisition</a>
|
||
</li>
|
||
</ul>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="container">
|
||
<div class="fitted content" id="anchors-container">
|
||
<div class="body" role="main">
|
||
|
||
<section id="scardac">
|
||
<span id="id1"></span><h1>scardac<a class="headerlink" href="#scardac" title="Permalink to this heading">¶</a></h1>
|
||
<p><strong>Waveform archive data availability collector.</strong></p>
|
||
<section id="description">
|
||
<h2>Description<a class="headerlink" href="#description" title="Permalink to this heading">¶</a></h2>
|
||
<p>scardac scans an <a class="reference internal" href="../base/glossary.html#term-SDS"><span class="xref std std-term">SDS waveform archive</span></a>, e.g.,
|
||
created by <a class="reference internal" href="slarchive.html#slarchive"><span class="std std-ref">slarchive</span></a> or <a class="reference internal" href="scart.html#scart"><span class="std std-ref">scart</span></a> for available
|
||
<a class="reference internal" href="../base/glossary.html#term-miniSeed"><span class="xref std std-term">miniSEED</span></a> data. It will collect information about</p>
|
||
<ul class="simple">
|
||
<li><p><code class="docutils literal notranslate"><span class="pre">DataExtents</span></code> – the earliest and latest times data is available
|
||
for a particular channel,</p></li>
|
||
<li><p><code class="docutils literal notranslate"><span class="pre">DataAttributeExtents</span></code> – the earliest and latest times data is available
|
||
for a particular channel, quality and sampling rate combination,</p></li>
|
||
<li><p><code class="docutils literal notranslate"><span class="pre">DataSegments</span></code> – continuous data segments sharing the same quality and
|
||
sampling rate attributes.</p></li>
|
||
</ul>
|
||
<p>scardac is intended to be executed periodically, e.g., as a cronjob.</p>
|
||
<p>The availability data information is stored in the SeisComP database under the
|
||
root element <a class="reference internal" href="../base/api-python.html#api-datamodel-python"><span class="std std-ref">DataAvailability</span></a>. Access to the
|
||
availability data is provided by the <a class="reference internal" href="fdsnws.html#fdsnws"><span class="std std-ref">fdsnws</span></a> module via the services:</p>
|
||
<ul class="simple">
|
||
<li><p><a class="reference internal" href="fdsnws.html#sec-station"><span class="std std-ref">/fdsnws/station</span></a> (extent information only, see
|
||
<code class="docutils literal notranslate"><span class="pre">matchtimeseries</span></code> and <code class="docutils literal notranslate"><span class="pre">includeavailability</span></code> request parameters).</p></li>
|
||
<li><p><a class="reference internal" href="fdsnws.html#sec-avail"><span class="std std-ref">/fdsnws/ext/availability</span></a> (extent and segment information
|
||
provided in different formats)</p></li>
|
||
</ul>
|
||
<section id="non-sds-archives">
|
||
<span id="scarcac-non-sds"></span><h3>Non-SDS archives<a class="headerlink" href="#non-sds-archives" title="Permalink to this heading">¶</a></h3>
|
||
<p>scardac can be extended by plugins to scan non-SDS archives. For example the
|
||
<code class="docutils literal notranslate"><span class="pre">daccaps</span></code> plugin provided by <span id="id2"><em>CAPS</em> [<a class="reference internal" href="../base/references.html#id91" title="CAPS. gempa module. URL: https://docs.gempa.de/caps/current/index.html.">3</a>]</span> allows scanning archives generated
|
||
by a CAPS server. Plugins are added to the global module configuration, e.g.:</p>
|
||
<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">plugins</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">${plugins}, daccaps</span>
|
||
</pre></div>
|
||
</div>
|
||
</section>
|
||
<section id="definitions">
|
||
<span id="scarcac-workflow"></span><h3>Definitions<a class="headerlink" href="#definitions" title="Permalink to this heading">¶</a></h3>
|
||
<ul class="simple">
|
||
<li><p><code class="docutils literal notranslate"><span class="pre">Record</span></code> – continuous waveform data of same sampling rate and quality bound
|
||
by a start and end time. scardac will only read the record’s meta data and not
|
||
the actual samples.</p></li>
|
||
<li><p><code class="docutils literal notranslate"><span class="pre">Chunk</span></code> – container for records, e.g., a <a class="reference internal" href="../base/glossary.html#term-miniSeed"><span class="xref std std-term">miniSEED</span></a> file,
|
||
with the following properties:</p>
|
||
<ul>
|
||
<li><p>overall, theoretical time range of records it may contain</p></li>
|
||
<li><p>contains at least one record, otherwise it must be absent</p></li>
|
||
<li><p>each record of a chunk must fulfill the following conditions:</p>
|
||
<ul>
|
||
<li><p><cite>chunk start <= record start < chunk end</cite></p></li>
|
||
<li><p><cite>chunk start < record end < next chunk end</cite></p></li>
|
||
</ul>
|
||
</li>
|
||
<li><p>chunks do not overlap, end time of current chunk equals start time of
|
||
successive chunk, otherwise a <code class="docutils literal notranslate"><span class="pre">chunk</span> <span class="pre">gap</span></code> is declared</p></li>
|
||
<li><p>records may occur unordered within a chunk or across chunk boundaries,
|
||
resulting in <cite>DataSegments</cite> marked as <code class="docutils literal notranslate"><span class="pre">outOfOrder</span></code></p></li>
|
||
</ul>
|
||
</li>
|
||
<li><p><code class="docutils literal notranslate"><span class="pre">Jitter</span></code> – maximum allowed deviation between the end time of the current
|
||
record and the start time of the next record in multiples of the current’s
|
||
record sampling rate. E.g., assuming a sampling rate of 100Hz and a jitter
|
||
of 0.5 will allow for a maximum end to start time difference of 50ms. If
|
||
exceeded a new <cite>DataSegment</cite> is created.</p></li>
|
||
<li><p><code class="docutils literal notranslate"><span class="pre">Mtime</span></code> – time the content of a chunk was last modified. It is used to</p>
|
||
<ul>
|
||
<li><p>decided whether a chunk needs to be read in a secondary application run</p></li>
|
||
<li><p>calculate the <code class="docutils literal notranslate"><span class="pre">updated</span></code> time stamp of a <cite>DataSegment</cite>,
|
||
<cite>DataAttributeExtent</cite> and <cite>DataExtent</cite></p></li>
|
||
</ul>
|
||
</li>
|
||
<li><p><code class="docutils literal notranslate"><span class="pre">Scan</span> <span class="pre">window</span></code> – time window limiting the synchronization of the archive
|
||
with the database configured via <a class="reference internal" href="#confval-filter.time.start"><code class="xref std std-confval docutils literal notranslate"><span class="pre">filter.time.start</span></code></a> and
|
||
<a class="reference internal" href="#confval-filter.time.end"><code class="xref std std-confval docutils literal notranslate"><span class="pre">filter.time.end</span></code></a> respectively <code class="xref std std-option docutils literal notranslate"><span class="pre">--start</span></code> and <code class="xref std std-option docutils literal notranslate"><span class="pre">--end</span></code>.
|
||
The scan window is useful to</p>
|
||
<ul>
|
||
<li><p>reduce the scan time of larger archives. Depending on the size and storage
|
||
type of the archive it may take some time to just list available chunks and
|
||
their mtime.</p></li>
|
||
<li><p>prevent deletion of availability information even though parts of the
|
||
archive have been deleted or moved to a different location</p></li>
|
||
</ul>
|
||
</li>
|
||
<li><p><code class="docutils literal notranslate"><span class="pre">Modification</span> <span class="pre">window</span></code> – the mtime of a chunk is compared with this time
|
||
window to decide whether it needs to be read or not. It is configured via
|
||
<a class="reference internal" href="#confval-mtime.start"><code class="xref std std-confval docutils literal notranslate"><span class="pre">mtime.start</span></code></a> and <a class="reference internal" href="#confval-mtime.end"><code class="xref std std-confval docutils literal notranslate"><span class="pre">mtime.end</span></code></a> repectively
|
||
<code class="xref std std-option docutils literal notranslate"><span class="pre">--modified-since</span></code> and <code class="xref std std-option docutils literal notranslate"><span class="pre">--modified-until</span></code>. If no lower bound
|
||
is defined then the <code class="docutils literal notranslate"><span class="pre">lastScan</span></code> time stored in the <cite>DataExtent</cite> is used
|
||
instead. The mtime check may be disabled using <a class="reference internal" href="#confval-mtime.ignore"><code class="xref std std-confval docutils literal notranslate"><span class="pre">mtime.ignore</span></code></a> or
|
||
<code class="xref std std-option docutils literal notranslate"><span class="pre">--deep-scan</span></code>.
|
||
<strong>Note:</strong> Chunks in front or right after a chunk gap are read in any case
|
||
regardless of the mtime settings.</p></li>
|
||
</ul>
|
||
</section>
|
||
<section id="workflow">
|
||
<h3>Workflow<a class="headerlink" href="#workflow" title="Permalink to this heading">¶</a></h3>
|
||
<ol class="arabic">
|
||
<li><p>Read existing <cite>DataExtents</cite> from database.</p></li>
|
||
<li><p>Collect a list of available stream IDs either by</p>
|
||
<ul class="simple">
|
||
<li><p>scanning the archive for available IDs or</p></li>
|
||
<li><p>reading an ID file defined by <a class="reference internal" href="#confval-nslcFile"><code class="xref std std-confval docutils literal notranslate"><span class="pre">nslcFile</span></code></a>.</p></li>
|
||
</ul>
|
||
</li>
|
||
<li><p>Identify extents to add, update or remove respecting <cite>scan window</cite>,
|
||
<a class="reference internal" href="#confval-filter.nslc.include"><code class="xref std std-confval docutils literal notranslate"><span class="pre">filter.nslc.include</span></code></a> and <a class="reference internal" href="#confval-filter.nslc.exclude"><code class="xref std std-confval docutils literal notranslate"><span class="pre">filter.nslc.exclude</span></code></a>.</p></li>
|
||
<li><p>Subsequently process the <cite>DataExtents</cite> using <a class="reference internal" href="#confval-threads"><code class="xref std std-confval docutils literal notranslate"><span class="pre">threads</span></code></a> number of
|
||
parallel threads. For each <cite>DataExtent</cite>:</p>
|
||
<ol class="arabic">
|
||
<li><p>Collect all available chunks within <cite>scan window</cite>.</p></li>
|
||
<li><p>If the <cite>DataExtent</cite> is new (no database entry yet), store a new and
|
||
empty <cite>DataExtent</cite> to database, else query existing <cite>DataSegments</cite> from
|
||
the database:</p>
|
||
<ul class="simple">
|
||
<li><p>count segments outside <cite>scan window</cite></p></li>
|
||
<li><p>create a database iterator for extents within <cite>scan window</cite></p></li>
|
||
</ul>
|
||
</li>
|
||
<li><p>Create two in-memory segment lists which collect segments to remove and
|
||
segments to add/update</p></li>
|
||
<li><p>For each chunk</p>
|
||
<ul>
|
||
<li><p>determine the <cite>chunk window</cite> and <cite>mtime</cite></p></li>
|
||
<li><p>decide whether the chunk needs to be read depending on the <cite>mtime</cite>
|
||
and a possible <cite>chunk gap</cite>. If necessary, read the chunk and</p>
|
||
<ul class="simple">
|
||
<li><p>create chunk segments by analyzing the chunk records for
|
||
gaps/overlaps defined by <a class="reference internal" href="#confval-jitter"><code class="xref std std-confval docutils literal notranslate"><span class="pre">jitter</span></code></a>, sampling rate or quality
|
||
changes</p></li>
|
||
<li><p>merge chunk segments with database segments and update the in-memory
|
||
segment lists.</p></li>
|
||
</ul>
|
||
<p>If not necessary, advance the database segment iterator to the end
|
||
of the chunk window.</p>
|
||
</li>
|
||
</ul>
|
||
</li>
|
||
<li><p>Remove and then add/update the collected segments.</p></li>
|
||
<li><p>Merge segment information into <cite>DataAttributeExtents</cite></p></li>
|
||
<li><p>Merge <cite>DataAttributeExtents</cite> into overall <cite>DataExtent</cite></p></li>
|
||
</ol>
|
||
</li>
|
||
</ol>
|
||
</section>
|
||
<section id="examples">
|
||
<h3>Examples<a class="headerlink" href="#examples" title="Permalink to this heading">¶</a></h3>
|
||
<ol class="arabic">
|
||
<li><p>Get command line help or execute scardac with default parameters and informative
|
||
debug output:</p>
|
||
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>scardac<span class="w"> </span>-h
|
||
scardac<span class="w"> </span>--debug
|
||
</pre></div>
|
||
</div>
|
||
</li>
|
||
<li><p>Synchronize the availability of waveform data files existing in the standard
|
||
<a class="reference internal" href="../base/glossary.html#term-SDS"><span class="xref std std-term">SDS</span></a> archive with the seiscomp database and create an XML file using
|
||
<a class="reference internal" href="scxmldump.html#scxmldump"><span class="std std-ref">scxmldump</span></a>:</p>
|
||
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>scardac<span class="w"> </span>-d<span class="w"> </span>mysql://sysop:sysop@localhost/seiscomp<span class="w"> </span>-a<span class="w"> </span><span class="nv">$SEISCOMP_ROOT</span>/var/lib/archive<span class="w"> </span>--debug
|
||
scxmldump<span class="w"> </span>-Yf<span class="w"> </span>-d<span class="w"> </span>mysql://sysop:sysop@localhost/seiscomp<span class="w"> </span>-o<span class="w"> </span>availability.xml
|
||
</pre></div>
|
||
</div>
|
||
</li>
|
||
<li><p>Synchronize the availability of waveform data files existing in the standard
|
||
<a class="reference internal" href="../base/glossary.html#term-SDS"><span class="xref std std-term">SDS</span></a> archive with the seiscomp database. Use <a class="reference internal" href="fdsnws.html#fdsnws"><span class="std std-ref">fdsnws</span></a> to fetch a flat file containing a list
|
||
of periods of available data from stations of the CX network sharing the same
|
||
quality and sampling rate attributes:</p>
|
||
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>scardac<span class="w"> </span>-d<span class="w"> </span>mysql://sysop:sysop@localhost/seiscomp<span class="w"> </span>-a<span class="w"> </span><span class="nv">$SEISCOMP_ROOT</span>/var/lib/archive
|
||
wget<span class="w"> </span>-O<span class="w"> </span>availability.txt<span class="w"> </span><span class="s1">'http://localhost:8080/fdsnws/ext/availability/1/query?network=CX'</span>
|
||
</pre></div>
|
||
</div>
|
||
<div class="admonition note">
|
||
<p class="admonition-title">Note</p>
|
||
<p>The <cite>SeisComP</cite> module <a class="reference internal" href="fdsnws.html#fdsnws"><span class="std std-ref">fdsnws</span></a> must be running for executing this
|
||
example.</p>
|
||
</div>
|
||
</li>
|
||
</ol>
|
||
</section>
|
||
</section>
|
||
<section id="module-configuration">
|
||
<span id="scardac-configuration"></span><h2>Module Configuration<a class="headerlink" href="#module-configuration" title="Permalink to this heading">¶</a></h2>
|
||
<div class="line-block">
|
||
<div class="line"><code class="file docutils literal notranslate"><span class="pre">etc/defaults/global.cfg</span></code></div>
|
||
<div class="line"><code class="file docutils literal notranslate"><span class="pre">etc/defaults/scardac.cfg</span></code></div>
|
||
<div class="line"><code class="file docutils literal notranslate"><span class="pre">etc/global.cfg</span></code></div>
|
||
<div class="line"><code class="file docutils literal notranslate"><span class="pre">etc/scardac.cfg</span></code></div>
|
||
<div class="line"><code class="file docutils literal notranslate"><span class="pre">~/.seiscomp/global.cfg</span></code></div>
|
||
<div class="line"><code class="file docutils literal notranslate"><span class="pre">~/.seiscomp/scardac.cfg</span></code></div>
|
||
</div>
|
||
<p>scardac inherits <a class="reference internal" href="global.html#global-configuration"><span class="std std-ref">global options</span></a>.</p>
|
||
<dl class="std confval">
|
||
<dt class="sig sig-object std" id="confval-archive">
|
||
<span class="sig-name descname"><span class="pre">archive</span></span><a class="headerlink" href="#confval-archive" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Default: <code class="docutils literal notranslate"><span class="pre">@SEISCOMP_ROOT@/var/lib/archive</span></code></p>
|
||
<p>Type: <em>string</em></p>
|
||
<p>The URL to the waveform archive where all data is stored.</p>
|
||
<p>Format: [<a class="reference external" href="service://">service://</a>]location[#type]</p>
|
||
<p>"service": The type of the archive. If not given,
|
||
"sds://" is implied assuming an SDS archive. The SDS
|
||
archive structure is defined as
|
||
YEAR/NET/STA/CHA/NET.STA.LOC.CHA.YEAR.DAYFYEAR, e.g.
|
||
2018/GE/APE/BHZ.D/GE.APE..BHZ.D.2018.125</p>
|
||
<p>Other archive types may be considered by plugins.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std confval">
|
||
<dt class="sig sig-object std" id="confval-threads">
|
||
<span class="sig-name descname"><span class="pre">threads</span></span><a class="headerlink" href="#confval-threads" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Default: <code class="docutils literal notranslate"><span class="pre">1</span></code></p>
|
||
<p>Type: <em>int</em></p>
|
||
<p>Number of threads scanning the archive in parallel.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std confval">
|
||
<dt class="sig sig-object std" id="confval-jitter">
|
||
<span class="sig-name descname"><span class="pre">jitter</span></span><a class="headerlink" href="#confval-jitter" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Default: <code class="docutils literal notranslate"><span class="pre">0.5</span></code></p>
|
||
<p>Type: <em>float</em></p>
|
||
<p>Acceptable derivation of end time and start time of successive
|
||
records in multiples of sample time.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std confval">
|
||
<dt class="sig sig-object std" id="confval-maxSegments">
|
||
<span class="sig-name descname"><span class="pre">maxSegments</span></span><a class="headerlink" href="#confval-maxSegments" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Default: <code class="docutils literal notranslate"><span class="pre">1000000</span></code></p>
|
||
<p>Type: <em>int</em></p>
|
||
<p>Maximum number of segments per stream. If the limit is reached
|
||
no more segments are added to the database and the corresponding
|
||
extent is flagged as too fragmented. Set this parameter to 0 to
|
||
disable any limits.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std confval">
|
||
<dt class="sig sig-object std" id="confval-nslcFile">
|
||
<span class="sig-name descname"><span class="pre">nslcFile</span></span><a class="headerlink" href="#confval-nslcFile" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Type: <em>string</em></p>
|
||
<p>Line-based text file of form NET.STA.LOC.CHA defining available
|
||
stream IDs. Depending on the archive type, size and storage
|
||
media used this file may offer a significant performance
|
||
improvement compared to collecting the available streams on each
|
||
startup. Filters defined under <cite>filter.nslc</cite> still apply.</p>
|
||
</dd></dl>
|
||
|
||
<div class="admonition note">
|
||
<p class="admonition-title">Note</p>
|
||
<p><strong>filter.*</strong>
|
||
<em>Parameters of this section limit the data processing to either</em>
|
||
**
|
||
<em>reduce the scan time of larger archives or to</em>
|
||
**
|
||
<em>prevent deletion of availability information even though parts</em>
|
||
<em>of the archive have been deleted or moved to a different</em>
|
||
<em>location.</em></p>
|
||
</div>
|
||
<div class="admonition note">
|
||
<p class="admonition-title">Note</p>
|
||
<p><strong>filter.time.*</strong>
|
||
<em>Limit the processing by record time.</em></p>
|
||
</div>
|
||
<dl class="std confval">
|
||
<dt class="sig sig-object std" id="confval-filter.time.start">
|
||
<span class="sig-name descname"><span class="pre">filter.time.start</span></span><a class="headerlink" href="#confval-filter.time.start" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Type: <em>string</em></p>
|
||
<p>Start of data availability check given as date string or
|
||
as number of days before now.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std confval">
|
||
<dt class="sig sig-object std" id="confval-filter.time.end">
|
||
<span class="sig-name descname"><span class="pre">filter.time.end</span></span><a class="headerlink" href="#confval-filter.time.end" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Type: <em>string</em></p>
|
||
<p>End of data availability check given as date string or
|
||
as number of days before now.</p>
|
||
</dd></dl>
|
||
|
||
<div class="admonition note">
|
||
<p class="admonition-title">Note</p>
|
||
<p><strong>filter.nslc.*</strong>
|
||
<em>Limit the processing by stream IDs.</em></p>
|
||
</div>
|
||
<dl class="std confval">
|
||
<dt class="sig sig-object std" id="confval-filter.nslc.include">
|
||
<span class="sig-name descname"><span class="pre">filter.nslc.include</span></span><a class="headerlink" href="#confval-filter.nslc.include" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Type: <em>list:string</em></p>
|
||
<p>Comma-separated list of stream IDs to process. If
|
||
empty all streams are accepted unless an exclude filter
|
||
is defined. The following wildcards are supported: ‘*’
|
||
and ‘?’.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std confval">
|
||
<dt class="sig sig-object std" id="confval-filter.nslc.exclude">
|
||
<span class="sig-name descname"><span class="pre">filter.nslc.exclude</span></span><a class="headerlink" href="#confval-filter.nslc.exclude" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Type: <em>list:string</em></p>
|
||
<p>Comma-separated list of stream IDs to exclude from
|
||
processing. Excludes take precedence over includes. The
|
||
following wildcards are supported: ‘*’ and ‘?’.</p>
|
||
</dd></dl>
|
||
|
||
<div class="admonition note">
|
||
<p class="admonition-title">Note</p>
|
||
<p><strong>mtime.*</strong>
|
||
<em>Parameters of this section control the rescan of data chunks.</em>
|
||
<em>By default the last update time of the extent is compared with</em>
|
||
<em>the record file modification time to read only files modified</em>
|
||
<em>since the list run.</em></p>
|
||
</div>
|
||
<dl class="std confval">
|
||
<dt class="sig sig-object std" id="confval-mtime.ignore">
|
||
<span class="sig-name descname"><span class="pre">mtime.ignore</span></span><a class="headerlink" href="#confval-mtime.ignore" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Default: <code class="docutils literal notranslate"><span class="pre">false</span></code></p>
|
||
<p>Type: <em>boolean</em></p>
|
||
<p>If set to true all data chunks are read independent of their
|
||
mtime.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std confval">
|
||
<dt class="sig sig-object std" id="confval-mtime.start">
|
||
<span class="sig-name descname"><span class="pre">mtime.start</span></span><a class="headerlink" href="#confval-mtime.start" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Type: <em>string</em></p>
|
||
<p>Only read chunks modified after specific date given as date
|
||
string or as number of days before now.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std confval">
|
||
<dt class="sig sig-object std" id="confval-mtime.end">
|
||
<span class="sig-name descname"><span class="pre">mtime.end</span></span><a class="headerlink" href="#confval-mtime.end" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Type: <em>string</em></p>
|
||
<p>Only read chunks modified before specific date given as date
|
||
string or as number of days before now.</p>
|
||
</dd></dl>
|
||
|
||
</section>
|
||
<section id="command-line-options">
|
||
<h2>Command-Line Options<a class="headerlink" href="#command-line-options" title="Permalink to this heading">¶</a></h2>
|
||
<p><strong class="program">scardac [OPTION]...</strong></p>
|
||
<section id="generic">
|
||
<h3>Generic<a class="headerlink" href="#generic" title="Permalink to this heading">¶</a></h3>
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-h">
|
||
<span id="cmdoption-scardac-help"></span><span class="sig-name descname"><span class="pre">-h</span></span><span class="sig-prename descclassname"></span><span class="sig-prename descclassname"><span class="pre">,</span> </span><span class="sig-name descname"><span class="pre">--help</span></span><span class="sig-prename descclassname"></span><a class="headerlink" href="#cmdoption-scardac-h" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Show help message.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-V">
|
||
<span id="cmdoption-scardac-version"></span><span class="sig-name descname"><span class="pre">-V</span></span><span class="sig-prename descclassname"></span><span class="sig-prename descclassname"><span class="pre">,</span> </span><span class="sig-name descname"><span class="pre">--version</span></span><span class="sig-prename descclassname"></span><a class="headerlink" href="#cmdoption-scardac-V" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Show version information.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-config-file">
|
||
<span class="sig-name descname"><span class="pre">--config-file</span></span><span class="sig-prename descclassname"> <span class="pre">arg</span></span><a class="headerlink" href="#cmdoption-scardac-config-file" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Use alternative configuration file. When this option is
|
||
used the loading of all stages is disabled. Only the
|
||
given configuration file is parsed and used. To use
|
||
another name for the configuration create a symbolic
|
||
link of the application or copy it. Example:
|
||
scautopick -> scautopick2.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-plugins">
|
||
<span class="sig-name descname"><span class="pre">--plugins</span></span><span class="sig-prename descclassname"> <span class="pre">arg</span></span><a class="headerlink" href="#cmdoption-scardac-plugins" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Load given plugins.</p>
|
||
</dd></dl>
|
||
|
||
</section>
|
||
<section id="verbosity">
|
||
<h3>Verbosity<a class="headerlink" href="#verbosity" title="Permalink to this heading">¶</a></h3>
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-verbosity">
|
||
<span class="sig-name descname"><span class="pre">--verbosity</span></span><span class="sig-prename descclassname"> <span class="pre">arg</span></span><a class="headerlink" href="#cmdoption-scardac-verbosity" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Verbosity level [0..4]. 0:quiet, 1:error, 2:warning, 3:info,
|
||
4:debug.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-v">
|
||
<span id="cmdoption-scardac-v"></span><span class="sig-name descname"><span class="pre">-v</span></span><span class="sig-prename descclassname"></span><span class="sig-prename descclassname"><span class="pre">,</span> </span><span class="sig-name descname"><span class="pre">--v</span></span><span class="sig-prename descclassname"></span><a class="headerlink" href="#cmdoption-scardac-v" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Increase verbosity level (may be repeated, eg. -vv).</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-q">
|
||
<span id="cmdoption-scardac-quiet"></span><span class="sig-name descname"><span class="pre">-q</span></span><span class="sig-prename descclassname"></span><span class="sig-prename descclassname"><span class="pre">,</span> </span><span class="sig-name descname"><span class="pre">--quiet</span></span><span class="sig-prename descclassname"></span><a class="headerlink" href="#cmdoption-scardac-q" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Quiet mode: no logging output.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-print-component">
|
||
<span class="sig-name descname"><span class="pre">--print-component</span></span><span class="sig-prename descclassname"> <span class="pre">arg</span></span><a class="headerlink" href="#cmdoption-scardac-print-component" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>For each log entry print the component right after the
|
||
log level. By default the component output is enabled
|
||
for file output but disabled for console output.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-component">
|
||
<span class="sig-name descname"><span class="pre">--component</span></span><span class="sig-prename descclassname"> <span class="pre">arg</span></span><a class="headerlink" href="#cmdoption-scardac-component" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Limit the logging to a certain component. This option can
|
||
be given more than once.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-s">
|
||
<span id="cmdoption-scardac-syslog"></span><span class="sig-name descname"><span class="pre">-s</span></span><span class="sig-prename descclassname"></span><span class="sig-prename descclassname"><span class="pre">,</span> </span><span class="sig-name descname"><span class="pre">--syslog</span></span><span class="sig-prename descclassname"></span><a class="headerlink" href="#cmdoption-scardac-s" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Use syslog logging backend. The output usually goes to
|
||
/var/lib/messages.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-l">
|
||
<span id="cmdoption-scardac-lockfile"></span><span class="sig-name descname"><span class="pre">-l</span></span><span class="sig-prename descclassname"></span><span class="sig-prename descclassname"><span class="pre">,</span> </span><span class="sig-name descname"><span class="pre">--lockfile</span></span><span class="sig-prename descclassname"> <span class="pre">arg</span></span><a class="headerlink" href="#cmdoption-scardac-l" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Path to lock file.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-console">
|
||
<span class="sig-name descname"><span class="pre">--console</span></span><span class="sig-prename descclassname"> <span class="pre">arg</span></span><a class="headerlink" href="#cmdoption-scardac-console" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Send log output to stdout.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-debug">
|
||
<span class="sig-name descname"><span class="pre">--debug</span></span><span class="sig-prename descclassname"></span><a class="headerlink" href="#cmdoption-scardac-debug" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Execute in debug mode.
|
||
Equivalent to --verbosity=4 --console=1 .</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-trace">
|
||
<span class="sig-name descname"><span class="pre">--trace</span></span><span class="sig-prename descclassname"></span><a class="headerlink" href="#cmdoption-scardac-trace" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Execute in trace mode.
|
||
Equivalent to --verbosity=4 --console=1 --print-component=1
|
||
--print-context=1 .</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-log-file">
|
||
<span class="sig-name descname"><span class="pre">--log-file</span></span><span class="sig-prename descclassname"> <span class="pre">arg</span></span><a class="headerlink" href="#cmdoption-scardac-log-file" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Use alternative log file.</p>
|
||
</dd></dl>
|
||
|
||
</section>
|
||
<section id="collector">
|
||
<h3>Collector<a class="headerlink" href="#collector" title="Permalink to this heading">¶</a></h3>
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-a">
|
||
<span id="cmdoption-scardac-archive"></span><span class="sig-name descname"><span class="pre">-a</span></span><span class="sig-prename descclassname"></span><span class="sig-prename descclassname"><span class="pre">,</span> </span><span class="sig-name descname"><span class="pre">--archive</span></span><span class="sig-prename descclassname"> <span class="pre">arg</span></span><a class="headerlink" href="#cmdoption-scardac-a" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Overrides configuration parameter <a class="reference internal" href="slarchive.html#confval-archive"><code class="xref std std-confval docutils literal notranslate"><span class="pre">archive</span></code></a>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-threads">
|
||
<span class="sig-name descname"><span class="pre">--threads</span></span><span class="sig-prename descclassname"> <span class="pre">arg</span></span><a class="headerlink" href="#cmdoption-scardac-threads" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Overrides configuration parameter <a class="reference internal" href="#confval-threads"><code class="xref std std-confval docutils literal notranslate"><span class="pre">threads</span></code></a>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-j">
|
||
<span id="cmdoption-scardac-jitter"></span><span class="sig-name descname"><span class="pre">-j</span></span><span class="sig-prename descclassname"></span><span class="sig-prename descclassname"><span class="pre">,</span> </span><span class="sig-name descname"><span class="pre">--jitter</span></span><span class="sig-prename descclassname"> <span class="pre">arg</span></span><a class="headerlink" href="#cmdoption-scardac-j" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Overrides configuration parameter <a class="reference internal" href="#confval-jitter"><code class="xref std std-confval docutils literal notranslate"><span class="pre">jitter</span></code></a>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-nslc">
|
||
<span class="sig-name descname"><span class="pre">--nslc</span></span><span class="sig-prename descclassname"> <span class="pre">arg</span></span><a class="headerlink" href="#cmdoption-scardac-nslc" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Overrides configuration parameter <a class="reference internal" href="#confval-nslcFile"><code class="xref std std-confval docutils literal notranslate"><span class="pre">nslcFile</span></code></a>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-start">
|
||
<span class="sig-name descname"><span class="pre">--start</span></span><span class="sig-prename descclassname"> <span class="pre">arg</span></span><a class="headerlink" href="#cmdoption-scardac-start" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Overrides configuration parameter <a class="reference internal" href="#confval-filter.time.start"><code class="xref std std-confval docutils literal notranslate"><span class="pre">filter.time.start</span></code></a>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-end">
|
||
<span class="sig-name descname"><span class="pre">--end</span></span><span class="sig-prename descclassname"> <span class="pre">arg</span></span><a class="headerlink" href="#cmdoption-scardac-end" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Overrides configuration parameter <a class="reference internal" href="#confval-filter.time.end"><code class="xref std std-confval docutils literal notranslate"><span class="pre">filter.time.end</span></code></a>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-include">
|
||
<span class="sig-name descname"><span class="pre">--include</span></span><span class="sig-prename descclassname"> <span class="pre">arg</span></span><a class="headerlink" href="#cmdoption-scardac-include" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Overrides configuration parameter <a class="reference internal" href="#confval-filter.nslc.include"><code class="xref std std-confval docutils literal notranslate"><span class="pre">filter.nslc.include</span></code></a>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-exclude">
|
||
<span class="sig-name descname"><span class="pre">--exclude</span></span><span class="sig-prename descclassname"> <span class="pre">arg</span></span><a class="headerlink" href="#cmdoption-scardac-exclude" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Overrides configuration parameter <a class="reference internal" href="#confval-filter.nslc.exclude"><code class="xref std std-confval docutils literal notranslate"><span class="pre">filter.nslc.exclude</span></code></a>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-deep-scan">
|
||
<span class="sig-name descname"><span class="pre">--deep-scan</span></span><span class="sig-prename descclassname"></span><a class="headerlink" href="#cmdoption-scardac-deep-scan" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Overrides configuration parameter <a class="reference internal" href="#confval-mtime.ignore"><code class="xref std std-confval docutils literal notranslate"><span class="pre">mtime.ignore</span></code></a>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-modified-since">
|
||
<span class="sig-name descname"><span class="pre">--modified-since</span></span><span class="sig-prename descclassname"> <span class="pre">arg</span></span><a class="headerlink" href="#cmdoption-scardac-modified-since" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Overrides configuration parameter <a class="reference internal" href="#confval-mtime.start"><code class="xref std std-confval docutils literal notranslate"><span class="pre">mtime.start</span></code></a>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-modified-until">
|
||
<span class="sig-name descname"><span class="pre">--modified-until</span></span><span class="sig-prename descclassname"> <span class="pre">arg</span></span><a class="headerlink" href="#cmdoption-scardac-modified-until" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Overrides configuration parameter <a class="reference internal" href="#confval-mtime.end"><code class="xref std std-confval docutils literal notranslate"><span class="pre">mtime.end</span></code></a>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="std option">
|
||
<dt class="sig sig-object std" id="cmdoption-scardac-generate-test-data">
|
||
<span class="sig-name descname"><span class="pre">--generate-test-data</span></span><span class="sig-prename descclassname"> <span class="pre">arg</span></span><a class="headerlink" href="#cmdoption-scardac-generate-test-data" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Do not scan the archive but generate test data for each
|
||
stream in the inventory. Format:
|
||
days,gaps,gapslen,overlaps,overlaplen. E.g., the following
|
||
parameter list would generate test data for 100 days
|
||
(starting from now()-100) which includes 150 gaps with a
|
||
length of 2.5s followed by 50 overlaps with an overlap of
|
||
5s: --generate-test-data=100,150,2.5,50,5</p>
|
||
</dd></dl>
|
||
|
||
</section>
|
||
</section>
|
||
</section>
|
||
|
||
|
||
<div id="anchors-bottom"></div>
|
||
</div>
|
||
|
||
<div class="sidebar" role="navigation" aria-label="main navigation">
|
||
<div id="anchors-top"></div>
|
||
<div id="anchors" class="content">
|
||
<div id="searchbox" style="display: none" role="search">
|
||
<h3 id="searchlabel">Quick search</h3>
|
||
<div class="searchformwrapper">
|
||
<form class="search" action="../search.html" method="get">
|
||
<input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
|
||
<input type="submit" value="Go" />
|
||
</form>
|
||
</div>
|
||
</div>
|
||
<script>document.getElementById('searchbox').style.display = "block"</script>
|
||
<div>
|
||
<h3><a href="../index.html">Table of Contents</a></h3>
|
||
<ul>
|
||
<li><a class="reference internal" href="#">scardac</a><ul>
|
||
<li><a class="reference internal" href="#description">Description</a><ul>
|
||
<li><a class="reference internal" href="#non-sds-archives">Non-SDS archives</a></li>
|
||
<li><a class="reference internal" href="#definitions">Definitions</a></li>
|
||
<li><a class="reference internal" href="#workflow">Workflow</a></li>
|
||
<li><a class="reference internal" href="#examples">Examples</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a class="reference internal" href="#module-configuration">Module Configuration</a></li>
|
||
<li><a class="reference internal" href="#command-line-options">Command-Line Options</a><ul>
|
||
<li><a class="reference internal" href="#generic">Generic</a></li>
|
||
<li><a class="reference internal" href="#verbosity">Verbosity</a></li>
|
||
<li><a class="reference internal" href="#collector">Collector</a></li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
|
||
</div>
|
||
<div>
|
||
<h4>Previous topic</h4>
|
||
<p class="topless"><a href="msrtsimul.html"
|
||
title="previous chapter">msrtsimul</a></p>
|
||
</div>
|
||
<div>
|
||
<h4>Next topic</h4>
|
||
<p class="topless"><a href="scwfas.html"
|
||
title="next chapter">scwfas</a></p>
|
||
</div>
|
||
<div role="note" aria-label="source link">
|
||
<h3>This Page</h3>
|
||
<ul class="this-page-menu">
|
||
<li><a href="../_sources/apps/scardac.rst.txt"
|
||
rel="nofollow">Show Source</a></li>
|
||
</ul>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="footer">
|
||
<div class="container">
|
||
<div class="horizontal layout content">
|
||
<a class="fade-in" href="https://www.gempa.de" target="_blank">
|
||
<img class="brand" src="../_static/brands/gempa.svg"/>
|
||
</a>
|
||
<div class="stretched align-center fitted content">
|
||
<div>
|
||
Version <b>6.9.0</b> Release
|
||
</div>
|
||
<div class="copyright">
|
||
Copyright © gempa GmbH, GFZ Potsdam.
|
||
</div>
|
||
</div>
|
||
<a class="fade-in" href="https://www.gfz-potsdam.de" target="_blank">
|
||
<img class="brand" src="../_static/brands/gfz.svg"/>
|
||
</a>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</body>
|
||
</html> |