.. highlight:: rst

.. _ngl2caps:

########
ngl2caps
########

**NGL CAPS plugin. Reads GNSS data in kenv format and sends it to CAPS.**


Description
===========

Read GNSS data in kenv format, convert to RAW and send to a CAPS server.


Waveform Data
=============

Format describtions
-------------------

* **kenv Format**:
  Final 5 min rapid solutions from stations [stat]:
  http://geodesy.unr.edu/gps_timeseries/rapids_5min/kenv/[stat]/

  * Sample rate: 1/300 sps
  * Example: http://geodesy.unr.edu/gps_timeseries/rapids_5min/kenv/0ALM/
  * Format (http://geodesy.unr.edu/gps_timeseries/README_kenv.txt):

    .. code-block:: properties

       ----------------------------------------
       .kenv format (east,north,up time series)
       ----------------------------------------
       Note: first line of .kenv contains header fields to help human interpretation.

       Column  Header          Example     Description
       ------  -----------     ---------   ------------------------------------
        1      site            JPLM        4-character station ID
        2      sec-J2000       3705278100  GPS seconds since 2000-01-01 12:00:00
        3      __MJD           55833       Modified Julian Day for GPS day
        4      year            2011        Year
        5      mm               9          Month
        6      dd              29          Day of month
        7      doy             272         Day of year
        8      s-day          10800        Seconds of the GPS day
        9      ___e-ref(m)    -0.202261    East from reference longitude in llh
       10      ___n-ref(m)     0.079096    North from reference latitude in llh
       11      ___v-ref(m)    -0.025883    Up from reference height in llh
       12      _e-mean(m)     -0.015904    East from daily mean position
       13      _n-mean(m)     -0.000944    North from daily mean position
       14      _v-mean(m)      0.000232    Up from daily mean position
       15      sig_e(m)       0.005700     Sigma east
       16      sig_n(m)       0.006875     Sigma north
       17      sig_v(m)       0.021739     Sigma up

* RINEX data web server: http://geodesy.unr.edu/magnet/rinex/


Data sources
------------

kenv data are provided by Nevada Geodetic Laboratory, NGL
(http://geodesy.unr.edu/):

* **Preferred:** Rapid data, 24 hours latency, 5 minutes sps, 1 ZIP file per year:
  http://geodesy.unr.edu/NGLStationPages/RapidStationList

  Archives:
  http://geodesy.unr.edu/gps_timeseries/kenv/

  Example for one station, 1 year:
  http://geodesy.unr.edu/gps_timeseries/kenv/0ABI/0ABI.2022.kenv.zip

* Ultra-rapid data, 1 hour latency, 5 minutes sps (may have many gaps and outages):
  http://geodesy.unr.edu/NGLStationPages/UltraStationList

  Hourly upload:
  http://geodesy.unr.edu/gps_timeseries/ultracombo/kenv/2022/141/

  Archives:
  http://geodesy.unr.edu/gps_timeseries/kenv/

* Final 24 h solutions from stations [stat]:
  http://geodesy.unr.edu/gps_timeseries/txyz/IGS14/[stat].txyz2

  Example: http://geodesy.unr.edu/gps_timeseries/txyz/IGS14/ARIS.txyz2


Fetch data
----------

.. note::

   Data on the NGL server are zipped files, one file per year.
   Geoffrey Blewitt why this is:

   "We zip into yearly files which are updated every week. The actual day it is
   updated varies depending on when other necessary inputs are ready, such as
   JPL orbit files, and weather model files from TU Vienna, and ocean loading
   files from Chalmers, Sweden.   So, you would not want to check every day, but
   I would say every Wednesday would typically work.
   It’s also important to note that while we process Final data every week,
   often we include newly discovered or late data that can go as far back as
   1994. So, we are not just incrementing with a new week of files.

   On the technical side, we zip the data into yearly files to reduce the number
   of “inodes” (files) on our server, which can slow things down when attempting
   to seek files, and in the past has overloaded the server with maximum number
   of inodes.

   Zipping files has speeded up our operations considerably and allows us to
   process all the world’s GPS geodetic data (currently > 18,000 stations).
   It also allows for efficient backup, which can take far too long with
   individual files."

For fetching data from the server above adjust:

* *year* year to consider as set in the file name. Use "" for all.
* *url*
* *outPut*: target directory of fetched files

in the Python script provided with the source code:
:file:`ngl/plugin/tools/fetchFilesFromWeb.py` and use it:

.. code-block:: sh

   python fetchFilesFromWeb.py

Change to the target directory and unpack the files:

.. code-block:: sh

   cd [target]
   unzip *
   gunzip *

Running ngl2caps converts data to RAW and sends them to the CAPS server. The
assumed physical unit of the output data is **nanometer** and the original data
are assumed in units of **meter**. The conversion is applied by the :confval:`gain`
which is therefore **1000,000,000** by default.
**Make sure to generate the inventory with the correct gain and gainUnit.**
If data are stored in "nm" and gainUnit of the stream in the inventory is "m" (SI unit),
the gain in the inventory must be :math:`10^{-09}`. Example:

.. code-block:: sh

   ngl2caps --debug -p [target directory] --gain 1000000000


Output streams
--------------

The plugin writes the following parameter to the header of the output data.

.. csv-table::
   :widths: 2 2 1 5
   :align: left
   :delim: ;
   :header: group, parameter, added, value/remark

   network; code; x; from input parameter
   network; start time; x; from input parameter

   station; code; x; from input file
   station; start time; x; same as network
   station; coordinates; x; from input file

   sensor location; code; x; from input parameter
   sensor location; start; x; same as network
   sensor location; elevation; x; same as station

   stream; code; x; from band + [XYZ][ZNE]
         ;     ;  ; from band + [X][ZNE] : derived data, daily mean removed
         ;     ;  ; from band + [Y][ZNE] : raw data
         ;     ;  ; from band + [Z][ZNE] : sigma data


Inventory
=========

* Access to different stations lists and formats: http://geodesy.unr.edu/PlugNPlayPortal.php
* Station list (HTML): http://geodesy.unr.edu/NGLStationPages/GlobalStationList
* Station list (TXT, 1/300 sps, 24 hour latency):
  http://geodesy.unr.edu/NGLStationPages/DataHoldingsRapid5min.txt
* Details about stations [stat]: http://geodesy.unr.edu/NGLStationPages/stations/[stat].sta

* Station table with coordinates:
  http://plugandplay.unavco.org:8080/unrgsac/gsacapi/site/search#tabId3153-1

:ref:`table2inv` can be used for conversion of a station table to :term:`SCML`.



Module Configuration
====================

| :file:`etc/defaults/global.cfg`
| :file:`etc/defaults/ngl2caps.cfg`
| :file:`etc/global.cfg`
| :file:`etc/ngl2caps.cfg`
| :file:`~/.seiscomp/global.cfg`
| :file:`~/.seiscomp/ngl2caps.cfg`

ngl2caps inherits :ref:`global options<global-configuration>`.

.. note::

   Modules/plugins may require a license file. The default path to license
   files is :file:`@DATADIR@/licenses/` which can be overridden by global
   configuration of the parameter :confval:`gempa.licensePath`. Example: ::

      gempa.licensePath = @CONFIGDIR@/licenses



.. _input:


.. note::

   **input.\***
   *Parameters controlling the input of event information and*
   *reception of GNSS data.*


.. confval:: input.readFrom

   Type: *string*

   Read input data from this file.


.. confval:: input.directory

   Type: *string*

   Watch this directory for incoming input files.


.. confval:: input.watchEvents

   Default: ``close_write``

   Type: *string*

   Listen for specific inotify event\(s\).
   If ommitted, close_write events are listened for. Events:
   
   access \- file or directory contents were read,
   
   modify \- file or directory contents were written,
   
   attrib \- file or directory attributes changed,
   
   close_write \- file or directory closed, after being opened in writable mode,
   
   close_nowrite \- file or directory closed, after being opened in read\-only mode,
   
   close \- file or directory closed, regardless of read\/write mode,
   
   open \- file or directory opened,
   
   moved_to \- file or directory moved to watched directory,
   
   moved_from \- file or directory moved from watched directory,
   
   move \- file or directory moved to or from watched directory,
   
   create \- file or directory created within watched directory,
   
   delete \- file or directory deleted within watched directory,
   
   delete_self \- file or directory was deleted,
   
   unmount \- file system containing file or directory unmounted.


.. confval:: input.watchPattern

   Type: *string*

   Process any events whose filename matches the specified regular expression


.. confval:: input.leapSecondsFile

   Default: ``@DATADIR@/caps/plugins/ngl2caps/leapseconds.txt``

   Type: *string*

   Name of file with leap seconds.


.. _streams:


.. note::

   **streams.\***
   *Parameters controlling the processing of received data.*


.. confval:: streams.networkCode

   Default: ``NG``

   Type: *string*

   Network code to use.


.. confval:: streams.locationCode

   Type: *string*

   Location code to use.


.. confval:: streams.bandCode

   Default: ``U``

   Type: *string*

   Band code of streams to use. Streams will be formed as
   [band][sensor type][component].


.. _output:


.. note::

   **output.\***
   *Parameters controlling the output of received data.*


.. confval:: output.host

   Default: ``localhost``

   Type: *string*

   Data output host.


.. confval:: output.port

   Default: ``18003``

   Type: *int*

   Data output port.


.. confval:: output.bufferSize

   Default: ``1048576``

   Type: *uint*

   Size \(bytes\) of the packet buffer.


.. confval:: output.gain

   Default: ``1000000``

   Type: *float*

   Apply given gain to samples.




Command-Line Options
====================


.. _Generic:


Generic
-------

.. option:: -h, --help

   Show help message.

.. option:: -V, --version

   Show version information.

.. option:: --config-file arg

   Use alternative configuration file. When this option is
   used the loading of all stages is disabled. Only the
   given configuration file is parsed and used. To use
   another name for the configuration create a symbolic
   link of the application or copy it. Example:
   scautopick \-> scautopick2.


.. _Verbosity:


Verbosity
---------

.. option:: --verbosity arg

   Verbosity level [0..4]. 0:quiet, 1:error, 2:warning, 3:info,
   4:debug.

.. option:: -v, --v

   Increase verbosity level \(may be repeated, eg. \-vv\).

.. option:: -q, --quiet

   Quiet mode: no logging output.

.. option:: --print-component arg

   For each log entry print the component right after the
   log level. By default the component output is enabled
   for file output but disabled for console output.

.. option:: --component arg

   Limit the logging to a certain component. This option can
   be given more than once.

.. option:: -s, --syslog

   Use syslog logging backend. The output usually goes to
   \/var\/lib\/messages.

.. option:: -l, --lockfile arg

   Path to lock file.

.. option:: --console arg

   Send log output to stdout.

.. option:: --debug

   Execute in debug mode.
   Equivalent to \-\-verbosity\=4 \-\-console\=1 .

.. option:: --trace

   Execute in trace mode.
   Equivalent to \-\-verbosity\=4 \-\-console\=1 \-\-print\-component\=1
   \-\-print\-context\=1 .

.. option:: --log-file arg

   Use alternative log file.


.. _Input event:


Input event
-----------

.. option:: -d, --directory arg

   Watch this directory for incoming event input files. By
   default the current directory is watched.

.. option:: --watch-pattern arg

   Process any event file which file name matches the specified
   regular expression.


.. _Input waveforms:


Input waveforms
---------------

.. option:: -f, --file arg

   Read kenv data directly from this given file.

.. option:: -p, --read-from arg

   Read all kenv files from this directory path. Only
   considered if file is not given.

.. option:: -l, --leap-file arg

   Path to leap seconds file.


.. _Filter:


Filter
------


.. _Streams:


Streams
-------

.. option:: --network-code arg

   Network code to use.

.. option:: --location-code arg

   Sensor location code to use.

.. option:: --location-code arg

   Band code to use. Streams will be formed as [band code][xyz][ZNE].


.. _Output:


Output
------

.. option:: -H, --host arg

   Data output host.

.. option:: -P, --port arg

   Data output port.

.. option:: -g, --gain arg

   Gain value multiplied to the data for unit conversion to nm.

.. option:: -u, --gain-unit arg

   Gain unit to write to data file.


