Programs of the LDM System

• The LDM Program ldmd
• The Utility pqact
• Data-Product Ingesters
• Data-Product Decoders
• The ldmadmin Script
• Ancillary Utilities of the LDM

The LDM Program ldmd

An LDM system's main program is the binary executable ldmd. This program

• Is typically started by the ldmadmin script.
• Configures itself according to the LDM configuration-file, ldmd.conf.
• Starts-up the LDM ingesters.
• Forks itself to create downstream LDMs.
• Starts-up the pqact processes.
• Validates FEEDME requests by downstream LDMs and forks itself to create corresponding upstream LDM processes.
• Validates HIYA connections by upstream LDMs and forks itself to create corresponding downstream LDMprocesses. This is an uncommon operation.

The Utility pqact

The pqact utility reads data-products from the product-queue and disposes of them according to user-configurable instructions. One or more instances of this program are, typically, started by the top-level ldmd process as the result of EXEC entries in the LDM configuration-file, ldmd.conf. Instructions for the disposition of data-products are contained in a pqact configuration-file.

Certain instructions in the pqact configuration-file cause the pqact utility to spawn child processs to aid in the disposition of data-products. These instructions are the pqact PIPE action and the pqact EXEC action.

Typically, pqact processes are terminated by reception of a SIGINT or SIGTERM from the top-level ldmd process.

When a pqact process terminates, it saves the insertion-time of the last, successfully-processed data-product in a file. The pathname of the file is that of the pqact configuration-file with ".state" appended. This allows a subsequent pqact process that uses the same configuration-file to start processing where the previous process stopped. It also means only one pqact process should use any configuration-file.

Data-Product Ingesters

An LDM data-product ingester is a program that creates data-products and inserts them into the LDM product-queue using the pq(3) API. Such a program has the following characteristics:

• It should send a SIGCONT signal to its process group whenever it inserts a data-product into the product-queue.
• In general, it should be started by the top-level ldmd process as the result of an EXEC entry in the LDM configuration-file, ldmd.conf, so that the the LDM process group will immediately act upon the new data-product. Failure to be started in this way will result in larger than necessary data-product latency.
• In general, it should never self-terminate; instead, it should terminate upon reception of a SIGINT or SIGTERM, which will be sent to it from the top-level ldmd process.
• It should use the ulog(3) API for logging messages, which is provided by the ulog module in the LDM library.

It is, ultimately, irrelevant how ingesters obtain the data from which they create data-products because such information is lost once the data-product has been created and inserted into the product-queue. Data sources that have been used include the following: serial ports, files, and pseudo-random number generators.

The LDM package comes with the following ingesters. While some are useful in their own right, their primary purpose is to provide examples of how to write ingesters.

pqing

Reads meteorological data in WMO or AFOS format from a file or serial port. This is a "classical" data-product ingester.

pqinsert

Inserts files as data-products. This is an example of an ingester that not only runs outside the LDM process group but also self-terminates.

Data-Product Decoders

An LDM data-product decoder is a program that reads data-product data from standard input and disposes of the data in some fashion -- often by writing the data into a file that is specific to an analysis package (e.g., McIDAS or GEMPAK). Such programs have the following characteristics:

• They are typically started by a pqact process as the result of PIPE actions in the process's pqact configuration-file.
• Their standard input is a read-only pipe with the parent pqact process but their standard output and standard error do not exist (but see below).
• They self-terminate if no data has been received within a certain period of time (e.g., 10 minutes) or upon reception of a SIGINT or SIGTERM from the pqact parent process.
• Due to a limitation on the number of output-files that the pqact program can have open at one time, decoders should
• Continue reading until the end-of-file is encountered and then exit; otherwise, the decoder process might become a zombie (alias defunct) process.
• Be written so that they can "start-up" in the middle of decoding something that spans multiple data-products (for example a single "sweep" of a NEXRAD radar actually comprises four data-products per parameter). The reason for this can be found in the documentation for the pqact configuration-file.

Because a decoder doesn't have a useful standard output or standard error, a decoder that depends them (to report errors, for example) will not function correctly. An easy workaround exists, however: wrap the decoder in a simple shell script that redirects such output into files. The following shell script is an example:

      #!/bin/sh
      exec 1><stdout>
      exec 2><stderr>
      <decoder> "$@"
    

where:

<stdout>

Is the pathname of the file to contain standard output

<stderr>

Is the pathname of the file to contain standard error

<decoder>

Is the pathname of the actual decoder

Decoders that are, themselves, shell scripts, may use the LDM package's ulogger utility to log error messages. For example, the following, simple shell script concatenates its standard input to a file -- reproducing the effect of the pqact(1) FILE action:

      progname=`basename $0`
      if test $# -ne 1; then
        ulogger -p FATAL "$progname: Usage: $progname <file>"
      else
        file=$1
        if ! cat >> "$file"; then
          ulogger -p FATAL "$progname: Couldn't append to file \"$file\""
        fi
      fi
    

By default, the ulogger utility logs at the NOTE level. It also uses the same logging mechanism as the other LDM programs; consequently, it will log to the LDM log file by default unless overridden by the "-l file" option.

The ldmadmin Script

The ldmadmin utility is a Perl-5 script intended for high-level administration of an LDM system. The script acts as a facade for more primitive utilities -- coordinating their use for the sake of convenience. The things you can do with the ldmadmin script include the following:

Ancillary Utilities of the LDM

Utilities that are included in the LDM package but are less important than the separately-documented ones above include the following: