PQACT

pqact - program to process products in a Unidata LDM product queue

pqact [-v] [-x] [-l logdest] [-d datadir] [-q pqfname] [-f feedtype] [-p pattern] [-i interval] [-t timeout] [-o time] [conf_file]

This program processes ldm data products from a local product queue (see pq(3)). Pattern matching is used to specify what actions are performed on each product.

A product’s feed type is used for coarse product discrimination, and pattern matching on product identifiers determines which of a set of general actions are applied to the product. A product identifier may match multiple patterns, and hence multiple actions may be applied to it. A pattern may be used to mark parts of a product identifier for use in the associated action. The actions supported include filing products for later scheduled decoding or for direct use, decoding products as they arrive, adding products to a database for quick retrieval by applications, or executing arbitrary programs on the arrival of selected products, e.g., mailing local severe weather warnings to users.

pqact uses the configuration file conf_file, (default $(regutil /pqact/config-path) ) to set up the table of patterns and associated actions for products. This file is human-readable and editable. It contains a list of pattern-action entries, where a pattern is a (feed type, regular expression) pair. The available actions are described in the ‘‘USAGE’’ section, below.

Although it can be run manually, this program is typically started as subprocess of the ldm server, ldmd(1), at system startup, from an exec line in the configuration file. This ensures that pqact is in the same process group as other programs sharing the queue.

When a pqact process terminates, it writes the insertion-time of the last successfully-processed data-product into a file. The pathname of the file is that of the configuration-file with ".state" appended. This allows a subsequent pqact process that executes the same configuration-file to start processing where the previous process stopped. It also means that a configuration-file should be associated with at most one pqact process. If you want to use the same configuration-file for multiple pqact processes, then create symbolic links to the configuration-file and execute each pqact with a unique link.

-v

Verbose logging. Informative messages (level LOG_INFO) are logged, including a line for each product read out of the product queue and a line for each action invoked on a product. By default, only messages of severity LOG_NOTICE and greater are logged.

-x

Debug logging. Debugging messages (level LOG_DEBUG) are logged.

−l logdest

Log to logdest. One of ’’ (system logging daemon), ’-’ (standard error stream), or file logdest. Default is the standard error stream if the process has a controlling terminal (i.e., the process isn’t a daemon); otherwise, either the LDM log file or the system logging daemon (execute this program with just the option ’-?’ to determine which).

−d datadir

The directory relative to which file names are specified in the configuration file. The default is $(regutil /pqact/datadir-path).

-q pqfname

The name of the product queue file. The default is $(regutil /queue/path) This configuration default can be overridden by setting the environment variable LDMPQFNAME. You can specify a null product queue /dev/null to just check the syntax of a configuration file for pqact, as in

pqact -v -q /dev/null pqact.conf

−f feedtype

Reads from the product queue only products that have a feedtype that is a member of the feedtype set. The default is ‘ANY’, specifying all feedtypes. See the ‘‘Usage’’ section below for a list of basic feedtypes and the use of feedtype expressions to specify combinations of basic feedtypes.

−p pattern

Reads from the product queue only products whose identifier matches pattern. The default is ‘.*’, specifying all products. A pattern that is longer than two characters and that has a ".*" prefix is deemed pathological and causes the program to log a warning message.

−i time

Polling interval, in seconds. Check for new products in the product queue every time seconds. When pqact is run in the same process group as the programs that insert products into the product queue, a signal informs pqact and all other interested processes in the process group whenever a new product is available, so polling is not necessary in this case. The default interval is 15 seconds. If time is 0, then the process will exit when it hits the end of the product-queue.

−o time

Offset time, in seconds. Begin scan with products created time seconds earlier than the current time. The default is to begin just after the time stamp of the oldest product in the queue at startup. This option might be used when manually processing data from an old queue.

−t timeout

Timeout, in seconds, for piping a buffer’s worth of data to a decoder before declaring the decoder unresponsive, closing the connection, and moving on to the next matching action.

conf_file

Configuration file. This is the pattern-action file that specifies what to do with each product whose feed type and product identifier match a specified pattern. The format of this file is described in the ‘‘Usage’’ section, below. The default is $(regutil /pqact/config-path).

SIGCONT

Wake up if sleeping, because there is a new product in the queue.

SIGINT

Immediate termination.

SIGHUP

Rereads configuration-file.

SIGTERM

Graceful termination after finishing actions on current product.

SIGUSR1

Refreshes logging if configure(1)-script executed without "--with-ulog" option.

SIGUSR2

Cyclically increment the verbosity of the program. Assumming the program was started without the -v or -x switches, the first SIGUSR2 will make it verbose and LOG_INFO priority messages will appear. The second will turn on the LOG_DEBUG priority messages as well. A third will take it back to the normal state.

Each pattern-action rule in the configuration file begins with a non-white-space character and consists of three fields, separated by one or more tab characters: a feedtype field, a pattern field, and an action field that may include one or more action arguments. An initial ‘#’ character indicates a comment line. An initial sequence of one or more blank or tab characters indicates a new field of the previous pattern-action line. This allows multiline pattern-action rules.

Feedtypes

Each data product has an associated feedtype and product identifier. The feedtype names the data stream from which the product originated and the product identifier distinguishes different products within a data stream. The feedtype field provides coarse discrimination among various data sources. It helps determine how the product identifiers should be interpreted, since each data source may use a different scheme for product identifiers. The feedtypes currently recognized are:

NONE

matches no feed

PPS

Public Products Service

DDS

Domestic Data Service

DDPLUS

Domestic Data Plus, a union of DDS and PPS.

IDS

International Product Service

HRS

High-Resolution Data Service, consisting of forecast model outputs from the U.S. National Meteorological Center and the European Center for Medium-Range Weather Forecasts.

WMO

any of the above WMO format products, so named because they use World Meteorological Organization standard headers for product identifiers.

MCIDAS

Unidata/Wisconsin broadcast

PCWS

Forecast Systems Lab PC DARE feed

FSL

Other FSL feeds

NMC

NMC feeds

NLDN

National Lighting Data Network

EXP

experimental data feed

ANY

wild card, matches any product feedtype

The feedtype field may also contain a feedtype expression specifying a combination of the above named feedtypes, using the union operator ‘|’, the difference operator ‘-’, and parentheses for grouping. For example, to specify products from either the IDS feed or the HRS feed, use

IDS|HRS

To specify any products from the WMO feeds except HRS products, use

WMO-HRS

Product Identifiers

A product identifier is typically extracted from characters at the beginning of a product in the data stream. When a product is first processed or created, the feedtype and product identifier are assigned. For each product it reads out of the product queue, pqact attempts to match the feedtype and identifier against entries in the configuration file. Whether a product is captured or processed further is determined by whether its feedtype and identifier match any of the pattern-action rules specified in the configuration file.

The identifier of a product is successively matched against each pattern in the configuration file that has a matching feedtype; whenever a match is found with such a pattern, the associated action is invoked.

Patterns

A pattern is an extended regular expression (see regex(5)) that matches product identifiers in the data stream. Since the end of the pattern field is delimited by a tab or newline character, the pattern cannot contain a literal tab or newline character but may contain blanks. Patterns longer than two characters and that start with a ".*" prefix are deemed pathological and cause the program to log warning messages.

Actions

The action field consists of an action command followed by a tab and one or more arguments separated by blanks or tabs. Valid actions in the LDM are:

FILE

Write a product into a file.

STDIOFILE

Buffered write of a product into a file.

PIPE

Run another process with the product as input.

EXEC

Run another process (no connection).

DBFILE

Store a product in a database.

ALLOW

Obsolete action to specify which hosts may request which products. This is now handled by the ldmd.access configuration file of ldmd(1) instead.

ACCEPT

Obsolete action to specify which hosts may feed which products. This is now handled by the ldmd.access configuration file of ldmd(1) instead.

For more information on these actions and their arguments, see the LDM Site Manager’s Guide.

Pattern-Action Rules

A product may match multiple pattern-action lines. In that case the specified actions are invoked for that product in the same order they appear in the configuration file.

Parts of a matching product identifier and components of the date of the data or the current date and time may be substituted into action arguments before the action is invoked. For more information about the notations for this, consult the section on the pqact(1) configuration-file in the ‘‘LDM Basics’’ webpages for your particular release at http://www.unidata.ucar.edu/software/ldm.

The following pattern-action rule will append all products from the DDS data stream that have product identifiers (in this case WMO headers) beginning with the characters ‘SAUS’ to hourly files named saus_mmdd.wmo (extracting the appropriate month and day fields from the product identifiers):

DDS

$(regutil /pqact/config-path)

Default configuration-file.

$(regutil /pqact/config-path).state

Default persistent-state file. Used to hold the insertion-time of the last, successfully-processed data-product.

runpg(1), ldmd(1), ulog(3), pq(3), syslogd(8), WWW URL http://www.unidata.ucar.edu/software/ldm/.

If you have problems with this program, then you should first examine the LDM email archive for similar problems and how they were solved. The email archive is available via the following World Wide Web URL:

http://www.unidata.ucar.edu/software/ldm

If this does not suffice and your site is a member of the Unidata program, then send an inquiry via email -- together will all relevant information -- to

support@unidata.ucar.edu