SCOUR

scour - utility for deleting old data files

scour[-d] [-e exclFile] [-l logfile] [-v|-x] [config-file]

scour is part of the Unidata Local Data Management system (LDM).

scour deletes old data files from a set of data directories. For each directory specified in a configuration file, scour deletes files in that directory and all of its subdirectories that are older than a specified number of days and that have a filename matching a specified pattern. If no new data files with filenames matching the specified pattern have been written in the directory or its subdirectories since the last time scour was invoked on that directory with that pattern, no files are deleted. This insures that the specified data directories will retain a set of data files, even if there is a failure in the system that generates or captures the data.

The directories, retention time in days, and an optional shell filename pattern for files to delete appear in a configuration file. scour is designed to be invoked periodically from a crontab(1) file to keep the directories of data captured by the Unidata LDM system from filling up. It can also be invoked manually as needed.

-d

Indicates that empty directories are to be deleted if they are or become empty.

-e exclFile

Pathname of file listing the directories to be excluded from scouring. Default is given by the command "regutil /scour/exclude-path".

-l logfile

Logs messages to logdest. One of ’’ (system logging daemon), ’-’ (standard error stream), or file logdest. The default is the standard error stream.

−v

Verbose flag. If present, the output will note when (in UTC time) scour starts and exits, and a message for each directory scoured will tell how many disk blocks were recovered and what the retention period for that directory was.

−x

Turn on debugging. This will produce extra debugging output in the log file.

[config-file]

The path name of the configuration file containing the table of data directories to be scoured and data retention times in days. If no configuration file is named, the default configuration file printed by the command "regutil /scour/config-path", is used.

The command

scour -v -l ~ldm/etc/scour.conf

will read the specified configuration file and delete old files as specified from the directories named in the configuration file, logging its messages to the same log file used by other LDM processes. Here is an example of a configuration file:

# Directory                     Days-old        Optional-filename-pattern

~ldm/var/data/forecasts         3               *.gdbm
~ldm/var/data/warnings          3               *.gdbm
~ldm/var/data/decoded           2               *.nc
~ldm/var/data/upperair          2               *.wmo
~ldm/var/data/GRIB              2
~ldm/var/data/GRIB              1               *eta.nc
# this contains sao, rad, buoy, syn, and front subdirectories
~ldm/var/data/surface           2
~ldm/var/data/gempak            2

Directory pathname that are not absolute are relative to the directory printed by the command "regutil /pqact/datadir-path".

Comment lines must begin with the ‘#’ character. Blank lines are ignored. Directory names may be absolute pathnames, or may use the ‘~user’ notation of csh(1). If a filename pattern is not specified, ‘*’ matching all files that don’t begin with the character ‘.’ is assumed.

The formats for the "Days-old" field are

D
-hh
-hh:mm
D-hh
D-hh:mm

where:

D

Is the number of days as one or more digits.

hh

Is the number of hours.

mm

Is the number of minutes.

In the example above, the directory name ‘~ldm/var/data/GRIB’ is specified twice, once without a pattern and once with a pattern. The intention is that all files in ~ldm/var/data/GRIB older than 2 days should be removed, but that in addition, files with names matching the pattern ‘*eta.nc’ that are older than 1 day should also be removed.

.scour*

Zero-size files created at the top of each scoured directory hierarchy to maintain the time the directory was last scoured with particular patterns.

ldmd(1)

In case a directory contains no new files since the last time scour was run on that directory, a message wil be logged since this may indicate a data ingest failure.