LDM-McIDAS Unidata-Wisconsin Changes

Unidata-Wisconsin Changes

Impending Unidata-Wisconsin datastream changes

As announced in the summer of 2003, the contents of the Unidata-Wisconsin datastream are scheduled to be changed. The full set of changes to be made is detailed in Proposed Unidata-Wisconsin Datastream Changes.

The first portion of the changes are scheduled to be made at 18 UTC on Wednesday, January 7. These changes will include:

the addition of Northern and Southern hemisphere Wildfire ABBA products from SSEC/CIMSS
an increase in the resolution of GOES-East Water Vapor images from 8 km to 4 km
the addition of short wavelength IR (3.9 um) images from GOES-East and GOES-West
the addition of long wavelength IR images from GOES-East (13.3 um) and GOES-West (12.0 um)

A second change is scheduled to be made on Friday, January 9. This change will consist of:

an increase in broadcast frequency from once-per-hour to twice-per-hour for the GOES-East and West IR (3.9 um, 6.5/6.8 um, 10.7 um, and13.3/12.0 um) and VIS (0.65 um) images.

A third change will occur in late January:

the expansion in geographical coverage of the GOES-East images to increase South America coverage whenever southern GOES-East scans are available

Things to consider to accommodate the Unidata-Wisconsin changes

Modify your LDM request for the Unidata-Wisconsin imagery

The datastream changes will result in an increase in the number of products in the Unidata-Wisconsin datastream and corresponding increase in the datastream volume and disk space needed to store the images.

You can choose to ingest all or a portion of the broadcast images by tailoring the data request line for the UNIWISC (also known as MCIDAS) feed in your LDM's ldmd.conf file. To help you understand how you can tailor the request to suit your needs, we offer the following:

Each UNIWISC IDD product has two fields that can be used to tailor data requests:

Qn - hour portion indicator (e.g., Q0, Q1, Q2, Q3, Q4)/

Un - McIDAS product routing code (e.g., UB, U5, U9, etc.)

Possible values for the hour portion indicator are as follows:

Q0	image products or composites; the time associated with the product has hourly resolution
Q1	images whose time lies between H+00 and H+15
Q2	images whose time lies between H+15 and H+30
Q3	images whose time lies between H+30 and H+45
Q4	images whose time lies between H+45 and H+60

McIDAS product code values for UNIWISC imagery are as follows (new products are highlighted in bold print):

CA	CIMSS Cloud Top Pressure
CB	CIMSS Precipitable Water
CC	CIMSS Sea Surface Temperature
CD	CIMSS Lifted Index
CE	CIMSS CAPE
CF	CIMSS Ozone
CG	CIMSS Wildfire ABBA Nth Hemishere
CH	CIMSS Wildfire ABBA Sth Hemishere
U1	Antarctic composite
U3	Manually Digitized Radar
U5	GOES-W 10.7 um Infrared
U7	GOES-W 3.9 um Infrared
U8	GOES-W 12.0 um Infrared
U9	GOES-W 0.65 um Visible
UA	Educational floater I
UB	GOES-W 6.8 um Water Vapor
UC	Educational floater II
UD	GOES-E 3.9 um Infrared
UE	GOES-E 13.3 um CO2
UI	GOES-E 10.7 um Infrared
UR	Research floater
UV	GOES-E 0.65 um Visible
UW	GOES-E 6.5 um Water Vapor
UX	Global Mollweide Infrared Composite
UY	Global Mollweide Water Vapor Composite

Some example UNIWISC headers:

      pnga2area Q0 CA 1100 GOES-10_SND UNKBAND 14km 20040104 2300
      pnga2area Q1 U1 191 GOES-9_IMG UNKBAND 4km 20040104 2100
      pnga2area Q1 U3 205 GRAPHICS UNKBAND 5km 20040104 2359
      pnga2area Q1 U9 120 GOES-10_IMG 0.65um 4km 20040105 0000
      pnga2area Q1 U5 130 GOES-10_IMG 10.7um 4km 20040105 0000
      pnga2area Q1 UB 170 GOES-10_IMG 6.8um 8km 20040105 0000
      pnga2area Q0 CF 1150 GOES-10_SND UNKBAND 20km 20040104 2300
      pnga2area Q1 UW 210 GOES-12_IMG 6.5um 8km 20040105 0015
      pnga2area Q1 UV 140 GOES-12_IMG 0.65um 4km 20040105 0015
      pnga2area Q1 UI 150 GOES-12_IMG 10.7um 4km 20040105 0015
      pnga2area Q1 UA 160 GOES-12_IMG 10.7um 4km 20040105 0015
      pnga2area Q1 UC 60 GOES-12_IMG 10.7um 4km 20040105 0015
      pnga2area Q0 CB 1110 GOES-10_SND UNKBAND 14km 20040104 2300
      pnga2area Q0 CD 1130 GOES-10_SND UNKBAND 14km 20040104 2300
      pnga2area Q0 CE 1140 GOES-10_SND UNKBAND 14km 20040104 2300
      pnga2area Q1 UX 100 GOES-8_IMG 10.7um 1km 20040105 0000
      pnga2area Q1 UY 110 GOES-8_IMG 6.8um 1km 20040105 0000

Images in the datastream before the changes all have hour portion indicators of Q0 or Q1. When the image broadcast frequency increases to twice-per-hour, the new, half hourly images will all have an hour portion indicator of Q3. This coupled with the routing code values in the image headers is enough for to completely tailor what you ingest.

The following are four possible ldmd.conf request lines. In all cases, upstream.host.name is the name of the IDD host you receive UNIWISC images from.

If you want to continue to just receive the hourly images that have been in the datastream before the changes and ignore new wavelength bands (e.g., 3.9 um, 12.0 um, and 13.3 um), you should set your ldmd.conf request to:
```
request	UNIWISC	"Q[01] (C[^GH]|U[^78DE])"	upstream.host.name
```
If you want to receive the half-hourly images for the wavelength channels that have been in the broadcast (i.e., ignore the new wavelength bands but get the more frequent images), you should set your ldmd.conf request to:
```
request  UNIWISC "Q. (C[^GH]|U[^78DE])"       upstream.host.name
```
If you want to ingest hourly images for all wavelength bands (i.e., you do not want to get two of each kind of image per hour), you should set your ldmd.conf request to:
```
request	UNIWISC	"Q[01]"	upstream.host.name
```
If you want to request all UNIWISC images for all times, you should set your ldmd.conf request to:
```
request	UNIWISC	".*"	upstream.host.name
```

NOTE: anytime you modify your ldmd.conf file, you must stop and restart your LDM for the changes to become active. To do so, login as ldm, and then type:

      ldmadmin stop
      ldmadmin start

Modify the decoding of Unidata-Wisconsin imagery

If you decide that you want to continue to ingest just the hourly imagery that is in the datastream before the changes (either of the first two options above), and if you are currently decoding that imagery with no problems, you need do nothing.

If you decide that you want the new wavelength band images, then you will most likely need to adjust the pnga2area decode entries in your LDM's pqact.conf file. There are two possibilities:

you use the McIDAS routing table approach to decoding images.
In this case, you will need to add new entries to your McIDAS routing table. This is done by the user mcidas. Type the following:
```
          cd workdata
          ftp ftp.unidata.ucar.edu
          <user> anonymous
          <pass> your_full_email_address
          binary
          cd pub/mcidas/data
          get UPRTE.BAT
          quit
          batch.k UPRTE.BAT
        
```
NOTE: This assumes that you have a McIDAS REDIRECTion for the copy of ROUTE.SYS that is located in the directory in which you decode LDM-MCIDAS imagery.
You use pnga2area entries in your LDM's pqact.conf file that write files into a directory hierarchy that is required by GEMPAK or used by McIDAS. In this case, you need to:
- incorporate new pnga2area entries from ldm-mcidas-pqact.conf.all into your pqact.conf file;
- check your modified pqact.conf file for integrity by running ldmadmin pqactcheck correcting any errors that may have been found;
- tell the LDM pqact application to reread pqact.conf using ldmadmin pqactHUP.
NOTE: An FTP-able copy of ldm-mcidas-pqact.conf.all can be found in the pub/ldm-mcidas/pqact directory of anonymous FTP on the Unidata FTP server (ftp.unidata.ucar.edu). Login as the ldm user, and then type:
```
            cd etc
            ftp ftp.unidata.ucar.edu
            <user> anonymous
            <pass> your_full_email_address
            binary
            cd pub/ldm-mcidas/pqact
            get ldm-mcidas-pqact.conf.all
            quit
          
```

Modify your procedures for scouring data

If you are only using the McIDAS routing table approach for decoding UNIWISC imagery, you should not need alter your image scouring procedures.

If you have been using the directory hierarchy decoding method offered by the pnga2area, you should not need to alter your image scouring procedures (assuming that your data scouring is currently working correctly).

If you switch from the McIDAS routing table to the directory hierarchy approach for decoding UNIWISC imagery (and this is what we recommend), you will need to establish new procedures for scouring imagery from your system. We have made an FTPable C-shell script that you can use to scour UNIWISC images ingested into a directory hierarchy. FTP this file to a directory in the PATH defined for your ldm user. Type the following:

      cd util             <- for example
      ftp ftp.unidata.ucar.edu
      <user> anonymous
      <pass> your_full_email_address
      cd pub/ldm/scour
      binary
      get prune_images.csh
      quit
      chmod +x prune_images.csh

You will need to edit prune_images.csh to set several items to match your LDM setup:

PATH - insure that the directory containing prune_images.csh is in PATH
KEEP - set the number of images you want to keep (same for all bands)
areadir - set to the directory under which all images/image subdirectories are found

After prune_images.csh is properly configured AND set to be executable, you will need to add a crontab entry to run the script at least once-per-hour. Here is an example entry that is used here at the Unidata Program Center:

15 * * * * util/prune_images.csh > /dev/null 2>&1

NOTE: this entry assumes that prune_images.csh was copied to the ~ldm/util directory.

Modify McIDAS ADDE serving of the data you ingest (McIDAS users only)

If you run McIDAS, you will want to update your ADDE setup to be able to serve the new images you ingest. How you do this depends on how you are decoding the UNIWISC images. In both cases, we have provided new ADDE data serving template BATCH files that you should FTP and use to update your ADDE setup.

You use the McIDAS routing table approach to decoding images. In this case, download the file DSSERVEOLD.BAT to the ~mcidas/data directory. Login as the mcidas user, then type:

          cd ~mcidas/data
          ftp ftp.unidata.ucar.edu
          <user> anonymous
          <pass> your_full_email_address
          cd pub/mcidas/data
          binary
          get DSSERVEOLD.BAT
          quit

You use pnga2area entries in your LDM's pqact.conf file that write files in to a directory hierarchy that is needed by GEMPAK or used by McIDAS. In this case, download the file DSSERVE.BAT to the ~mcidas/data directory.Login as the mcidas user, then type:
```
          cd ~mcidas/data
          ftp ftp.unidata.ucar.edu
          <user> anonymous
          <pass> your_full_email_address
          cd pub/mcidas/data
          binary
          get DSSERVE.BAT
          quit
        
```

In both cases, use the file you just downloaded to create a new LSSERVE.BAT adjusting entries to match how you are decoding the UNIWISC imagery:

      cp LSSERVE.BAT LSSERVE.BAT.bak
      cp DSSERVEOLD.BAT LSSERVE.BAT
        - or -
      cp DSSERVE.BAT LSSERVE.BAT

Then edit LSSERVE.BAT and set entries to match your decoding setup>.

IMPORTANT: make sure that you include any changes you made in your previous version of LSSERVE.BAT into the new copy!

After making the changes to LSSERVE.BAT, make the changes active for McIDAS ADDE serving:

      cd ~mcidas/workdata
      batch.k LSSERVE.BAT

After running batch.k LSSERVE.BAT, make sure that you can access the new imagery:

      dsinfo.k IMAGE CIMSS
      dsinfo.k IMAGE RTIMAGES

      imglist.k RTIMAGES/GE-39
      imglist.k RTIMAGES/GE-CO2
      imglist.k RTIMAGES/GW-39
      imglist.k RTIMAGES/GW-12

Send questions/comments to <support-ldm-mcidas AT unidata.ucar.edu>.