Unidata-Wisconsin Changes

Impending Unidata-Wisconsin datastream changes

As announced in the summer of 2003, the contents of the Unidata-Wisconsin datastream are scheduled to be changed. The full set of changes to be made is detailed in Proposed Unidata-Wisconsin Datastream Changes.

The first portion of the changes are scheduled to be made at 18 UTC on Wednesday, January 7. These changes will include:

A second change is scheduled to be made on Friday, January 9. This change will consist of:

A third change will occur in late January:

Things to consider to accommodate the Unidata-Wisconsin changes

Modify your LDM request for the Unidata-Wisconsin imagery

The datastream changes will result in an increase in the number of products in the Unidata-Wisconsin datastream and corresponding increase in the datastream volume and disk space needed to store the images.

You can choose to ingest all or a portion of the broadcast images by tailoring the data request line for the UNIWISC (also known as MCIDAS) feed in your LDM's ldmd.conf file. To help you understand how you can tailor the request to suit your needs, we offer the following:

Each UNIWISC IDD product has two fields that can be used to tailor data requests:

Qn - hour portion indicator (e.g., Q0, Q1, Q2, Q3, Q4)/

Un - McIDAS product routing code (e.g., UB, U5, U9, etc.)

Possible values for the hour portion indicator are as follows:

Q0 image products or composites; the time associated with the product has hourly resolution
Q1 images whose time lies between H+00 and H+15
Q2 images whose time lies between H+15 and H+30
Q3 images whose time lies between H+30 and H+45
Q4 images whose time lies between H+45 and H+60

McIDAS product code values for UNIWISC imagery are as follows (new products are highlighted in bold print):

CA CIMSS Cloud Top Pressure
CB CIMSS Precipitable Water
CC CIMSS Sea Surface Temperature
CD CIMSS Lifted Index
CE CIMSS CAPE
CF CIMSS Ozone
CG CIMSS Wildfire ABBA Nth Hemishere
CH CIMSS Wildfire ABBA Sth Hemishere
U1 Antarctic composite
U3 Manually Digitized Radar
U5 GOES-W 10.7 um Infrared
U7 GOES-W 3.9 um Infrared
U8 GOES-W 12.0 um Infrared
U9 GOES-W 0.65 um Visible
UA Educational floater I
UB GOES-W 6.8 um Water Vapor
UC Educational floater II
UD GOES-E 3.9 um Infrared
UE GOES-E 13.3 um CO2
UI GOES-E 10.7 um Infrared
UR Research floater
UV GOES-E 0.65 um Visible
UW GOES-E 6.5 um Water Vapor
UX Global Mollweide Infrared Composite
UY Global Mollweide Water Vapor Composite

Some example UNIWISC headers:

      pnga2area Q0 CA 1100 GOES-10_SND UNKBAND 14km 20040104 2300
      pnga2area Q1 U1 191 GOES-9_IMG UNKBAND 4km 20040104 2100
      pnga2area Q1 U3 205 GRAPHICS UNKBAND 5km 20040104 2359
      pnga2area Q1 U9 120 GOES-10_IMG 0.65um 4km 20040105 0000
      pnga2area Q1 U5 130 GOES-10_IMG 10.7um 4km 20040105 0000
      pnga2area Q1 UB 170 GOES-10_IMG 6.8um 8km 20040105 0000
      pnga2area Q0 CF 1150 GOES-10_SND UNKBAND 20km 20040104 2300
      pnga2area Q1 UW 210 GOES-12_IMG 6.5um 8km 20040105 0015
      pnga2area Q1 UV 140 GOES-12_IMG 0.65um 4km 20040105 0015
      pnga2area Q1 UI 150 GOES-12_IMG 10.7um 4km 20040105 0015
      pnga2area Q1 UA 160 GOES-12_IMG 10.7um 4km 20040105 0015
      pnga2area Q1 UC 60 GOES-12_IMG 10.7um 4km 20040105 0015
      pnga2area Q0 CB 1110 GOES-10_SND UNKBAND 14km 20040104 2300
      pnga2area Q0 CD 1130 GOES-10_SND UNKBAND 14km 20040104 2300
      pnga2area Q0 CE 1140 GOES-10_SND UNKBAND 14km 20040104 2300
      pnga2area Q1 UX 100 GOES-8_IMG 10.7um 1km 20040105 0000
      pnga2area Q1 UY 110 GOES-8_IMG 6.8um 1km 20040105 0000
    

Images in the datastream before the changes all have hour portion indicators of Q0 or Q1. When the image broadcast frequency increases to twice-per-hour, the new, half hourly images will all have an hour portion indicator of Q3. This coupled with the routing code values in the image headers is enough for to completely tailor what you ingest.

The following are four possible ldmd.conf request lines. In all cases, upstream.host.name is the name of the IDD host you receive UNIWISC images from.

NOTE: anytime you modify your ldmd.conf file, you must stop and restart your LDM for the changes to become active. To do so, login as ldm, and then type:

      ldmadmin stop
      ldmadmin start
    

Modify the decoding of Unidata-Wisconsin imagery

If you decide that you want to continue to ingest just the hourly imagery that is in the datastream before the changes (either of the first two options above), and if you are currently decoding that imagery with no problems, you need do nothing.

If you decide that you want the new wavelength band images, then you will most likely need to adjust the pnga2area decode entries in your LDM's pqact.conf file. There are two possibilities:

  1. you use the McIDAS routing table approach to decoding images.

    In this case, you will need to add new entries to your McIDAS routing table. This is done by the user mcidas. Type the following:

              cd workdata
              ftp ftp.unidata.ucar.edu
              <user> anonymous
              <pass> your_full_email_address
              binary
              cd pub/mcidas/data
              get UPRTE.BAT
              quit
              batch.k UPRTE.BAT
            

    NOTE: This assumes that you have a McIDAS REDIRECTion for the copy of ROUTE.SYS that is located in the directory in which you decode LDM-MCIDAS imagery.

  2. You use pnga2area entries in your LDM's pqact.conf file that write files into a directory hierarchy that is required by GEMPAK or used by McIDAS. In this case, you need to:

    NOTE: An FTP-able copy of ldm-mcidas-pqact.conf.all can be found in the pub/ldm-mcidas/pqact directory of anonymous FTP on the Unidata FTP server (ftp.unidata.ucar.edu). Login as the ldm user, and then type:

                cd etc
                ftp ftp.unidata.ucar.edu
                <user> anonymous
                <pass> your_full_email_address
                binary
                cd pub/ldm-mcidas/pqact
                get ldm-mcidas-pqact.conf.all
                quit
              

Modify your procedures for scouring data

If you are only using the McIDAS routing table approach for decoding UNIWISC imagery, you should not need alter your image scouring procedures.

If you have been using the directory hierarchy decoding method offered by the pnga2area, you should not need to alter your image scouring procedures (assuming that your data scouring is currently working correctly).

If you switch from the McIDAS routing table to the directory hierarchy approach for decoding UNIWISC imagery (and this is what we recommend), you will need to establish new procedures for scouring imagery from your system. We have made an FTPable C-shell script that you can use to scour UNIWISC images ingested into a directory hierarchy. FTP this file to a directory in the PATH defined for your ldm user. Type the following:

      cd util             <- for example
      ftp ftp.unidata.ucar.edu
      <user> anonymous
      <pass> your_full_email_address
      cd pub/ldm/scour
      binary
      get prune_images.csh
      quit
      chmod +x prune_images.csh
    

You will need to edit prune_images.csh to set several items to match your LDM setup:

After prune_images.csh is properly configured AND set to be executable, you will need to add a crontab entry to run the script at least once-per-hour. Here is an example entry that is used here at the Unidata Program Center:

15 * * * * util/prune_images.csh > /dev/null 2>&1

NOTE: this entry assumes that prune_images.csh was copied to the ~ldm/util directory.

Modify McIDAS ADDE serving of the data you ingest (McIDAS users only)

If you run McIDAS, you will want to update your ADDE setup to be able to serve the new images you ingest. How you do this depends on how you are decoding the UNIWISC images. In both cases, we have provided new ADDE data serving template BATCH files that you should FTP and use to update your ADDE setup.

  1. You use the McIDAS routing table approach to decoding images. In this case, download the file DSSERVEOLD.BAT to the ~mcidas/data directory. Login as the mcidas user, then type:
              cd ~mcidas/data
              ftp ftp.unidata.ucar.edu
              <user> anonymous
              <pass> your_full_email_address
              cd pub/mcidas/data
              binary
              get DSSERVEOLD.BAT
              quit
            
  2. You use pnga2area entries in your LDM's pqact.conf file that write files in to a directory hierarchy that is needed by GEMPAK or used by McIDAS. In this case, download the file DSSERVE.BAT to the ~mcidas/data directory.Login as the mcidas user, then type:
              cd ~mcidas/data
              ftp ftp.unidata.ucar.edu
              <user> anonymous
              <pass> your_full_email_address
              cd pub/mcidas/data
              binary
              get DSSERVE.BAT
              quit
            

In both cases, use the file you just downloaded to create a new LSSERVE.BAT adjusting entries to match how you are decoding the UNIWISC imagery:

      cp LSSERVE.BAT LSSERVE.BAT.bak
      cp DSSERVEOLD.BAT LSSERVE.BAT
        - or -
      cp DSSERVE.BAT LSSERVE.BAT
    

Then edit LSSERVE.BAT and set entries to match your decoding setup>.

IMPORTANT: make sure that you include any changes you made in your previous version of LSSERVE.BAT into the new copy!

After making the changes to LSSERVE.BAT, make the changes active for McIDAS ADDE serving:

      cd ~mcidas/workdata
      batch.k LSSERVE.BAT
    

After running batch.k LSSERVE.BAT, make sure that you can access the new imagery:

      dsinfo.k IMAGE CIMSS
      dsinfo.k IMAGE RTIMAGES

      imglist.k RTIMAGES/GE-39
      imglist.k RTIMAGES/GE-CO2
      imglist.k RTIMAGES/GW-39
      imglist.k RTIMAGES/GW-12
    

Send questions/comments to <support-ldm-mcidas AT unidata.ucar.edu>.