Table of contents Previous: Displaying non-AREA Imagery Next: Unidata McIDAS-X Useful Information

15.7 Importing Foreign Datasets Into McIDAS

Textual Point Source Data

Observed point source data in ASCII text format may be imported into a McIDAS-compatible data file (MDXXnnnn) using the routine TXT2MD.

MD files can have a variety of shapes. A schematic depiction of a general MD file is shown in:

The exact shape of an MD file is determined by its schema. In order to specify which schema to use when creating an MD file, on must first:

Detailed information on MD files and schemas may be found in MD Files - Basic Concepts.

To illustrate the process of importing textual observation data into McIDAS, we detail the process for a dataset we grabbed from the CODIAC data management system that is maintained by the Joint Office for Field Support (JOSS) program of UCAR.

First, we need to grab the dataset. We do this from the Unix shell prompt: 

     mkdir /data/workshop/comet
     cd /data/workshop/comet

     ftp ftp.unidata.ucar.edu
       <user> umcidas
       <pass> uni_mci!
       cd data/textdata
       binary
       get STORM.TXT
       quit

After grabbing the dataset, we need to make sure that it is accessible from our McIDAS session. We do this by adding a file REDIRECTion to the file, STORM.TXT, which is located in the /var/data/mcidas directory:

Next, we make sure that we entered the REDIRECT command correctly: If McIDAS can find the file using DMAP, then we should take a quick look at it to see the form of the contents: You can see by the SEE listing that the first two lines are filled with meta data. We added these lines after grabbing the CODIAC dataset. The purpose of the two lines is to identify the contents of the file through a mapping of parameter Key name and corresponding column, and by declaring the units, if not American meteorological convention, for the paramters in the file: 

SEE STORM.TXT NLINES=10
TYPE MOD DAY TIME ID  LAT     LON    ZS   PSL     PRE       T       TD       SPD DIR    PCP
UNIT=X X X X X X X X X X C C X X X
0    0   95080 00 NEW 30.03   90.03    3. 1015.90 1016.00   21.75   17.25    3.60  15 0.00  -9999
0    0   95080 00 LCH 30.12   93.22    5. 1010.90 1011.00   22.25   19.45    8.20  15 0.00  -9999
0    0   95080 00 LFT 30.20   91.98   11. 1013.20 1014.20   22.85   18.35    5.10  15 0.00  -9999
0    0   95080 00 AUS 30.28   97.70  179.  987.60 1007.70   22.85   18.35    6.70  13
SEE: Done
The correspondence in names and column entries is such that the first column refers to a Key named TYPE, the second MOD, the third DAY, etc. The ordering of the parameters is not important as long as each line in the file has the same correspondence with the Key names.

The second line, the line that begins UNIT=, is optional. If included it defines the units of the parameters. This line is needed when the units of the input data is different than the units that are to be stored in the MD file. One word of warning is in order: McIDAS implicitly expects data to be stored in American meteorological convention units.

From the listing, you can see that the parameters identified as T and TD (temperature and dew point, respectively), are given in degrees Centigrade. Since temperatures will be stored in the output MD file in K, we need to alert TXT2MD to do the necessary conversion.

The naming of the keys is not random and is not free-form. The Key names listed must match those that will be stored in the MD file. Furthermore, MD file Key names must be no more than four characters in lenght.

To understand how we chose the key names listed, we must look at the file schema that will be used to create the file. In McIDAS file schemas are stored in so-called LW files. LW files are files with fixed, 80-character records that are not separated with line feeds. The naming convention for McIDAS schema files is DCxxxx. Here, xxxx refers to a name that is four or less characters in length.

Since the data set contained in STORM.TXT is surface data taken from a mesonet, we will use the McIDAS surface MD file form to store our data. The current schema for McIDAS surface data (such as that created by XCD decoders is named ISFC. Its template is contained in the file DCISFC which is distributed in the /home/mcidas/data directory.

Let's take a look at the ISFC template file: 

" ISFC (JSP) 0289 SCHEMA -- GENERAL SURFACE SCHEMA
"        NAME VSN DATE  ID "TEXTID
"        ---- --- ----- -- -------
  SCHEMA ISFC  7 99085 0 "SURFACE HOURLY OBSERVATIONS

  ROWS 72                "24 HOURS, 3 REPORTS/HOUR

  TYPE                   "0=HOURLY,1=SPECIAL 1,2=SPECIAL 2
  DAY        CYD         "YEAR AND JULIAN DAY (CCYYDDD)
  TIME       HMS         "NOMINAL TIME (HH0000)
  NREC                   "APPROX. # OF RECORDS WRITTEN IN THIS ROW

  COLUMNS 3500           "MAXIMUM OF 3500 REPORTING STATIONS

  ID         CHAR        "STATION CALL LETTERS
  LAT  4     DEG         "RANGE (-90 => +90) POSITIVE NORTH
  LON  4     DEG         "RANGE (-180 => +180) POSITIVE WEST
  ZS         M           "ELEVATION
  ST         CHAR        "STATE ID
  CO         CHAR        "COUNTRY ID

  DATA

  MOD                    "USER MODIFICATION FLAG (1=MODIFIED)
  HMS        HMS         "ACTUAL TIME OF OBSERVATION
  CIGC                   "CEILING CLOUD COVER (0=CLEAR,1=SCATTERED,
  CC1                    "FIRST NON-CEILING 2=BROKEN,3=OVERCAST)
  CC2                    "SECOND NON-CEILING CLOUD COVER
  CIGH -2    FT          "HEIGHT OF CLOUD CEILING
  ZCL1 -2    FT          "HEIGHT OF 1ST (LOWEST) NON-CEILING CLOUD LAYER
  ZCL2 -2    FT          "HEIGHT OF 2ND NON-CEILING CLOUD LAYER
  VIS  1     MI          "VISIBILITY
  WX1        CHAR        "WEATHER TYPE (FIRST FOUR CHARACTERS)
  WX2        CHAR        "             (NEXT FOUR CHARACTERS)
  T    2     K           "TEMPERATURE
  TD   2     K           "DEW POINT TEMPERATURE
  DIR        DEG         "WIND DIRECTION
  SPD  1     MPS         "     SPEED
  GUS  1     MPS         "     GUSTS
  PSL  2     MB          "ALTIMETER SETTING REDUCED TO SEA LEVEL BY STD ATM
  PCP  2     IN          "3 HOURLY PRECIPITATION TOTAL
  SNO        IN          "CUMULATIVE SNOW DEPTH
  PRE  2     MB          "SEA LEVEL PRESSURE
  P24  2     IN          "24 HOUR PRECIPITATION TOTAL
  WXC1       CODE        "WMO WEATHER CODE 1
  WXC2       CODE        "WMO WEATHER CODE 2
  WXC3       CODE        "WMO WEATHER CODE 3
  WXC4       CODE        "WMO WEATHER CODE 4

ENDSCHEMA
SEE: Done...EOF Encountered.
The listing illustrates the set of Key names include in the file. You should notice that the number of Keys in the schema template is greater than those specified in the data we are going to import. The parameters that are missing from the data set will be filed in the MD as MISSing values.

The second column in the schema tells us the power of 10 that the data value will be multiplied by before it is written into the MD file. All numeric data are stored as integers in McIDAS data files.

The third column specifies the unit of the data values as they will be stored in the MD file. You will see that temperatures T and TD will be stored/are expected to be stored in K.

Any information including and following the " is treated as a comment.

As was mentioned earlier, the format for the schema template file is 80-character, records that are not separated by carriage returns or line feeds. This means that this file type is not editable by a regular ASCII text editor (such as vi or emacs). McIDAS provides a pair of conversion utilities that are used to convert from LW files to ASCII text and back again: LWODOS and DOSTOLW. We talk more about this later.

In order to use this or any other schema, TXT2MD needs it stored in the global schema repository file, SCHEMA. SCHEMA must be locatable in your session, how would you verify that it is?

To see if the schema that you wish to use with TXT2MD is available for use, you need to interrogate SCHEMA:

If you get a listing like: 

NAME: ISFC  VERSION:  7  DATE: 1999085  TEXTID: "SURFACE HOURLY OBSERVATIONS
----        -------
   DEFAULT NUMBER OF ROWS:   72                        INTEGER ID: 0
                     COLS:   3500                   MISSING DATA VALUE:  -2139062144
   REPEAT GROUP:  NUMBER OF REPETITIONS:   1
                      STARTING POSITION:  11
                                   SIZE:  25
   NUMBER OF KEYS IN ROW HEADER:     4
                     COL HEADER:     6   STARTING AT POSITION   5
                    DATA RECORD:    25   STARTING AT POSITION  11
                                  ----
                                    35 TOTAL

        KEY  SCALE UNIT     KEY  SCALE UNIT     KEY  SCALE UNIT
        ---- ----- ----     ---- ----- ----     ---- ----- ----
        TYPE   0            DAY    0   CYD      TIME   0   HMS
        NREC   0            ID     0   CHAR     LAT    4   DEG
        LON    4   DEG      ZS     0   M        ST     0   CHAR
        CO     0   CHAR     MOD    0            HMS    0   HMS
        CIGC   0            CC1    0            CC2    0
        CIGH  -2   FT       ZCL1  -2   FT       ZCL2  -2   FT
        VIS    1   MI       WX1    0   CHAR     WX2    0   CHAR
        T      2   K        TD     2   K        DIR    0   DEG
        SPD    1   MPS      GUS    1   MPS      PSL    2   MB
        PCP    2   IN       SNO    0   IN       PRE    2   MB
        P24    2   IN       WXC1   0   CODE     WXC2   0   CODE
        WXC3   0   CODE     WXC4   0   CODE
LSCHE: DONE
you are ready to proceed. If not, then you need to register the ISFC schema in your SCHEMA file: Next, you must decide the name of the MD file that you want to write the data from STORM.TXT into. McIDAS users MD file numbers in the range from 1 to 100 for realtime datasets. The Learning Guide data set uses MD file numbers in the 8000s. For this exercise, let's create the output data file MDXX1000, but first, let's make sure that it does not already exist: If it does, delete it before proceeding: Finally, we are ready to import the data from STORM.TXT and store it in MDXX1000: Make sure that the file was created and is readable: After successful creation, you can use McIDAS display, list, and analysis tools to manipulate your data. First, let's check to see that we have defined a dataset that includes the file MDXX1000: You should have a listing that looks something like: 

Group Name                    Server IP Address
--------------------         ----------------------------------------
BLIZZARD                     <LOCAL-DATA>
MYDATA                       <LOCAL-DATA>
RTGRIDS                      ADDE.UCAR.EDU
RTIMAGES                     ADDE.UCAR.EDU
RTNEXRAD                     ADDE.UCAR.EDU
NEXRCOMP                     ADDE.UCAR.EDU
RTPTSRC                      ADDE.UCAR.EDU
RTWXTEXT                     ADDE.UCAR.EDU
TOPO                         CTTCSUN.UNIDATA.UCAR.EDU

<LOCAL-DATA> indicates that data will be accessed from the local data directory.
DATALOC -- done
The dataset of interest for this exercise is MYDATA. Let's take a closer look: 

        Dataset Names of Type: POINT in Group: MYDATA

Name         NumPos   Content
------------ ------   --------------------------------------
PTSRCS        9999    All MDXX files

DSINFO -- done
The file we created, MDXX1000, will occupy the 1000th position in this dataset. We can, therefore, refer to it directly as MYDATA/PTSRCS.1000. We use this identifier in a number of commands to list out information from/about the data: Let's plot some of the data: This plotted temperature (T) over the continental US. Note that the temperature was plotted in K, the unit of storage in the MD file.

Plot the same data in Farenheit:

Notice that we first erased only the temperatures from the first plot!

Since the mesonet was located in the central US, lets focus our attention there, plot the temperature for 3 UTC, and reduce the size of the plotted values:

You can see that more values were plotted this time. This is a result of the plot decluttering algorithm that McIDAS users.

Finally, plot the temperature data as integers over the same region:

To exercise your growing skills with McIDAS, try plotting other fields from the MD file instead of and in combination with temperature.

You will notice that the MD file created by the above process was rather large give the amount of data that was imported. The reason for this is the schema that was used in its creation. In McIDAS you do not have to choose an existing schema when importing data. Instead, you can create your own, customized schema that is tailored to the data that you want to work with. This is especially important if the data that you want to import doesn't match any existing schema. A good example of this was the NLDN lightning data that Unidata sites can get free-of-charge from the State University of New York at Albany.

Let's repeat the exercise of importing the data in STORM.TXT, but this time, let's create a custom schema to use. Since the ISFC schema listed previously was a pretty good match for our data, let's use it as a starting point.

First, make an ASCII text copy of the schema repository file DCISFC From the McIDAS command mode:

From a Unix command window (xterm), cd to your McIDAS working directory (/home/mcidas/workdata for user mcidas; $HOME/mcidas/data for other users) and edit the ASCII text file JSFC.TXT to look like the following: 

" JSFC (TCY) 0289 SCHEMA -- GENERAL SURFACE SCHEMA
"        NAME VSN DATE  ID "TEXTID
"        ---- --- ----- -- -------
  SCHEMA JSFC  1 98264 0 "CODIAC MESONET OBSERVATIONS

  ROWS 24                "24 HOURS, 3 REPORTS/HOUR

  TYPE                   "0=HOURLY,1=SPECIAL 1,2=SPECIAL 2
  DAY        SYD         "YEAR AND JULIAN DAY (YYDDD)
  TIME       HMS         "NOMINAL TIME (HH0000)
  NREC                   "APPROX. # OF RECORDS WRITTEN IN THIS ROW

  COLUMNS 500            "MAXIMUM OF 3500 REPORTING STATIONS

  ID         CHAR        "STATION CALL LETTERS
  LAT  4     DEG         "RANGE (-90 => +90) POSITIVE NORTH
  LON  4     DEG         "RANGE (-180 => +180) POSITIVE WEST
  ZS         M           "ELEVATION

  DATA

  MOD                    "USER MODIFICATION FLAG (1=MODIFIED)
  T    2     K           "TEMPERATURE
  TD   2     K           "DEW POINT TEMPERATURE
  DIR        DEG         "WIND DIRECTION
  SPD  1     MPS         "     SPEED
  PSL  2     MB          "ALTIMETER SETTING
  PCP  2     IN          "3 HOURLY PRECIPITATION TOTAL
  PRE  2     MB          "SEA LEVEL PRESSURE

ENDSCHEMA
Notice that we have modified a few entries in the file and deleted a number of items that are not needed.

The next job is to turn the ASCII file back into a McIDAS LW file:

We are not ready to register the JSFC schema in the schema registry file SCHEMA: At this point, we are ready to create an MD file using the new schema: Compare the size of the two MD files: 

-rw-rw-r--   1 mcidas   ustaff 5734256 Sep 21 09:17 MDXX1000
-rw-rw-r--   1 mcidas   ustaff  357856 Sep 21 09:25 MDXX1001
See how much smaller the new file is? Rerun the list and display commands presented earlier to verify that the information in the second, smaller MD file is the same as in the first.


Table of contents Previous: Displaying non-AREA Imagery Next: Unidata McIDAS-X Useful Information