What Data Can I Actually Access from My McIDAS Session?
The McIDAS data products that are contained in the Unidata-Wisconsin datastream are:
Source Product Resolution Size Broadcast Frequency --------+----------------------+------------+----------+------------------- CIMSS Clour Top Pressure 11 km 320 x 960 hourly CIMSS Precipitable Water 11 km 320 x 752 hourly CIMSS Sea Surface Temp 4 km 360 x 720 daily CIMSS Lifted Index 11 km 320 x 752 hourly CIMSS CAPE 11 km 400 x 800 hourly CIMSS Ozone 16 km 480 x 640 hourly CIMSS Wildfire ABBA - NA 5 km 1190 x 1172 hourly CIMSS Wildfire ABBA - SA 5 km 1700 x 1128 hourly GOES-E Visible 0.65 um 4 km 2726 x 1732 half hourly GOES-E Near Infrared 3.9 um 4 km 2726 x 1732 half hourly GOES-E Water Vapor 6.5 um 4 km 2726 x 1732 half hourly GOES-E Infrared 10.7 um 4 km 2726 x 1732 half hourly GOES-E CO2 13.3 um 4 km 2726 x 1732 half hourly GOES-W Visible 0.65 um 4 km 1400 x 1740 half hourly GOES-W Near Infrared 3.9 um 4 km 1400 x 1740 half hourly GOES-W Water Vapor 6.8 um 8 km 700 x 872 half hourly GOES-W Infrared 10.7 um 4 km 1400 x 1740 half hourly GOES-W Far Infrared 12.0 um 4 km 1400 x 1740 half hourly GRID MDR 9 km 350 x 640 hourly MULTI Antarctic 10.7 um 21 km 512 x 512 3 hourly MULTI Mollweide WV 6.8 um 20 km 350 x 640 3 hourly MULTI Mollweide IR 10.7 um 20 km 350 x 640 3 hourlyA more detailed description of these products and examples of their areal coverage (where appropriate) can be found at:
http://www.unidata.ucar.edu/data/datastream.html
McIDAS uses a combination of two facilities to locate data and information files: MCPATH and file REDIRECTions.
For example, if MCPATH is set to:
/home/fred/mcidas/data:/home/mcidas/data:/home/mcidas/help
and McIDAS-X wants to read from a data file, it first looks for the file in /home/fred/mcidas/data. If the file is not found there, McIDAS-X checks for the file in /home/mcidas/data then /home/mcidas/help. If the file is not located in any of the directories, the system outputs a message saying that the file was not found.
You can set MCPATH in two ways:
$HOME/mcidas/data:$HOME/mcidas/help:~mcidas/data:~mcidas/help
To change MCPATH's value, exit McIDAS-X, set MCPATH to the new value and restart McIDAS-X. If you use the Korn or Bourne shell, the Unix command to explicitly set MCPATH is similar to:
export MCPATH;MCPATH=$HOME/data:$HOME/mcidas/data:/home/mcidas/data:/home/mcidas/help
If you use the C shell, the command is similar to:
setenv MCPATH $HOME/data:$HOME/mcidas/data:/home/mcidas/data:/home/mcidas/help
To set the value of MCPATH each time you login to your account, add the command above to your shell-specific configuration file: .cshrc for the C and Tcsh shells, and .profile for the Bourne and Korn shells, .bash_profile for BASH, etc..
When you start McIDAS-X, a temporary directory is automatically created for you, and its full path is added to MCPATH. For example, $HOME/.mctmp/356052, is appended to the MCPATH value, whether the McIDAS-X start-up script set MCPATH or you set its value.
If you run a McIDAS-X command that writes data, the directory in which the data file is written will be put in the first MCPATH directory for which you have write permission.
If you run a command that writes data to a named file and there are files of the same name residing in one or more MCPATH directories, the command will output to the first writable directory in MCPATH.
For example, if a file named AREA0001 exists in the third and fourth MCPATH directories, the McIDAS-X command LWU POKE AREA0001 0 1 will attempt to write to the copy of AREA0001 in the third MCPATH directory. If you do not have write permissions for that file, you may get unexpected results; for example, the command stops due to write failure.
**CAUTION**
If you run a command that rewrites a file, and a file of the same name exists in more than one MCPATH directory, then you will most likely get unexpected results. The reason for this is that the process of rewriting a file is typically one of deleting the file followed by writing a new copy of it. The delete operation will attempt to remove the file from the first MCPATH directory it is found in. When the command goes to write the new copy of the file, it will again look through the MCPATH directories to see where it should be written. Upon finding a file of the name that it is trying to create, it will attempt to overwrite the existing file. If the write permission on that file is not set, then the write will fail.
File redirection is the specification of the directory in which a file or group of files will be found, read, and written by McIDAS applications. Files that are REDIRECTed with the McIDAS REDIRECT command will be used in precedence over files found in directories listed in MCPATH. This lets you explicitly set the location of a particular file or group of files represented by a file mask (regular expression).
For example, if MCPATH is set to:
/home/fred/mcidas/data:~mcidas/data:~mcidas/helpand each of these directories contains a file named AREA0001, the McIDAS command LWU LIST AREA0001 will use the file in /home/fred/mcidas/data if the file is not redirected to another directory.
However, if you have the redirection entry:
AREA00* /home/dave/aircraft/datathe LWU command uses the file AREA0001 stored in /home/dave/aircraft/data instead of the one stored in /home/fred/mcidas/data.
Instructions for creating redirection files are contained in the Unidata McIDAS Users Guide.
Unidata provides a set of example redirects in the file EXAMPLE.NAM. This file is put in the user mcidas' data directory (~mcidas/data) by the McIDAS-X installation process.
EXAMPLE.NAM is intended to be copied to LOCAL.NAM and then edited by the McIDAS administrator. The editing changes needed are the setting of directories to match your site's McIDAS installation. The entries in EXAMPLE.NAM should be all that are necessary to insure access to McIDAS data files produced by ldm-mcidas decoders or by McIDAS-XCD.
SSEC began recasting McIDAS into a client-server model in early 1993. The Distributed Data Environment (DDE) allowed McIDAS-OS2 systems (now archaic) acting as clients to access data from McIDAS-MVS servers.
Over the years since 1993, the DDE has been transitioned into the Abstract Data Distribution Environment (ADDE) that is now an integral part of McIDAS-X.
In non-ADDE McIDAS commands, all image, grid and point source files are referenced by file numbers. If you don't know the the file numbers or don't understand the McIDAS file naming conventions, finding and using data could be difficult. ADDE commands, on the other hand, access information from named datasets. Dataset names are assigned to sets of files by the McIDAS system administrator on the machine that is running the ADDE server. End users then run ADDE client commands to interrogate these ADDE servers to find out what data is available to their sessions. They no longer need to be concerned about which files the data actually resides in.
The current ADDE implementation allows Unix workstations to act as clients that can access data from any other workstation, accessible over TCP/IP ethernet, on which McIDAS-X has been installed and on which the McIDAS remote ADDE server has been installed. ADDE servers do not have to only exist on remote machines, however. When ADDE clients make requests for data not served by ADDE remote servers, a local server is automatically started in the user's session to provide the data.
The concepts that form the basis for ADDE clients are discussed in detail in the ADDE section.
Instructions for installing an ADDE remote server are contained in Installing the McIDAS-X ADDE Remote Server.
McIDAS provides a variety of routines to list data available to your McIDAS session. Each data type (IMAGE or AREA, GRID, and POINT or MD) have utilities that list the datasets (ADDE) or files (non-ADDE) that are available for use. Which data are available for individual users is, however, dependent on how McIDAS was installed on their system and how their McIDAS environment is configured. For more information on configuring user's McIDAS sessions, please refer to: Customizing McIDAS-X Sessions.
In the ADDE environment, one uses the DATALOC command to list dataset group names and the Internet name or IP address of the ADDE servers that can supply the data. This list is known as as the client routing table. Typically, the McIDAS site administrator defines the contents of a globally accessible (through MCPATH) client routing table for all users. Knowledgeable users can, however, use DATALOC to manage private client routing tables in addition to accessing the globally defined one.
In its simplest form, DATALOC is run as follows:
DATALOCOn a system that has not had client routing tables setup, this would result in a empty listing:
Group Name Server IP Address -------------------- ---------------------------------------- DATALOC -- doneAs one works through the Unidata McIDAS Learning Guide, their individual client routing table will be built so that the DATALOC listing will begin to look something like:
Group Name Server IP Address -------------------- ---------------------------------------- AREAS <LOCAL-DATA> BLIZZARD ADDE.UCAR.EDU MYDATA <LOCAL-DATA> RTGRIDS ADDE.UCAR.EDU RTIMAGES ADDE.UCAR.EDU RTPTSRC ADDE.UCAR.EDU RTWXTEXT ADDE.UCAR.EDU TEST <LOCAL-DATA> <LOCAL-DATA> indicates that data will be accessed from the local data directory. DATALOC -- doneThis listing tells us that the dataset groups named AREAS, MYDATA, and TEST will be served by a local ADDE server while BLIZZARD, RTGRIDS, RTIMAGES, and RTPTSRC datasets will be serviced by a machine whose Internet name is ADDE.UCAR.EDU.
A local ADDE server will attempt to supply data from all datasets that are not named in the client routing table. It is our recommendation, however, that the McIDAS system administrator defines ALL datasets explicitly so that users will be able to easily see all datasets that exist by running DATALOC.
After running DATALOC, one only knows about the dataset group names that are accessible from servers. To see all of the types of data available that a particular group, you would run the DSINFO command.
The command syntax for DSINFO is:
SINFO type groupThe choices for type are:
IMAGE - imagery (e.g. satellite imagery) POINT - point source data (e.g. surface observations, profiler data, etc.) GRID - gridded data (e.g. model output) TEXT - textual data (e.g. raw observations) NAV - navigation information (e.g. navigation codicils) ALL - any of the aboveThe default data type to list is ALL.
To see what data types are included in the group BLIZZARD, you would run:
DSINFO X BLIZZARD
Dataset Names of Type: IMAGE in Group: BLIZZARD
Name NumPos Content
------------ ------ --------------------------------------
G7-IR-4K 3 Storm of the Century G7 IR
G7-VIS-1K 1 Storm of the Century G7 1K
G7-VIS-4K 3 Storm of the Century G7 Vis
G7-VIS-FD 1 Storm of the Century G7 16K
IMAGES 14 Storm of the Century Images
M3-IR 3 Storm of the Century M3 IR
M3-VIS 3 Storm of the Century M3 VIS
Dataset Names of Type: POINT in Group: BLIZZARD
Name NumPos Content
------------ ------ --------------------------------------
PTSRCS 5 Storm of the Century Point Source Data
SFCHOURLY 1 Storm of the Century Surface Observations
SHIPBUOY 1 Storm of the Century Ship/Buoy Observations
SYNOPTIC 1 Storm of the Century Synoptic Observations
UPPERMAND 1 Storm of the Century Upper Air (Mandatory Levels)
UPPERSIG 1 Storm of the Century Upper Air (Significant Levels)
Dataset Names of Type: GRID in Group: BLIZZARD
Name NumPos Content
------------ ------ --------------------------------------
GRIDS 5 Storm of the Century Grids
No Datasets found of Type: TEXT in Group: BLIZZARD
No Datasets found of Type: NAV in Group: BLIZZARD
DSINFO -- done
The entries listed under Name are generically referred to as
descriptors. Descriptors are not data files; they are
names that identify sets of data files. Descriptors are very
flexible. Different descriptors within a group can point to the exact
same collection of data, or to various subsets. For instance, the
descriptor IMAGES was defined to point at all of the data in all
of the other descriptors for group BLIZZARD.
McIDAS provides the DMAP command for listing out the location of any/all files accessible to McIDAS sessions. DMAP uses information defined in the user's file REDIRECTion table and in the directories defined in the users's MCPATH.
The McIDAS data file naming convention uses the following prefixes to identify particular data, and auxiliary types:
AREA - imagery (e.g. satellite imagery) MDXX - point source data (e.g., surface observations, profiler data, etc.) GRID - gridded data (e.g., model output) TEXT - textual data (e.g., raw observations) VIRT - virtual graphics (e.g., map outlines)
DMAP invoked without file specification:
DMAPwill list out all files that are accessible to you McIDAS session. These are files that are be contained in the session's file redirection table and in the directories detailed in the session's MCPATH. **Caution** this listing could be extensive!
DMAP can be instructed to look for individual or sets of files that have similar names. DMAP uses the characters specified on the command line as a mask for the file names that it should list. For instance:
DMAP AREAwill list out the locations of all files whose names begin with AREA.
DMAP AREA01will list out the locations of all files whose names begin with AREA01 and so on.
DMAP can also list files based on specified sufficies. For example:
DMAP *.IDXwill list out all files that have IDX as their suffix.
DMAP is the quickest way of finding out where one or more files is located on the system. DMAP does not, however, tell you anything about what is contained in the file. The McIDAS utilities for listing the contents of data files are presented below.
After you ran DSINFO X BLIZZARD above, you saw that the dataset whose group name is BLIZZARD has a variety of imagery data in it. To list more detailed information about the contents of the various descriptors in group BLIZZARD, you would use the ADDE IMGLIST command.
The simpliest invocation syntax for IMGLIST is:
IMGLIST dataset eposHere dataset is the full dataset name (group name and descriptor separated by a /) to search and epos is the ending position in the dataset.
To get a short listing of the most recent image in the set of 4 km GOES-7 Infrared images in group BLIZZARD, you would run:
IMGLIST BLIZZARD/G7-IR-4K
Image file directory listing for:BLIZZARD/G7-IR-4K
Pos Satellite/ Date Time Center Band(s)
sensor Lat Lon
--- ------------- ------------ -------- ---- ---- ------------
3 GOES-7 IR 13 MAR 93072 18:01:00 26 87 8
IMGLIST: done
To list out all of the images in the set defined by the descriptor
G7-IR-4K, you would run:
IMGLIST BLIZZARD/G7-IR-4K.ALL
Image file directory listing for:BLIZZARD/G7-IR-4K
Pos Satellite/ Date Time Center Band(s)
sensor Lat Lon
--- ------------- ------------ -------- ---- ---- ------------
1 GOES-7 IR 13 MAR 93072 17:01:00 26 87 8
2 GOES-7 IR 13 MAR 93072 17:31:00 26 87 8
3 GOES-7 IR 13 MAR 93072 18:01:00 26 87 8
IMGLIST: done
To list out all of the images in the group BLIZZARD you would run:
IMGLIST BLIZZARD/IMAGES.ALL
Image file directory listing for:BLIZZARD/IMAGES
Pos Satellite/ Date Time Center Band(s)
sensor Lat Lon
--- ------------- ------------ -------- ---- ---- ------------
1 GOES-7 13 MAR 93072 17:01:00 26 87 1
2 GOES-7 13 MAR 93072 17:31:00 26 87 1
3 GOES-7 13 MAR 93072 18:01:00 26 87 1
4 GOES-7 IR 13 MAR 93072 17:01:00 26 87 8
5 GOES-7 IR 13 MAR 93072 17:31:00 26 87 8
6 GOES-7 IR 13 MAR 93072 18:01:00 26 87 8
7 GOES-7 13 MAR 93072 18:01:00 1 113 1
8 GOES-7 13 MAR 93072 18:01:00 34 102 1
9 METEOSAT3 13 MAR 93072 17:00:00 26 84 1
10 METEOSAT3 13 MAR 93072 17:30:00 26 84 1
11 METEOSAT3 13 MAR 93072 18:00:00 26 84 1
12 METEOSAT3 13 MAR 93072 17:00:00 26 84 8
13 METEOSAT3 13 MAR 93072 17:30:00 26 84 8
14 METEOSAT3 13 MAR 93072 18:00:00 26 84 8
IMGLIST: done
IMGLIST will list out more detailed information if you tell it to:
IMGLIST BLIZZARD/G7-IR-4K FORM=EXP
Image file directory listing for:BLIZZARD/G7-IR-4K
Pos Satellite/ Date Time Center Res (km) Image_Size
sensor Lat Lon Lat Lon
--- ------------- ------------ -------- ---- ---- ----- ----- ------------
3 GOES-7 IR 13 MAR 93072 18:01:00 26 87
Band: 8 11.2 um Night cloud detection, sfc temp 4.0 4.0 1337 x 1604
proj: 0 created: 93208 192750 memo: AAA-MSI TO IR16-S VIA ARCHIVE
type:AAA cal type:RAW
offsets: data= 1280 navigation= 256 calibration= 768 auxillary= 0
doc length: 512 cal length: 116 lev length: 4 PREFIX= 636
valcod: 1104213119 zcor: 1 avg-smp: N
pdl: 87 87 3 0 0 0 8A 8A 0 0 0 0 0 0 0 0
0 0 0 0 0 0 10 0 0 73 6 33 59 1 20 2C
start yyddd:93072 start time:180105 start scan: 56
lcor: 1123 ecor: 7606 bytes per pixel: 2 ss: 33
Image Center Point Res (derived) Lat: 4.34 (km) Lon: 3.62 (km)
IMGLIST: done
It can also tell you more explicit information on the band that the image
represents:
IMGLIST BLIZZARD/G7-IR-4K FORM=BAND
Image file directory listing for:BLIZZARD/G7-IR-4K
Pos Satellite/ Date Time Center Res (km) Image_Size
sensor Lat Lon Lat Lon
--- ------------- ------------ -------- ---- ---- ----- ----- ------------
3 GOES-7 IR 13 MAR 93072 18:01:00 26 87
Band: 8 11.2 um Night cloud detection, sfc temp 4.0 4.0 1337 x 1604
IMGLIST: done
In ADDE, there is a single command GRDLIST that is used for listing out the grid datasets members and the contents of these members.
To see what members exist in the BLIZZARD/GRIDS dataset, you would run:
GRDLIST BLIZZARD/GRIDS.ALL FORM=FILE
DATASET NAME: BLIZZARD/GRIDS
Dataset Position Creation Date Max Grids Directory Title
---------------- ------------- --------- -------------------------------
1 93070 159 12Z 93070 NMC NGM TRAINING GRID
2 93071 159 00Z 93071 NMC NGM TRAINING GRID
3 93071 159 12Z 93071 NMC NGM TRAINING GRID
4 93072 159 00Z 93072 NMC NGM TRAINING GRID
5 93072 159 12Z 93072 NMC NGM TRAINING GRID
GRDLIST - done
To list out the the first five grids from the first member of the
BLIZZARD/GRIDS dataset, you would run:
GRDLIST BLIZZARD/GRIDS.1 FORM=SHORT GRID=1 5 Dataset position 1 Directory Title= 12Z 93070 NMC NGM TRAINING GRID PARAM LEVEL DAY TIME SRC FHOUR FDAY FTIME GRID PRO ----- ------- ------------ -------- ---- ----- ------------ -------- ------ ---- Z 1000 MB 11 MAR 93070 12:00:00 ROI 0 11 MAR 93070 12:00:00 1 MERC T 1000 MB 11 MAR 93070 12:00:00 NGM 0 11 MAR 93070 12:00:00 2 MERC U 1000 MB 11 MAR 93070 12:00:00 ROI 0 11 MAR 93070 12:00:00 3 MERC V 1000 MB 11 MAR 93070 12:00:00 ROI 0 11 MAR 93070 12:00:00 4 MERC Z 850 MB 11 MAR 93070 12:00:00 ROI 0 11 MAR 93070 12:00:00 5 MERC Number of grids listed = 5 GRDLIST - doneTo list out the contents of the first member of the BLIZZARD/GRIDS dataset, you would run:
GRDLIST BLIZZARD/GRIDS.1 FORM=SHORT GRID=1 LASTYou can also list all grids that match search conditions that you specify. Suppose you would like to get a list of all 700 mb temperature grids for forecast hours 12 through 36 for the 12Z run on March 11 . To produce this list, you would run:
GRDLIST BLIZZARD/GRIDS NUM=ALL LEV=700 PARAM=T FRANGE=12 36 TIME=12 DAY=93070 Dataset position 1 Directory Title= 12Z 93070 NMC NGM TRAINING GRID PARAM LEVEL DAY TIME SRC FHOUR FDAY FTIME GRID PRO ----- ------- ------------ -------- ---- ----- ------------ -------- ------ ---- T 700 MB 11 MAR 93070 12:00:00 NGM 12 12 MAR 93071 00:00:00 30 MERC T 700 MB 11 MAR 93070 12:00:00 NGM 24 12 MAR 93071 12:00:00 50 MERC T 700 MB 11 MAR 93070 12:00:00 NGM 36 13 MAR 93072 00:00:00 70 MERC Number of grids listed = 3 GRDLIST - done
PTLIST can be used to list out general information on a dataset member:
PTLIST BLIZZARD/SFCHOURLY.ALL FORM=FILE
Pos Description Schema NRows NCols
------ -------------------------------- ------ ----- -----
1 SURFACE HOURLY OBSERVATIONS ISFC 72 3500
PTLIST: Done
Since BLIZZARD has a descriptor, OBS, that contains all of
the point source data types, one can list out general information for all of
the dataset members with a single command:
PTLIST BLIZZARD/OBS.ALL FORM=FILE
Pos Description Schema NRows NCols
------ -------------------------------- ------ ----- -----
1 SURFACE HOURLY OBSERVATIONS ISFC 72 3500
2 INTL. RADIOSONDE OBS (UPPER AIR IRAB 8 1500
3 INTL. RADIOSONDE OBS--SIG LEVELS IRSG 16 8000
4 SHIP/BUOY OBSERVATIONS ISHP 24 3000
5 SURFACE SYNOPTICS SYN 8 6000
PTLIST: Done
PTLIST can also be used to list out the names of the data stored within
the dataset member:
PTLIST BLIZZARD/SFCHOURLY.ALL FORM=PARAM PARAMETER UNIT STORAGE TYPE DEFAULT FORMAT --------- ---- ------------ -------------- TYPE integer I9 DAY SYD integer I9 TIME HMS integer I9 NREC integer I9 ID character A4 LAT DEG real F9.4 LON DEG real F9.4 ZS M integer I9 ST character A4 CO character A4 MOD integer I9 HMS HMS integer I9 CIGC integer I9 CC1 integer I9 CC2 integer I9 CIGH FT real F9 ZCL1 FT real F9 ZCL2 FT real F9 VIS MI real F9.1 WX1 character A4 WX2 character A4 T K real F9.2 TD K real F9.2 DIR DEG integer I9 SPD MPS real F9.1 GUS MPS real F9.1 PSL MB real F9.2 PCP IN real F9.2 SNO IN integer I9 PRE MB real F9.2 PTLIST: DoneThis listing tells us not only the names of the data elements of the data set member, but also their units, how they are stored, and how their values will be printed by default for other PTLIST options.
You can also simply find out the values for all of the data for the first entry in a dataset member:
PTLIST BLIZZARD/SFCHOURLY.ALL FORM=DATA Row : 1 Col : 2 TYPE = 0 | DAY = 93072 SYD | TIME = 0 HMS | NREC = 1924 | ID = OAR | LAT = 36.6833 DEG | LON = 121.7666 DEG | ZS = 41 M | ST = CA | CO = US | MOD = 0 | HMS = 235500 HMS | CIGC = 2 | CC1 = 2 | CC2 = _missing_ | CIGH = 15000. FT | ZCL1 = 20000. FT | ZCL2 = _missing_ | VIS = 25.0 MI | WX1 = _missing_ | WX2 = _missing_ | T = 289.27 K | TD = 284.83 K | DIR = 270 DEG | SPD = 6.2 MPS | GUS = _missing_ | PSL = 1019.31 MB | PCP = _missing_ | SNO = _missing_ | PRE = _missing_ | --------------------------------------------------------------------------- Number of matches found = 1 PTLIST: DoneOne can also selection criteria by which parameters are listed. For instance, if you want to list out the station ID, report TIME, and Temperature for all of the stations in South Carolina that are reporting temperatures between 45 and 70 F at 15 Z, you would run:
PTLIST BLIZZARD/SFCHOURLY.1 FORM=DATA NUM=ALL PARAM=ID TIME T[F] SELECT='ST SC;T[F] 45 70;TIME 15' ID TIME[HMS] T[F] ---- --------- --------- CHS 150000 55.00 NBC 150000 50.99 MMT 150000 50.00 FLO 150000 64.99 SSC 150000 53.01 MYR 150000 63.00 CRE 150000 61.99 MYR 150000 61.99 Number of matches found = 8 PTLIST: Done