Section: 4.5.1 Text Point Data

4.5.1 Text Point Data

Text (ASCII) Point Data Format If your point (in situ) data is not available in one of the file formats that the IDV can handle, but you can output it into a column-oriented ASCII text file, you may still be able to use the IDV displays. The IDV supports reading in point observation data in a text comma separated value (CSV) file format.

You need to specify the metadata of this point data. This can be done in two ways. First, the file can have two extra header lines that defines for the IDV the fields that are within the file and the types of the fields. The only other requirement is that there must be a latitude, longitude and time field in the data. See below.

Secondly, if there are no metadata header lines defined, the Text Point Data Source will show the Metadata GUI described below.

Examples:

Comma separated numeric values

In the simplest form, each line line is an observation at one time and one location:

(index -> (Time,Latitude,Longitude,Altitude,PSL,GUS,SPD,DIR,TD,T))
Time[fmt="yyyy-MM-dd HH:mm:ss"],Latitude[unit="deg"],Longitude[unit="degrees west"],Altitude[unit="m"],PSL[unit="hPa"],GUS[unit="m.s-1"],SPD[unit="m/s"],DIR[uni
t="deg"],TD[unit="celsius"],T[unit="celsius"]
2007-01-07 16:00:00Z,32.9,117.1,145.0,1026.1,NaN,0.0,0.0,-2.8,12.8
2007-01-07 16:00:00Z,48.3,92.5,341.0,1003.7,NaN,1.5,170.0,-2.0,-0.99
2007-01-07 16:00:00Z,36.8,98.7,449.0,1024.0,12.4,9.8,330.0,-3.0,3.0
2007-01-07 16:00:00Z,44.3,121.2,938.0,1030.1,NaN,2.1,110.0,-3.3,-1.7

Note also that the first line's structure is quite rigid -- you must have a variable (e.g., index, recNum) that is the domain parameter; this should map to the range values. The second line defines the formatting and units of the parameters. Unit names should be standard international unit specifications (udunits compatible). A list of valid names can be found here. (A complete description of this format is contained in the VisAD README.text file.)

Also note that you need to set the name of the variable for time as "Time", as well as the locations as "Latitude", "Longitude" and "Altitude" (if needed).

Text fields

If you have text fields (i.e., non-numeric data) in your observation just do this:

(index -> (Time,Latitude,Longitude,Altitude,ST(Text),T))
Time[fmt="yyyy-MM-dd HH:mm:ss z"],Latitude[unit="deg"],Longitude[unit="degrees west"],Altitude[unit="m"],ST(Text),T[unit="celsius"]
2007-02-16 11:00:00 MST,32.9,117.1,145.0,CA  ,20.6

Here we have a ST field (State from metars). Its field name is defined as "ST(Text)" and its entry in the second line is the same.

Time in multiple columns

If you have time fields that span multiple columns, you can use the colspan keyword:

(index -> (IDN, Latitude, Longitude, Time, WDIR, WSPD, GST))
IDN, Latitude[unit="deg"], Longitude[unit="deg"], Time[fmt="yyyy MM dd HH mm" colspan="5"], WDIR[unit="deg" miss="MM"], WSPD[unit="m/s" miss="MM"], GST[unit="m/s" miss="MM"]
41001 34.68 -72.66 2007 07 17 20 50 210 4.0 6.0
41004 32.5 -79.09 2007 07 17 20 50 210 6.0 MM
41008 31.4 -80.87 2007 07 17 21 50 170 7.0 8.0

Skipping columns

You can use the "skip" parameter if you want to ignore (not read) values in your text file that you don't want to use:

(recNum->(Latitude, Longitude, Altitude, type(Text), time, turb_intensity))
Time[fmt=yyyyMMddHH], obtime, skip, type(Text), skip, skip, Latitude, Longitude[scale=-1], Altitude[unit=ft], Altitude2[unit=ft], turb_intensity, skip
2004050100 0005    34     C210 1 T  38.82   92.22   7000   7000  0 -9
2004050100 0004    35     PA32 0 T  35.40   98.62   4000   4000  0 -9
2004050100 0008    58      A36 0 T  29.18   81.05   5000   5000  2 -9

Fixed values for several observations

If you have values (like time or location) that are fixed for several observations, you may use this construct:

(index -> (Longitude,Latitude,Time,ST(Text),SPD,DIR,TD,T))
Longitude[unit="degrees west"],Latitude[unit="deg"],Time[fmt="yyyy-MM-dd HH:mm:ss z"],ST(Text),SPD[unit="m/s"],DIR[unit="deg"],TD[unit="celsius"],T[unit="celsius"]

Longitude=-117.1
Latitude=32.9
ST=MSN
2007-02-20 11:00:00 MST  ,0.0,0.0,8.9,13.3
2007-02-20 12:00:00 MST  ,0.0,0.0,11.9,15.0

Longitude=-89.4
Latitude=43.1
ST=DEN
2007-02-20 11:00:00 MST  ,1.5,160.0,-7.0,-2.0
2007-02-20 12:00:00 MST  ,1.5,160.0,-7.0,-2.0

Longitude=-121.2
Latitude=44.3
ST=ORD
2007-02-20 11:00:00 MST  ,10.8,230.0,-1.1,6.7

Loading into the IDV

Finally, after you have created your file, you will want to tailor your IDV display in two ways:

Create a Layout (station) model for those variables using the IDV's menu "Tools->Layout Model Editor"

Then, when you run the IDV to read your data:

Use Data Source Type as "Text Point Data files"
Use the "Point Data Plot" for the Display
Select your newly created Layout model

Using the Text Point Metadata Gui

If your text data does not have the metadata header lines the Text Point Data Source will show the following dialog which allows you to specify the meta data.

Skipping lines
At the top a number of the initial lines from the text data are shown. The arrow keys allow you to specify the start line. For example, if you had some other header information you can skip over those lines.

Specifying metadata
For each column of text data there is a row shown in the bottom of the dialog. This shows the sampled value and allows you to enter a name, unit, date format, missing value and extra information. There are some names that the IDV treats special: "Latitude", "Longitude", "Altitude", and "Time". You must have at least Latitude, Longitude and Time specified.

The Unit/Date Format field allows you to specify the Unit for data fields and the date format. For text fields choose Misc→Text as the unit.

The Extra fields must be of the form:

name="value"

Don't forget the quotes!

Skipping columns
You can skip over certain columns by entering the Name: "skip"

Saving this as a preference
After all of your hard work to keep from having to do this again next time you load a new text point data of the form simply press the "Preferences" button and select Save Current. This allows you to save these metadata settings and reapply them later using the "Preferences" button.

Unidata IDV Workshop for version 6.3 > Advanced Topics > Miscellaneous Items