A Forecast Model Run Collection (FMRC) is a collection of forecast model runs, which can be uniquely identified by the start of the model run, called the model run time, also called the analysis time or generating time, or reference time. Each model run has a series of forecast times. A collection of these runs therefore has two time coordinates, the run time and the forecast time. An FMRC creates a 2D time collection dataset, and then creates various 1D time subsets out of it. See this poster for a detailed example.
Previously this functionality was provided using FMRC Aggregation through NcML and the <fmrcDataset> element in the TDS configuration catalog. As of TDS 4.2, that implementation is now deprecated and <featureCollection> elements are the correct way to provide this functionality. As of 4.3, one should only serve GRIB files with featureCollection=GRIB, eg not with FMRC. Typically, FMRC is is used for collections of model runs stored in netCDF/CF files.
Defines options on feature collections with featureType="FMRC".
<fmrcConfig regularize="true" datasetTypes="TwoD Best Files Runs" />
<fmrcConfig regularize="true" datasetTypes="Files"> <dataset name="Best" offsetsGreaterEqual="0"/> </fmrcConfig>
where:
- regularize: If true, then the runs for a given hour (from 0Z) are assumed to have the same forecast time coordinates. For example, if you have 4 model runs per day (eg 0, 6, 12, 18Z) and many days of model runs, then all the 6Z runs for all days will have the same time coordiantes, etc. This "regularizes" time coordinates, and is useful when there may be missing forecast times, which may result in creating a new time coordinate. This obviates the need for the FMRC definition files which previously were used on motherlode for models in the IDD feed.
- datasetTypes: list the dataset types that are exposed in the TDS catalog. The possible values are:
- TwoD: dataset with two time dimensions (run time and forecast time), which contains all the data in the collection.
- Best: dataset using the latest model data available for each possible forecast hour.
- Files: each component file of the collection is available seperately, as in a datasetScan. A "latest" file will be added if there is a "latest" Resolver service in the catalog.
- Runs: A model run dataset contains all the data for one run time.
- ConstantForecasts: A constant forecast dataset is created from all the data that have the same forecast time. This kind of dataset has successively shorter forecasts of the same endpoint.
- ConstantOffsets: A constant offset dataset is created from all the data that have the same offset from the beginning of the run.
- dataset: you can define your own "best dataset". This uses the same algorithm as the Best dataset above, but excludes data based on its offset hour. In the above example, a Best dataset is created with offset hours less than 0 excluded.
- name: the name of the dataset, must be unique within the fmrcConfig element. Do not use "best.ncd", "fmrc.ncd", "runs", "files", "forecast", or "offset".
- offsetsGreaterEqual: forecast offset hours (forecast time - run time) less than this value are excluded.
Notes:
The ToolsUI FMRC tab allows you to view internal strurctures of an FMRC. You can pass it a collection specification string or a file with a featureCollection element in it.
Use static method on ucar.nc2.ft.fmrc.Fmrc:
public static Fmrc open(String collection, Formatter errlog, Formatter debug);
The collection may be one of:
If a dateFormatMark is given, a DateExtractor extracts the rundate from the filename or URL. Otherwise there must be global attributes _CoordinateModelBaseDate or _CoordinateModelRunDate inside each dataset. The GRIB IOSP reader automatically adds this global attribute.
Each file is opened as a GridDataset:
gds = GridDataset.open( mfile.getPath());
and the forecast time coordinates are extracted from the grid coordinate system.
There is no need to specify forecastModelRunCollection vs forecastModelRunSingleCollectionc, nor timeUnitsChange. This is detected automatically.
If true, then all runs for a given offset hour (from 0Z) are assumed to have the same forecast time coordinates. This obviates the need for the FMRC definition files which previously were used on motherlode. This evens out time coordinates, and compensates for missing forecast times in the IDD feed.
An fmrInv.xml file is made which records the essential grid information from each file. It is cached in a persistent Berkeley Database (bdb) key/value store, so that it only has to be done the first time the file is accessed in an FMRC. Each collection becomes a seperate bdb database, and each file in the collection is an element in the database, with the filename as the key and the fmrInv.xml as the value. When a collection is scanned, any filenames already in the database are reused. Any new ones are read and added to the database. Any entries in the database that no longer have a filename associated with them are deleted.
ToolsUI collections tab allows you to delete database or individual elements.
There is no need to specify forecastModelRunCollection vs forecastModelRunSingleCollection, nor timeUnitsChange. This is detected automatically.
Old way:
1) <datasetFmrc name="NCEP-GFS-CONUS_80km" collectionType="ForecastModelRuns" harvest="true" path="fmrc/NCEP/GFS/CONUS_80km">
<metadata inherited="true">
<documentation type="summary">good stuff</documentation>
</metadata>
2) <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" enhance="true">
<aggregation dimName="run" type="forecastModelRunCollection" fmrcDefinition="NCEP-GFS-CONUS_80km.fmrcDefinition.xml" recheckEvery="15 min">
<scan location="/data/ldm/pub/native/grid/NCEP/GFS/CONUS_80km/" suffix=".grib1"
dateFormatMark="GFS_CONUS_80km_#yyyyMMdd_HHmm" subdirs="true" olderThan="5 min"/>
</aggregation>
</netcdf>
3) <fmrcInventory location="/data/ldm/pub/native/grid/NCEP/GFS/CONUS_80km/" suffix=".grib1" fmrcDefinition="NCEP-GFS-CONUS_80km.fmrcDefinition.xml" />
<addTimeCoverage datasetNameMatchPattern="GFS_CONUS_80km_([0-9]{4})([0-9]{2})([0-9]{2})_([0-9]{2})00.grib1$"
startTimeSubstitutionPattern="$1-$2-$3T$4:00:00"
duration="240 hours"/>
</datasetFmrc>
Old way:
<datasetFmrc name="RTOFS Forecast Model Run Collection" path="fmrc/rtofs">
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
1) <variable name="mixed_layer_depth">
<attribute name="long_name" value="mixed_layer_depth @ surface"/>
<attribute name="units" value="m"/>
</variable>
<aggregation dimName="runtime" type="forecastModelRunSingleCollection" timeUnitsChange="true" recheckEvery="10 min">
2) <variable name="time">
<attribute name="units" value="hours since "/>
</variable>
3) <scanFmrc location="c:/rps/cf/rtofs" regExp=".*ofs_atl.*\.grib2$"
runDateMatcher="#ofs.#yyyyMMdd" forecastOffsetMatcher="HHH#.grb.grib2#" subdirs="true"
olderThan="10 min"/>
</aggregation>
</netcdf>
</datasetFmrc>
where:
New way:
<featureCollection name="NCEP-GFS-CONUS_80km" featureType="FMRC" harvest="true" path="fmrc/NCEP/GFS/CONUS_80km">
<metadata inherited="true">
<documentation type="summary">good stuff</documentation>
</metadata>
1) <collection spec="/data/ldm/pub/native/grid/NCEP/GFS/CONUS_80km/GFS_CONUS_80km_#yyyyMMdd_HHmm#.grib1"
recheckAfter="15 min"
olderThan="5 min"/>
2) <update startup="true" rescan="0 5 3 * * ? *" />
3) <protoDataset choice="Penultimate" change="0 2 3 * * ? *" />
4) <fmrcConfig regularize="true" datasetTypes="TwoD Best Files Runs ConstantForecasts ConstantOffsets" />
</featureCollection>
This document was last updated Mar 2015.