CDM Remote is a web service implemented in the CDM library (client) and TDS (server), providing remote access to CDM datasets, using ncstream as the on-the-wire protocol. It provides access at the NetcdfFile and FeatureDataset levels of the CDM API, so there are two levels of services:
The CDM Remote services and protocol are experimental and should not be used outside of the CDM stack for now.
The client forms requests of the form endpoint?query. The possible query parameters are:
req=( CDL | NcML | capabilities | header | data) var=vars where: vars := varspec | varspec[';' varspec] varspec := varname[subsetSpec] varname := escaped variable name subsetSpec := '(' fortran-90 arraySpec ')' fortran-90 arraySpec := dim | dim ',' dims dim := ':' | slice | start ':' end | start ':' end ':' stride slice := INTEGER start := INTEGER stride := INTEGER end := INTEGER
Example service requests are:
Request | Response |
---|---|
http://server:8080/thredds/cdmremote/data.nc?req=CDL |
dataset CDL |
http://server:8080/thredds/cdmremote/data.nc?req=NcML | dataset NcML |
http://server:8080/thredds/cdmremote/data.nc?req=capabilities | capabilities XML |
http://server:8080/thredds/cdmremote/data.nc?req=header | header ncstream message, contains the structural metadata |
http://server:8080/thredds/cdmremote/data.nc?req=data&var=Temp(0:99:10,0:127,:);lat;lon | data ncstream message |
ucar.nc2.stream.CdmRemote is a subclass of NetcdfFile which provides index subsetting on remote CDM datasets. NetcdfDataset.openOrAcquireFile() looks for cdmremote:url prefix, and calls new CdmRemote(url) if found. The url must be an endpoint for a cdmremote service.
<cdmRemoteCapabilities location="http://localhost:8080/thredds/cdmremote/testAll/testdata/2004050300_eta_211.nc">
<featureDataset url="http://localhost:8080/thredds/cdmremote/testAll/testdata/2004050300_eta_211.nc" />
</cdmRemoteCapabilities>
The client forms requests of the form endpoint?query. The possible query parameters are:
req=( capabilities | data | form | stations) accept= (csv | xml | ncstream | netcdf ) time_start,time_end=time range north,south,east,west=bounding box var=vars stn=stns where: vars := varName | varName[,varName] stns := stnName | stnName[,stnName] varName := valid variable name stnName := valid station name
Possible requests are:
req=( capabilities | data | form)
Example service requests are:
Request | Response | Meaning |
---|---|---|
http://server:8080/thredds/cdmrfeature/data.nc/point?req=form | HTML form | form interface |
http://server:8080/thredds/cdmrfeature/data.nc/point?req=capabilities | capabilities XML | describe dataset |
http://server:8080/thredds/cdmrfeature/data.nc/point?req=data | pointFeatureList message | get all the data |
http://server:8080/thredds/cdmrfeature/data.nc/point?req=data&north=40.3&south=22.8&east=-80&west=-105 | pointFeatureList message | get data in bounding box |
http://server:8080/thredds/cdmrfeature/data.nc/point?req=data&time_start=&time_end= | pointFeatureList message | get data in time range |
http://server:8080/thredds/cdmfeature/data.nc/point?req=data&var=Temp,lat,lon | pointFeatureList message | get data for listed variables |
Possible query parameters are:
req=( capabilities | data | form | stations)
Example service requests are:
Request | Response | Meaning |
---|---|---|
http://server:8080/thredds/cdmrfeature/data.nc/station?req=form | HTML form | form interface |
http://server:8080/thredds/cdmrfeature/data.nc/station?req=capabilities | capabilities XML | describe dataset |
http://server:8080/thredds/cdmrfeature/data.nc/station?req=stations | stationListMessage | get all stations |
http://server:8080/thredds/cdmrfeature/data.nc/station?req=stations&north=40.3&south=22.8&east=-80&west=-105 | stationListMessage | get stations in bounding box |
http://server:8080/thredds/cdmrfeature/data.nc/station?req=stations&stn=KDEN,KLOG,MOAS | stationListMessage | get stations in list |
http://server:8080/thredds/cdmrfeature/data.nc/station?req=data&time_start=&time_end=&stn=KDEN | pointFeatureList message | get data in time range |
http://server:8080/thredds/cdmrfeature/data.nc/station?req=data&stn=KDEN,KLOG,MOAS | pointFeatureList message | get data for station list |
http://server:8080/thredds/cdmrfeature/data.nc/station?req=data&north=40.3&south=22.8&east=-80&west=-105 | pointFeatureList message | get data in bounding box |
http://server:8080/thredds/cdmrfeature/data.nc/station?req=data&var=Temp,lat,lon&stn=KDEN | pointFeatureList message | get data for listed variables |
Questions:
ucar.nc2.stream.CdmRemoteFeatureDataset.factory() makes an HTTP request to endpoint+"?req=capabilities" to obtain the GetCapabilities XML document. This document describes what feature type(s) the dataset supports, which determines what are the valid requests, along with the endpoints for those feature types, which may be different than the original endpoint.
FeatureDatasetFactoryManager.open() looks for cdmremote:url prefix, and calls CdmRemoteFeatureDataset.factory(url) if found. ThreddsDataFactory.openFeatureDataset() looks for catalog InvDatasets with featureType.isPointFeatureType() and ServiceType.CdmRemote, and if found, calls CdmRemoteFeatureDataset.factory().
This is an ad-hoc format (for now), example:
<?xml version="1.0" encoding="UTF-8"?> <capabilities location="http://localhost:8080/thredds/cdmremote/idd/metar/ncdecodedLocalHome"> <featureDataset type="station" url="http://localhost:8080/thredds/cdmremote/idd/metar/ncdecodedLocalHome/station" />
<variable name="record.air_pressure_at_sea_level" type="float">
<attribute name="long_name" value="Air pressure at sea level" />
<attribute name="standard_name" value="air_pressure_at_sea_level" />
<attribute name="_FillValue" type="float" value="-99999.0" />
<attribute name="units" value="hectoPascal" />
</variable>
<variable name="record.air_temperature" type="float">
<attribute name="long_name" value="Air temperature at 2 meters" />
<attribute name="standard_name" value="air_temperature" />
<attribute name="_FillValue" type="float" value="-99999.0" />
<attribute name="units" value="Celsius" />
</variable>
...
<LatLonBox>
<west>-180.0000</west>
<east>180.0000</east>
<south>-90.0000</south>
<north>82.5199</north>
</LatLonBox>
<AcceptList>
<accept>raw</accept>
<accept>xml</accept>
<accept>csv</accept>
<accept>netcdf</accept>
<accept>ncstream</accept>
</AcceptList>
</capabilities>
An pointstream is a sequence of one or more messages:
pointstream = {message} message = stationListMessage | pointFeatureListMessage | errorMessage | endMessage stationListMessage = MAGIC_StationList, vlen, PointStreamProto.StationList pointFeatureListMessage = pointFeatureCollectionMessage, {pointFeatureMessage}, (endMessage | errorMessage) pointFeatureCollectionMessage = MAGIC_PointFeatureCollection, vlen, PointStreamProto.PointFeatureCollection pointFeatureMessage = MAGIC_PointFeature, vlen, PointStreamProto.PointFeature endMessage = MAGIC_END errorMessage = MAGIC_ERR, vlen, NcStreamProto.Error vlen = variable length encoded positive integer == length of the following object in bytes // 8 byte constants MAGIC_StationList = 0fe, 0xec, 0xce, 0xda MAGIC_PointFeatureCollection = 0xfi, 0xec, 0xce, 0xba
MAGIC_PointFeature = 0xab, 0xec, 0xce, 0xba
MAGIC_END = 0xed, 0xed, 0xde, 0xde
MAGIC_ERR = 0xab, 0xad, 0xba, 0xda
The protobuf messages are defined by
1) web.xml has a org.springframework.web.servlet.DispatcherServlet cdmRemote with a mapping:
<servlet-mapping> <servlet-name>cdmRemote</servlet-name> <url-pattern>/cdmremote/*</url-pattern> </servlet-mapping>
2) Temporarily, we have the configuration of the datasets in thredds\tds\src\main\webapp\WEB-INF\cdmRemote-servlet.xml: which configures the
collectionController. We will replacce this with pure catalog configuration in 4.2+. This is now being shown in serverStartup.log
<bean id="collectionManager" class="thredds.server.cdmremote.CollectionManager"> <property name="collections"> <list> <ref bean="ncMetars" /> <ref bean="gempakMetars" /> <ref bean="quickScatWinds" /> <ref bean="ncMetarsLocal" /> <ref bean="ncMetarsLocalHome" /> <ref bean="gempakMetarsLocal" /> <ref bean="gempakMetarsLocalHome" /> </list> </property> </bean>
<bean id="ncMetars" class="thredds.server.cdmremote.CollectionBean"> <property name="path" value="/idd/metar/ncdecoded"/> <property name="spec" value="/data/ldm/pub/decoded/netcdf/surface/metar/Surface_METAR_#yyyyMMdd_HHmm#.nc"/> <property name="recheck" value="15 min"/> <property name="featureType" value="STATION"/> <property name="raw" value="report"/> <property name="resolution" value="20 min"/> </bean> etc
This is now being shown in serverStartup.log:
serverStartup: CdmRemoteController collections /idd/metar/gempak == /data/ldm/gempak/surface/#yyyyMMdd#_sao.gem /idd/bufr/quickScat == /data/ldm/gempak/surface/**/#yyyyMMdd#.gem /idd/metar/ncdecodedLocalHome == C:/data/datasets/metars/Surface_METAR_#yyyyMMdd_HHmm#.nc /idd/metar/ncdecoded == /data/ldm/pub/decoded/netcdf/surface/metar/Surface_METAR_#yyyyMMdd_HHmm#.nc /idd/metar/gempakLocalHome == C:/data/formats/gempak/surface/#yyyyMMdd#_sao.gem /idd/metar/gempakLocal == D:/formats/gempak/surface2/#yyyyMMdd#_sao.gem /idd/metar/ncdecodedLocal == Q:/station/ldm/metar/Surface_METAR_#yyyyMMdd_HHmm#.nc
So the URLs are
http://server:port/thredds/cdmremote/path?req eg: http://motherlode.ucar.edu:8081/thredds/cdmremote/idd/metar/gempak http://motherlode.ucar.edu:8081/thredds/cdmremote/idd/metar/ncdecoded http://motherlode.ucar.edu:8081/thredds/cdmremote/idd/bufr/quickScat http://localhost:8080/thredds/cdmremote/idd/metar/ncdecodedLocal http://localhost:8080/thredds/cdmremote/idd/metar/ncdecodedLocalHome?req=form
3) The default output is the dataset description: should be getCapabilities ????
FeatureDataset on location= http://localhost:8080/thredds/cdmremote/local/metars/collection featureType= STATION title= null desc= null range= start= 2006-03-25 00:00:00Z end= 2006-09-25 00:00:00Z duration= 6.0453 months resolution= null start= 2006-03-25 00:00:00Z end = 2006-09-25 00:00:00Z bb = ll: 90.0S 180.0W+ ur: 82.51N 180.0E bb = lat= [-90.00,82.51] lon= [-180.00,180.00] has netcdf = true Data Variables (0) parseInfo= FeatureCollection 0 STATION Q:/station/ldm/metar/Surface_METAR_#yyyyMMdd_HHmm#.nc npts = -1 bb = lat= [-90.00,82.51] lon= [-180.00,180.00]
4) In ToolsUI, FeatureType/PointFeature Tab:
cdmremote:http://localhost:8080/thredds/cdmremote/local/metars/collection
FeatureDataset on location= cdmremote:http://localhost:8080/thredds/cdmremote/local/metars/collection featureType= STATION title= METAR Data from NWS desc= Metar Data from NWS distributed through the Unidata IDD realtime datastream. 1 day's worth of data range= null start= Unknown end = Unknown bb = null has netcdf = true Attributes title = "METAR Data from NWS" version = 2.3 processor = "metar2nc version v1.2" Conventions = "Unidata Observation Dataset v1.0" standard_name_vocabulary = "CF-1.0" description = "Metar Data from NWS distributed through the Unidata IDD\n realtime datastream. 1 day\'s worth of data" time_coordinate = "time_observation" cdm_datatype = "Station" stationDimension = "station" station_id = "station_id" station_description = "station_description" latitude_coordinate = "latitude" longitude_coordinate = "longitude" altitude_coordinate = "altitude" geospatial_lat_max = "90.0" geospatial_lat_min = "-90.0" geospatial_lon_max = "360.0" geospatial_lon_min = "0.0" time_coverage_start = "1143243900" time_coverage_end = "1143330240" observationDimension = "recNum" Data Variables (0) parseInfo= FeatureCollection 0 STATION cdmremote:http://localhost:8080/thredds/cdmremote/local/metars/collection npts = -1 bb = lat= [-90.00,82.51] lon= [-180.00,180.00]
Ask for station data, on the server, its rather slow:
CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060325_0000.nc CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060326_0000.nc CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060327_0000.nc CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060328_0000.nc CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060329_0000.nc CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060330_0000.nc CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060331_0000.nc CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060401_0000.nc CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060402_0000.nc CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060629_0000.nc sent 481 features to 20V (check to see what the strategy is for stations: coomplete scan or ??)
5) cdmremoteCatalog.xml defines datasets for the collections we have working so far.