CDM Remote Web Service (Experimental)

CDM Remote is a web service implemented in the CDM library (client) and TDS (server), providing remote access to CDM datasets, using ncstream as the on-the-wire protocol. It provides access at the NetcdfFile and FeatureDataset levels of the CDM API, so there are two levels of services:

1. CdmRemote provides index subsetting on remote CDM datasets, with the same functionality that NetcdfFile provides for CDM local datasets and DODSNetcdfFile provides for remote OPeNDAP datasets. CdmRemote supports the full CDM data model.
2. CdmrFeature provides coordinate subsetting on remote CDM Feature Datasets, with similar functionality to WCS and Unidata's experimental NetCDF Subset Service (NCSS). Currently (version 4.2) it provides a remote API for Point and StationTimeSeries Feature Datasets.

The CDM Remote services and protocol are experimental and should not be used outside of the CDM stack for now.

CdmRemote Protocol (Data Access and Coordinate Systems)

The client forms requests of the form endpoint?query. The possible query parameters are:

  req=( CDL | NcML | capabilities | header | data)
  var=vars
where:
  vars := varspec | varspec[';' varspec]
  varspec := varname[subsetSpec]
  varname := escaped variable name
  subsetSpec := '(' fortran-90 arraySpec ')'

  fortran-90 arraySpec := dim | dim ',' dims
  dim := ':' | slice | start ':' end | start ':' end ':' stride
  slice := INTEGER
  start := INTEGER
  stride := INTEGER
  end := INTEGER

• Request parameter values are case-insensitive
• The var parameter is only used for data requests (req=data)
• The variable names are case-sensitive and must be backslash-escaped
• Nonterminals are in lower case, terminals are in upper case, literals are in single quotes.
• Optional components are enclosed between square braces '[' and ']'.

Example service requests are:

• Data request uses Section specification (same as Fortran-90 array notation) to ask for a subset in index space.
• Variable names are case-sensitive and must be backslash-escaped
• Capabilities response indicates which (if any) feature types the dataset can be opened as, along with the URLs of those services. TBD.
• The protobuf messages are defined by thredds\cdm\src\main\java\ucar\nc2\stream\ncStream.proto.

Client implementation

ucar.nc2.stream.CdmRemote is a subclass of NetcdfFile which provides index subsetting on remote CDM datasets. NetcdfDataset.openOrAcquireFile() looks for cdmremote:url prefix, and calls new CdmRemote(url) if found. The url must be an endpoint for a cdmremote service.

• upon opening, req=header is called and the NetcdfFile objects are read from the response
• when data is requested, and the data is not already stored (immediate mode). then req=data is called

Capabilities

<cdmRemoteCapabilities location="http://localhost:8080/thredds/cdmremote/testAll/testdata/2004050300_eta_211.nc">
  <featureDataset url="http://localhost:8080/thredds/cdmremote/testAll/testdata/2004050300_eta_211.nc" />
</cdmRemoteCapabilities>

CdmrFeature Protocol (Point Feature API)

The client forms requests of the form endpoint?query. The possible query parameters are:

  req=( capabilities | data | form | stations)
  accept= (csv | xml | ncstream | netcdf )
  time_start,time_end=time range
  north,south,east,west=bounding box
  var=vars
  stn=stns

where:
  vars := varName | varName[,varName]
  stns := stnName | stnName[,stnName]
  varName := valid variable name
  stnName := valid station name

• request names are case-insensitive
• capabilities lists the possible accept values
• variable names are case-sensitive and must be UTF8-escape encoded
• station names are case-sensitive and must be UTF8-escape encoded
• literals are in bold

PointDataset

Possible requests are:

  req=( capabilities | data | form)

Example service requests are:

• data requests return an unordered list of observation as StructureData
• The protobuf messages are defined by thredds\cdm\src\main\java\ucar\nc2\ft\point\remote\pointStream.proto

StationTimeSeries

Possible query parameters are:

  req=( capabilities | data | form | stations)

Example service requests are:

• a list of stations or a bounding box must always be provided (?)
• data requests return an unordered list of observation as StructureData
• The protobuf messages are defined by thredds\cdm\src\main\java\ucar\nc2\ft\point\remote\pointStream.proto

Questions:

• stationList ncstream vs XML message

Client implementation

ucar.nc2.stream.CdmRemoteFeatureDataset.factory() makes an HTTP request to endpoint+"?req=capabilities" to obtain the GetCapabilities XML document. This document describes what feature type(s) the dataset supports, which determines what are the valid requests, along with the endpoints for those feature types, which may be different than the original endpoint.

FeatureDatasetFactoryManager.open() looks for cdmremote:url prefix, and calls CdmRemoteFeatureDataset.factory(url) if found. ThreddsDataFactory.openFeatureDataset() looks for catalog InvDatasets with featureType.isPointFeatureType() and ServiceType.CdmRemote, and if found, calls CdmRemoteFeatureDataset.factory().

GetCapabilities XML document

This is an ad-hoc format (for now), example:

<?xml version="1.0" encoding="UTF-8"?>
<capabilities location="http://localhost:8080/thredds/cdmremote/idd/metar/ncdecodedLocalHome">
  <featureDataset type="station" url="http://localhost:8080/thredds/cdmremote/idd/metar/ncdecodedLocalHome/station" />
  <variable name="record.air_pressure_at_sea_level" type="float">
    <attribute name="long_name" value="Air pressure at sea level" />
    <attribute name="standard_name" value="air_pressure_at_sea_level" />
    <attribute name="_FillValue" type="float" value="-99999.0" />
    <attribute name="units" value="hectoPascal" />
  </variable>
  <variable name="record.air_temperature" type="float">
    <attribute name="long_name" value="Air temperature at 2 meters" />
    <attribute name="standard_name" value="air_temperature" />
    <attribute name="_FillValue" type="float" value="-99999.0" />
    <attribute name="units" value="Celsius" />
  </variable>
  ...

  <LatLonBox>
    <west>-180.0000</west>
    <east>180.0000</east>
    <south>-90.0000</south>
    <north>82.5199</north>
  </LatLonBox>
  <AcceptList>
    <accept>raw</accept>
    <accept>xml</accept>
    <accept>csv</accept>
    <accept>netcdf</accept>
    <accept>ncstream</accept>
  </AcceptList>
</capabilities>

PointStream Grammer

An pointstream is a sequence of one or more messages:

   pointstream = {message}
   message = stationListMessage | pointFeatureListMessage | errorMessage | endMessage
   stationListMessage = MAGIC_StationList, vlen, PointStreamProto.StationList 
   pointFeatureListMessage = pointFeatureCollectionMessage, {pointFeatureMessage}, (endMessage | errorMessage)


   pointFeatureCollectionMessage = MAGIC_PointFeatureCollection, vlen, PointStreamProto.PointFeatureCollection
   pointFeatureMessage = MAGIC_PointFeature, vlen, PointStreamProto.PointFeature
   endMessage = MAGIC_END
   errorMessage = MAGIC_ERR, vlen, NcStreamProto.Error
   
   vlen = variable length encoded positive integer == length of the following object in bytes


   // 8 byte constants

   MAGIC_StationList            =  0fe, 0xec, 0xce, 0xda 
   MAGIC_PointFeatureCollection =  0xfi, 0xec, 0xce, 0xba 

   MAGIC_PointFeature           =  0xab, 0xec, 0xce, 0xba 
   MAGIC_END                    =  0xed, 0xed, 0xde, 0xde

   MAGIC_ERR                    =  0xab, 0xad, 0xba, 0xda 

The protobuf messages are defined by

• thredds\cdm\src\main\java\ucar\nc2\ft\point\remote\pointStream.proto

Current notes on motherlode dev

• List of cdm datasets: http://thredds.ucar.edu/thredds/idd/newPointObs.html

Nov-11-2010

1) web.xml has a org.springframework.web.servlet.DispatcherServlet cdmRemote with a mapping:

 <servlet-mapping>
   <servlet-name>cdmRemote</servlet-name>
   <url-pattern>/cdmremote/*</url-pattern>
 </servlet-mapping>

2) Temporarily, we have the configuration of the datasets in thredds\tds\src\main\webapp\WEB-INF\cdmRemote-servlet.xml: which configures the collectionController. We will replacce this with pure catalog configuration in 4.2+. This is now being shown in serverStartup.log

 <bean id="collectionManager" class="thredds.server.cdmremote.CollectionManager">
 <property name="collections">
  <list>
   <ref bean="ncMetars" />
   <ref bean="gempakMetars" />
   <ref bean="quickScatWinds" />
   <ref bean="ncMetarsLocal" />
   <ref bean="ncMetarsLocalHome" />
   <ref bean="gempakMetarsLocal" />
   <ref bean="gempakMetarsLocalHome" />
  </list>
 </property>
</bean>

 <bean id="ncMetars" class="thredds.server.cdmremote.CollectionBean">
   <property name="path" value="/idd/metar/ncdecoded"/>
   <property name="spec" value="/data/ldm/pub/decoded/netcdf/surface/metar/Surface_METAR_#yyyyMMdd_HHmm#.nc"/>
   <property name="recheck" value="15 min"/>
   <property name="featureType" value="STATION"/>
   <property name="raw" value="report"/>
   <property name="resolution" value="20 min"/>
 </bean>

etc

This is now being shown in serverStartup.log:

serverStartup: CdmRemoteController collections
   /idd/metar/gempak == /data/ldm/gempak/surface/#yyyyMMdd#_sao.gem
   /idd/bufr/quickScat == /data/ldm/gempak/surface/**/#yyyyMMdd#.gem
   /idd/metar/ncdecodedLocalHome == C:/data/datasets/metars/Surface_METAR_#yyyyMMdd_HHmm#.nc
   /idd/metar/ncdecoded == /data/ldm/pub/decoded/netcdf/surface/metar/Surface_METAR_#yyyyMMdd_HHmm#.nc
   /idd/metar/gempakLocalHome == C:/data/formats/gempak/surface/#yyyyMMdd#_sao.gem
   /idd/metar/gempakLocal == D:/formats/gempak/surface2/#yyyyMMdd#_sao.gem
   /idd/metar/ncdecodedLocal == Q:/station/ldm/metar/Surface_METAR_#yyyyMMdd_HHmm#.nc

So the URLs are

 http://server:port/thredds/cdmremote/path?req
eg:
 http://motherlode.ucar.edu:8081/thredds/cdmremote/idd/metar/gempak
 http://motherlode.ucar.edu:8081/thredds/cdmremote/idd/metar/ncdecoded
 http://motherlode.ucar.edu:8081/thredds/cdmremote/idd/bufr/quickScat
 http://localhost:8080/thredds/cdmremote/idd/metar/ncdecodedLocal
 http://localhost:8080/thredds/cdmremote/idd/metar/ncdecodedLocalHome?req=form

3) The default output is the dataset description: should be getCapabilities ????

FeatureDataset on location= http://localhost:8080/thredds/cdmremote/local/metars/collection
  featureType= STATION
  title= null
  desc= null
  range= start= 2006-03-25 00:00:00Z end= 2006-09-25 00:00:00Z duration= 6.0453 months resolution= null
  start= 2006-03-25 00:00:00Z
  end  = 2006-09-25 00:00:00Z
  bb   =  ll: 90.0S 180.0W+ ur: 82.51N 180.0E
  bb   =  lat= [-90.00,82.51] lon= [-180.00,180.00]
  has netcdf = true
  Data Variables (0)

parseInfo=


FeatureCollection 0 
 STATION Q:/station/ldm/metar/Surface_METAR_#yyyyMMdd_HHmm#.nc
   npts = -1 
     bb =  lat= [-90.00,82.51] lon= [-180.00,180.00] 

4) In ToolsUI, FeatureType/PointFeature Tab:

cdmremote:http://localhost:8080/thredds/cdmremote/local/metars/collection

 
FeatureDataset on location= cdmremote:http://localhost:8080/thredds/cdmremote/local/metars/collection
  featureType= STATION
  title= METAR Data from NWS
  desc= Metar Data from NWS distributed through the Unidata IDD
    realtime datastream. 1 day's worth of data
  range= null
  start= Unknown
  end  = Unknown
  bb   = null
  has netcdf = true
  Attributes
    title = "METAR Data from NWS"
    version = 2.3
    processor = "metar2nc  version v1.2"
    Conventions = "Unidata Observation Dataset v1.0"
    standard_name_vocabulary = "CF-1.0"
    description = "Metar Data from NWS distributed through the Unidata IDD\n    realtime datastream. 1 day\'s worth of data"
    time_coordinate = "time_observation"
    cdm_datatype = "Station"
    stationDimension = "station"
    station_id = "station_id"
    station_description = "station_description"
    latitude_coordinate = "latitude"
    longitude_coordinate = "longitude"
    altitude_coordinate = "altitude"
    geospatial_lat_max = "90.0"
    geospatial_lat_min = "-90.0"
    geospatial_lon_max = "360.0"
    geospatial_lon_min = "0.0"
    time_coverage_start = "1143243900"
    time_coverage_end = "1143330240"
    observationDimension = "recNum"
  Data Variables (0)

parseInfo=


FeatureCollection 0 
 STATION cdmremote:http://localhost:8080/thredds/cdmremote/local/metars/collection
   npts = -1 
     bb =  lat= [-90.00,82.51] lon= [-180.00,180.00] 

Ask for station data, on the server, its rather slow:

CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060325_0000.nc
CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060326_0000.nc
CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060327_0000.nc
CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060328_0000.nc
CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060329_0000.nc
CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060330_0000.nc
CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060331_0000.nc
CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060401_0000.nc
CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060402_0000.nc
CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060629_0000.nc
 sent 481 features to 20V 
 

  (check to see what the strategy is for stations: coomplete scan or ??)

5) cdmremoteCatalog.xml defines datasets for the collections we have working so far.

This document is maintained by Unidata. Send comments to THREDDS support. Last updated: April 2015