CDMRemote

CDM Remote Web Service

CDM Remote is a web service implemented in the CDM library (client) and TDS (server), providing remote access to CDM datasets, using ncstream as the on-the-wire protocol. It provides access at the NetcdfFile and FeatureDataset levels of the CDM API, so there are two levels of services:

1. CdmRemote provides index subsetting on remote CDM datasets, with the same functionality that NetcdfFile provides for CDM local datasets and DODSNetcdfFile provides for remote OPeNDAP datasets. CdmRemote_ supports the full CDM data model.
2. CDM Remote Feature_ provides coordinate subsetting on remote CDM Feature Datasets, with similar functionality to WCS and Unidata’s experimental NetCDF Subset Service (NCSS). It provides a remote API for Point and StationTimeSeries Feature Datasets.

The CDM Remote services and protocol are experimental and should not be used outside of the CDM stack for now.

CdmRemote Protocol (Data Access and Coordinate Systems)

The client forms requests of the form endpoint?query. The possible query parameters are:

  req=( CDL | NcML | capabilities | header | data)
  var=vars
where:
  vars := varspec | varspec[';' varspec]
  varspec := varname[subsetSpec]
  varname := escaped variable name
  subsetSpec := '(' fortran-90 arraySpec ')'

  fortran-90 arraySpec := dim | dim ',' dims
  dim := ':' | slice | start ':' end | start ':' end ':' stride
  slice := INTEGER
  start := INTEGER
  stride := INTEGER
  end := INTEGER

• Request parameter values are case-insensitive
• The var parameter is only used for data requests (req=data)
• The variable names are case-sensitive and must be backslash-escaped
• Nonterminals are in lower case, terminals are in upper case, literals are in single quotes.
• Optional components are enclosed between square braces ‘[’ and ‘]’.

Example service requests are:

• Data request uses Section specification (same as Fortran-90 array notation) to ask for a subset in index space.
• Variable names are case-sensitive and must be backslash-escaped
• Capabilities response indicates which (if any) feature types the dataset can be opened as, along with the URLs of those services. TBD.
• The protobuf messages are defined by thredds\cdm\src\main\java\ucar\nc2\stream\ncStream.proto.

Client implementation

ucar.nc2.stream.CdmRemote is a subclass of NetcdfFile which provides index subsetting on remote CDM datasets. NetcdfDataset.openOrAcquireFile() looks for cdmremote:url prefix, and calls new CdmRemote(url) if found. The url must be an endpoint for a cdmremote service.

• upon opening, req=header is called and the NetcdfFile objects are read from the response
• when data is requested, and the data is not already stored (immediate mode). then req=data is called

Capabilities

<cdmRemoteCapabilities location="http://localhost:8080/thredds/cdmremote/testAll/testdata/2004050300_eta_211.nc">
  <featureDataset url="http://localhost:8080/thredds/cdmremote/testAll/testdata/2004050300_eta_211.nc" />
</cdmRemoteCapabilities>

CdmrFeature Protocol (Point Feature API)

The client forms requests of the form endpoint?query. The possible query parameters are:

  req=( capabilities | data | form | stations)
  accept= (csv | xml | ncstream | netcdf )
  time_start,time_end=time range
  north,south,east,west=bounding box
  var=vars
  stn=stns

where:
  vars := varName | varName[,varName]
  stns := stnName | stnName[,stnName]
  varName := valid variable name
  stnName := valid station name

• request names are case-insensitive
• capabilities lists the possible accept values
• variable names are case-sensitive and must be UTF8-escape encoded
• station names are case-sensitive and must be UTF8-escape encoded
• literals are in bold

PointDataset

Possible requests are:

  req=( capabilities | data | form)

Example service requests are:

• data requests return an unordered list of observation as StructureData
• The protobuf messages are defined by thredds\cdm\src\main\java\ucar\nc2\ft\point\remote\pointStream.proto

StationTimeSeries

Possible query parameters are:

  req=( capabilities | data | form | stations)

Example service requests are:

• a list of stations or a bounding box must always be provided (?)
• data requests return an unordered list of observation as StructureData
• The protobuf messages are defined by thredds\cdm\src\main\java\ucar\nc2\ft\point\remote\pointStream.proto

Questions:

• stationList ncstream vs XML message

Client implementation

ucar.nc2.stream.CdmRemoteFeatureDataset.factory() makes an HTTP request to endpoint+”?req=capabilities” to obtain the GetCapabilities XML document. This document describes what feature type(s) the dataset supports, which determines what are the valid requests, along with the endpoints for those feature types, which may be different than the original endpoint.

FeatureDatasetFactoryManager.open() looks for cdmremote:url prefix, and calls -CdmRemoteFeatureDataset.factory(url)_ if found. ThreddsDataFactory.openFeatureDataset() looks for catalog InvDatasets with featureType.isPointFeatureType() and ServiceType.CdmRemote, and if found, calls CdmRemoteFeatureDataset.factory().

GetCapabilities XML document

This is an ad-hoc format (for now), example:

<?xml version="1.0" encoding="UTF-8"?>
<capabilities location="http://localhost:8080/thredds/cdmremote/idd/metar/ncdecodedLocalHome">
  <featureDataset type="station" url="http://localhost:8080/thredds/cdmremote/idd/metar/ncdecodedLocalHome/station" />
  <variable name="record.air_pressure_at_sea_level" type="float">
    <attribute name="long_name" value="Air pressure at sea level" />
    <attribute name="standard_name" value="air_pressure_at_sea_level" />
    <attribute name="_FillValue" type="float" value="-99999.0" />
    <attribute name="units" value="hectoPascal" />
  </variable>
  <variable name="record.air_temperature" type="float">
    <attribute name="long_name" value="Air temperature at 2 meters" />
    <attribute name="standard_name" value="air_temperature" />
    <attribute name="_FillValue" type="float" value="-99999.0" />
    <attribute name="units" value="Celsius" />
  </variable>
  ...

  <LatLonBox>
    <west>-180.0000</west>
    <east>180.0000</east>
    <south>-90.0000</south>
    <north>82.5199</north>
  </LatLonBox>
  <AcceptList>
    <accept>raw</accept>
    <accept>xml</accept>
    <accept>csv</accept>
    <accept>netcdf</accept>
    <accept>ncstream</accept>
  </AcceptList>
</capabilities>

PointStream Grammer

An pointstream is a sequence of one or more messages:

   pointstream = {message}
   message = stationListMessage | pointFeatureListMessage | errorMessage | endMessage
   stationListMessage = MAGIC_StationList, vlen, PointStreamProto.StationList 
   pointFeatureListMessage = pointFeatureCollectionMessage, {pointFeatureMessage}, (endMessage | errorMessage)


   pointFeatureCollectionMessage = MAGIC_PointFeatureCollection, vlen, PointStreamProto.PointFeatureCollection
   pointFeatureMessage = MAGIC_PointFeature, vlen, PointStreamProto.PointFeature
   endMessage = MAGIC_END
   errorMessage = MAGIC_ERR, vlen, NcStreamProto.Error
   
   vlen = variable length encoded positive integer == length of the following object in bytes


   // 8 byte constants

   MAGIC_StationList            =  0fe, 0xec, 0xce, 0xda 
   MAGIC_PointFeatureCollection =  0xfi, 0xec, 0xce, 0xba 

   MAGIC_PointFeature           =  0xab, 0xec, 0xce, 0xba 
   MAGIC_END                    =  0xed, 0xed, 0xde, 0xde

   MAGIC_ERR                    =  0xab, 0xad, 0xba, 0xda 

The protobuf messages are defined by

• thredds\cdm\src\main\java\ucar\nc2\ft\point\remote\pointStream.proto