Dataset Subsetting

Resources For This Session:

Test Servers used to test NcSS, CDMRemote, Protocols:

1. Thredds Test Server https://thredds-test.unidata.ucar.edu/thredds/catalog.html

2. Test Servers used to test DAP2 and DAP4 Protocols:
   • DAP2 Test Server http://remotetest.unidata.ucar.edu/dts

   • DAP4 Test Server http://remotetest.unidata.ucar.edu/d4ts

3. Thredds Test Server providing DAP4 Services http://remotetest.unidata.ucar.edu/thredds

What this presentation covers

• Client Server Architecture

• What is Dataset Subsetting?

• The NetCDF Subset Service (NcSS) Protocol

• The DAP2 and DAP4 Protocol

• The CDM Remote Protocol

Client Server Architecture

• We all routinely use the Client Server Architecture when we use a web browser.
  

• Key Elements:
  • Client Program (Web Browser)
  • Server Program (Web Page Server)
  • Protocol (HTTP)
  • Message Requests and Replies (HTML)

• The Thredds Data Server is a Server in a client-server architecture. It differs from a web browser in supporting a number of protocols and associated message formats.

  

What is Dataset Subsetting?

• Why not just use FTP? Because Downloading a whole dataset can take a long time and can use significant memory or disk locally.

• Better is to be able to ask the server to provide a subset (a piece) of the dataset.

  • Because that is all the client needs

  • Or because we can process the dataset locally piece by piece.

• Other names for subsetting: query, constraint.

• Many of the protocols supported by the thredds server can provide subsetting; however, there is no standard, so the form of the subset request differs between protocols.

• What can be subsetted is determined by the "data model" presented to the user:
  1. variable arrays indexed by integer dimensions
  2. variables indexed by (latitude, longitude)

• So Typical elements that can be subsetted:
  • Subset of variables (i.e. give me temperature, velocity)

  • Subset of the data within a one or more variables:
    
    • index-based: [first:stride:last] or (first:stride:last+1)
      e.g var[0:1:10][3:2:9]
      Sometimes (e.g. stride and last are reversed.

    • coordinate-based:
      e.g. [43e:45e][10s:85n]

The NetCDF Subset Service (NcSS)

• NcSS uses HTTP forms to send a request and get results back in a variety of formats: XML, netcdf-3, netcdf-4, XML, etc.

• Mostly uses coordinate subsets, but can support others like stations.

• The request is encoded as a url query specifying the variables and e.g. a bounding box.

• Examples:

  http://thredds.ucar.edu/thredds/catalog/grib/NCEP/DGEX/CONUS_12km/catalog.html?dataset=grib/NCEP/DGEX/CONUS_12km/Best

• Reference:
  /tds/4.6/adminguide/reference/NetcdfSubsetServiceReference.html

OPenDAP and DAP2 and DAP4

• DAP version 2 (aka DAP2) is a widely supported protocol and standard data format for accessing remote data

• The DAP2 protocol was expressly designed to serve as intermediate format for accessing a wide variety of data sources

• The new DAP version 4 protocol (DAP4) provides a richer data model and a more powerful constraint language (aka subset language) than DAP2.

• DAP2 and DAP4 (currently) are index based.

• Examples:

  Sources of test data

  http://remotetest.unidata.ucar.edu/dts

  http://remotetest.unidata.ucar.edu/d4ts

  Request Single variable

  BASE: http://remotetest.unidata.ucar.edu/thredds/dodsC/testdods/rtofs.nc.dds

  SUBSET: http://remotetest.unidata.ucar.edu/thredds/dodsC/testdods/rtofs.nc.asc?X

  SUBSET: http://remotetest.unidata.ucar.edu/thredds/dodsC/testdods/rtofs.nc.asc?X[0:10]

  Form-Based Access

  FORM: http://remotetest.unidata.ucar.edu/thredds/dodsC/testdods/rtofs.nc.html

• DAP2 Reference:
  http://www.opendap.org/pdf/ESE-RFC-004v1.2.pdf

• DAP4 Reference:
  http://docs.opendap.org/index.php/DAP4:_Specification_Volume_1

CDMRemote

• CDMRemote is a protocol developed by Unidata to provide faster access to data than DAP2.

• Uses the Google protobuf serialization technology to encode/decode metadata. Data is serialized using a simpler protocol.

• Supports both coordinate queries and index queries.

• The request is encoded as a url query specifying various kinds of info. That is, it is of the form
  <url>?req=... or <url>/<kind>?req=...

• Possible kinds include: point, station
  • No kind => index based subsetting
  • point => coordinate based subsetting
  • station => coordinate based subsetting

• Two important "req"s are:

  • req=form —return the dataset's metadata in netcdf cdl format

  • req=data —return extra information about the dataset

• Some requests require additional fields, e.g:
  • req=data requires an additional query element of the form
    var=<list of vars with subsetting>

• Examples:

  BASE: http://thredds.ucar.edu/thredds/cdmremote/grib/NCEP/DGEX/CONUS_12km/Best?req=form

  SUBSET: http://thredds.ucar.edu/thredds/cdmremote/grib/NCEP/DGEX/CONUS_12km/Best?req=data&var=x(0:10:3);isobaric2"

  SUBSET: <url>.../point?req=data&north=40.3&south=22.8&east=-80&west=-105

  SUBSET: <url>.../station?req=data&stn=KDEN,KLOG,MOAS

• Reference: CdmRemote reference.