DAP4 Protocol Support

Table of Contents

• DAP4 Introduction
• Accessing Data Using the DAP4 Prototocol
• Defined Client Parameters
• DAP4 Fetch Process
• Notes on Debugging DAP4 Access
  • HTTP Configuration.
• Point of Contact

DAP4 Introduction

Beginning with netCDF version 4.5.0, optional support is provided for accessing data from servers supporting the DAP4 protocol.

DAP4 support is enabled if the –enable-dap option is used with './configure'. If DAP4 support is enabled, then a usable version of libcurl must be specified using the LDFLAGS environment variable (similar to the way that the HDF5 libraries are referenced). Refer to the installation manual for details. By default DAP4 support is enabled if libcurl is found. DAP4 support can be disabled using the –disable-dap.

DAP4 uses a data model that is, by design, similar to, but – for historical reasons – not identical with, the netCDF-Enhanced (aka netcdf-4) model. Generically, the DAP4 data model is encoded in XML document called a DMR. For detailed information about the DAP4 DMR, refer to the DAP4 specification Volume 1: http://docs.opendap.org/index.php/DAP4:_Specification_Volume_1

Accessing Data Using the DAP4 Prototocol

In order to access a DAP4 data source through the netCDF API, the file name normally used is replaced with a URL with a specific format. The URL is composed of three parts.

• URL - this is a standard form URL with specific markers to indicate that it refers to a DAP4 encoded dataset. The markers are of the form "dap4", "mode=dap4", or "/thredds/dap4". The following examples show how they are specified. Note that the "/thredds/dap4" case will only work when accessing a Thredds-based server.
  • [dap4]http://remotetest.unidata.ucar.edu/d4ts/test.01
  • [mode=dap4]http://remotetest.unidata.ucar.edu/d4ts/test.01
  • http://remotetest.unidata.ucar.edu/d4ts/test.01#dap4
  • http://remotetest.unidata.ucar.edu/d4ts/test.01#mode=dap4
  • http://thredds.ucar.edu/thredds/dap4/...
• Constraints - these are suffixed to the URL and take the form “?dap4.ce=<expression>”. The form of the constraint expression is somewhat complicated, and the specification should be consulted.
• Client parameters - these may be specified in either of two ways. The older, deprecated form prefixes text to the front of the url and is of the the general form [<name>] or [<name>=value]. Examples include [show=fetch] or [noprefetch]. The newer, preferred form prefixes the parameters to the end of the url using the semi-standard '#' format: e.g. http://....#show=fetch&noprefetch.

It is possible to see what the translation does to a particular DAP4 data source in two steps. First, one can examine the DMR source through a web browser and then second, one can examine the translation using the "ncdump -h" command to see the corresponding netCDF-4 representation.

For example, if a web browser is given the following (fictional) URL, it will return the DMR for the specified dataset (in XML format).

http://remotetest.unidata.ucar.edu/d4ts/test.01.dmr.xml#dap4

By using the following ncdump command, it is possible to see the equivalent netCDF-4 translation.

ncdump -h '[dap4]http://remotetest.unidata.ucar.edu/d4ts/test.01'

Defined Client Parameters

Currently, a limited set of client parameters is recognized. Parameters not listed here are ignored, but no error is signalled.

Parameter Name Legal Values Semantics

• "log" | "log=<file>" - Turn on logging and send the log output to the specified file. If no file is specified, then output is sent to standard error.
• "show=fetch" - This parameter causes the netCDF code to log a copy of the complete url for every HTTP get request. If logging is enabled, then this can be helpful in checking to see the access behavior of the netCDF code.
• "translate=nc4" - This parameter causes the netCDF code to look for specially named elements in the DMR XML in order to achieve a better translation of the DAP4 meta-data to NetCDF enhanced metadata.
• "opaquesize=<integer>" - This parameter causes the netCDF code to convert DAP4 variable size OPAQUE objects to netcdf-4 fixed size objects and forces all of them to be of the size specified.
• "fillmismatch" - Unfortunately, a number of servers sometimes fail to make sure that the type of the "_FillValue" attribute of a variable is the same as the type of the containing variable. Setting this tag caused the netcdf translation to attempt to fix this mismatch. If not set, then an error will occur. This defaults to allowing mismatch.

DAP4 Fetch Process

When a DAP4 client is given a URL, it operates as follows.

1. The URL is used by client program to fetch the DMR. This avoids reading possibly large amounts of data until absolutely necessary. It also means that, for example, "ncdump -h" will operate reasonably fast.
2. The first time a client program actually requests data – using the "nc_get_vara()" procedure, for example – the data is fetched from the server and stored locally.
3. Subsequent requests for data will extract it from the data retrieved in step 2, so no new server requests are needed.

One must be careful about step 2. As a rule, the URL should specify a constraint. Otherwise, the initial fetch will fetch all of the data associated with the dataset. This can be quite large. This means that it will be slow. Additionally, many servers limit how much data can be returned and attempts to read more than that limit will result in an error.

Notes on Debugging DAP4 Access

The DAP4 support has a logging facility. Turning on this logging can sometimes give important information. Logging can be enabled by using the client parameter "log" or "log=filename", where the first case will send log output to standard error and the second will send log output to the specified file.

Users should also be aware that if one is accessing data over an NFS mount, one may see some .nfsxxxxx files; those can be ignored.

HTTP Configuration.

Limited support for configuring the http connection is provided via parameters in the “.daprc” configuration file (aka ".dodsrc"). The relevant .daprc file is located by first looking in the current working directory, and if not found, then looking in the directory specified by the “$HOME” environment variable.

Entries in the .daprc file are of the form:

['['<url>']']<key>=<value>

That is, it consists of a key name and value pair and optionally preceded by a url enclosed in square brackets.

For given KEY and URL strings, the value chosen is as follows:

If URL is null, then look for the .daprc entry that has no url prefix and whose key is same as the KEY for which we are looking.

If the URL is not null, then look for all the .daprc entries that have a url, URL1, say, and for which URL1 has the same host and port as URL. All parts of the url's except host and port are ignored. For example, if URL = http//x.y/a, then it will match entries of the form [http//x.y/a]KEY=VALUE or [http//x.y/b]KEY=VALUE. It will not match an entry of the form _[http//x.y:8080]KEY=VALUE because the second has a port number (8080) different than the URL. Finally from the set so constructed, choose the first matching entry.

Currently, the supported set of keys (with descriptions) are as follows.

- HTTP.VERBOSE
        Type: boolean ("1"/"0")
        Description: Produce verbose output, especially using SSL.
        Related CURL Flags: CURLOPT_VERBOSE
 
- HTTP.DEFLATE
        Type: boolean ("1"/"0")
        Description: Allow use of compression by the server.
        Related CURL Flags: CURLOPT_ENCODING
 
- HTTP.COOKIEJAR
        Type: String representing file path
        Description: Specify the name of file into which to store cookies. Defaults to in-memory storage.
        Related CURL Flags:CURLOPT_COOKIEJAR
 
- HTTP.CREDENTIALS.USER
        Type: String representing user name
        Description: Specify the user name for Digest and Basic authentication.
        Related CURL Flags:
 
- HTTP.CREDENTIALS.PASSWORD
        Type: String representing password
        Type: boolean ("1"/"0")
        Description: Specify the password for Digest and Basic authentication.
        Related CURL Flags:
 
- HTTP.SSL.CERTIFICATE
        Type: String representing file path
        Description: Path to a file containing a PEM cerficate.
        Related CURL Flags: CURLOPT_CERT
 
- HTTP.SSL.KEY
        Type: String representing file path
        Description: Same as HTTP.SSL.CERTIFICATE, and should usually have the same value.
        Related CURL Flags: CURLOPT_SSLKEY
 
- HTTP.SSL.KEYPASSWORD
        Type: String representing password
        Description: Password for accessing the HTTP.SSL.KEY/HTTP.SSL.CERTIFICATE
        Related CURL Flags: CURLOPT_KEYPASSWORD
 
- HTTP.SSL.CAPATH
        Type: String representing directory
        Description: Path to a directory containing trusted certificates for validating server certificates.
        Related CURL Flags: CURLOPT_CAPATH
 
- HTTP.SSL.VALIDATE
        Type: boolean ("1"/"0")
        Description: Cause the client to verify the server's presented certificate.
        Related CURL Flags: CURLOPT_SSL_VERIFYPEER, CURLOPT_SSL_VERIFYHOST
 
- HTTP.TIMEOUT
        Type: String ("dddddd")
        Description: Specify the maximum time in seconds that you allow the http transfer operation to take.
        Related CURL Flags: CURLOPT_TIMEOUT, CURLOPT_NOSIGNAL, CURLOPT_CONNECTIONTIMEOUT
 
- HTTP.PROXY_SERVER
        Type: String representing url to access the proxy: (e.g.http://[username:password@]host[:port])
        Description: Specify the needed information for accessing a proxy.
        Related CURL Flags: CURLOPT_PROXY, CURLOPT_PROXYHOST, CURLOPT_PROXYUSERPWD
 
- HTTP.READ.BUFFERSIZE
        Type: String ("dddddd")
        Description: Specify the the internal buffer size for curl reads.
        Related CURL Flags: CURLOPT_BUFFERSIZE, CURL_MAX_WRITE_SIZE (16kB),
                            CURL_MAX_READ_SIZE (512kB).
 
- HTTP.KEEPALIVE
        Type: String ("on|n/m")
        Description: Specify that TCP KEEPALIVE should be enabled and that the associated idle wait time is n and that the associated repeat interval is m. If the value is of the form is the string "on", then turn on keepalive, but do not set idle or interval.
        Related CURL Flags: CURLOPT_TCP_KEEPALIVE, CURLOPT_TCP_KEEPIDLE,
                            CURLOPT_TCP_KEEPINTVL.
 
- HTTP.CONNECTIONTIMEOUT
        Type: String ("dddddd")
        Description: Specify the maximum time in seconds that you allow the http connection to complete.
        Related CURL Flags: CURLOPT_TIMEOUT, CURLOPT_NOSIGNAL

The related curl flags line indicates the curl flags modified by this key. See the libcurl documentation of the curl_easy_setopt() function for more detail (http://curl.haxx.se/libcurl/c/curl_easy_setopt.html).

For servers that require client authentication, as a rule, the following .daprc entries should be specified.

HTTP.SSL.VALIDATE
HTTP.COOKIEJAR
HTTP.SSL.CERTIFICATE
HTTP.SSL.KEY
HTTP.SSL.CAPATH

Additionally, the HTTP.SSL.CERTIFICATE and HTTP.SSL.KEY entries should have same value, which is the file path for a certificate. The HTTP.SSL.CAPATH entry should be the path to a directory containing validation "certificates".

Point of Contact

Author: Dennis Heimbigner
Email: dmh at ucar dot edu
Initial Version: 6/5/2017
Last Revised: 06/18/2020