THREDDS Configuration Catalogs

TDS Content Directory

Location

All TDS configuration information is stored under the TDS content directory. The location of the directory is controlled by the tds.content.root.path Java system property. The default location is ${tomcat_home}/content/, however we strongly recommend that you explicitly set this value in your Tomcat startup script.

Layout

The content directory is created and populated with default files the first time the TDS is deployed. It is persistent even when a TDS installation is upgraded or re-deployed. All your configuration, modifications, and additions should be made in this directory. Do not place files containing passwords or anything else with security issues in this directory. Typically you will only be adding and modifying catalogs and configuration files.

For now, we will focus on the following subset of the content directory:

• <tds.content.root.path>/thredds/
  • catalog.xml - the main TDS configuration catalog (root catalog for TDS configuration)
  • enhancedCatalog.xml - an example catalog [Note: It is referenced from catalog.xml.]
  • threddsConfig.xml - configuration file for allowing non-default services, configuring caching, etc (more details available here).
  • logs/
    • catalogInit.log - log file for messages generated while reading TDS configuration catalogs during TDS initialization and reinitialization.
    • threddsServlet.log - log messages about individual TDS requests, including any error messages. Useful for debugging problems.
  • cache/ - various cache directories
    • agg/
    • cdm/
    • collection/
    • ehcache/
    • ncss/
    • wcs/

TDS Root Catalog

The main TDS configuration catalog is at <tds.content.root.path>/thredds/catalog.xml. We ship a simple test catalog:

<?xml version="1.0" encoding="UTF-8"?>
<catalog name="THREDDS Server Default Catalog : You must change this to fit your server!"
    xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
    xmlns:xlink="http://www.w3.org/1999/xlink"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0
      https://schemas.unidata.ucar.edu/thredds/InvCatalog.1.0.6.xsd">

  <service name="all" base="" serviceType="compound">
    <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />
    <service name="dap4" serviceType="DAP4" base="/thredds/dap4/" />
    <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
    <!--service name="wcs" serviceType="WCS" base="/thredds/wcs/" /-->
    <!--service name="wms" serviceType="WMS" base="/thredds/wms/" /-->
    <!--service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/" /-->
  </service>
  
  <service name="dap" base="" serviceType="compound">
    <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />
    <service name="dap4" serviceType="DAP4" base="/thredds/dap4/" />
  </service>
  
  <datasetRoot path="test" location="content/testdata/" />
  
  <dataset name="Test Single Dataset" ID="testDataset"
      serviceName="dap"  urlPath="test/testData.nc" dataType="Grid"/>
  
  <dataset name="Test Single Dataset 2" ID="testDataset2"
      serviceName="odap" urlPath="test/testData2.grib2" dataType="Grid"/>

  <datasetScan name="Test all files in a directory" ID="testDatasetScan"
      path="testAll" location="content/testdata">
    <metadata inherited="true">
      <serviceName>all</serviceName>
      <dataType>Grid</dataType>
    </metadata>
    
    <filter>
      <include wildcard="*eta_211.nc"/>
      <include wildcard="testgrid*.nc"/>
    </filter>
  </datasetScan>
  
  <catalogRef xlink:title="Test Enhanced Catalog" xlink:href="enhancedCatalog.xml" name=""/>
</catalog>

When the TDS starts, this root configuration catalog is read, as are all catalogs in the catalog tree defined by catalogRef elements. The resulting tree of catalogs are used as the top-level catalogs served by the TDS. In the case of our test catalog, the tree looks like:

catalog.xml
    |
    |-- enhancedCatalog.xml

The tree of configuration catalogs can be as deeply nested as desired.

Additional Root Catalogs

Additional root configuration catalogs can be defined in

<tds.content.root.path>/thredds/threddsConfig.xml 

file. For instance, to add a test catalog add the following line:

<catalogRoot>myTestCatalog.xml</catalogRoot>

Each additional root configuration catalog can be the root of another tree of configuration catalogs.

TDS Configuration Catalogs

TDS configuration catalogs are like THREDDS Client catalogs with extensions. They contain information detailing the datasets the TDS will serve and what services will be available for each dataset:

• The datasetRoot and datasetScan elements are extensions that:
  • provide mappings between incoming URL requests and directories on disk; and
  • are used in the detailing of the datasets the TDS will serve.
• Available services are indicated in the normal THREDDS catalog manner with service name references.

The TDS configuration catalogs represent the top-level catalogs served by the TDS:

• The configuration information is only needed by the server.
• The client view of the catalogs does not contain any configuration information.

datasetRoot Element

Each datasetRoot element defines a single mapping between a URL base path and a directory. The URL base path so defined can then be used in accessible datasets for files under the mapped directory. For instance, you have a directory /machine/tds/data/my/test that contains:

afile.nc
testData.nc
junk.nc
grib/
  data1.grib2
  data2.grib2

You can serve the "testData.nc" and "grib1/data1.grib2" files with the following:

...
<service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />

<datasetRoot path="my/test" location="/machine/tds/data/my/test" />    <!-- 1 -->

<dataset name="A Test Dataset" ID="testDataset" urlPath="my/test/testData.nc" >    <!-- 2 -->
  <serviceName>odap</serviceName>
</dataset>

<dataset name="A Test Dataset 2" ID="testDataset2" urlPath="my/test/grib/data1.grib2" >    <!-- 3 -->
  <serviceName>odap</serviceName>
</dataset>
...

The datasetRoot element above (1) maps the "my/test" path to the "/machine/tds/data/my/test/" directory. The URLs to access the datasets (2 & 3) are

http://hostname:port/thredds/dodsC/my/test/testData.nc
http://hostname:port/thredds/dodsC/my/test/grib/data1.grib2

When the server receives a request for one of the above URLs, it uses the URL path to look for a matching dataset root. In this case it finds the mapping provided by the datasetRoot element above and looks in the /machine/tds/data/my/test directory for the file.

The client catalog that results from this catalog is the same as the above without the datasetRoot element.

datasetScan Element

Each datasetScan element also defines a single mapping between a URL base path and a directory. Unlike the datasetRoot element which works with dataset elements to define the datasets served, the datasetScan element will automatically serve some or all of the datasets found in the mapped directory. So, all the files in the above listing could be served with the following:

...
<service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />

<datasetScan name="Test all files in a directory" ID="testDatasetScan"
    path="my/test/all" location="/machine/tds/data/my/test">
  <metadata inherited="true">
    <serviceName>odap</serviceName>
  </metadata>
</datasetScan>
...

In the client view of a configuration catalog, datasetScan elements are converted to catalogRef elements. So, the resulting client view of this catalog looks like:

...
<service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />

<catalogRef xlink:title="Test all files in a directory" ID="testDatasetScan"
    xlink:href="/thredds/catalog/my/test/all/catalog.xml" name="" />
...

The generation of the catalog referenced by the catalogRef element is deferred until a request is made for that catalog. When the catalog is requested the location directory is scanned, directories are represented as catalogRef elements and files are represented as dataset elements. The scanning of each subdirectory is deferred until a request is made for the corresponding catalog. The catalog referenced above would look like:

...
<service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />

<dataset name="Test all files in a directory" ID="testDatasetScan" >
  <metadata inherited="true">
          <serviceName>odap</serviceName>
  </metadata>

  <dataset name="afile.nc" ID="testDatasetScan/afile.nc" urlPath="my/test/all/afile.nc" />
  <dataset name="testData.nc" ID="testDatasetScan/testData.nc"
      urlPath="my/test/all/testData.nc" />
  <dataset name="junk.nc" ID="testDatasetScan/junk.nc" urlPath="my/test/all/junk.nc" />

  <catalogRef xlink:title="grib" ID="testDatasetScan/grib" name=""
      xlink:href="/thredds/catalog/my/test/all/grib/catalog.xml" />
</dataset>
...

Note: The datasetScan element provides ways for limiting the datasets that are included in the scan, changing the names of datasets, sorting datasets, etc. We will go into more detail on this later.