TDS Configuration File Reference (`threddsConfig.xml`)

TDS Administrator's Guide TDS Configuration File Reference (`threddsConfig.xml`)

The TDS configuration file ${tds.content.root.path}/thredds/threddsConfig.xml allows the TDS administrator to control the behavior of the TDS.

Note: As of 5.0, if this file is missing, then default settings will be used.

We strongly encourage you to fill in the Server Information fields, but all other fields are optional, and default settings are used whenever you don’t override. Generally, its good practice to only include the elements you want to change from the defaults.

When changing threddsConfig.xml, you must restart the thredds webapp (e.g., through the Tomcat manager application) for them to take effect.

Note: View an annotated example of threddsConfig.xml.

TDS Global Configuration Options

Server Information

In the serverInformation element, you provide basic information about your server including contact information, and information about the group hosting the server.

<serverInformation>
    <name>Initial TDS Installation</name>
    <logoUrl>/thredds/threddsIcon.png</logoUrl>
    <logoAltText>Initial TDS Installation</logoAltText>

    <abstract>Scientific Data</abstract>
    <keywords>meteorology, atmosphere, climate, ocean, earth science</keywords>

    <contact>
      <name>Support</name>
      <organization>My Group</organization>
      <email>support@my.group</email>
      <phone></phone>
    </contact>
    <hostInstitution>
      <name>My Group</name>
      <webSite>http://www.my.group/</webSite>
      <logoUrl>myGroup.gif</logoUrl>
      <logoAltText>My Group</logoAltText>
    </hostInstitution>
</serverInformation>

The information provided in the serverInformation element is used in:

the headers of all generated HTML pages (they contain the names and logos of the server and host institution)
the Server section of the WMS GetCapabilities response
the server information documents (see below)
the Server section of the WCS GetCapabilities response
all generated THREDDS catalogs that don’t override this information

The best way to use your own logo is to put it in the ${tds.content.root.path}/thredds/public/ directory, and specify it in serverInformation as /thredds/<name>, e.g.:

<logoUrl>/thredds/yourIcon.gif</logoUrl>

Server Information Documents

The TDS supports the following requests for server information:

/thredds/serverInfo.html (sample response)
/thredds/serverInfo.xml (sample response)
/thredds/serverVersion.txt (sample response)

Generated HTML Pages

In the htmlSetup element, you can configure which CSS documents are used in the HTML pages generated by the TDS. Default CSS files are provided, and should not be modified. Instead, these can be overridden by placing the appropriate CSS files in the ${tds.content.root.path}/thredds/public/ directory and pointing to them here:

<htmlSetup>
    <standardCssUrl>standard.css</standardCssUrl> <!--1-->
    <catalogCssUrl>catalog.css</catalogCssUrl>    <!--2-->
    <datasetCssUrl>dataset.css</datasetCssUrl>    <!--3-->
    <openDapCssUrl>tdsDap.css</openDapCssUrl>     <!--4-->
    <googleTrackingCode>XX-XXXXXXXX-X</googleTrackingCode>  <!--5-->
    <generateDatasetJsonLD>false</generateDatasetJsonLD>    <!--6-->
</htmlSetup>

The CSS used in all TDS generate html pages (except the OPeNDAP form).
The CSS used in TDS catalogs pages
The CSS used in TDS Dataset catalogs pages
The CSS used in the OPeNDAP form.
Google Analytics Tracking Code (GATC) enables tracking catalog use. Obtain the GATC from Google and enter it here to enable this feature.
If set to true, schema.org/Dataset objects will be encoded using json-ld and embedded into the <head> element of the generated dataset HTML pages.

Controlling THREDDS Catalog Output

<catalogWriting>
  <useBytesForDataSize>false<useBytesForDataSize/>
</catalogWriting>

if true, in a TDS catalog, output the Data Size with exact byte count. By default, it will output 4 significant digits, choosing units appropriately.

Extra Catalog Roots

<catalogRoot>tempCatalog.xml</catalogRoot>
<catalogRoot>idd/catalog.xml</catalogRoot>
<catalogRoot>catgen/subdir/enhancedCatalog-1.0.xml</catalogRoot>

These elements name catalog roots that are not referenced from your default catalog root ( ${tds.content.root.path}/thredds/catalog.xml). On start up, the TDS reads the default catalog root and any root catalogs you list here, plus any catalogs that are referenced by them in a catalogRef. Data roots and other needed information are found and cached. All the catalogs found in this way are called static catalogs, and all static catalogs must live within the ${tds.content.root.path}/thredds directory tree.

Adding Viewers

<Viewer>my.package.MyViewer</Viewer>

You can place a link to your own Viewer on the TDS HTML page, by loading a Viewers at runtime. This line is needed in the config file only if you are writing your own Java class.

Adding Dataset Sources

<datasetSource>my.package.DatasetSourceImpl</datasetSource>

You can add a DataSource - essentially an IOSP with access to Servlet request parameters, by loading a Dataset Source Plugin at runtime.

Checking For Updates

<TdsUpdateConfig>
  <logVersionInfo>true</logVersionInfo>
</TdsUpdateConfig>

The TdsUpdateConfig element controls if the TDS checks with Unidata regarding possible updates. The default (true) is for the TDS to check for the current stable and development release versions, and to log that information in the TDS serverStartup.log file as INFO entries. If you do not want the TDS to check for this on startup, set this to false.

Controlling Data Services

Remote Catalog Service For Catalogs

Catalog services are available by default for catalogs served by the local TDS. However, for remote catalogs, these services must be explicitly enabled in threddsConfig.xml:

<CatalogServices>
  <allowRemote>true</allowRemote>
</CatalogServices>

OPeNDAP Service

<Opendap>
  <ascLimit>50</ascLimit>
  <binLimit>500</binLimit>
  <serverVersion>opendap/3.7</serverVersion>
</Opendap>

This controls the OPeNDAP data service. Because its easy for a user to inadvertently request very large amounts of data, the TDS limits the size of the data response. In our experience legitimate requests ask for subset sizes that are well below the defaults.

ascLimit: maximum size of an ascii data request , in Megabytes. Default 50 Mbytes.
binLimit: maximum size of a binary data request , in Megabytes. Default is 500 Mbytes.
serverVersion: this is the String returned by the OPeNDAP getVersion request, and placed into the XDOS-Server HTTP Header on all OPeNDAP responses.

WCS Service

The OGC WCS service provided as part of the TDS is described in more detail here. By default this service is enabled, and can be disabled by including the following in the threddsConfig.xml file:

<WCS>
  <allow>false</allow>
</WCS>

The following shows all the configuration options available in the WCS section of the threddsConfig.xml file with the default values shown:

<WCS>
  <allow>true</allow>
  <dir>(see the note below)</dir>
  <scour>15 min</scour>
  <maxAge>30 min</maxAge>
</WCS>

We recommend that you include in the threddsConfig.xml file only the options you want to change. Here is the description of the various options:

allow: a value of false disables the WCS service.
dir: the working directory where generated files are cached before being sent to the client (see choosing a cache directory). If not otherwise set, the TDS will use the ${tds.content.root.path}/thredds/cache/wcs/ directory. We recommend that you do not specify a WCS.dir element, and use the default.
scour: how often to scour the working directory, to delete files that were not successfully downloaded.
maxAge: how long to leave the files in the working directory while the download is occurring. The files are deleted after a successful download. Do not set to <= 0.

WMS Service

The OGC WMS service provided as part of the TDS is described in more detail here (TDS Web Map Service (WMS)). By default, this service is enabled, and can be disabled by including the following in the threddsConfig.xml file:

<WMS>
  <allow>false</allow>
</WMS>

The following shows all the configuration options available in the WMS section of the threddsConfig.xml file with the default values shown:

<WMS>
  <allow>true</allow>
  <allowRemote>false</allowRemote>
  <paletteLocationDir>/WEB-INF/palettes</paletteLocationDir>
  <maxImageWidth>2048</maxImageWidth>
  <maxImageHeight>2048</maxImageHeight>
</WMS>

We recommend that you include in the threddsConfig.xml file only the options you want to change. Here is the description of the various options:

allow: a value of false disables the WMS service.
allowRemote: a value of true enables the WMS service for datasets available from a remote server.
paletteLocationDir: optionally specify the location of the directory containing your own palette files, by specifying the directory where they are contained. If the directory location starts with a /, the path is absolute, otherwise it is relative to ${tds.content.root.path}/thredds/. If you don’t specify it, or specify it incorrectly, the default palettes will be used, which are in the war file under WEB-INF/palettes.
maxImageWidth: the maximum image width in pixels that this WMS service will return.
maxImageHeight: the maximum image height in pixels that this WMS service will return.

NetCDF Subset Service (NCSS)

The NetCDF Subset Service provided as part of the TDS is described in more detail here (NetCDF Subset Service Reference). By default this service is enabled, and can be disabled by including the following in the threddsConfig.xml file:

<NetcdfSubsetService>
  <allow>false</allow>
</NetcdfSubsetService>

The following shows all the configuration options available in the NetcdfSubsetService section of the threddsConfig.xml file with the default values shown:

<NetcdfSubsetService>
  <allow>true</allow>
  <dir>(see the note below)</dir>
  <scour>15 min</scour>
  <maxAge>30 min</maxAge>
  <maxFileDownloadSize>300 MB</maxFileDownloadSize>
</NetcdfSubsetService>

We recommend that you include in the threddsConfig.xml file only the options you want to change. Here is the description of the various options:

allow: a value of false disables the NetCDF Subset Service.
dir: the working directory for creating files for download (see choosing a cache directory). If not otherwise set, the TDS will use the ${tds.content.root.path}/thredds/cache/ncss/ directory. We recommend that you do not specify a NetcdfSubsetService.dir element, and use the default.
scour: how often to scour the working directory, to delete files that were not successfully downloaded.
maxAge: how long to leave the files in the working directory while the download is occurring. The files are deleted after a successful download. Do not set to <= 0.
maxFileDownloadSize: maximum size of file that can be requested. Optional; default is that there is no size limitation. If the file is > 2 GB, large format netCDF will be written.

ncISO Service

By default, these services are enabled, and can be disabled by including the following in the threddsConfig.xml file:

<NCISO>
  <ncmlAllow>false</ncmlAllow>
  <uddcAllow>false</uddcAllow>
  <isoAllow>false</isoAllow>
</NCISO>

Each of the allow elements above enables the corresponding ncISO service (NCML, UDDC, and ISO). The ncISO services are described in more detail on the ncISO reference page (TDS and ncISO: Metadata Services).

CDM Configuration

NetCDF-4 C Library Loading

<Netcdf4Clibrary>
  <libraryPath>/usr/local/lib</libraryPath>
  <libraryName>netcdf</libraryName>
  <useForReading>false</useForReading>
</Netcdf4Clibrary>

In order to write netCDF-4 files, you must have the netCDF-4 C library version 4.3.1 or above—compiled and available on your system, along with all supporting libraries (HDF5, zlib, and curl). The details of this differ for each operating system. The elements above allow you to configure how the library is discovered and used.

libraryPath: The directory in which the native library is installed.
libraryName: The name of the native library. This will be used to locate the proper .DLL, .SO, or .DYLIB file within the libraryPath directory.
useForReading: By default, the native library is only used for writing NetCDF-4 files; a pure-Java layer is responsible for reading them. However, if this property is set to true, then it will be used for reading NetCDF-4 (and HDF5) files as well.

For TDS users, we recommend setting the library path and name in threddsConfig.xml as in the above example.

NetCDF-Java runtime Loading

<nj22Config>
  <ioServiceProvider class="edu.univ.ny.stuff.FooFiles"/>
  <coordSysBuilder convention="foo" class="test.Foo"/>
  <coordTransBuilder name="atmos_ln_sigma_coordinates" type="vertical" class="my.stuff.atmosSigmaLog"/>
  <typedDatasetFactory datatype="Point" class="gov.noaa.obscure.file.Flabulate"/>
  <table type="GRIB1" filename="/home/rkambic/grib/tables/userlookup.lst"/>
  <table type="GRIB2" filename="/home/rkambic/grib/tables/grib2userparameters"/>
</nj22Config>

These elements allow you to specify runtime parameters for the NetCDF-Java library from the threddsConfig file. See the NetCDF-Java tutorial for an overview.

Aggregation

<Aggregation>
  <typicalDataset>penultimate</typicalDataset>
</Aggregation>

You can control how NcML Aggregation chooses its typical/template dataset — the one it uses to populate the metadata for the resulting aggregated dataset. Valid values are first, random, latest, and penultimate (latest but one). The default is penultimate.

Disk Caching and temporary files

The various cache directory locations are all under /{tds.content.root.path}/thredds/ by default:

cache	location	description
AggregationCache.dir	`cache/agg/`	for joinExisting aggregations only: write XML files here.
CdmRemote.dir	`cache/cdmr/`	temporary files for cdmremote
CdmValidatorService.dir	`cache/cdmValidate/`	temporary files for cdmvalidator (seperate war)
DiskCache.dir	`cache/cdm/`	only used when non-writeable data directory or `alwaysUse` = true; puts CDM indexes, decompressed files, etc. into this directory
GribIndex.dir	`cache/grib/`	put GRIB Index files (gbx9, ncx4) in this directory
FeatureCollectionCache.dir	`cache/collection/`	when we read GridDataset for FMRC, write an XML summary, store in BDB in this directory
NetcdfSubsetService.dir	`cache/ncss/`	temporary files for NCSS
WCS.dir	`cache/wcs/`	temporary files for WCS

We recommend that you use these defaults, by not specifying them in the threddsConfig.xml file. If you need to move the cache location, move all of them by using a symbolic file link in ${tds.content.root.path}/thredds/. At Unidata, we move the entire content directory by creating a symbolic link:

cd {tomcat_home}
ln -s /data/thredds/content content

These various caches at times may contain large amounts of data. You should choose a location that will not fill up (especially if that location affects other important locations like /opt, /home, etc). If you have a large disk for your data, that may be a good location for the cache directories. On unix-like machines, you can run df to get a listing of disks on your machine. The listing includes size and mount location.

The following elements allow fine grain control over the location and scouring of each of these.

CDM library Disk cache

<DiskCache>
  <alwaysUse>false</alwaysUse>
  <dir>/temp/cache/</dir>
  <scour>1 hour</scour>
  <maxSize>10 Gb</maxSize>
</DiskCache>

These elements control where the CDM/NetCDF-Java library writes temporary files, for example when it needs to unzip files, or write GRIB index files, etc. If alwaysUse is true, these temporary files will always be written to the cache directory specified by dir (see choosing a cache directory). If alwaysUse is false, TDS will try to write them to the same directory as the original file, and if the TDS doesn’t have write-permission it will then write the files to the cache directory. Write permission will be determined by what rights the Tomcat user has (the user that starts up and runs the Tomcat servlet container). For security reasons, you want to carefully limit the file permissions of the Tomcat user.

When opening a file, if alwaysUse is true, TDS looks only in the cache directory for the temporary file. If alwaysUse is false, TDS wil first look for the temporary file in the same directory as the original file, and if not found, then will look in the cache.

Every scour amount of time, the largest items in the cache will be deleted, until the directory has less than maxSize bytes. Note that the directory will sometimes exceed maxSize, and will only be knocked back to maxSize when the scour thread runs. To turn off scouring, set the scour time to 0 (e.g.: 0 secs).

If not otherwise set, the TDS will use the ${tds.content.root.path}/thredds/cache/cdm directory. We recommend that you use this default, by not specifying the DiskCache.dir element.

Aggregation Cache

<AggregationCache>
  <dir>${tds.content.root.path}/thredds/cache/agg/</dir>
  <scour>24 hours</scour>
  <maxAge>90 days</maxAge>
  <cachePathPolicy>nestedDirectory</cachePathPolicy>
</AggregationCache>

If you have joinExisting Aggregations, coordinate information will be written to a cache directory specified by dir (see choosing a cache directory). If not otherwise set, the TDS will use the ${tds.content.root.path}/thredds/cache/agg/ directory. We recommend that you use this default, by not specifying a AggregationCache.dir element.

Every scour amount of time, any item that hasn’t been changed since maxAge time will be deleted. If you have aggregations that never change, set scour to -1 to disable the operation. Otherwise, make maxAge longer than the longest time between changes. Basically, you don’t want to remove active aggregations.

cachePathPolicy controls how cache files are stored in dir. It must be set to one of oneDirectory or nestedDirectory (the default). oneDirectory will put all cache files into the same directory, while nestedDirectory will preserve their directory structure. Use nestedDirectory for large aggregations, as some file systems struggle when a directory contains thousands of files.

This cache information is intended to be permanent; it stores coordinate information from each file in the aggregation, so that the file does not have to be opened each time the dataset is opened. If you have large joinExisting aggregations, there will be a very pronounced difference with and without this cache.

The cache information is updated based on the recheckEvery field in the joinExisting aggregation element.

FeatureCollection Cache

This is where persistent information is kept about FMRCs, in order to speed them up. We recommend that you use the default settings, by not specifying this option.

 <FeatureCollection>
   <dir>${tds.content.root.path}/thredds/cache/collection/</dir>
   <maxSize>20 Mb</maxSize>
   <jvmPercent>2</jvmPercent>
 </FeatureCollection>

dir: location of Feature Collection cache, currently implemented with Berkeley DB. If not otherwise set, the TDS will use the${tds.content.root.path}/thredds/cache/collection/ directory. We recommend that you use this default, by not specifying a FeatureCollection.dir element.
maxSize: maximum amount of memory to be used for this cache.
jvmPercent: alternately, set the memory use as a percent of JVM memory, i.e. -Xmx value. maxSize will override if present. Default is 2 %.

GRIB Index Redirection

<GribIndex>
  <alwaysUse>false</alwaysUse>
  <neverUse>false</neverUse>
  <dir>${tds.content.root.path}/thredds/cache/grib/</dir>
  <policy>nestedDirectory</policy>
  <scour>0 hours</scour>
  <maxAge>90 days</maxAge>
</GribIndex>

These elements control where GRIB index files are written.

If alwaysUse is true, grib index files will always be written to the index directory specified by dir (see choosing a cache directory). If neverUse is true, the index directory will never be used. If neither is set, the TDS will try to write grib indexes to the same directory as the original file, and if the TDS doesnt have write permission it will then write the files to the index directory. Write permission will be determined by what rights the Tomcat user has (the user that starts up Tomcat). For security reasons, you want to carefully limit the file permissions of the Tomcat user.
The policy must be set to one of oneDirectory or nestedDirectory (the default). oneDirectory will put all index files into the same directory, while nestedDirectory will preserve the directory structure of the index files. Use nestedDirectory for large collections of files, as some file systems struggle when a directory contains thousands of files.
Every scour amount of time, any files in the cache that are older than maxAge will be removed. To turn off scouring, set the scour time to 0 (e.g.: 0 hours), or leave out the <scour> element. Typically you do not want to scour the indices.

Managing the GRIB indices is an important task, and can be difficult if the files are changing, as in a rolling archive, or for very large collections. There are two typical ways to do this:

For rolling archives, allow the indices to be written in the same directory as the data files by specifying <neverUse>true</neverUse> or by not using a <neverUse> or <alwaysUse> element (which uses the default behavior). When you delete the data files, delete the corresponding indices.
If you need to keep the index files separate from your data files, set <alwaysUse>true</alwaysUse>, and use <policy>nestedDirectory</policy>. There is currently no way to specify different cache directories for different datasets. All GRIB indices, both gbx9 and ncx4, are kept in the same cache.

A good rule of thumb is that the index files will need disk space between 500 and 1000 times smaller than the size of the grib data files. So, a 1 Terabyte collection of GRIB data will need up to 2 GB of indices.

Object Caching

The default settings will work well enough, and you should only tune them if you have performance problems, and are able to monitor their effect.

File Handle Caching

<RandomAccessFile>
  <minFiles>400</minFiles>
  <maxFiles>500</maxFiles>
  <scour>11 min</scour>
</RandomAccessFile>

There is a pool of shared RandomAccessFile objects, each of which stores an open OS file handle. Since each OS has a maximum on the number of open file handles per process, you must make sure that the sum of the maxFiles does not exceed your OS maximum. For better performance, make these numbers as high as possible.

NetcdfFile Objects

<NetcdfFileCache>
  <minFiles>100</minFiles>
  <maxFiles>150</maxFiles>
  <scour>12 min</scour>
</NetcdfFileCache>

<TimePartition>
  <minFiles>100</minFiles>
  <maxFiles>150</maxFiles>
  <scour>13 min</scour>
</TimePartition>

These elements control the size of the TDS cache for objects for 1) NetcdfFile objects, and 2) GRIB Partition files, respectively. Up to maxFiles objects will be cached, and every scour amount of time, older items in the cache will be released, until only minFiles objects are left. The scour element uses any valid udunits time string, such as sec, min, hour, day. To disable the cache, set maxFiles to 0.

Catalog Processing

Configuration Catalog

<ConfigCatalog>
  <keepInMemory>100</keepInMemory>
  <reread>always</reread>
  <dir>/tomcat_home/content/thredds/cache/catalog/</dir>
  <maxDatasets>10000</maxDatasets>
</ConfigCatalog>

keepInMemory: Configuration catalogs are always cached in memory, for performance reasons. You can set the maximum number of catalogs in the cache. The amount of memory used by a catalog can be approximated simply by the size in bytes of the catalog.xml file itself.
reread:
- always: on startup, all catalogs are read. (default). safest, use if there are a small number of catalogs.
- check: on startup, catalogs that have changed will be reread.
- trigger: after initial read, config catalogs will only be read again if user explicitly triggers it. fastest startup if catalogs rarely change.
dir: The location where the database is written. Default is ${tds.content.root.path}/thredds/cache/catalog/. We recommend that you leave the default and use a symbolic link to move it if needed.
maxDatasets: The maximum number of datasets.

Several files will be created in the directory, including one large memory-mapped file about 500 bytes * maxDatasets. These files are the persistent catalog cache, and can be deleted (when the TDS is stopped), which forces a complete read of the configuration catalogs the next time TDS starts up.

If the maximum number of datasets increases over the limit you have already set, shut down TDS, delete the catalog cache files, change maxDatasets in threddsConfig.xml, and restart. The catalogs will be reread.

Windows may have problems with memory-mapped large files (> 4 GBytes, thus > 7M datastes), and so Linux is preferred for large installations.

User Triggering

You must have Remote Managenment enabled (enable SSL/TLS in Tomcat, and login as a user with the tdsConfig user-role).

From Admin page https://server/thredds/admin/debug:

https://localhost:8443/thredds/admin/debug?Catalogs/reinit (Read all catalogs)
https://localhost:8443/thredds/admin/debug?Catalogs/recheck (Read changed catalogs)

From a program or script, an authenticated user can make a GET HTTP call to:

https://server/thredds/admin/catalog?req=readAll
https://server/thredds/admin/catalog?req=readChanged

Watch Mode (NOT IMPLEMENTED YET)

In this mode, the TDS will not read all catalogs in when starting, but will only read in root catalogs and catalogs that have changed. If using catalogScan, the catalogScan directories will be watched and any changes made while TDS is running will be detected.

To use this mode, the following conditions must be met:

all dataRoot elements must be in a root catalog. A root catalog is the top catalog (${tds.content.root.path}/thredds/catalog.xml), plus any other catalogs specified in a catalogRoot element.
all catalogs are referenced in a catalogRef element in a root catalog, or you are using catalogScan from a root catalog.