Requirements
- Java and tomcat versions listed here
-
On the command line when starting up Tomcat/TDS, you must specify
-Dtds.content.root.path=<content root>
where<content root>
points to the top of the content directory. Note, in this example, that this is/data/content/
, not/data/content/thredds/
. Don’t forget the trailing slash. For example:-Dtds.content.root.path=/data/content/
Quick Navigation
5.7 Upgrade
The following changes will impact the upgrade process from v5.6 to v5.7. If you are upgrading to v5.7 from a version prior to v5.6, please see the summary of changes from v4.x through v5.6 Going forward, we will document significant changes relative to the previous release and not as a grand summary of changes since 4.x.
Caching
Chronicle-Map has been replaced with a two-level cache–an entry-limited in-memory Guava cache (L1) and a persisted EclipseStore disk cache (L2).
Since Chronicle-Map has been removed, the need to add special java options to export specific JVM packages is no longer necessary.
We used to document these in setEnvh.sh
as the CHRONICLE_CACHE
variable:
CHRONICLE_CACHE="--add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-exports java.base/sun.nio.ch=ALL-UNNAMED --add-exports jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.lang.reflect=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED"
You should remove these from the java options used to start the TDS. For more details, please see the Running Tomcat documentation.
Additionally, because of the different caching stack used, a few of the cache related parameters defined in threddsConfig.xml
have been removed.
Specifically, the maxBloatFactor
and averageValueSize
configuration options have been removed from the FeatureCollection
cache.
For more information, please see the TDS Configuration File Reference documentation.
Finally, you will want to remove any of the old chronicle-map based caches:
${tds.content.root.path}/thredds/cache/catalog/*
${tds.content.root.path}/thredds/cache/collection/*
threddsIso
The threddsIso service is once again bundled with the TDS war file, eliminating the need to manually obtain and install the plug-in.
wms
New (old) style
The version of the WMS service shipped with TDS 4.x contained a default style called colored_fat_arrows
.
This style is not part of the new edal-java
library used in TDS 5.x, and as such, has not been usable out of the box.
While it was possible to manually add this style, it is once again distributed as part of the TDS.
Mandatory config changes
If you have not modified the default wmsConfig.xml
file (${tds.content.root.path}/thredds/wmsConfig.xml
), you will need to delete it and allow the TDS to create a new one on startup.
If you have modified the wmsConfig.xml
file, you will need to make some changes before starting TDS v5.7.
First, open the file and ensure that any references to the wmsConfig dtd are updated to reflect the new 2.0 dtd:
<!DOCTYPE wmsConfig SYSTEM "https://schemas.unidata.ucar.edu/thredds/dtd/ncwms/wmsConfig_2_0.dtd">
Note that older versions may reference the dtd using the hostname www.unidata.ucar.edu
- these should be updated to use the schemas hostname exactly as shown above.
Additionally, four new default WMS configuration options have been added, and these will need to be added to your existing configuration:
<wmsConfig>
<global>
<defaults>
...
<defaultAboveMaxColor>#000000</defaultAboveMaxColor>
<defaultBelowMinColor>#000000</defaultBelowMinColor>
<defaultNoDataColor>extend</defaultNoDataColor>
<defaultOpacity>100</defaultOpacity>
</defaults>
</global>
...
</wmsConfig>
A description of these parameters can be found in the wms customization documentation.
TDM / TDS Local API
A new, local API has been added to the TDS to simplify the process of running the TDM in situations where the TDS and TDM are running on the same system. Previously, you needed to configure a tdm tomcat user with authentication to send triggers to the TDS to initiate GRIB related updates. With the introduction of the new local API, the TDM can send triggers to a TDS located on the same host without the need for authentication. See the TDM documentation for more details.
Zarr compatible datasetScan
Starting with TDS 5.7, version 2 zarr datasets (represented by a directory or S3 key with appropriate delimiter) will be recognized by datasetScan
s.
These zarr datasets will have two entries in a catalog: a catalogRef
that points to a catalog that exposes the underlying directory content (previous behavior), as well as a direct access dataset
entry (new).
Component files or objects within a zarr dataset (e.g. .zgroup
, .zarrts
, .zarray
) will only be served using the HTTPServer
.
A zarr dataset can be served by any service other than HTTPServer
.
Note that many zarr stores utilize blosc
for compression, which is not supported by netCDF-Java (but is scheduled to be part of the next release), so data access services may fail, but metadata services are expected to work.
Information on how to describe object storage based datasets can be found in the netCDF-Java DatasetUrl documentation.
Examples of exposing object store hosted data can be seen here and here (note S3 GRIB collections are a work in progress and not currently supported).
More documentation to come in the future.
Upgrading from 4.x
The configuration catalogs and internal state of the TDS has been extensively re-worked to be able to scale to large numbers of catalogs, datasets, and internal objects without excessive use of memory.
A running TDS can be triggered to reread the configuration catalogs without having to restart.
It can be configured to reread only changed catalogs, for fast incremental updates. Other features have been added to make writing configuration catalogs more maintainable, including the <catalogScan>
element, and default and standard services.
The other major enhancement is that GridDataset
is replaced by FeatureDatasetCoverage
, to better support very large feature collections.
The Coverage API works with coordinate values (not array indices), which solves various intractable problems that arise when using array index subsetting on large collections.
A number of API enhancements have been made to take advantage of evolution in the Java language, for example try-with-resource and foreach constructs. The use of these make code simpler and more reliable.
Deprecated classes and methods have been removed, and the module structure and third-party jar use has been improved.
TDS Data Services
Netcdf Subset Service (NCSS)
NCSS queries and responses have been improved and clarified. Generally the previous queries are backwards compatible. See NCSS Reference for details.
Catalog (config and client) base changes:
- NCSS service has been split and now has two different paths, depending on the feature type of data being served.
- For
GRID
, use:<service name="ncssGrid" serviceType="NetcdfSubset" base="/thredds/ncss/grid/" />
and for
POINT
orSTATION
, use:<service name="ncssPoint" serviceType="NetcdfSubset" base="/thredds/ncss/point/" />
New functionality:
- 2D time can now be handled for gridded datasets, with addition of
runtime
andtimeOffset
parameters. - Handling of interval coordinates has been clarified.
- Use
ensCoord
to select an ensemble member.
Minor syntax changes:
- Use
time=all
instead oftemporal=all
- For station datasets,
subset=stns
orsubset=bb
is not needed. Just definestns
or a bounding box.
ThreddsConfig.xml
You no longer turn catalog caching on or off, but you can control how many catalogs are cached (see here for the new syntax).
The following is no longer used:
<Catalog>
<cache>false</cache>
</Catalog>
- By default, most services are enabled, but may still be turned off in
threddsConfig.xml
.
WMS/ Godiva
TDS 5.x uses the edal-java library (formerly ncWMS 1.x).
As this is a major version change to that library, there may be some breaking changes.
See also the edal user guide and the
changes from ncWMS 1.x.
Starting with TDS 5.6, wmsConfig.xml
has been extended to support four new default options:
- defaultAboveMaxColor
- defaultBelowMinColor
- defaultNoDataColor
- defaultOpacity
As a result, the Document Type Definition used in the DOCTYPE
element of the config file has been updated from wmsConfig.dtd
to wmsConfig_2_0.dtd
.
Also note, TDS 5.6 includes an updated release of edal-java.
As part of that update, the version of the h2 database, which holds coordinate reference system definitions, was updated.
If you are upgrading from a previous version of TDS 5, you may encounter database incompatibility errors that will prevent startup of the TDS.
You will need to manually remove the directory ${tds.content.root.path}/hredds/cache/edal-java/epsg/.h2
which contains the previous database files.
Java Web Start
Java Web Start has been deprecated as of Java 9, and has been removed in Java 11, which is the Long-term Release post-Java 8. Due to these changes, the netCDF-Java project no longer provide Java Web Start files as of version 5.0. Following suite, the TDS no longer provide any Web Start based Viewers on Dataset pages out of the box.
Catalogs
Catalog Schema Changes
Schema version is now 1.2
.
Client Catalogs
<service>
elements may not be nested inside of<dataset>
elements, they must be directly contained in the<catalog>
element.
Server Configuration Catalogs
- The
<catalogScan>
element is now available, which scans a directory for catalog files (any file ending in xml) - The
<datasetFmrc>
element is no longer supported <datasetRoot>
elements may not be contained inside of service elements, they must be directly contained in the<catalog>
element<service>
elements may not be nested inside of<dataset>
elements, they must be directly contained in the<catalog>
element.<service>
elements no longer need to be explicitly defined in each config catalog, but may reference user defined global services- If the
datatype/featureType
is defined for a dataset, then the<service>
element may be omitted, and the default set of services for thatdatatype
will be used. - The
expires
attribute is no longer used.
Viewers
thredds.servlet.Viewer
hasInvDatasetImpl
changed toDataset
thredds.servlet.ViewerLinkProvider
hasInvDatasetImpl
changed toDataset
thredds.server.viewer.dataservice.ViewerService
hasInvDatasetImpl
changed toDataset
DatasetScan
addID
is no longer needed, ids are always addedaddDatasetSize
is no longer needed, the dataset size is always added- With
addLatest
, theservice
name is no longer used, it is alwaysResolver
, and the correct service is automatically added. UseaddLatest
attribute for simple case. fileSort
: by default, datasets at each collection level are listed in increasing order by filename. To change to decreasing order, use the fileSort element.sort
: deprecated in favor offilesSort
- User pluggable classes implementing
UserImplType
(crawlableDatasetImpl
,crawlableDatasetFilterImpl
,crawlableDatasetLabelerImpl
,crawlableDatasetSorterImpl
) are no longer supported. (This was never officially released or documented). DatasetScan
details are here
Standard Services
- The TDS provides standard service elements, which know which services are appropriate for each Feature Type.
- User defined services in the root catalog are global and can be referenced by name in any other config catalog.
- User defined services in non-root catalogs are local to that catalog and override (by name) any global services.
- All services are enabled unless explicitly disabled. ** Except for remote catalog services.
- Standard service details are here)
Feature Collections
- The
update
element default is nowstartup="never"
, meaning do not update collection on start up, and use existing indices when the collection is accessed. -
The
fileSort
element is now inside thefeatureCollection
itself, so it can be processed uniformly for all types of feature collections. When a collection shows a list of files, the files will be sorted by increasing name. To use a decreasing sort, use the element<filesSort increasing="false" />
inside thefeatureCollection
element. This supersedes the old way of placing that element in the<gribConfig>
element, or the older verboselexigraphicByName
element:<filesSort> <lexigraphicByName increasing="false" /> // deprecated </filesSort>
- Feature Collection details are here
Recommendations for 5.x catalogs
- Put all
<datasetRoot>
elements in root catalog. - Put all
<catalogScan>
elements in root catalog. - Use
StandardServices
when possible. Annotate your datasets withfeatureType
/dataType
. - Put all user-defined
<service>
elements in root catalog. - Only use user-defined
<service>
elements in non-root catalogs when they are experimental or truly a special case.
Recommendations for ESGF
You must determine the number of datasets that are contained in all of your catalogs.
To get a report, enable Remote Management, and from https://server/thredds/admin/debug
, select “Make Catalog Report”.
This may take 5-20 minutes, depending on the numbers of catalogs.
Add the <ConfigCatalog>
element to threddsConfig.xml
:
<ConfigCatalog>
<keepInMemory>100</keepInMemory>
<reread>check</reread>
<dir>/tomcat_home/content/thredds/cache/catalog/</dir>
<maxDatasets>1000000</maxDatasets>
</ConfigCatalog>
where:
keepInMemory
: using the default value of 100 is probably good enough.reread
: use value of check to only read changed catalogs when restarting TDS.dir
is where the catalog cache files are kept. Use the default directory (or symlink to another place) unless you have a good reason to change.maxDatasets
: this is the number you found in step 1. Typical values for ESGF are 1 - 7 million. This is a maximum, so it’s ok to make it bigger than you need.
Here are some additional, optional changes you can make to increase maintainability:
- Place all
datasetRoot
elements in the top catalog - Place all
service
elements in the root catalog (catalog.xml). These can be referenced from any catalog. - Remove
<service>
elements from non-root catalogs. - Add a
catalogScan
element to the root catalog, replacing the list ofcatalogRef
s listing all the other catalogs.- This assumes that other catalogs live in a subdirectory under the root, for example
${tds.content.root.path}/thredds/esgcet/**
.
- This assumes that other catalogs live in a subdirectory under the root, for example
For example:
<?xml version='1.0' encoding='UTF-8'?>
<catalog name="ESGF Master Catalog" version="1.2"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
xsi:schemaLocation="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0 https://schemas.unidata.ucar.edu/thredds/InvCatalog.1.2.xsd">
<datasetRoot location="/esg/data" path="esg_testroot"/>
<datasetRoot location="/esg/arc/data/" path="esg_obs4MIPs"/>
<datasetRoot location="/esg/cordex/data/" path="esg_cordex"/>
<datasetRoot location="/esg/specs/data/" path="esg_specs"/>
<service base="/thredds/dodsC/" desc="OpenDAP" name="gridded" serviceType="OpenDAP">
<property name="requires_authorization" value="false"/>
<property name="application" value="Web Browser"/>
</service>
<service base="" name="fileservice" serviceType="Compound">
<service base="/thredds/fileServer/" desc="HTTPServer" name="HTTPServer" serviceType="HTTPServer">
<property name="requires_authorization" value="true"/>
<property name="application" value="Web Browser"/>
<property name="application" value="Web Script"/>
</service>
<service base="gsiftp://cmip-bdm1.badc.rl.ac.uk/" desc="GridFTP" name="GridFTPServer" serviceType="GridFTP">
<property name="requires_authorization" value="true"/>
<property name="application" value="DataMover-Lite"/>
</service>
<service base="/thredds/dodsC/" desc="OpenDAP" name="OpenDAPFiles" serviceType="OpenDAP">
<property name="requires_authorization" value="false"/>
<property name="application" value="Web Browser"/>
</service>
</service>
<catalogScan name="ESGF catalogs" path="esgcet" location="esgcet" />
</catalog>