Using The TdsMonitor Tool

link to modifying access logs

TdsMonitor Tool

The TdsMonitor tool allows you to fetch remote TDS log files onto your local machine, and analyze them.

Setup

1. Make sure you have access logging enabled.
   • Open ${tomcat_home}/conf/server.xml and uncomment the AccessLogValve element in the Host element. Modify the pattern attribute as per these instructions.
   • Browse your installation of the TDS to generate some entries in the access logs.
2. Make sure you have the TDS servlet logging enabled.
   • This is turned on by default in the log4j.xml file.

3. Give yourself the access to the TdsMonitor tool. Modify ${tomcat_home}/conf/tomcat-users.xml to add a new role with the rolename attribute of tdsMonitor and add this role to your list of roles:

   <tomcat-users>      
     <role rolename="manager"/>
     <role rolename="tdsConfig"/>      
     <role rolename="tdsMonitor"/>
     <user name="admin" password="e5e9fa1ba31ecd1ae84f75caaa474f3a663f05f4"
           roles="manager,tdsConfig,tdsMonitor"/>
   </tomcat-users>
   

4. Download the ToolsUI jar file and start the TdsMonitor tool. Download the ToolsUI.jar file to some convenient place on your local drive.

   $ pwd
   /home/tds
   
   $ ls -l
   -rw-r--r-- 1 tds ustaff 26235155 Oct 18 20:44 toolsUI-5.5.jar
   

   Start the TdsMonitor program by running the following command:

   $ java -Xmx4g -classpath toolsUI-5.5.jar thredds.ui.monitor.TdsMonitor
   

   You should see TdsMonitor start:

Download the current logs files into the TdsMonitor tool.

1. Type localhost:8080 into the server comboBox
2. Select all 3 buttons access logs, server logs, and data roots.
3. Press Download You should get a security challenge:

   

\4. Type in your username and password. \5. You should see messages showing up in the text area:

The files will be stored in $USER_HOME/tdsMonitor/localhost%3A8080. For production servers, you will need to manage the large number of logs stored here.

The server name will be stored for future use. To remove unwanted servers, right click inside the server comboBox, select Delete, and then click on the server names you want to delete.

Viewing Tomcat Access Logs

Switch to the AccessLogs tab, and select localhost:8080, which will cause the Start Date and End Date fields get populated.

Press the Get Logs Button , and the logs in the selected time range are read in:

Each log entry in the access logs is a row in the table. You can rearrange the columns by dragging on the column headers. You can sort by column by clicking on the column headers.

Select a row, and right click to bring up the context menu. Choose Resend URL which takes you to the UrlDump screen. Choose Get and the URL will be resent to the server, and the results shown in the text area:

Go to the User tab, and reverse sort by count (click on count column header twice), to see a summary of log access records by IP address. Press the lookup DNS button to do reverse DNS lookup on the IP address (this is very slow, and you have to scroll the table to force a refresh).

Select a user, then right click to get the context menu. Select User requests, which takes you back to the LogTable screen, but only shows the logs that are from the selected user. Press Show All Logs button to get back to showing all the logs.

The Service tab shows the logs summarized by service, and you can drill down to see the individual access calls for a specific service.

Similarly, the DataRoot tab shows the logs summarized by data root, and you can drill down to see the individual access calls for a specific data root. For this to work correctly, you must have selected data roots in the ManageLogs tab when downloading. This will download the dataroot file to $USER_HOME/tdsMonitor/localhost%3A8080/roots.txt. You should refresh it whenever the data roots change on your server.

Go to the TimeSeries tab:

This gives a summary of activity on your server for the selected time period, using 5 minute intervals. The total number of requests, total Mbytes sent, and Average Latency in each 5 minute interval are shown.

Viewing TDS Servlet Logs

The TDS Servlet logs are for debugging and are more low-level than the access logs. They are also bigger and you will want to be careful about examining a manageable subset of them at any one time.

Switch to the ServletLogs tab, and select localhost:8080, which will trigger the Start Date and End Date fields be populated. Press the Get Logs Button , and the logs in the selected time range are read in:

The LogTable shows the raw data. You almost always want to look just at the Merged logs, by choosing the Merge tab. For illustration purposes, we have switched to a larger set of logs from thredds.ucar.edu server:

The top table shows groups of logs which are continuous in their reqSeq numbers, so each group indicates that the TDS web application (and/or Tomcat) was restarted between the endDate and startDate of the subsequent group. Select one of these groups to see the logs that are in that group in the lower table.

As with the access log tables, you can rearrange and sort on the columns. Reverse sort on the extra column (by clicking on it twice) to show the logs that have extra information. In this example:

Select one of the rows with extra information, and click on it to bring up the information about that request:

This shows all the information about request 294. The first line shows the information when the request first arrived at the server:

2010-11-03T18:14:15	ISO date/time of the request
[312059]	msecs since server start
[294]	request number
INFO	type of log message
thredds.server.opendap.OpendapServlet	servlet handling the request
(174.100.87.24)	IP address
/thredds/dodsC/nexrad/level2/IDD/KCLE/ 20101103/Level2_KCLE_20101103_2356.ar2v.dds	request path

The second two lines show warning messages from the IOSP that opened the file. In general, problems will be recorded here, and there may be times when Unidata support will ask you to forward this information to them in order to diagnose issues.

The last line shows the closing message after the request has been completed:

2010-11-03T18:14:15	ISO date/time the request was completed
[312336]	msecs since server start
[294]	request number
INFO	type of log message
thredds.server.opendap.OpendapServlet	servlet handling the request
200	HTTP status return
-1	size in bytes of returned content
227	total request time in msecs

In this case, the result was handled successfully (status return 200), and took 227 msecs. The size is often not known in these logs, indicated by a -1. As you can see, all of the pertinent information is summarized in the table row. Looking at the complete information is usually only done to see what the “extra” log messages are.

The Undone and Misc tabs are used to record messages that don’t have a proper start and end message, and are generally only useful to TDS support.