ucar.nc2.util.cache

Class FileCache

java.lang.Object

ucar.nc2.util.cache.FileCache

All Implemented Interfaces:

FileCacheIF

Direct Known Subclasses:

FileCacheNOP

@ThreadSafe
public class FileCache
extends java.lang.Object
implements FileCacheIF

Keep cache of open FileCacheable objects, for example NetcdfFile. The FileCacheable object typically contains a RandomAccessFile object that wraps a system resource like a file handle. These are left open when the FileCacheable is in the cache. The maximum number of these is bounded, though not strictly. A cleanup routine reduces cache size to a minimum number. This cleanup is called periodically in a background thread, and also when the maximum cache size is reached.

1. The FileCacheable object must not be modified.
2. The hashKey must uniquely define the FileCacheable object.
3. The location must be usable in a FileFactory.open().
4. If the FileCacheable is acquired from the cache (ie already open), getLastModified() is used to see if it has changed, and is discarded if so.
5. Make sure you call shutdown() when exiting the program, in order to shut down the cleanup thread.

Normal usage is through the NetcdfDataset interface:

 NetcdfDataset.initNetcdfFileCache(...); // on application startup
 ...
 NetcdfFile ncfile = null;
 try {
   ncfile = NetcdfDataset.acquireFile(location, cancelTask);
   ...
 } finally {
   if (ncfile != null) ncfile.close();
 }
 ...
 NetcdfDataset.shutdown();  // when terminating the application
 

All methods are thread safe. Cleanup is done automatically in a background thread, using LRU algorithm.

Since:

May 30, 2008

Field Summary

Fields

Modifier and Type	Field and Description
protected java.util.concurrent.ConcurrentHashMap<java.lang.Object,ucar.nc2.util.cache.FileCache.CacheElement>	cache
protected static org.slf4j.Logger	cacheLog
protected java.util.concurrent.atomic.AtomicInteger	cleanups
protected java.util.concurrent.ConcurrentHashMap<FileCacheable,ucar.nc2.util.cache.FileCache.CacheElement.CacheFile>	files
protected int	hardLimit
protected java.util.concurrent.atomic.AtomicBoolean	hasScheduled
protected java.util.concurrent.atomic.AtomicInteger	hits
protected static org.slf4j.Logger	log
protected int	minElements
protected java.util.concurrent.atomic.AtomicInteger	miss
protected java.lang.String	name
protected long	period
protected int	softLimit
protected java.util.concurrent.ConcurrentHashMap<java.lang.Object,ucar.nc2.util.cache.FileCache.Tracker>	track
protected boolean	trackAll

Constructor Summary

Constructors

Constructor and Description
FileCache(int minElementsInMemory, int maxElementsInMemory, int period) Constructor.
FileCache(int minElementsInMemory, int softLimit, int hardLimit, int period) Constructor.
FileCache(java.lang.String name, int minElementsInMemory, int softLimit, int hardLimit, int period) Constructor.
FileCache(java.lang.String name, int minElementsInMemory, int softLimit, int hardLimit, int period, boolean removeDeleted) Constructor.

Method Summary

Modifier and Type	Method and Description
FileCacheable	acquire(FileFactory factory, DatasetUrl durl)
FileCacheable	acquire(FileFactory factory, DatasetUrl durl, CancelTask cancelTask) Acquire a FileCacheable, and lock it so no one else can use it.
FileCacheable	acquire(FileFactory factory, java.lang.Object hashKey, DatasetUrl location, int buffer_size, CancelTask cancelTask, java.lang.Object spiObject) Acquire a FileCacheable from the cache, and lock it so no one else can use it.
void	clearCache(boolean force) Remove all cache entries.
void	disable() Disable the cache, and force release all files.
void	eject(java.lang.Object hashKey) Remove all instances of object from the cache
void	enable() Enable the cache, with the current set of parameters.
java.lang.String	getInfo(FileCacheable ncfile)
boolean	release(FileCacheable ncfile) Release the file.
void	resetTracking()
java.util.List<java.lang.String>	showCache()
void	showCache(java.util.Formatter format) Show individual cache entries, add to formatter.
void	showStats(java.util.Formatter format) Add stat report (hits, misses, etc) to formatter.
void	showTracking(java.util.Formatter format)
static void	shutdown() You must call shutdown() to shut down the background threads in order to get a clean process shutdown.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

protected static final org.slf4j.Logger log

cacheLog

protected static final org.slf4j.Logger cacheLog

name

protected java.lang.String name

softLimit

protected final int softLimit

minElements

protected final int minElements

hardLimit

protected final int hardLimit

period

protected final long period

hasScheduled

protected final java.util.concurrent.atomic.AtomicBoolean hasScheduled

cache

protected final java.util.concurrent.ConcurrentHashMap<java.lang.Object,ucar.nc2.util.cache.FileCache.CacheElement> cache

files

protected final java.util.concurrent.ConcurrentHashMap<FileCacheable,ucar.nc2.util.cache.FileCache.CacheElement.CacheFile> files

cleanups

protected final java.util.concurrent.atomic.AtomicInteger cleanups

hits

protected final java.util.concurrent.atomic.AtomicInteger hits

miss

protected final java.util.concurrent.atomic.AtomicInteger miss

track

protected java.util.concurrent.ConcurrentHashMap<java.lang.Object,ucar.nc2.util.cache.FileCache.Tracker> track

trackAll

protected boolean trackAll

Constructor Detail

FileCache

public FileCache(int minElementsInMemory,
                 int maxElementsInMemory,
                 int period)

Constructor.

Parameters:

minElementsInMemory - keep this number in the cache

maxElementsInMemory - trigger a cleanup if it goes over this number.

period - (secs) do periodic cleanups every this number of seconds.

FileCache

public FileCache(int minElementsInMemory,
                 int softLimit,
                 int hardLimit,
                 int period)

Constructor.

Parameters:

minElementsInMemory - keep this number in the cache

softLimit - trigger a cleanup if it goes over this number.

hardLimit - if > 0, never allow more than this many elements. This causes a cleanup to be done in the calling thread.

period - if > 0, do periodic cleanups every this number of seconds.

FileCache

public FileCache(java.lang.String name,
                 int minElementsInMemory,
                 int softLimit,
                 int hardLimit,
                 int period,
                 boolean removeDeleted)

Constructor.

Parameters:

name - of file cache

minElementsInMemory - keep this number in the cache

softLimit - trigger a cleanup if it goes over this number.

hardLimit - if > 0, never allow more than this many elements. This causes a cleanup to be done in the calling thread.

period - if > 0, do periodic cleanups every this number of seconds.

removeDeleted - if true, then remove deleted files from the cache when a cleanup is performed.

FileCache

public FileCache(java.lang.String name,
                 int minElementsInMemory,
                 int softLimit,
                 int hardLimit,
                 int period)

Constructor.

Parameters:

name - of file cache

minElementsInMemory - keep this number in the cache

softLimit - trigger a cleanup if it goes over this number.

hardLimit - if > 0, never allow more than this many elements. This causes a cleanup to be done in the calling thread.

period - if > 0, do periodic cleanups every this number of seconds.

Method Detail

shutdown

public static void shutdown()

You must call shutdown() to shut down the background threads in order to get a clean process shutdown.

disable

public void disable()

Disable the cache, and force release all files. You must still call shutdown() before exiting the application.

Specified by:

disable in interface FileCacheIF

enable

public void enable()

Enable the cache, with the current set of parameters.

Specified by:

enable in interface FileCacheIF

acquire

public FileCacheable acquire(FileFactory factory,
                             DatasetUrl durl,
                             CancelTask cancelTask)
                      throws java.io.IOException

Acquire a FileCacheable, and lock it so no one else can use it. call FileCacheable.close when done.

Parameters:

factory - use this factory to open the file; may not be null

durl - file location, also used as the cache name, will be passed to the NetcdfFileFactory

cancelTask - user can cancel, ok to be null.

Returns:

NetcdfFile corresponding to location.

Throws:

java.io.IOException - on error

acquire

public FileCacheable acquire(FileFactory factory,
                             DatasetUrl durl)
                      throws java.io.IOException

Specified by:

acquire in interface FileCacheIF

Throws:

java.io.IOException

acquire

public FileCacheable acquire(FileFactory factory,
                             java.lang.Object hashKey,
                             DatasetUrl location,
                             int buffer_size,
                             CancelTask cancelTask,
                             java.lang.Object spiObject)
                      throws java.io.IOException

Acquire a FileCacheable from the cache, and lock it so no one else can use it. If not already in cache, open it the FileFactory, and put in cache.

App should call FileCacheable.close when done, and the file is then released instead of closed.

If cache size goes over maxElement, then immediately (actually in 100 msec) schedule a cleanup in a background thread. This means that the cache should never get much larger than maxElement, unless you have them all locked.

Specified by:

acquire in interface FileCacheIF

Parameters:

factory - use this factory to open the file if not in the cache; may not be null

hashKey - unique key for this file. If null, the location will be used

location - file location, may also used as the cache name, will be passed to the NetcdfFileFactory

buffer_size - RandomAccessFile buffer size, if <= 0, use default size

cancelTask - user can cancel, ok to be null.

spiObject - passed to the factory if object needs to be recreated

Returns:

FileCacheable corresponding to location.

Throws:

java.io.IOException - on error

eject

public void eject(java.lang.Object hashKey)

Remove all instances of object from the cache

Specified by:

eject in interface FileCacheIF

Parameters:

hashKey - the object

release

public boolean release(FileCacheable ncfile)
                throws java.io.IOException

Release the file. This unlocks it, updates its lastAccessed date. Normally applications need not call this, just close the file as usual. The FileCacheable has to do tricky stuff.

Specified by:

release in interface FileCacheIF

Parameters:

ncfile - release this file.

Returns:

true if file was in cache, false if it was not

Throws:

java.io.IOException

getInfo

public java.lang.String getInfo(FileCacheable ncfile)

clearCache

public void clearCache(boolean force)

Remove all cache entries.

Specified by:

clearCache in interface FileCacheIF

Parameters:

force - if true, remove them even if they are currently locked.

showCache

public void showCache(java.util.Formatter format)

Show individual cache entries, add to formatter.

Specified by:

showCache in interface FileCacheIF

Parameters:

format - add to this

showCache

public java.util.List<java.lang.String> showCache()

Specified by:

showCache in interface FileCacheIF

showStats

public void showStats(java.util.Formatter format)

Add stat report (hits, misses, etc) to formatter.

Specified by:

showStats in interface FileCacheIF

Parameters:

format - add to this

showTracking

public void showTracking(java.util.Formatter format)

Specified by:

showTracking in interface FileCacheIF

resetTracking

public void resetTracking()

Specified by:

resetTracking in interface FileCacheIF