HTTP Services

Table of Contents

1. Introduction
2. HTTPSession and Session Semantics
3. HTTPMethod
4. HTTPMethodStream
5. HTTPException
6. HTTPFactory
7. ucar.nc2.util.net.HttpClientManager
8. Appendix A: AuthScope

Introduction

In order to provide some API stability and to simplify common use, the Apache libraries have been wrapped. The wrapping code is in the ucar.httpservices package. The new public classes are: HTTPFactory.java, HTTPFormBuilder.java, HTTPSession.java, HTTPMethod.java, HTTPMethodStream.java, and HTTPException.java. Additionally, the class ucar.nc2.util.net.HttpClientManager is closely tied to the ucar.httpservices package.

This document describes some of the methods for each class. For a complete description of the API, the javadoc files should be referenced.

Note that when the code was switched to using httpclient version 4.5, the API was changed somewhat to reflect experience with previous versions. Some methods exist to support Backward compatibility, but are deprecated.

HTTPSession and Session Semantics

The primary key for an HTTPsession object is an instance of AuthScope, which consists of a host (an ip address or name such as unidata.ucar.edu), a port number, and a scheme (e.g. "http" or "https").

Effectively, an AuthScope is intended to map to a specific server somewhere in the internet. Authentication information (credentials) are tied to that scope and all http methods (put, get, etc.) operate in the context of an AuthScope.

Notes:

• It is possible to convert a URL to an AuthScope by taking just the hostname,port, and scheme part of the URL and ignoring the rest.
• Once set, it is not possible to change the AuthScope for an HTTPSession object: it is immutable in this respect.
• HTTPSession objects do NOT necessarily correspond to the Http notion of a session (SESSIONID or JSESSIONID). There are security issues involved with using SESSIONID that have not yet been resolved.
• It is impossible to create an HTTPSession object without specifying an AuthScope.

HTTPSession API

The non authentication related global methods of general interest are as follows.

• static public void setGlobalUserAgent(String userAgent)

• static public void setGlobalCompression(String compressors)

  As a rule, the only allowable compressors are "gzip" and "deflate".

• static public void setMaxRedirects(int n)

• static public void setFollowRedirects(boolean tf)

• public void close()

  HTTPSession is AutoCloseable so try-with-resources can be used.

• public void setCompression(String compressors)

• public void removeCompression()

• public void setUserAgent(String agent)

• public void setMaxRedirects(int n)

• public void setFollowRedirects(boolean tf)

• public void clearCookies()

• public List getCookies()

• static public void setGlobalCredentialsProvider(String url, CredentialsProvider provider) throws HTTPException

  This is no longer supported because only the host+port part of the URL would be used and that was provided at session creation.

• static public void setGlobalCredentials(String url, Credentials creds) throws HTTPException

  Same reason as immediate prior case.

• public void setCredentials(String url, Credentials creds) throws HTTPException

  Same reason as immediate prior case.

• public void setCredentialsProvider(String url, CredentialsProvider provider) throws HTTPException

  Same reason as immediate prior case.

Authentication-Related HTTPSession Methods

To see this, consider the case where we want to access a number of datasets on a server, where each dataset might be protected by some combination of username+password. This is often called BASIC authentication. When a request is made for one of these datasets, the CredentialsProvider is invoked and it may do something like pop up a window to ask the user for the needed username+password for this dataset. When another dataset is accessed, it may pop up another window. Thus the same CredentialsProvider may be used repeatedly. In each case, it would return a specific instance of Credentials that contains the requisite username+password for that specific dataset.

HTTPSession supports credentials caching so that if the client repeatedly accesses the same dataset, it will be asked for the username+password only once.

As a rule, it is best to use the methods here that take CredentialsProvider argument. The case of a Credentials argument is only for convenience when only a specific set of credentials is known a-priori.

• static public void setGlobalCredentialsProvider(CredentialsProvider provider) throws HTTPException

• static public void setGlobalCredentials(Credentials creds) throws HTTPException

• public void setCredentialsProvider(CredentialsProvider provider) throws HTTPException

• public void setCredentials(Credentials creds) throws HTTPException

SSL HTTPSession Methods

• static public void setGlobalSSLAuth(String keypath, String keypassword, String trustpath, String trustpassword)

For this case, the client must provide access to a password protected "keystore" in which the client certificate is stored. The first two parameters of the above method specify these.

In addition, the client may wish to use a so-called trust store in which the set of certificates for trusted servers is stored. the last two arguments specify these.

The above method is not strictly necessary since the relevant values can be specified on the java VM command line using the following JVM parameters.

• "-Dkeystore=..."
• "-Dkeystorepassword=..."
• "-Dtruststore=..."
• "-Dtruststorepassword=..."

Proxy Support

• static public void setGlobalProxy(String proxyurl)

Notes:

• The path, query, and fragment parts of the URL are ignored.
• If the proxy needs authentication, then the URL needs to include it in the form of "http(s)://username:password@host:port".

• "-Dproxyurl=..."

Miscellaneous -D Flags

HTTPMethod

An instance of HTTPMethod always operates in the context of a parent HTTPSession object. It is possible to create an HTTPMethod object without specifying a parent session object. In that case, a temporary HTTPSession object is created by the method object. The lifetime of the session object is the same as the method object: that is, when the method object is closed, the temporary session object will also be closed.

HTTPMethod takes a URL string as its primary argument. Optionally (but normally) , it may take an HTTPSession object as an additional argument.

An important point about the URL argument is that it must be compatible with the AuthScope of the parent HTTPSession object. The term "compatible" means the following. The method url is compatible if all of the following are true.

1. the URL's host+port is the same as the session scope's host+port.
2. the URL's scheme is compatible with the session scope's scheme, where compatible means either that the two schemes are identical or one is the same as the other with a trailing 's' character removed — so "http" is compatible with "https".

HTTPMethodStream

HTTPException

HTTPFactory

For session creation, the following methods are provided:

• static public HTTPSession newSession(String host, int port) throws HTTPException

• static public HTTPSession newSession(String url) throws HTTPException

• static public HTTPSession newSession(AuthScope target) throws HTTPException

For method creattion, the supported methods are: GET, HEAD, PUT, POST, and OPTIONS. For each method Xxx, the following factory methods are provided.

• static public HTTPMethod Xxx(HTTPSession session, String url) throws HTTPException

• static public HTTPMethod Xxx(String url) throws HTTPException

ucar.nc2.util.net.HttpClientManager

This class contains only static methods and these methods perform the following utility operations.

• Return the contents of a URL as a String

  String getUrlContentsAsString(String urlencoded, int maxKbytes)
  

  Make a request on the specified url and return its response as a String of max length maxKbytes.

• Get the contents of URL and store in a file.

  void copyUrlContentsToFile(String urlencoded, File file) throws HTTPException
  

  Make a request on the specified url, obtain its contents and write to the specified file.

• Append the contents of a URL to the end of a File

  long appendUrlContentsToFile(String url, File file, long start, long end)
  

  Make a request on the specified url, obtain its contents and append to the specified file.

• Initialize the Client Manager class

  void init(CredentialsProvider provider, String userAgent)
  

  Initialize to use a specified credentials provider and user agent when making requests.

Examples

Example 1: Create/Use/Release Cycle

public class Main
{
    public static void main(String[] argv)
    {
	String url = argv[0];
	try (HTTPSession session = new HTTPSession(url);
	     HTTPMethod method = HTTPMethod.Get(session,url);
	     int status = method.execute();
             System.out.printf("Execute: status code = %d\n", status);
	} // implicitly calls method.close() then session.close()
    }
}

Example 2: Using HttpClientManager

public class Main
{
    public static void main(String[] argv)
    {
	String url = argv[0[];
	string content = HttpClientManager.getUrlContentsAsString(url,1024);
    }
}

Example 3: Setting Some Global Parameters

public class Main
{
    public static void main(String[] argv)
    {
	String url = argv[0];
	String userpwd = argv[1];

	// Set some parameters at the global level
        HTTPSession.setGlobalCredentials(new UsernamePasswordCredentials(userpwd));
	HTTPSession.setGlobalUserAgent("netcdf/java");
	HTTPSession.setMaxConnections(4);
	HTTPSession.setGlobalAuthenticationPreemptive(true);

        try (HTTPMethod method = HTTPMethod.Get(url)) {//implicit session
	    int status = method.execute();
            System.out.printf("Execute: status code = %d\n", status);
	}
    }
}

Example 4: Setting Some Local Parameters

public class Main
{
    public static void main(String[] argv)
    {
	String url = argv[0];
	String userpwd = argv[1];

        try (HTTPSession session = new HTTPSession(url)) {
            session.setCredentials(new UsernamePasswordCredentials(userpwd));
	    session.setUserAgent("agent");
	    session.setConnectionManagerTimeout(475);
	    session.setSoTimeout(475);
	    try (HTTPMethod method = HTTPMethod.Get(session)) {
		int status = method.execute();
		System.out.printf("Execute: status code = %d\n", status);
	    } // method.close()
	} // session.close()
    }
}

Appendices

Appendix A: AuthScope

• AuthScope(String hostname, int port)

• AuthScope(String hostname, int port, String realm, String scheme)

Author