Sample http/https connection with Apache HttpComponents in java

Hi All
While I am building common project for our team. I think it would be worth it to share with others. even it just started to build though.

I remember few times that url connection's destination server has been down for few hours and our service also effected due to the destination server down. because, we didn't set any time out for url connection. which consumed all of JVM thread and the result was, Stuck Thread occured, no thread from thread pool.

For building url connection. better set time out.

There are few things more consider to make a URL connection.
- stable, stable, stable
- in case of no response from destination server for long time, it should close connection
- http connection pooling
- https connection
- connection via proxy
- via credential access


The reason to make this source is, Apache httpclient is very powerful for making url connection, but without looking into detail, it is difficult to use right a way.



Source code structure :

Maven Dependency : pom.xml file

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>com.devtrigger</groupId>
 <artifactId>sample</artifactId>
 <packaging>jar</packaging>
 <version>1.0-SNAPSHOT</version>
 <name>sample</name>
 <url>http://maven.apache.org</url>
 <dependencies>
  <dependency>
   <groupId>junit</groupId>
   <artifactId>junit</artifactId>
   <version>4.10</version>
   <scope>test</scope>
  </dependency>
  <dependency>
   <groupId>org.apache.httpcomponents</groupId>
   <artifactId>httpclient</artifactId>
   <version>4.2.2</version>
  </dependency>
  <dependency>
   <groupId>org.apache.commons</groupId>
   <artifactId>commons-lang3</artifactId>
   <version>3.1</version>
  </dependency>
  <dependency>
   <groupId>commons-configuration</groupId>
   <artifactId>commons-configuration</artifactId>
   <version>1.9</version>
  </dependency> 
  <dependency>
   <groupId>org.codehaus.jackson</groupId>
   <artifactId>jackson-mapper-asl</artifactId>
   <version>1.9.11</version>
  </dependency>   
 </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.3.2</version>
                <configuration>
                    <source>1.6</source>
                    <target>1.6</target>
                    <encoding>utf-8</encoding>
                </configuration>
            </plugin>
   <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-eclipse-plugin</artifactId>
    <version>2.8</version>
    <configuration>
     <downloadJavadocs>true</downloadJavadocs>
     <downloadSources>true</downloadSources>
    </configuration>
   </plugin>            
        </plugins>
    </build>  
</project>

Configuration : conf.xml

<?xml version="1.0" encoding="ISO-8859-1" ?>
<sample>
 <proxyServer>
  192.168.1.1
 </proxyServer>
 <proxyPort>
  8080
 </proxyPort>
</sample>

HttpClientService.java

package com.devtrigger;

import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;

import org.apache.commons.configuration.ConfigurationException;
import org.apache.commons.configuration.XMLConfiguration;
import org.apache.commons.lang3.StringUtils;
import org.apache.http.HttpHost;
import org.apache.http.HttpVersion;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.conn.ClientConnectionManager;
import org.apache.http.conn.params.ConnRoutePNames;
import org.apache.http.conn.scheme.PlainSocketFactory;
import org.apache.http.conn.scheme.Scheme;
import org.apache.http.conn.scheme.SchemeRegistry;
import org.apache.http.conn.ssl.SSLSocketFactory;
import org.apache.http.conn.ssl.TrustStrategy;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.impl.conn.PoolingClientConnectionManager;
import org.apache.http.params.BasicHttpParams;
import org.apache.http.params.HttpConnectionParams;
import org.apache.http.params.HttpParams;
import org.apache.http.params.HttpProtocolParams;

public class HttpClientService {

 private static XMLConfiguration config;

 private static final int CONNECTION_TIMEOUT = 10000;
 private static final int SOCKET_TIMEOUT = 60000;

 private static HttpClientService httpClientService = new HttpClientService(); 
 
 private HttpClientService() {
  try {
   config = new XMLConfiguration("config.xml");
  } catch (ConfigurationException e) {
   e.printStackTrace();
  }
 }
 
 public static HttpClientService getInstance(){
  return httpClientService;
 }
 
 /**
  * @return
  */
 DefaultHttpClient getHttpClient() {
  return getHttpClient(CONNECTION_TIMEOUT, SOCKET_TIMEOUT);
 }

 /**
  * @param domain
  * @param userName
  * @param password
  * @return
  */
 DefaultHttpClient getHttpClient(String domain, String userName, String password){
  
  DefaultHttpClient httpClient = getHttpClient();
  
  httpClient.getCredentialsProvider().setCredentials(
                new AuthScope(domain, 443),
                new UsernamePasswordCredentials(userName, password));
  
  return httpClient;
 }
 /**
  * @param connectionTimeOut
  * @param sockeTimeOut
  * @return
  */
 private DefaultHttpClient getHttpClient(int connectionTimeOut, int sockeTimeOut) {
  try {

   HttpParams httpParams = new BasicHttpParams();
   
   setParams(httpParams, connectionTimeOut, sockeTimeOut);
   
   SSLSocketFactory sf = new SSLSocketFactory(new TrustStrategy() {
    public boolean isTrusted(X509Certificate[] chain, String authType) throws CertificateException {
     return true;
    }
   }, SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
   
   SchemeRegistry registry = new SchemeRegistry();
   registry.register(new Scheme("http", 80, PlainSocketFactory.getSocketFactory()));
   registry.register(new Scheme("https", 443, sf));

   ClientConnectionManager ccm = new PoolingClientConnectionManager(registry);
   
   return new DefaultHttpClient(ccm, httpParams);
  } catch (Exception e) {
   return new DefaultHttpClient();
  }
 } 
 
 /**
  * @param httpParams
  * @param connectionTimeOut
  * @param sockeTimeOut
  */
 private void setParams(HttpParams httpParams, int connectionTimeOut, int sockeTimeOut){
  
  setHttpProxy(httpParams);
  setHttpProtocolParams(httpParams);
  setHttpConnectionParams(httpParams, connectionTimeOut, sockeTimeOut);
  
  httpParams.setBooleanParameter("http.protocol.expect-continue", false);
 }
 
 /**
  * @param httpParams
  */
 private void setHttpProxy(HttpParams httpParams) {
  
  String proxyHost = config.getString("proxyServer");

  if (!StringUtils.isEmpty(proxyHost)) {
   int proxyPort = Integer.parseInt(config.getString("proxyPort"));
   HttpHost proxy = new HttpHost(proxyHost, proxyPort);
   httpParams.setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
  }  
  
 }

 /**
  * @param httpParams
  */
 private void setHttpProtocolParams(HttpParams httpParams){
  HttpProtocolParams.setVersion(httpParams, HttpVersion.HTTP_1_1);
  HttpProtocolParams.setContentCharset(httpParams, "utf-8");
 }
 
 /**
  * @param httpParams
  * @param connectionTimeOut
  * @param sockeTimeOut
  */
 private void setHttpConnectionParams(HttpParams httpParams, int connectionTimeOut, int sockeTimeOut){
  HttpConnectionParams.setConnectionTimeout(httpParams, connectionTimeOut);
  HttpConnectionParams.setSoTimeout(httpParams, sockeTimeOut);
 }
  
}

HttpRequestService.java

package com.devtrigger;

import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.UnsupportedEncodingException;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.http.Header;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.HttpStatus;
import org.apache.http.NameValuePair;
import org.apache.http.ParseException;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.methods.HttpRequestBase;
import org.apache.http.entity.StringEntity;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.util.EntityUtils;
import org.codehaus.jackson.JsonGenerationException;
import org.codehaus.jackson.map.JsonMappingException;
import org.codehaus.jackson.map.ObjectMapper;

public class HttpRequestService {

 HttpClientService httpClientService = HttpClientService.getInstance();
 
 private final Log log = LogFactory.getLog(this.getClass());

 ObjectMapper objectMapper = new ObjectMapper();
 
 private boolean useCredential = false; 
 
 private String domain;
 
 private String userName;
 
 private String password;
 
 
 private boolean isUseCredential() {
  return useCredential;
 }

 private void setUseCredential(boolean useCredential) {
  this.useCredential = useCredential;
 }

 private String getDomain() {
  return domain;
 }

 private void setDomain(String domain) {
  this.domain = domain;
 }

 private String getUserName() {
  return userName;
 }

 private void setUserName(String userName) {
  this.userName = userName;
 }

 private String getPassword() {
  return password;
 }

 private void setPassword(String password) {
  this.password = password;
 }

 public void enableUseCredential(String domain, String userName, String password){
  
  setDomain(domain);
  setUserName(userName);
  setPassword(password);
  setUseCredential(true);
 }
 
 public String doGetRequestNgetResponseBody(String url) {
  return doGetRequestNgetResponseBody(url, new ArrayList
());
 }
 
 /**
  * @param context
  * @param url
  * @param headerList
  * @return
  */
 public String doGetRequestNgetResponseBody(String url, List
 headerList) {

  URI uri = null;

  try {
   uri = new URI(url);
  } catch (URISyntaxException e) {
   e.printStackTrace();
  }

  HttpGet get = new HttpGet(uri);

  for (Header header : headerList) {
   get.addHeader(header);
  }

  return getResponseBody(get);

 }

 /**
  * @param context
  * @param url
  * @param headerList
  * @return
  */
 public HttpResponse doGetRequestNgetResponse(String url, List
 headerList) {

  URI uri = null;

  try {
   uri = new URI(url);
  } catch (URISyntaxException e) {
   e.printStackTrace();
  }

  HttpGet get = new HttpGet(uri);

  for (Header header : headerList) {
   get.addHeader(header);
  }

  return getResponse(get);

 }

 /**
  * @param context
  * @param url
  * @return
  */
 public HttpResponse doGetRequestNgetResponse(String url) {

  URI uri = null;

  try {
   uri = new URI(url);
  } catch (URISyntaxException e) {
   e.printStackTrace();
  }

  HttpGet get = new HttpGet(uri);

  return getResponse(get);

 }

 /**
  * @param context
  * @param url
  * @param params
  * @return
  */
 public String doPostRequestNgetResponseBody(String url, List params) {

  HttpPost post = new HttpPost(url);

  UrlEncodedFormEntity ent = null;

  try {
   ent = new UrlEncodedFormEntity(params);
  } catch (UnsupportedEncodingException e) {
   e.printStackTrace();
  }

  if (ent != null)
   post.setEntity(ent);

  return getResponseBody(post);

 }

 /**
  * @param context
  * @param url
  * @param params
  * @return
  */
 public HttpResponse doPostRequestNgetResponse(String url, List params) {

  HttpPost post = new HttpPost(url);

  UrlEncodedFormEntity ent = null;

  try {
   ent = new UrlEncodedFormEntity(params);
  } catch (UnsupportedEncodingException e) {
   e.printStackTrace();
  }

  if (ent != null)
   post.setEntity(ent);

  return getResponse(post);

 }

 /**
  * @param context
  * @param url
  * @param jsonParams
  * @return
  */
 public String doPostRequestNgetResponseBody(String url, Object jsonObject) {

  HttpPost post = new HttpPost(url);
  StringEntity ent = null;

   try {
    log.debug(objectMapper.writeValueAsString(jsonObject));
    ent = new StringEntity(objectMapper.writeValueAsString(jsonObject));
    ent.setContentType("application/json");
   } catch (JsonGenerationException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
   } catch (JsonMappingException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
   } catch (UnsupportedEncodingException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
   } catch (IOException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
   }

  if (ent != null)
   post.setEntity(ent);

  return getResponseBody(post);

 }

 /**
  * @param context
  * @param httpRequest
  * @return
  */
 private String getResponseBody(HttpRequestBase httpRequest) {

  HttpResponse response = getResponse(httpRequest);

  return getResponseBody(response);
 }

 /**
  * It returns response body string
  * 
  * @param response
  * @return
  */
 public String getResponseBody(HttpResponse response) {

  HttpEntity resEntity = null;
  if (response != null)
   resEntity = response.getEntity();

  if (resEntity != null) {
   try {
    return EntityUtils.toString(resEntity);
   } catch (ParseException e) {
    e.printStackTrace();
   } catch (IOException e) {
    e.printStackTrace();
   }
  }

  return "";
 }

 public boolean saveResponseBodyStream(HttpResponse response, String filePath) {

  boolean result = false;

  HttpEntity resEntity = null;
  if (response != null)
   resEntity = response.getEntity();

  if (resEntity != null) {

   BufferedInputStream bis;
   try {
    bis = new BufferedInputStream(resEntity.getContent());
    BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(new File(filePath)));
    int inByte;
    while ((inByte = bis.read()) != -1)
     bos.write(inByte);
    bis.close();
    bos.close();
   } catch (IllegalStateException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
   } catch (IOException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
   }

  }

  return result;
 }

 /**
  * @param context
  * @param url
  * @param headerList
  * @return
  */
 public HashMap doGetRequestNgetResponseHeader(String url, List
 headerList) {

  URI uri = null;

  try {
   uri = new URI(url);
  } catch (URISyntaxException e) {
   e.printStackTrace();
  }

  HttpGet get = new HttpGet(uri);

  for (Header header : headerList) {
   get.addHeader(header);
  }

  return getResponseHeader(get);

 }

 /**
  * @param context
  * @param url
  * @param params
  * @return
  */
 public HashMap doPostRequestNgetResponseHeader(String url, List params) {

  HttpPost post = new HttpPost(url);

  UrlEncodedFormEntity ent = null;

  try {
   ent = new UrlEncodedFormEntity(params);
  } catch (UnsupportedEncodingException e) {
   e.printStackTrace();
  }

  if (ent != null)
   post.setEntity(ent);

  return getResponseHeader(post);

 }

 /**
  * @param context
  * @param httpRequest
  * @return
  */
 private HashMap getResponseHeader(HttpRequestBase httpRequest) {

  HttpResponse response = getResponse(httpRequest);

  return getResponseHeader(response);
 }

 /**
  * @param response
  * @param headerName
  * @return
  */
 public String getResponseHeaderValue(HttpResponse response, String headerName) {

  HashMap header = getResponseHeader(response);
  return (String) header.get(headerName);

 }

 /**
  * @param response
  * @return
  */
 public HashMap getResponseHeader(HttpResponse response) {

  if (response != null) {
   Header[] headers = response.getAllHeaders();
   return converHeaders2Map(headers);
  } else {
   return new HashMap();
  }

 }

 /**
  * when response parameter is null, it will return 410 Gone status.
  * 
  * @param response
  * @return
  */
 public int getResponseCode(HttpResponse response) {

  if (response != null) {
   return response.getStatusLine().getStatusCode();
  } else {
   return HttpStatus.SC_GONE;
  }

 }

 /**
  * @param headers
  * @return
  */
 private HashMap converHeaders2Map(Header[] headers) {

  HashMap hashMap = new HashMap();

  for (Header header : headers) {
   hashMap.put(header.getName(), header.getValue());
  }

  return hashMap;

 }

 /**
  * @param context
  * @param httpRequest
  * @return
  */
 public HttpResponse getResponse(HttpRequestBase httpRequest) {

  HttpClient httpClient = null;
  
  if(!isUseCredential()){
   httpClient = httpClientService.getHttpClient();
  }else{
   httpClient = httpClientService.getHttpClient(getDomain(), getUserName(), getPassword());
   setUseCredential(false);
  }

  HttpResponse response = null;
  try {
   response = httpClient.execute(httpRequest);
  } catch (ClientProtocolException e1) {
   e1.printStackTrace();
  } catch (IOException e1) {
   e1.printStackTrace();
  }

  return response;
 }
}

HttpRequestTest.java for Junit test run.

package com.devtrigger;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.junit.Assert;
import org.junit.Test;

public class HttpRequestTest {

  private final Log logger = LogFactory.getLog(this.getClass());
 
 @Test
 public void testRequest(){
 
  HttpRequestService service = new HttpRequestService();
  String response = service.doGetRequestNgetResponseBody("http://www.google.com/");
  
  logger.debug("response : " + response);
  
  Assert.assertTrue(response.substring(0, 20).matches(".*(?i)(HtMl).*"));
  
 }
 
}

Test Success

In the end, you can just use like this on your source code.
HttpRequestService service = new HttpRequestService();
String response = service.doGetRequestNgetResponseBody("http://www.google.com/");
further more, you can do post request with header vales or json http body request so on. probably you would need to extend more method.

I hope it helps :) even though, it is not optimized and refactored.



@Add Dec 4th 2012
I have simply looped the test cases 1000 times to my local server.
It worked well and look into regarding to if it is threadsafe and concurrent. It is thread safe. I need to look into more, it's not easy for me to follow up this. you can get more detail info below
http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html
"HttpClient assumes complete control over the process of connection initialization and termination as well as I/O operations on active connections. However various aspects of connection operations can be influenced using a number of parameters."

http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/conn/PoolingClientConnectionManager.html
"PoolingConnectionManager maintains a maximum limit of connection on a per route basis and in total. Per default this implementation will create no more than than 2 concurrent connections per given route and no more 20 connections in total. For many real-world applications these limits may prove too constraining, especially if they use HTTP as a transport protocol for their services. Connection limits, however, can be adjusted using HTTP parameters."