This document provides an overview of how HttpClient handles character encodings and how to use HttpClient in an encoding safe way. There are three main sections: HTTP Headers, Request/Response Body and URLs.
The headers of a HTTP request or response must be in US-ASCII format. It is not possible to use non US-ASCII characters in the header of a request or response. Generally this is not an issue however, because the HTTP headers are designed to facilite the transfer of data rather than to actually transfer the data itself.
One exception however are cookies. Since cookies are transfered as HTTP Headers they are confined to the US-ASCII character set. See the Cookie Guide for more information.
The request or response body can be any encoding, but by default is ISO-8859-1. The encoding may be specified in the Content-Type header, for example:
Content-Type: text/html; charset=UTF-8
In this case the application should be careful to use UTF-8 encoding
when converting the body to a String or some characters may be corrupt.
You can set the content type header for a request with the
addRequestHeader
method in each method and retrieve the
encoding for the response body with the getResponseCharSet
method.
If the response is known to be a String, you can use the
getResponseBodyAsString
method which will automatically use
the encoding specified in the Content-Type header or
ISO-8859-1 if no charset is specified.
Note that some document types, such as HTML and XML allow the author to specify the content type of the file. In this case, you should consult the appropriate standards regarding how to resovle any conflicts in the reported charsets.
The standard for URLs (RFC1738) explictly
states that URLs may only contain graphic printable characters of the
US-ASCII coded character set and is defined in terms of octets.
The octets 80-FF
hexadecimal are not used in US-ASCII and the octets
OO-1F
hexadecimal represent control characters; characters in these
ranges must be encoded.
Characters which cannot be represented by an 8-bit ASCII code, can not be used in an URL as there is no way to reliably encode them (the encoding scheme for URLs is based off of octets). Despite this, some servers do support varying means of encoding double byte characters in URLs, the most common technique seems to be to use UTF-8 encoding and encode each octet separately even if a pair of octets represents one character. This however, is not specified by the standard and is highly prone to error, so it is recommended that URLs be restricted to the 8-bit ASCII range.