class HTTPClient

The HTTPClient class provides several methods for accessing Web resources via HTTP.

HTTPClient instance is designed to be MT-safe. You can call a HTTPClient instance from several threads without synchronization after setting up an instance.

clnt = HTTPClient.new
clnt.set_cookie_store('/home/nahi/cookie.dat')
urls.each do |url|
  Thread.new(url) do |u|
    p clnt.head(u).status
  end
end

How to use

At first, how to create your client. See initialize for more detail.

  1. Create simple client.

    clnt = HTTPClient.new
    
  2. Accessing resources through HTTP proxy. You can use environment variable ‘http_proxy’ or ‘HTTP_PROXY’ instead.

    clnt = HTTPClient.new('http://myproxy:8080')
    

How to retrieve web resources

See get and get_content.

  1. Get content of specified URL. It returns HTTP::Message object and calling ‘body’ method of it returns a content String.

    puts clnt.get('http://dev.ctor.org/').body
    
  2. For getting content directly, use get_content. It follows redirect response and returns a String of whole result.

    puts clnt.get_content('http://dev.ctor.org/')
    
  3. You can pass :follow_redirect option to follow redirect response in get.

    puts clnt.get('http://dev.ctor.org/', :follow_redirect => true)
    
  4. Get content as chunks of String. It yields chunks of String.

    clnt.get_content('http://dev.ctor.org/') do |chunk|
      puts chunk
    end
    

Invoking other HTTP methods

See head, get, post, put, delete, options, propfind, proppatch and trace.

It returns a HTTP::Message instance as a response.

  1. Do HEAD request.

    res = clnt.head(uri)
    p res.header['Last-Modified'][0]
    
  2. Do GET request with query.

    query = { 'keyword' => 'ruby', 'lang' => 'en' }
    res = clnt.get(uri, query)
    p res.status
    p res.contenttype
    p res.header['X-Custom']
    puts res.body
    

    You can also use keyword argument style.

    res = clnt.get(uri, :query => { :keyword => 'ruby', :lang => 'en' })
    

How to POST

See post.

  1. Do POST a form data.

    body = { 'keyword' => 'ruby', 'lang' => 'en' }
    res = clnt.post(uri, body)
    

    Keyword argument style.

    res = clnt.post(uri, :body => ...)
  2. Do multipart file upload with POST. No need to set extra header by yourself from httpclient/2.1.4.

    File.open('/tmp/post_data') do |file|
      body = { 'upload' => file, 'user' => 'nahi' }
      res = clnt.post(uri, body)
    end
    
  3. Do multipart with custom body.

    File.open('/tmp/post_data') do |file|
      body = [{ 'Content-Type' => 'application/atom+xml; charset=UTF-8',
                :content => '<entry>...</entry>' },
              { 'Content-Type' => 'video/mp4',
                'Content-Transfer-Encoding' => 'binary',
                :content => file }]
      res = clnt.post(uri, body)
    end
    

Accessing via SSL

Ruby needs to be compiled with OpenSSL.

  1. Get content of specified URL via SSL. Just pass an URL which starts with ‘https://’.

    https_url = 'https://www.rsa.com'
    clnt.get(https_url)
    
  2. Getting peer certificate from response.

    res = clnt.get(https_url)
    p res.peer_cert #=> returns OpenSSL::X509::Certificate
    
  3. Configuring OpenSSL options. See HTTPClient::SSLConfig for more details.

    user_cert_file = 'cert.pem'
    user_key_file = 'privkey.pem'
    clnt.ssl_config.set_client_cert_file(user_cert_file, user_key_file)
    clnt.get(https_url)
    
  4. Revocation check. On JRuby you can set following options to let HTTPClient to perform revocation check with CRL and OCSP:

    -J-Dcom.sun.security.enableCRLDP=true -J-Dcom.sun.net.ssl.checkRevocation=true
    ex. jruby -J-Dcom.sun.security.enableCRLDP=true -J-Dcom.sun.net.ssl.checkRevocation=true app.rb
    Revoked cert example: https://test-sspev.verisign.com:2443/test-SSPEV-revoked-verisign.html

    On other platform you can download CRL by yourself and set it via SSLConfig#add_crl.

Handling Cookies

  1. Using volatile Cookies. Nothing to do. HTTPClient handles Cookies.

    clnt = HTTPClient.new
    res = clnt.get(url1) # receives Cookies.
    res = clnt.get(url2) # sends Cookies if needed.
    p res.cookies
    
  2. Saving non volatile Cookies to a specified file. Need to set a file at first and invoke save method at last.

    clnt = HTTPClient.new
    clnt.set_cookie_store('/home/nahi/cookie.dat')
    clnt.get(url)
    ...
    clnt.save_cookie_store
    
  3. Disabling Cookies.

    clnt = HTTPClient.new
    clnt.cookie_manager = nil
    

Configuring authentication credentials

  1. Authentication with Web server. Supports BasicAuth, DigestAuth, and Negotiate/NTLM (requires ruby/ntlm module).

    clnt = HTTPClient.new
    domain = 'http://dev.ctor.org/http-access2/'
    user = 'user'
    password = 'user'
    clnt.set_auth(domain, user, password)
    p clnt.get('http://dev.ctor.org/http-access2/login').status
    
  2. Authentication with Proxy server. Supports BasicAuth and NTLM (requires win32/sspi)

    clnt = HTTPClient.new(proxy)
    user = 'proxy'
    password = 'proxy'
    clnt.set_proxy_auth(user, password)
    p clnt.get(url)
    

Invoking HTTP methods with custom header

Pass a Hash or an Array for header argument.

header = { 'Accept' => 'text/html' }
clnt.get(uri, query, header)

header = [['Accept', 'image/jpeg'], ['Accept', 'image/png']]
clnt.get_content(uri, query, header)

Invoking HTTP methods asynchronously

See head_async, get_async, post_async, put_async, delete_async, options_async, propfind_async, proppatch_async, and trace_async. It immediately returns a HTTPClient::Connection instance as a returning value.

connection = clnt.post_async(url, body)
print 'posting.'
while true
  break if connection.finished?
  print '.'
  sleep 1
end
puts '.'
res = connection.pop
p res.status
p res.body.read # res.body is an IO for the res of async method.

Shortcut methods

You can invoke get_content, get, etc. without creating HTTPClient instance.

ruby -rhttpclient -e 'puts HTTPClient.get_content(ARGV.shift)' http://dev.ctor.org/
ruby -rhttpclient -e 'p HTTPClient.head(ARGV.shift).header["last-modified"]' http://dev.ctor.org/

HTTPClient - HTTP client library. Copyright © 2000-2015 NAKAMURA, Hiroshi <nahi@ruby-lang.org>.

This program is copyrighted free software by NAKAMURA, Hiroshi. You can redistribute it and/or modify it under the same terms of Ruby’s license; either the dual license version in 2003, or any later version.

HTTPClient - HTTP client library. Copyright © 2000-2015 NAKAMURA, Hiroshi <nahi@ruby-lang.org>.

This program is copyrighted free software by NAKAMURA, Hiroshi. You can redistribute it and/or modify it under the same terms of Ruby’s license; either the dual license version in 2003, or any later version.

Constants

CookieManager
DEFAULT_AGENT_NAME

Default User-Agent header

GSSAPIEnabled
LIB_NAME
NO_PROXY_HOSTS
NTLMEnabled
PROPFIND_DEFAULT_EXTHEADER

Default header for PROPFIND request.

RUBY_VERSION_STRING
SSLEnabled
SSLSocket
SSPIEnabled
VERSION

Attributes

base_url[RW]

Base url of resources.

default_header[RW]

Default request header.

follow_redirect_count[RW]

How many times get_content and post_content follows HTTP redirect. 10 by default.

proxy_auth[R]
HTTPClient::ProxyAuth

Proxy authentication handler.

request_filter[R]

An array of request filter which can trap HTTP request/response. See HTTPClient::WWWAuth to see how to use it.

ssl_config[R]
HTTPClient::SSLConfig

SSL configurator.

test_loopback_response[R]

An array of response HTTP message body String which is used for loop-back test. See test/* to see how to use it. If you want to do loop-back test of HTTP header, use test_loopback_http_response instead.

www_auth[R]
HTTPClient::WWWAuth

WWW authentication handler.

Public Class Methods

new(*args, &block) click to toggle source

Creates a HTTPClient instance which manages sessions, cookies, etc.

HTTPClient.new takes optional arguments as a Hash.

* :proxy - proxy url string
* :agent_name - User-Agent String
* :from - from header String
* :base_url - base URL of resources
* :default_header - header Hash all HTTP requests should have
* :force_basic_auth - flag for sending Authorization header w/o gettin 401 first

User-Agent and From are embedded in HTTP request Header if given. From header is not set without setting it explicitly.

proxy = 'http://myproxy:8080'
agent_name = 'MyAgent/0.1'
from = 'from@example.com'
HTTPClient.new(proxy, agent_name, from)

After you set :base_url, all resources you pass to get, post and other methods are recognized to be prefixed with base_url. Say base_url is ‘api.example.com/v1/, get(’users’) is the same as get(‘api.example.com/v1/users’) internally. You can also pass full URL from ‘http://’ even after setting base_url.

The expected base_url and path behavior is the following. Please take care of ‘/’ in base_url and path.

The last ‘/’ is important for base_url:

1. http://foo/bar/baz/ + path -> http://foo/bar/baz/path
2. http://foo/bar/baz + path -> http://foo/bar/path

Relative path handling:

3. http://foo/bar/baz/ + ../path -> http://foo/bar/path
4. http://foo/bar/baz + ../path -> http://foo/path
5. http://foo/bar/baz/ + ./path -> http://foo/bar/baz/path
6. http://foo/bar/baz + ./path -> http://foo/bar/path

The leading ‘/’ of path means absolute path:

7. http://foo/bar/baz/ + /path -> http://foo/path
8. http://foo/bar/baz + /path -> http://foo/path

:default_header is for providing default headers Hash that all HTTP requests should have, such as custom ‘Authorization’ header in API. You can override :default_header with :header Hash parameter in HTTP request methods.

:force_basic_auth turns on/off the BasicAuth force flag. Generally HTTP client must send Authorization header after it gets 401 error from server from security reason. But in some situation (e.g. API client) you might want to send Authorization from the beginning.

# File lib/httpclient.rb, line 428
def initialize(*args, &block)
  proxy, agent_name, from, base_url, default_header, force_basic_auth =
    keyword_argument(args, :proxy, :agent_name, :from, :base_url, :default_header, :force_basic_auth)
  @proxy = nil        # assigned later.
  @no_proxy = nil
  @no_proxy_regexps = []
  @base_url = base_url
  @default_header = default_header || {}
  @www_auth = WWWAuth.new
  @proxy_auth = ProxyAuth.new
  @www_auth.basic_auth.force_auth = @proxy_auth.basic_auth.force_auth = force_basic_auth
  @request_filter = [@proxy_auth, @www_auth]
  @debug_dev = nil
  @redirect_uri_callback = method(:default_redirect_uri_callback)
  @test_loopback_response = []
  @session_manager = SessionManager.new(self)
  @session_manager.agent_name = agent_name || DEFAULT_AGENT_NAME
  @session_manager.from = from
  @session_manager.ssl_config = @ssl_config = SSLConfig.new(self)
  @cookie_manager = CookieManager.new
  @follow_redirect_count = 10
  load_environment
  self.proxy = proxy if proxy
  keep_webmock_compat
  instance_eval(&block) if block
end
timeout_scheduler() click to toggle source

CAUTION: caller must aware of race condition.

# File lib/httpclient/timeout.rb, line 116
def timeout_scheduler
  @timeout_scheduler ||= TimeoutScheduler.new
end

Private Class Methods

attr_proxy(symbol, assignable = false) click to toggle source
# File lib/httpclient.rb, line 304
def attr_proxy(symbol, assignable = false)
  name = symbol.to_s
  define_method(name) {
    @session_manager.__send__(name)
  }
  if assignable
    aname = name + '='
    define_method(aname) { |rhs|
      @session_manager.__send__(aname, rhs)
    }
  end
end

Public Instance Methods

body() click to toggle source
# File lib/httpclient.rb, line 460
def body
  def (o = self.content).content
    self
  end
  o
end
cookies() click to toggle source

Returns stored cookies.

# File lib/httpclient.rb, line 614
def cookies
  if @cookie_manager
    @cookie_manager.cookies
  end
end
debug_dev() click to toggle source

Returns debug device if exists. See debug_dev=.

# File lib/httpclient.rb, line 471
def debug_dev
  @debug_dev
end
debug_dev=(dev) click to toggle source

Sets debug device. Once debug device is set, all HTTP requests and responses are dumped to given device. dev must respond to << for dump.

Calling this method resets all existing sessions.

# File lib/httpclient.rb, line 479
def debug_dev=(dev)
  @debug_dev = dev
  reset_all
  @session_manager.debug_dev = dev
end
default_redirect_uri_callback(uri, res) click to toggle source

A default method for redirect uri callback. This method is used by HTTPClient instance by default. This callback allows relative redirect such as

Location: ../foo/

in HTTP header.

# File lib/httpclient.rb, line 722
def default_redirect_uri_callback(uri, res)
  newuri = urify(res.header['location'][0])
  if !http?(newuri) && !https?(newuri)
    warn("#{newuri}: a relative URI in location header which is not recommended")
    warn("'The field value consists of a single absolute URI' in HTTP spec")
    newuri = uri + newuri
  end
  if https?(uri) && !https?(newuri)
    raise BadResponseError.new("redirecting to non-https resource")
  end
  puts "redirect to: #{newuri}" if $DEBUG
  newuri
end
delete(uri, *args, &block) click to toggle source

Sends DELETE request to the specified URL. See request for arguments.

# File lib/httpclient.rb, line 779
def delete(uri, *args, &block)
  request(:delete, uri, argument_to_hash(args, :body, :header, :query), &block)
end
delete_async(uri, *args) click to toggle source

Sends DELETE request in async style. See request_async2 for arguments. It immediately returns a HTTPClient::Connection instance as a result.

# File lib/httpclient.rb, line 907
def delete_async(uri, *args)
  request_async2(:delete, uri, argument_to_hash(args, :body, :header, :query))
end
force_basic_auth=(force_basic_auth) click to toggle source

Turn on/off the BasicAuth force flag. Generally HTTP client must send Authorization header after it gets 401 error from server from security reason. But in some situation (e.g. API client) you might want to send Authorization from the beginning.

# File lib/httpclient.rb, line 592
def force_basic_auth=(force_basic_auth)
  @www_auth.basic_auth.force_auth = @proxy_auth.basic_auth.force_auth = force_basic_auth
end
get(uri, *args, &block) click to toggle source

Sends GET request to the specified URL. See request for arguments.

# File lib/httpclient.rb, line 742
def get(uri, *args, &block)
  request(:get, uri, argument_to_hash(args, :query, :header, :follow_redirect), &block)
end
get_async(uri, *args) click to toggle source

Sends GET request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

# File lib/httpclient.rb, line 868
def get_async(uri, *args)
  request_async2(:get, uri, argument_to_hash(args, :query, :header))
end
get_content(uri, *args, &block) click to toggle source

Retrieves a web resource.

uri

a String or an URI object which represents an URL of web resource.

query

a Hash or an Array of query part of URL. e.g. { “a” => “b” } => ‘host/part?a=b’. Give an array to pass multiple value like

[“a”, “b”], [“a”, “c”]

> ‘host/part?a=b&a=c’.

header

a Hash or an Array of extra headers. e.g. { ‘Accept’ => ‘text/html’ } or [[‘Accept’, ‘image/jpeg’], [‘Accept’, ‘image/png’]].

&block

Give a block to get chunked message-body of response like get_content(uri) { |chunked_body| … }. Size of each chunk may not be the same.

get_content follows HTTP redirect status (see HTTP::Status.redirect?) internally and try to retrieve content from redirected URL. See redirect_uri_callback= how HTTP redirection is handled.

If you need to get full HTTP response including HTTP status and headers, use get method. get returns HTTP::Message as a response and you need to follow HTTP redirect by yourself if you need.

# File lib/httpclient.rb, line 653
def get_content(uri, *args, &block)
  query, header = keyword_argument(args, :query, :header)
  success_content(follow_redirect(:get, uri, query, nil, header || {}, &block))
end
head(uri, *args) click to toggle source

Sends HEAD request to the specified URL. See request for arguments.

# File lib/httpclient.rb, line 737
def head(uri, *args)
  request(:head, uri, argument_to_hash(args, :query, :header, :follow_redirect))
end
head_async(uri, *args) click to toggle source

Sends HEAD request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

# File lib/httpclient.rb, line 862
def head_async(uri, *args)
  request_async2(:head, uri, argument_to_hash(args, :query, :header))
end
keep_webmock_compat() click to toggle source

webmock 1.6.2 depends on HTTP::Message#body.content to work. let’s keep it work iif webmock is loaded for a while.

# File lib/httpclient.rb, line 457
def keep_webmock_compat
  if respond_to?(:do_get_block_with_webmock)
    ::HTTP::Message.module_eval do
      def body
        def (o = self.content).content
          self
        end
        o
      end
    end
  end
end
no_proxy() click to toggle source

Returns NO_PROXY setting String if given.

# File lib/httpclient.rb, line 521
def no_proxy
  @no_proxy
end
no_proxy=(no_proxy) click to toggle source

Sets NO_PROXY setting String. no_proxy must be a comma separated String. Each entry must be ‘host’ or ‘host:port’ such as; HTTPClient#no_proxy = ‘example.com,example.co.jp:443’

‘localhost’ is treated as a no_proxy site regardless of explicitly listed. HTTPClient checks given URI objects before accessing it. ‘host’ is tail string match. No IP-addr conversion.

You can use environment variable ‘no_proxy’ or ‘NO_PROXY’ for it.

Calling this method resets all existing sessions.

# File lib/httpclient.rb, line 536
def no_proxy=(no_proxy)
  @no_proxy = no_proxy
  @no_proxy_regexps.clear
  if @no_proxy
    @no_proxy.scan(/([^\s:,]+)(?::(\d+))?/) do |host, port|
      if host[0] == ?.
        regexp = /#{Regexp.quote(host)}\z/i
      else
        regexp = /(\A|\.)#{Regexp.quote(host)}\z/i
      end
      @no_proxy_regexps << [regexp, port]
    end
  end
  reset_all
end
options(uri, *args, &block) click to toggle source

Sends OPTIONS request to the specified URL. See request for arguments.

# File lib/httpclient.rb, line 784
def options(uri, *args, &block)
  new_args = argument_to_hash(args, :header, :query, :body)
  request(:options, uri, new_args, &block)
end
options_async(uri, *args) click to toggle source

Sends OPTIONS request in async style. See request_async2 for arguments. It immediately returns a HTTPClient::Connection instance as a result.

# File lib/httpclient.rb, line 913
def options_async(uri, *args)
  request_async2(:options, uri, argument_to_hash(args, :header, :query, :body))
end
patch(uri, *args, &block) click to toggle source

Sends PATCH request to the specified URL. See request for arguments.

# File lib/httpclient.rb, line 747
def patch(uri, *args, &block)
  if hashy_argument_has_keys(args, :query, :body)
    new_args = args[0]
  else
    new_args = argument_to_hash(args, :body, :header)
  end
  request(:patch, uri, new_args, &block)
end
patch_async(uri, *args) click to toggle source

Sends PATCH request in async style. See request_async2 for arguments. It immediately returns a HTTPClient::Connection instance as a result.

# File lib/httpclient.rb, line 874
def patch_async(uri, *args)
  if hashy_argument_has_keys(args, :query, :body)
    new_args = args[0]
  else
    new_args = argument_to_hash(args, :body, :header)
  end
  request_async2(:patch, uri, new_args)
end
post(uri, *args, &block) click to toggle source

Sends POST request to the specified URL. See request for arguments. You should not depend on :follow_redirect => true for POST method. It sends the same POST method to the new location which is prohibited in HTTP spec.

# File lib/httpclient.rb, line 759
def post(uri, *args, &block)
  if hashy_argument_has_keys(args, :query, :body)
    new_args = args[0]
  else
    new_args = argument_to_hash(args, :body, :header, :follow_redirect)
  end
  request(:post, uri, new_args, &block)
end
post_async(uri, *args) click to toggle source

Sends POST request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

# File lib/httpclient.rb, line 885
def post_async(uri, *args)
  if hashy_argument_has_keys(args, :query, :body)
    new_args = args[0]
  else
    new_args = argument_to_hash(args, :body, :header)
  end
  request_async2(:post, uri, new_args)
end
post_content(uri, *args, &block) click to toggle source

Posts a content.

uri

a String or an URI object which represents an URL of web resource.

body

a Hash or an Array of body part. e.g.

{ "a" => "b" } => 'a=b'

Give an array to pass multiple value like

[["a", "b"], ["a", "c"]] => 'a=b&a=c'

When you pass a File as a value, it will be posted as a multipart/form-data. e.g.

{ 'upload' => file }

You can also send custom multipart by passing an array of hashes. Each part must have a :content attribute which can be a file, all other keys will become headers.

[{ 'Content-Type' => 'text/plain', :content => "some text" },
 { 'Content-Type' => 'video/mp4', :content => File.new('video.mp4') }]
=> <Two parts with custom Content-Type header>
header

a Hash or an Array of extra headers. e.g.

{ 'Accept' => 'text/html' }

or

[['Accept', 'image/jpeg'], ['Accept', 'image/png']].
&block

Give a block to get chunked message-body of response like

post_content(uri) { |chunked_body| ... }.

Size of each chunk may not be the same.

post_content follows HTTP redirect status (see HTTP::Status.redirect?) internally and try to post the content to redirected URL. See redirect_uri_callback= how HTTP redirection is handled. Bear in mind that you should not depend on post_content because it sends the same POST method to the new location which is prohibited in HTTP spec.

If you need to get full HTTP response including HTTP status and headers, use post method.

# File lib/httpclient.rb, line 690
def post_content(uri, *args, &block)
  if hashy_argument_has_keys(args, :query, :body)
    query, body, header = keyword_argument(args, :query, :body, :header)
  else
    query = nil
    body, header = keyword_argument(args, :body, :header)
  end
  success_content(follow_redirect(:post, uri, query, body, header || {}, &block))
end
propfind(uri, *args, &block) click to toggle source

Sends PROPFIND request to the specified URL. See request for arguments.

# File lib/httpclient.rb, line 790
def propfind(uri, *args, &block)
  request(:propfind, uri, argument_to_hash(args, :header), &block)
end
propfind_async(uri, *args) click to toggle source

Sends PROPFIND request in async style. See request_async2 for arguments. It immediately returns a HTTPClient::Connection instance as a result.

# File lib/httpclient.rb, line 919
def propfind_async(uri, *args)
  request_async2(:propfind, uri, argument_to_hash(args, :body, :header))
end
proppatch(uri, *args, &block) click to toggle source

Sends PROPPATCH request to the specified URL. See request for arguments.

# File lib/httpclient.rb, line 795
def proppatch(uri, *args, &block)
  request(:proppatch, uri, argument_to_hash(args, :body, :header), &block)
end
proppatch_async(uri, *args) click to toggle source

Sends PROPPATCH request in async style. See request_async2 for arguments. It immediately returns a HTTPClient::Connection instance as a result.

# File lib/httpclient.rb, line 925
def proppatch_async(uri, *args)
  request_async2(:proppatch, uri, argument_to_hash(args, :body, :header))
end
proxy() click to toggle source

Returns URI object of HTTP proxy if exists.

# File lib/httpclient.rb, line 486
def proxy
  @proxy
end
proxy=(proxy) click to toggle source

Sets HTTP proxy used for HTTP connection. Given proxy can be an URI, a String or nil. You can set user/password for proxy authentication like HTTPClient#proxy = ‘user:passwd@myproxy:8080

You can use environment variable ‘http_proxy’ or ‘HTTP_PROXY’ for it. You need to use ‘cgi_http_proxy’ or ‘CGI_HTTP_PROXY’ instead if you run HTTPClient from CGI environment from security reason. (HTTPClient checks ‘REQUEST_METHOD’ environment variable whether it’s CGI or not)

Calling this method resets all existing sessions.

# File lib/httpclient.rb, line 500
def proxy=(proxy)
  if proxy.nil? || proxy.to_s.empty?
    @proxy = nil
    @proxy_auth.reset_challenge
  else
    @proxy = urify(proxy)
    if @proxy.scheme == nil or @proxy.scheme.downcase != 'http' or
        @proxy.host == nil or @proxy.port == nil
      raise ArgumentError.new("unsupported proxy #{proxy}")
    end
    @proxy_auth.reset_challenge
    if @proxy.user || @proxy.password
      @proxy_auth.set_auth(@proxy.user, @proxy.password)
    end
  end
  reset_all
  @session_manager.proxy = @proxy
  @proxy
end
put(uri, *args, &block) click to toggle source

Sends PUT request to the specified URL. See request for arguments.

# File lib/httpclient.rb, line 769
def put(uri, *args, &block)
  if hashy_argument_has_keys(args, :query, :body)
    new_args = args[0]
  else
    new_args = argument_to_hash(args, :body, :header)
  end
  request(:put, uri, new_args, &block)
end
put_async(uri, *args) click to toggle source

Sends PUT request in async style. See request_async2 for arguments. It immediately returns a HTTPClient::Connection instance as a result.

# File lib/httpclient.rb, line 896
def put_async(uri, *args)
  if hashy_argument_has_keys(args, :query, :body)
    new_args = args[0]
  else
    new_args = argument_to_hash(args, :body, :header)
  end
  request_async2(:put, uri, new_args)
end
redirect_uri_callback=(redirect_uri_callback) click to toggle source

Sets callback proc when HTTP redirect status is returned for get_content and post_content. default_redirect_uri_callback is used by default.

If you need strict implementation which does not allow relative URI redirection, set strict_redirect_uri_callback instead.

clnt.redirect_uri_callback = clnt.method(:strict_redirect_uri_callback)
# File lib/httpclient.rb, line 628
def redirect_uri_callback=(redirect_uri_callback)
  @redirect_uri_callback = redirect_uri_callback
end
request(method, uri, *args, &block) click to toggle source

Sends a request to the specified URL.

method

HTTP method to be sent. method.to_s.upcase is used.

uri

a String or an URI object which represents an URL of web resource.

query

a Hash or an Array of query part of URL. e.g. { “a” => “b” } => ‘host/part?a=b’ Give an array to pass multiple value like

[“a”, “b”], [“a”, “c”]

> ‘host/part?a=b&a=c

body

a Hash or an Array of body part. e.g.

{ "a" => "b" }
=> 'a=b'

Give an array to pass multiple value like

[["a", "b"], ["a", "c"]]
=> 'a=b&a=c'.

When the given method is ‘POST’ and the given body contains a file as a value, it will be posted as a multipart/form-data. e.g.

{ 'upload' => file }

You can also send custom multipart by passing an array of hashes. Each part must have a :content attribute which can be a file, all other keys will become headers.

[{ 'Content-Type' => 'text/plain', :content => "some text" },
 { 'Content-Type' => 'video/mp4', :content => File.new('video.mp4') }]
=> <Two parts with custom Content-Type header>

See HTTP::Message.file? for actual condition of ‘a file’.

header

a Hash or an Array of extra headers. e.g. { ‘Accept’ => ‘text/html’ } or [[‘Accept’, ‘image/jpeg’], [‘Accept’, ‘image/png’]].

&block

Give a block to get chunked message-body of response like get(uri) { |chunked_body| … }. Size of each chunk may not be the same.

You can also pass a String as a body. HTTPClient just sends a String as a HTTP request message body.

When you pass an IO as a body, HTTPClient sends it as a HTTP request with chunked encoding (Transfer-Encoding: chunked in HTTP header) if IO does not respond to :size. Bear in mind that some server application does not support chunked request. At least cgi.rb does not support it.

# File lib/httpclient.rb, line 842
def request(method, uri, *args, &block)
  query, body, header, follow_redirect = keyword_argument(args, :query, :body, :header, :follow_redirect)
  if method == :propfind
    header ||= PROPFIND_DEFAULT_EXTHEADER
  else
    header ||= {}
  end
  uri = to_resource_url(uri)
  if block
    filtered_block = adapt_block(&block)
  end
  if follow_redirect
    follow_redirect(method, uri, query, body, header, &block)
  else
    do_request(method, uri, query, body, header, &filtered_block)
  end
end
request_async(method, uri, query = nil, body = nil, header = {}) click to toggle source

Sends a request in async style. request method creates new Thread for HTTP connection and returns a HTTPClient::Connection instance immediately.

Arguments definition is the same as request.

# File lib/httpclient.rb, line 939
def request_async(method, uri, query = nil, body = nil, header = {})
  uri = to_resource_url(uri)
  do_request_async(method, uri, query, body, header)
end
request_async2(method, uri, *args) click to toggle source

new method that has same signature as ‘request’

# File lib/httpclient.rb, line 945
def request_async2(method, uri, *args)
  query, body, header = keyword_argument(args, :query, :body, :header)
  if [:post, :put].include?(method)
    body ||= ''
  end
  if method == :propfind
    header ||= PROPFIND_DEFAULT_EXTHEADER
  else
    header ||= {}
  end
  uri = to_resource_url(uri)
  do_request_async(method, uri, query, body, header)
end
reset(uri) click to toggle source

Resets internal session for the given URL. Keep-alive connection for the site (host-port pair) is disconnected if exists.

# File lib/httpclient.rb, line 961
def reset(uri)
  uri = to_resource_url(uri)
  @session_manager.reset(uri)
end
reset_all() click to toggle source

Resets all of internal sessions. Keep-alive connections are disconnected.

# File lib/httpclient.rb, line 967
def reset_all
  @session_manager.reset_all
end
set_auth(domain, user, passwd) click to toggle source

Sets credential for Web server authentication.

domain

a String or an URI to specify where HTTPClient should use this

credential.  If you set uri to nil, HTTPClient uses this credential
wherever a server requires it.
user

username String.

passwd

password String.

You can set multiple credentials for each uri.

clnt.set_auth('http://www.example.com/foo/', 'foo_user', 'passwd')
clnt.set_auth('http://www.example.com/bar/', 'bar_user', 'passwd')

Calling this method resets all existing sessions.

# File lib/httpclient.rb, line 565
def set_auth(domain, user, passwd)
  uri = to_resource_url(domain)
  @www_auth.set_auth(uri, user, passwd)
  reset_all
end
set_basic_auth(domain, user, passwd) click to toggle source

Deprecated. Use set_auth instead.

# File lib/httpclient.rb, line 572
def set_basic_auth(domain, user, passwd)
  uri = to_resource_url(domain)
  @www_auth.basic_auth.set(uri, user, passwd)
  reset_all
end
set_proxy_auth(user, passwd) click to toggle source

Sets credential for Proxy authentication.

user

username String.

passwd

password String.

Calling this method resets all existing sessions.

# File lib/httpclient.rb, line 583
def set_proxy_auth(user, passwd)
  @proxy_auth.set_auth(user, passwd)
  reset_all
end
strict_redirect_uri_callback(uri, res) click to toggle source

A method for redirect uri callback. How to use:

clnt.redirect_uri_callback = clnt.method(:strict_redirect_uri_callback)

This callback does not allow relative redirect such as

Location: ../foo/

in HTTP header. (raises BadResponseError instead)

# File lib/httpclient.rb, line 705
def strict_redirect_uri_callback(uri, res)
  newuri = urify(res.header['location'][0])
  if https?(uri) && !https?(newuri)
    raise BadResponseError.new("redirecting to non-https resource")
  end
  if !http?(newuri) && !https?(newuri)
    raise BadResponseError.new("unexpected location: #{newuri}", res)
  end
  puts "redirect to: #{newuri}" if $DEBUG
  newuri
end
trace(uri, *args, &block) click to toggle source

Sends TRACE request to the specified URL. See request for arguments.

# File lib/httpclient.rb, line 800
def trace(uri, *args, &block)
  request('TRACE', uri, argument_to_hash(args, :query, :header), &block)
end
trace_async(uri, *args) click to toggle source

Sends TRACE request in async style. See request_async2 for arguments. It immediately returns a HTTPClient::Connection instance as a result.

# File lib/httpclient.rb, line 931
def trace_async(uri, *args)
  request_async2(:trace, uri, argument_to_hash(args, :query, :header))
end

Private Instance Methods

adapt_block(&block) click to toggle source
# File lib/httpclient.rb, line 1083
def adapt_block(&block)
  return block if block.arity == 2
  proc { |r, str| block.call(str) }
end
create_boundary() click to toggle source
# File lib/httpclient.rb, line 1190
def create_boundary
  Digest::SHA1.hexdigest(Time.now.to_s)
end
create_request(method, uri, query, body, header) click to toggle source
# File lib/httpclient.rb, line 1145
def create_request(method, uri, query, body, header)
  method = method.to_s.upcase
  if header.is_a?(Hash)
    header = @default_header.merge(header).to_a
  else
    header = @default_header.to_a + header.dup
  end
  boundary = nil
  if body
    _, content_type = header.find { |key, value|
      key.to_s.downcase == 'content-type'
    }
    if content_type
      if /\Amultipart/ =~ content_type
        if content_type =~ /boundary=(.+)\z/
          boundary = $1
        else
          boundary = create_boundary
          content_type = "#{content_type}; boundary=#{boundary}"
          header = override_header(header, 'content-type', content_type)
        end
      end
    else
      if file_in_form_data?(body)
        boundary = create_boundary
        content_type = "multipart/form-data; boundary=#{boundary}"
      else
        content_type = 'application/x-www-form-urlencoded'
      end
      header << ['Content-Type', content_type]
    end
  end
  req = HTTP::Message.new_request(method, uri, query, body, boundary)
  header.each do |key, value|
    req.header.add(key.to_s, value)
  end
  if @cookie_manager
    cookie_value = @cookie_manager.cookie_value(uri)
    if cookie_value
      req.header.add('Cookie', cookie_value)
    end
  end
  req
end
do_get_block(req, proxy, conn, &block) click to toggle source

!! CAUTION !!

Method 'do_get*' runs under MT conditon. Be careful to change.
# File lib/httpclient.rb, line 1229
def do_get_block(req, proxy, conn, &block)
  @request_filter.each do |filter|
    filter.filter_request(req)
  end
  if str = @test_loopback_response.shift
    dump_dummy_request_response(req.http_body.dump, str) if @debug_dev
    res = HTTP::Message.new_response(str, req.header)
    conn.push(res)
    return res
  end
  content = block ? nil : ''
  res = HTTP::Message.new_response(content, req.header)
  @debug_dev << "= Request\n\n" if @debug_dev
  sess = @session_manager.query(req, proxy)
  res.peer_cert = sess.ssl_peer_cert
  @debug_dev << "\n\n= Response\n\n" if @debug_dev
  do_get_header(req, res, sess)
  conn.push(res)
  sess.get_body do |part|
    set_encoding(part, res.body_encoding)
    if block
      block.call(res, part.dup)
    else
      content << part
    end
  end
  # there could be a race condition but it's OK to cache unreusable
  # connection because we do retry for that case.
  @session_manager.keep(sess) unless sess.closed?
  commands = @request_filter.collect { |filter|
    filter.filter_response(req, res)
  }
  if commands.find { |command| command == :retry }
    raise RetryableResponse.new(res)
  end
  res
end
do_get_header(req, res, sess) click to toggle source
# File lib/httpclient.rb, line 1298
def do_get_header(req, res, sess)
  res.http_version, res.status, res.reason, headers = sess.get_header
  res.header.set_headers(headers)
  if @cookie_manager
    res.header['set-cookie'].each do |cookie|
      @cookie_manager.parse(cookie, req.header.request_uri)
    end
  end
end
do_get_stream(req, proxy, conn) click to toggle source
# File lib/httpclient.rb, line 1267
def do_get_stream(req, proxy, conn)
  @request_filter.each do |filter|
    filter.filter_request(req)
  end
  if str = @test_loopback_response.shift
    dump_dummy_request_response(req.http_body.dump, str) if @debug_dev
    conn.push(HTTP::Message.new_response(StringIO.new(str), req.header))
    return
  end
  piper, pipew = IO.pipe
  pipew.binmode
  res = HTTP::Message.new_response(piper, req.header)
  @debug_dev << "= Request\n\n" if @debug_dev
  sess = @session_manager.query(req, proxy)
  res.peer_cert = sess.ssl_peer_cert
  @debug_dev << "\n\n= Response\n\n" if @debug_dev
  do_get_header(req, res, sess)
  conn.push(res)
  sess.get_body do |part|
    set_encoding(part, res.body_encoding)
    pipew.write(part)
  end
  pipew.close
  @session_manager.keep(sess) unless sess.closed?
  _ = @request_filter.collect { |filter|
    filter.filter_response(req, res)
  }
  # ignore commands (not retryable in async mode)
  res
end
do_request(method, uri, query, body, header, &block) click to toggle source
# File lib/httpclient.rb, line 998
def do_request(method, uri, query, body, header, &block)
  res = nil
  if HTTP::Message.file?(body)
    pos = body.pos rescue nil
  end
  retry_count = @session_manager.protocol_retry_count
  proxy = no_proxy?(uri) ? nil : @proxy
  previous_request = previous_response = nil
  while retry_count > 0
    body.pos = pos if pos
    req = create_request(method, uri, query, body, header)
    if previous_request
      # to remember IO positions to read
      req.http_body.positions = previous_request.http_body.positions
    end
    begin
      protect_keep_alive_disconnected do
        # TODO: remove Connection.new
        # We want to delete Connection usage in do_get_block but Newrelic gem depends on it.
        # https://github.com/newrelic/rpm/blob/master/lib/new_relic/agent/instrumentation/httpclient.rb#L34-L36
        conn = Connection.new
        res = do_get_block(req, proxy, conn, &block)
        # Webmock's do_get_block returns ConditionVariable
        if !res.respond_to?(:previous)
          res = conn.pop
        end
      end
      res.previous = previous_response
      break
    rescue RetryableResponse => e
      previous_request = req
      previous_response = res = e.res
      retry_count -= 1
    end
  end
  res
end
do_request_async(method, uri, query, body, header) click to toggle source
# File lib/httpclient.rb, line 1036
def do_request_async(method, uri, query, body, header)
  conn = Connection.new
  t = Thread.new(conn) { |tconn|
    begin
      if HTTP::Message.file?(body)
        pos = body.pos rescue nil
      end
      retry_count = @session_manager.protocol_retry_count
      proxy = no_proxy?(uri) ? nil : @proxy
      while retry_count > 0
        body.pos = pos if pos
        req = create_request(method, uri, query, body, header)
        begin
          protect_keep_alive_disconnected do
            do_get_stream(req, proxy, tconn)
          end
          break
        rescue RetryableResponse
          retry_count -= 1
        end
      end
    rescue Exception => e
      conn.push e
    end
  }
  conn.async_thread = t
  conn
end
dump_dummy_request_response(req, res) click to toggle source
# File lib/httpclient.rb, line 1308
def dump_dummy_request_response(req, res)
  @debug_dev << "= Dummy Request\n\n"
  @debug_dev << req
  @debug_dev << "\n\n= Dummy Response\n\n"
  @debug_dev << res
end
file_in_form_data?(body) click to toggle source
# File lib/httpclient.rb, line 1194
def file_in_form_data?(body)
  HTTP::Message.multiparam_query?(body) &&
    body.any? { |k, v| HTTP::Message.file?(v) }
end
follow_redirect(method, uri, query, body, header, &block) click to toggle source
# File lib/httpclient.rb, line 1088
def follow_redirect(method, uri, query, body, header, &block)
  uri = to_resource_url(uri)
  if block
    b = adapt_block(&block)
    filtered_block = proc { |r, str|
      b.call(r, str) if r.ok?
    }
  end
  if HTTP::Message.file?(body)
    pos = body.pos rescue nil
  end
  retry_number = 0
  previous = nil
  request_query = query
  while retry_number < @follow_redirect_count
    body.pos = pos if pos
    res = do_request(method, uri, request_query, body, header, &filtered_block)
    res.previous = previous
    if res.redirect?
      if res.header['location'].empty?
        raise BadResponseError.new("Missing Location header for redirect", res)
      end
      method = :get if res.see_other? # See RFC2616 10.3.4
      uri = urify(@redirect_uri_callback.call(uri, res))
      # To avoid duped query parameter. 'location' must include query part.
      request_query = nil
      previous = res
      retry_number += 1
    else
      return res
    end
  end
  raise BadResponseError.new("retry count exceeded", res)
end
getenv(name) click to toggle source
# File lib/httpclient.rb, line 1079
def getenv(name)
  ENV[name.downcase] || ENV[name.upcase]
end
hashy_argument_has_keys(args, *key) click to toggle source
# File lib/httpclient.rb, line 991
def hashy_argument_has_keys(args, *key)
  # if the given arg is a single Hash and...
  args.size == 1 and Hash === args[0] and
    # it has any one of the key
    key.all? { |e| args[0].key?(e) }
end
load_environment() click to toggle source
# File lib/httpclient.rb, line 1065
def load_environment
  # http_proxy
  if getenv('REQUEST_METHOD')
    # HTTP_PROXY conflicts with the environment variable usage in CGI where
    # HTTP_* is used for HTTP header information.  Unlike open-uri, we
    # simply ignore http_proxy in CGI env and use cgi_http_proxy instead.
    self.proxy = getenv('cgi_http_proxy')
  else
    self.proxy = getenv('http_proxy')
  end
  # no_proxy
  self.no_proxy = getenv('no_proxy')
end
no_proxy?(uri) click to toggle source
# File lib/httpclient.rb, line 1213
def no_proxy?(uri)
  if !@proxy or NO_PROXY_HOSTS.include?(uri.host)
    return true
  end
  @no_proxy_regexps.each do |regexp, port|
    if !port || uri.port == port.to_i
      if regexp =~ uri.host
        return true
      end
    end
  end
  false
end
override_header(header, key, value) click to toggle source
# File lib/httpclient.rb, line 1199
def override_header(header, key, value)
  result = []
  header.each do |k, v|
    if k.to_s.downcase == key
      result << [key, value]
    else
      result << [k, v]
    end
  end
  result
end
protect_keep_alive_disconnected() { || ... } click to toggle source
# File lib/httpclient.rb, line 1131
def protect_keep_alive_disconnected
  begin
    yield
  rescue KeepAliveDisconnected
    # Force to create new connection
    Thread.current[:HTTPClient_AcquireNewConnection] = true
    begin
      yield
    ensure
      Thread.current[:HTTPClient_AcquireNewConnection] = false
    end
  end
end
set_encoding(str, encoding) click to toggle source
# File lib/httpclient.rb, line 1315
def set_encoding(str, encoding)
  str.force_encoding(encoding) if encoding
end
success_content(res) click to toggle source
# File lib/httpclient.rb, line 1123
def success_content(res)
  if res.ok?
    return res.content
  else
    raise BadResponseError.new("unexpected response: #{res.header.inspect}", res)
  end
end
to_resource_url(uri) click to toggle source
# File lib/httpclient.rb, line 1319
def to_resource_url(uri)
  u = urify(uri)
  if @base_url && u.scheme.nil? && u.host.nil?
    urify(@base_url) + uri
  else
    u
  end
end