NTLM authentication with httpclient

I have rewritten my net/http script that I had questions on a couple of days ago using httpclient but am stuck on the NTLM authentication piece. This is what I have so far:

#!c:/ruby/bin/ruby.exe
CERT_FILE = "c:/certs/jim_nopw2.pem"
CA_CERT = "c:/ca_certs/servers_ca.cer"

require 'rubygems'
require 'httpclient' # using v2.1.2 and rubyntlm 0.1.1 installed

client = HTTPClient.new
client.ssl_config.set_trust_ca(CA_CERT)
client.ssl_config.set_client_cert_file(CERT_FILE, CERT_FILE)

# This fetches the page but 401 error because not authenticated through NTLM
resp = client.get("https://some_website.com/default1.asp")

# Not sure how to turn on NTLM authentication and feed arguments. I tried this but still get 401 error
#resp = client.request('GET', "https://some_website.com/default1.asp", {'username' => 'DOMAIN\username', 'password' => 'password'})
puts resp.content
puts resp.status

Thanks again for any and all help.

-Jim

There are examples included in the rubyntlm gem. I'm running Windows
so they're in C:\ruby\lib\ruby\gems\1.8\gems\rubyntlm-0.1.1\examples\.
YMMV.

···

On Dec 1, 1:02 am, Jim Clark <diegosl...@gmail.com> wrote:

I have rewritten my net/http script that I had questions on a couple of
days ago using httpclient but am stuck on the NTLM authentication piece.
This is what I have so far:

#!c:/ruby/bin/ruby.exe
CERT_FILE = "c:/certs/jim_nopw2.pem"
CA_CERT = "c:/ca_certs/servers_ca.cer"

require 'rubygems'
require 'httpclient' # using v2.1.2 and rubyntlm 0.1.1 installed

client = HTTPClient.new
client.ssl_config.set_trust_ca(CA_CERT)
client.ssl_config.set_client_cert_file(CERT_FILE, CERT_FILE)

# This fetches the page but 401 error because not authenticated through NTLM
resp = client.get("https://some_website.com/default1.asp")

# Not sure how to turn on NTLM authentication and feed arguments. I
tried this but still get 401 error
#resp = client.request('GET', "https://some_website.com/default1.asp",
{'username' => 'DOMAIN\username', 'password' => 'password'})
puts resp.content
puts resp.status

Thanks again for any and all help.

-Jim

FWIW, I've worked around this in the past by using a download of curl
that supports ntlm, and wrapped that in a shell command.

Note that if you want binary data (which it doesn't sound like you do)
and are on windows, you'll need to go through a temporary file to work
around this bug:
http://rubyforge.org/tracker/?func=detail&aid=14565&group_id=426&atid=1698

···

On Dec 1, 12:02 am, Jim Clark <diegosl...@gmail.com> wrote:

I have rewritten my net/http script that I had questions on a couple of
days ago using httpclient but am stuck on the NTLM authentication piece.
This is what I have so far:

yermej wrote:

There are examples included in the rubyntlm gem. I'm running Windows
so they're in C:\ruby\lib\ruby\gems\1.8\gems\rubyntlm-0.1.1\examples\.
YMMV.
    

I'm really trying to do this in httpclient versus getting down into rubyntlm which shouldn't be necessary. I took another look at the httpclient.rb source and found what I was looking for on the get method which allows an extheader hash to be passed with arguments. Now I feel I am real close but I am still getting a 401 error. The script is now looking like:

#!c:/ruby/bin/ruby.exe
CERT_FILE = "c:/certs/jim_nopw2.pem"
CA_CERT = "c:/ca_certs/servers_ca.cer"

require 'rubygems'
require 'httpclient' # using v2.1.2 and rubyntlm 0.1.1 installed

client = HTTPClient.new
client.ssl_config.set_trust_ca(CA_CERT)
client.ssl_config.set_client_cert_file(CERT_FILE, CERT_FILE)

resp = client.get("https://some_website.com/default1.asp"), nil, {'username' => 'DOMAIN\username', 'password' => 'password'})
puts resp.content
puts resp.status

When I look at resp.content in my debugger, I see that the @header_item array has the normal info (content type, content length, server, etc.) and also has ["WWW-Authenticate", "Negotiate"] and the second ["WWW-Authenticate", "NTLM"]. When I trace through the get method, I see that @negotiate_auth has a hash @auth which == {} and @auth_default which == null. I guess if I can figure out how to populate these correctly then I'm probably good to go.

This will have to wait until tomorrow or Sunday night and a few more drinks. However, I'll send another S.O.S. that if someone has already figured this out and can show me a good code snippet, I'd be more than willing to share a few drinks (if you are around Seattle) or offer high praise and many thanks from afar. :slight_smile:

-Jim

Keep in mind that NTLM requires at least two round trips to the server in
each connection. Your first GET/POST/whatever request needs to have an NTLM
"type 1" message in the Authorize header. The server will then respond with
a 401, but the response will contain an NTLM "type 2" message that you use
to create an NTLM "type 3" response. You then send your GET/POST again, with
the type 3 response in the Authorize header. At that point, if all goes
well, you get a 2xx from the server.

Your best bet is to use rubyntlm as a previous commenter suggested.

···

On Dec 1, 2007 4:08 AM, Jim Clark <diegoslice@gmail.com> wrote:

When I look at resp.content in my debugger, I see that the @header_item
array has the normal info (content type, content length, server, etc.)
and also has ["WWW-Authenticate", "Negotiate"] and the second
["WWW-Authenticate", "NTLM"]. When I trace through the get method, I see
that @negotiate_auth has a hash @auth which == {} and @auth_default
which == null. I guess if I can figure out how to populate these
correctly then I'm probably good to go.

Francis Cianfrocca wrote:

Keep in mind that NTLM requires at least two round trips to the server in
each connection. Your first GET/POST/whatever request needs to have an NTLM
"type 1" message in the Authorize header. The server will then respond with
a 401, but the response will contain an NTLM "type 2" message that you use
to create an NTLM "type 3" response. You then send your GET/POST again, with
the type 3 response in the Authorize header. At that point, if all goes
well, you get a 2xx from the server.

Your best bet is to use rubyntlm as a previous commenter suggested

I admit that I'm a bit stubborn sometimes. If my wife subscribed to this list, I'm sure she would provide ample evidence. :slight_smile:

Since the httpclient2 2.1.2 release (2007-09-22) specifically says it supports "NTLM auth for WWW-Authenticate", I'm reluctant to give up on it. When I look at the httpclient.rb source, the request method may loop up to 5 times depending and the NegotiateAuth class has all the logic in it to handle the NTLM. Tracing through, the @www_auth.set_auth method never seems to be called at least using the client.get call. I tried to force the set_auth call like this:

  def create_request(method, uri, query, body, extheader, proxy)
    if extheader.is_a?(Hash)
      extheader = extheader.to_a
    end
    if cookies = @cookie_manager.find(uri)
      extheader << ['Cookie', cookies]
    end
    boundary = nil
    content_type = extheader.find { |key, value|
      key.downcase == 'content-type'
    }
    if content_type && content_type[1] =~ /boundary=(.+)\z/
      boundary = $1
    end
    req = HTTP::Message.new_request(method, uri, query, body, proxy, boundary)
+ myuser, mypassword = nil, nil
    extheader.each do |key, value|
      req.header.set(key, value)
+ myuser = value if key == "username"
+ mypassword = value if key == "password"
    end
+ @www_auth.set_auth(uri, myuser, mypassword) if !myuser.nil? && !mypassword.nil?
    if content_type.nil? and !body.nil?
      req.header.set('content-type', 'application/x-www-form-urlencoded')
    end
    req
  end

And another change in line 1703 to allow access to @www_auth,
- attr_reader :www_auth
+ attt_accessor :www_auth

Sorry for not providing a real diff file. I don't have it on my windows machine and I don't feel like fetching it at this late hour.

Point is that @www_auth.set_auth is now called within each of the do_get_block calls on line 1956 of the request method but I am still getting a 401.1 not authorized error.

I'm about done on following up on this though. In working through the certificate problems for my httpclient script yesterday, I tried the same changes in my Perl LWP script tonight and it is now working with SSL, client certificates and NTLM authentication. Since I need a working script, I'm moving forward now with my Perl script and dropping any more efforts on the Ruby version. I realize I may get a few boo/hisses by saying that on a Ruby list but I've already lost too many hours sleep already!

If Hiroshi or anyone else gets NTLM working, please provide an example script to show how it's done. I'd love to use this library because I like what I see in httpclient.

Regards,
Jim

I don't have an NTLM server to test against at this point, but from
looking at the code you might want to try (with the original
httpclient.rb) calling set_auth on the client rather than passing in
an extra header, thusly:

CERT_FILE = "c:/certs/jim_nopw2.pem"
CA_CERT = "c:/ca_certs/servers_ca.cer"

require 'rubygems'
require 'httpclient' # using v2.1.2 and rubyntlm 0.1.1 installed

client = HTTPClient.new
client.ssl_config.set_trust_ca(CA_CERT)
client.ssl_config.set_client_cert_file(CERT_FILE, CERT_FILE)

client.set_auth(nil, 'DOMAIN\username', 'password')

resp = client.get("https://some_website.com/default1.asp")
puts resp.content
puts resp.status

I think I'll be able to play with this once I'm back at work tomorrow.
I'll post more if I can get it working. As you said, all the pieces
are there, it's just a matter of figuring out how to use them
correctly.

···

On Dec 2, 5:07 am, Jim Clark <diegosl...@gmail.com> wrote:

If Hiroshi or anyone else gets NTLM working, please provide an example
script to show how it's done. I'd love to use this library because I
like what I see in httpclient.

Regards,
Jim

I've played with this some and there seem to be a couple problems. Or
I'm just doing things completely wrong.

It seems that calling set_auth with the first parameter as nil should
end up setting the @auth_default instance variable in the
NegotiateAuth object. It does not. So, using the target URL as the
first parameter is the first step to getting things almost working.

Once I did that, I was getting to the response stage of the
authentication, but it wasn't successful. At this point, that seems
like it might be a problem with the way HTTPClient uses rubyntlm or it
might not support the version our server is using or...

It's a bit soon to say, but it seems that there might be a bug in
HTTPClient. I'll post again once I've figured it out.

···

On Dec 2, 3:11 pm, yermej <yer...@gmail.com> wrote:

On Dec 2, 5:07 am, Jim Clark <diegosl...@gmail.com> wrote:

> If Hiroshi or anyone else getsNTLMworking, please provide an example
> script to show how it's done. I'd love to use this library because I
> like what I see in httpclient.

> Regards,
> Jim

I don't have anNTLMserver to test against at this point, but from
looking at the code you might want to try (with the original
httpclient.rb) calling set_auth on the client rather than passing in
an extra header, thusly:

CERT_FILE = "c:/certs/jim_nopw2.pem"
CA_CERT = "c:/ca_certs/servers_ca.cer"

require 'rubygems'
require 'httpclient' # using v2.1.2 and rubyntlm 0.1.1 installed

client = HTTPClient.new
client.ssl_config.set_trust_ca(CA_CERT)
client.ssl_config.set_client_cert_file(CERT_FILE, CERT_FILE)

client.set_auth(nil, 'DOMAIN\username', 'password')

resp = client.get("https://some_website.com/default1.asp")
puts resp.content
puts resp.status

I think I'll be able to play with this once I'm back at work tomorrow.
I'll post more if I can get it working. As you said, all the pieces
are there, it's just a matter of figuring out how to use them
correctly.

Hi

I'm sorry to be asking questions on such and old thread. But I am having
similar problems. I've gotten the CA_CERT by exporting it with Firefox
from the EWS server that I am trying to connect to. The thing I am stuck
on is the CERT_FILE. How do I get this file. And why is the same file
used in both of the parameters. The method seems to be expecting a key
and a certificate.

I've tried generating my own private key but no luck. If I leave the
set_client_cert_file line out I don't get any errors regarding the
certificate, but I get a 401 response from the server.

Any advice would be appreciated.

···

CERT_FILE = "c:/certs/jim_nopw2.pem"
CA_CERT = "c:/ca_certs/servers_ca.cer"

require 'rubygems'
require 'httpclient' # using v2.1.2 and rubyntlm 0.1.1 installed

client = HTTPClient.new
client.ssl_config.set_trust_ca(CA_CERT)
client.ssl_config.set_client_cert_file(CERT_FILE, CERT_FILE)

client.set_auth(nil, 'DOMAIN\username', 'password')

resp = client.get("https://some_website.com/default1.asp")
puts resp.content
puts resp.status

--
Posted via http://www.ruby-forum.com/\.

It works now! I just made a stupid mistake in my testing. This is an
example that should work:

> CERT_FILE = "c:/certs/jim_nopw2.pem"
> CA_CERT = "c:/ca_certs/servers_ca.cer"

> require 'rubygems'
> require 'httpclient' # using v2.1.2 and rubyntlm 0.1.1 installed

> client = HTTPClient.new
> client.ssl_config.set_trust_ca(CA_CERT)
> client.ssl_config.set_client_cert_file(CERT_FILE, CERT_FILE)

> client.set_auth("https://some_website.com/default1.asp", 'DOMAIN\username', 'password')

> resp = client.get("https://some_website.com/default1.asp")
> puts resp.content
> puts resp.status

I forgot to include the domain in my previous attempts as it isn't
required in the browser. I'm guessing the browser grabs the target/
domain during the ntlm process and prepends it to the username. It
wouldn't be hard to modify httpclient to do the same, but isn't
completely necessary.

I think it should also work to call set_auth with nil as the first
parameter and have that set the default credentials, but that doesn't
work the way httpclient is currently coded. NegotiateAuth#set seems to
allow for it though. I hope this helps and will allow you to continue
on your way with Ruby.

yermej wrote:

It works now! I just made a stupid mistake in my testing. This is an
example that should work

Very cool and thank you!

Regards,
Jim