Installing on RHEL 4 64bit

I used the rpms to install ruby but couldn't get gem to work because of
a bug in ruby-1.8.1 so I decided to install from the source. I let it
install in /usr/local but told it to install libs in /usr/local/lib64
or else the libraries won't be found. I did that and ruby runs. So I
run the setup for the latest rubygems and it gives me this error:
setup.rb:52:in `require': no such file to load -- rbconfig (LoadError)
        from setup.rb:52

So I tried rubygems-0.8.10 and get the same thing. I ran irb and got
the same thing also. Whats going on? Any ideas?