Gems not working

ruby-talk
1

I am currently not able to install any gems. I was using Gem 1.1.1 but
kept receiving the following errors:

ERROR: While executing gem ... (Gem::RemoteSourceException)
    Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
http://gems.rubyonrails.org/yaml

That seemed to be the same problem discussed here:
http://www.ruby-forum.com/topic/149595

That solution does me no good since I am using 32bit and running FreeBSD
not Linux.

I decided to install Rubygem 1.1.0 and tried to install rails and got
this:

gem install rails
Bulk updating Gem source index for: http://gems.rubyonrails.org/
Bulk updating Gem source index for: http://gems.rubyforge.org/
ERROR: While executing gem ... (Gem::RemoteFetcher::FetchError)
    Errno::ECONNRESET: Connection reset by peer reading
http://gems.rubyforge.org/gems/rake-0.8.1.gem

Additionally this is stopping me from using the Freebsd rubygem-* ports
due to the same error.

my gem env is:

RubyGems Environment:
  - RUBYGEMS VERSION: 1.1.0 (1.1.0)
  - RUBY VERSION: 1.8.6 (2007-09-24 patchlevel 111) [i386-freebsd6]
  - INSTALLATION DIRECTORY: /usr/local/lib/ruby/gems/1.8
  - RUBY EXECUTABLE: /usr/local/bin/ruby18
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86-freebsd-6
  - GEM PATHS:
     - /usr/local/lib/ruby/gems/1.8
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :benchmark => false
     - :backtrace => false
     - :bulk_threshold => 1000
     - :sources => ["http://gems.rubyonrails.org",
"http://gems.rubyforge.org"]
  - REMOTE SOURCES:
     - http://gems.rubyonrails.org
     - http://gems.rubyforge.org

Any Ideas? I am at the end of my rope on this.

···

--
Posted via http://www.ruby-forum.com/.

2

Have you tried a couple of times? The errors just look like a slow
line that is disconnecting.

I have had strange proxy issues with Rubyforge before, but that
was always when trying to log in, never with getting gems.

Leslie

···

On Thu, May 29, 2008 at 8:55 PM, Jim Hoskins <jimh@avatar-intl.com> wrote:

I am currently not able to install any gems. I was using Gem 1.1.1 but
kept receiving the following errors:

ERROR: While executing gem ... (Gem::RemoteSourceException)
   Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
http://gems.rubyonrails.org/yaml

That seemed to be the same problem discussed here:
Rails gem site seems borked - Ruby - Ruby-Forum

That solution does me no good since I am using 32bit and running FreeBSD
not Linux.

I decided to install Rubygem 1.1.0 and tried to install rails and got
this:

gem install rails
Bulk updating Gem source index for: http://gems.rubyonrails.org/
Bulk updating Gem source index for: http://gems.rubyforge.org/
ERROR: While executing gem ... (Gem::RemoteFetcher::FetchError)
   Errno::ECONNRESET: Connection reset by peer reading
RubyGems.org | your community gem host

Additionally this is stopping me from using the Freebsd rubygem-* ports
due to the same error.

my gem env is:

RubyGems Environment:
- RUBYGEMS VERSION: 1.1.0 (1.1.0)
- RUBY VERSION: 1.8.6 (2007-09-24 patchlevel 111) [i386-freebsd6]
- INSTALLATION DIRECTORY: /usr/local/lib/ruby/gems/1.8
- RUBY EXECUTABLE: /usr/local/bin/ruby18
- RUBYGEMS PLATFORMS:
   - ruby
   - x86-freebsd-6
- GEM PATHS:
    - /usr/local/lib/ruby/gems/1.8
- GEM CONFIGURATION:
    - :update_sources => true
    - :verbose => true
    - :benchmark => false
    - :backtrace => false
    - :bulk_threshold => 1000
    - :sources => ["http://gems.rubyonrails.org",
"http://gems.rubyforge.org"]
- REMOTE SOURCES:
    - http://gems.rubyonrails.org
    - http://gems.rubyforge.org

Any Ideas? I am at the end of my rope on this.

3

Hm, that "yaml" file is 20 MB now... is your client timing out, or maybe
refusing to fetch that much data?

Yours,

Tom

···

On Fri, 2008-05-30 at 03:55 +0900, Jim Hoskins wrote:

I am currently not able to install any gems. I was using Gem 1.1.1 but
kept receiving the following errors:

ERROR: While executing gem ... (Gem::RemoteSourceException)
    Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
http://gems.rubyonrails.org/yaml

4

Leslie Viljoen wrote:

···

On Thu, May 29, 2008 at 8:55 PM, Jim Hoskins <jimh@avatar-intl.com> > wrote:

That solution does me no good since I am using 32bit and running FreeBSD
RubyGems.org | your community gem host
- INSTALLATION DIRECTORY: /usr/local/lib/ruby/gems/1.8
    - :backtrace => false
    - :bulk_threshold => 1000
    - :sources => ["http://gems.rubyonrails.org",
"http://gems.rubyforge.org"]
- REMOTE SOURCES:
    - http://gems.rubyonrails.org
    - http://gems.rubyforge.org

Any Ideas? I am at the end of my rope on this.

Have you tried a couple of times? The errors just look like a slow
line that is disconnecting.

I have had strange proxy issues with Rubyforge before, but that
was always when trying to log in, never with getting gems.

Leslie

That's what I thought, but other win32 machines on the same subnet work
just fine. It seems contained to the one FreeBSD Box.

A day later, still the same results, even after running a barrage of
network diagnostics and tests.

--
Posted via http://www.ruby-forum.com/\.

5

The gem servers at rubyforge and whoever hosts for ruby on rails seem to
be badly overloaded. I get these broken pipe errors all the time to the
point that I wrap all uses of gem in "unless" blocks in bash to make
sure things actually go through.

What would help is if the --no-update-sources flag actually did
something. Half the time the point of failure lies in updating the
local source cache from the remote site, but whether or not I specify
--no-update-sources the source cache is downloaded at tremendous
expense. Sometimes it's downloaded multiple times in a single
invocation (if I install a lot of gems at the command line at once).

···

On Sun, 2008-06-01 at 09:20 +0900, Tom Copeland wrote:

On Fri, 2008-05-30 at 03:55 +0900, Jim Hoskins wrote:
> I am currently not able to install any gems. I was using Gem 1.1.1 but
> kept receiving the following errors:
>
> ERROR: While executing gem ... (Gem::RemoteSourceException)
> Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
> http://gems.rubyonrails.org/yaml

Hm, that "yaml" file is 20 MB now... is your client timing out, or maybe
refusing to fetch that much data?

--
Michael T. Richter <ttmrichter@gmail.com> (GoogleTalk:
ttmrichter@gmail.com)
It's OK to figure out murder mysteries, but you shouldn't need to figure
out code. You should be able to read it. (Steve McConnell)

6

Are you running IPV6 on this system? If so, I'd try disabling that -- no
good reason, just a WAG :slight_smile:

If that doesn't help, use a packet sniffer to examine the wire traffic.

Good luck,

···

On Fri, May 30, 2008 at 7:11 AM, Jim Hoskins <jimh@avatar-intl.com> wrote:

A day later, still the same results, even after running a barrage of
network diagnostics and tests.

--
Hassan Schroeder ------------------------ hassan.schroeder@gmail.com

7

Since I've had to reinstall all my gems after the upgrade to 1.8.7,
here's an idea of what's happening with the gem servers right now:

$ for a in actionmailer RedCloth rspec rubygems-update rails fxruby
bindata actionpack gem_plugin rush ruby-debug-ide activeresource opod ;
do echo $a ; until gem install $a ; do echo Retrying $a... ; done ; done
actionmailer
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyonrails.org/
ERROR: While executing gem ... (Gem::RemoteSourceException)
    Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
http://gems.rubyonrails.org/yaml
Retrying actionmailer...
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyonrails.org/
ERROR: While executing gem ... (Gem::RemoteSourceException)
    Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
http://gems.rubyonrails.org/yaml
Retrying actionmailer...
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyonrails.org/
ERROR: While executing gem ... (Gem::RemoteSourceException)
    Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
http://gems.rubyonrails.org/yaml
Retrying actionmailer...
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyonrails.org/
ERROR: While executing gem ... (Gem::RemoteSourceException)
    Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
http://gems.rubyonrails.org/yaml
Retrying actionmailer...
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyonrails.org/
ERROR: While executing gem ... (Gem::RemoteSourceException)
    Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
http://gems.rubyonrails.org/yaml
Retrying actionmailer...
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyonrails.org/
ERROR: While executing gem ... (Gem::RemoteSourceException)
    Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
http://gems.rubyonrails.org/yaml
Retrying actionmailer...
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyonrails.org/
ERROR: While executing gem ... (Gem::RemoteSourceException)
    Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
http://gems.rubyonrails.org/yaml
Retrying actionmailer...
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyforge.org/

Note that the "bulk updating" thing happens twice for
gems.rubyforge.org. Double the opportunity for failure. Notice how the
gems.rubyonrails.org site always fails while downloading that massive
YAML file.

Wouldn't it be nice if there were some kind of, oh I don't know,
distributed mechanism for gem retrieval?

···

On Sun, 2008-06-01 at 22:53 +0900, Michael T. Richter wrote:

On Sun, 2008-06-01 at 09:20 +0900, Tom Copeland wrote:

> On Fri, 2008-05-30 at 03:55 +0900, Jim Hoskins wrote:
> > I am currently not able to install any gems. I was using Gem 1.1.1 but
> > kept receiving the following errors:
> >
> > ERROR: While executing gem ... (Gem::RemoteSourceException)
> > Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
> > http://gems.rubyonrails.org/yaml
>
> Hm, that "yaml" file is 20 MB now... is your client timing out, or maybe
> refusing to fetch that much data?

The gem servers at rubyforge and whoever hosts for ruby on rails seem
to be badly overloaded. I get these broken pipe errors all the time
to the point that I wrap all uses of gem in "unless" blocks in bash to
make sure things actually go through.

What would help is if the --no-update-sources flag actually did
something. Half the time the point of failure lies in updating the
local source cache from the remote site, but whether or not I specify
--no-update-sources the source cache is downloaded at tremendous
expense. Sometimes it's downloaded multiple times in a single
invocation (if I install a lot of gems at the command line at once).

--
Michael T. Richter <ttmrichter@gmail.com> (GoogleTalk:
ttmrichter@gmail.com)
If there's one thing that computers do well, it's to make the same
mistake uncountable times at inhuman speed. (Peter Coffee)

8

The gem servers at rubyforge and whoever hosts for ruby on rails seem to be badly overloaded. I get these broken pipe errors all the time to the point that I wrap all uses of gem in "unless" blocks in bash to make sure things actually go through.

I do not have this problem, and my usage of RubyGems with remote sources is excessive. Today I installed 500 gems over the internet, 55MB of gems and 1000 files during the afternoon Pacific Time, which should be some of the peak usage times for the RubyGems mirrors.

Even on crappy coffee-shop internet, or 40K/s send-my-packets-across-the-country EVDO internet-on-a-stick I never get Errno::EPIPE, so I think the problem is on your end of the internet, and not on the RubyForge or gem mirror end.

What would help is if the --no-update-sources flag actually did something. Half the time the point of failure lies in updating the local source cache from the remote site, but whether or not I specify --no-update-sources the source cache is downloaded at tremendous expense. Sometimes it's downloaded multiple times in a single invocation (if I install a lot of gems at the command line at once).

This code has been replaced in trunk, trunk no longer performs updates of specs it doesn't need.

···

On Jun 1, 2008, at 06:53 AM, Michael T. Richter wrote:

9

Since I've had to reinstall all my gems after the upgrade to 1.8.7,
here's an idea of what's happening with the gem servers right now:

$ for a in actionmailer RedCloth rspec rubygems-update rails fxruby
bindata actionpack gem_plugin rush ruby-debug-ide activeresource
opod ; do echo $a ; until gem install $a ; do echo Retrying $a... ;
done ; done

I think an easier way to do this might be:

sudo gem install actionmailer RedCloth [... etc ...]

actionmailer
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyforge.org/

I've noticed this double bulk update sometimes as well. You might want
to post a note to the rubygems-developers list about it.

Yours,

tom

···

On Sun, 2008-06-01 at 23:53 +0900, Michael T. Richter wrote:

10

No, there is no problem with the rubyforge mirrors.

gem sources -r http://gems.rubyonrails.org/

Will probably fix your problem.

···

On Jun 1, 2008, at 07:53 AM, Michael T. Richter wrote:

Since I've had to reinstall all my gems after the upgrade to 1.8.7, here's an idea of what's happening with the gem servers right now:

$ for a in actionmailer RedCloth rspec rubygems-update rails fxruby bindata actionpack gem_plugin rush ruby-debug-ide activeresource opod ; do echo $a ; until gem install $a ; do echo Retrying $a... ; done ; done
actionmailer
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyonrails.org/
ERROR: While executing gem ... (Gem::RemoteSourceException)
    Error fetching remote gem cache: Errno::EPIPE: Broken pipe readinghttp://gems.rubyonrails.org/yaml

11

The problem with this is that if -- scratch that, when -- it fails, I
either have to go into the backscroll and figure out which pieces
installed and which didn't, or I wrap that all up in an until and if
(when) any single gem fails, the whole thing gets restarted.

My way, while less efficient overall, at least doesn't reinstall gems a
dozen times over because the next-to-last one failed to install. Were
the gems servers actually reliable, I wouldn't have to do even the until
wrap, not to mention the grotesquely inefficient one-at-a-time approach
I took above.

···

On Mon, 2008-06-02 at 04:07 +0900, Tom Copeland wrote:

> $ for a in actionmailer RedCloth rspec rubygems-update rails fxruby
> bindata actionpack gem_plugin rush ruby-debug-ide activeresource
> opod ; do echo $a ; until gem install $a ; do echo Retrying $a... ;
> done ; done

I think an easier way to do this might be:

sudo gem install actionmailer RedCloth [... etc ...]

--
Michael T. Richter <ttmrichter@gmail.com> (GoogleTalk:
ttmrichter@gmail.com)
All really first class designers are both artists, engineers, and men of
a powerful and intolerant temper, quick to resist the least modification
of the plans, energetic in fighting the least infringement upon what
they regard as their own sphere of action. (Nevil Shute)

12

Eric Hodel wrote:

···

On Jun 1, 2008, at 07:53 AM, Michael T. Richter wrote:

Bulk updating Gem source index for: http://gems.rubyonrails.org/
ERROR: While executing gem ... (Gem::RemoteSourceException)
    Error fetching remote gem cache: Errno::EPIPE: Broken pipe
readinghttp://gems.rubyonrails.org/yaml

No, there is no problem with the rubyforge mirrors.

gem sources -r http://gems.rubyonrails.org/

Will probably fix your problem.

appears to be a 403 if i go directly to

http://gems.rubyonrails.org
--
Posted via http://www.ruby-forum.com/\.

13

> > $ for a in actionmailer RedCloth rspec rubygems-update rails fxruby
> > bindata actionpack gem_plugin rush ruby-debug-ide activeresource
> > opod ; do echo $a ; until gem install $a ; do echo Retrying $a... ;
> > done ; done

> I think an easier way to do this might be:
>
> sudo gem install actionmailer RedCloth [... etc ...]

The problem with this is that if -- scratch that, when -- it fails, I
either have to go into the backscroll and figure out which pieces
installed and which didn't, or I wrap that all up in an until and if
(when) any single gem fails, the whole thing gets restarted.

Ah right, because the gem client is timing out, gotcha.

My way, while less efficient overall, at least doesn't reinstall gems
a dozen times over because the next-to-last one failed to install.
Were the gems servers actually reliable, I wouldn't have to do even
the until wrap, not to mention the grotesquely inefficient
one-at-a-time approach I took above.

What surprises me is seeing the gem client fetch the uncompressed index
(that is, the yaml file) rather than fetching the compressed index. The
compressed version is only 1MB, and seems like that would be much less
likely to time out. It might be worth posting to the
rubygems-developers list and seeing if they could help you hunt down
this problem.

Yours,

Tom

···

On Mon, 2008-06-02 at 08:58 +0900, Michael T. Richter wrote:

On Mon, 2008-06-02 at 04:07 +0900, Tom Copeland wrote:

14

There's a lot that surprises me in the recent gem releases (1.0+). The
fact that there seems to be absolutely no caching of metadata is the
first point of surprise. Even if I tell it NOT to go looking for the
metadata it goes looking for the metadata which indicates to me that
it's not caching locally at all and has to. Then the fact, as you point
out, that it needs to download the whole YAML file instead of the
compressed version is suspicious. I'm seriously unimpressed with the
1.0+ versions of gem, frankly.

And yet another mailing list to sign up for? God DAMN! How about just
a bug tracker somewhere? Is there an URL for that? I really don't want
to increase my incoming mail any more than I already have it. :frowning:

···

On Mon, 2008-06-02 at 10:31 +0900, Tom Copeland wrote:

What surprises me is seeing the gem client fetch the uncompressed index
(that is, the yaml file) rather than fetching the compressed index. The
compressed version is only 1MB, and seems like that would be much less
likely to time out. It might be worth posting to the
rubygems-developers list and seeing if they could help you hunt down
this problem.

--
Michael T. Richter <ttmrichter@gmail.com> (GoogleTalk:
ttmrichter@gmail.com)
We should sell bloat credits, the way the government sells pollution
credits. Everybody's assigned a certain amount of bloat, and if they go
over, they have to purchase bloat credits from some other group that's
been more careful. (Bent Hagemark)

15

Yup, there's the usual complement of RubyForge trackers and such here:

http://rubyforge.org/projects/rubygems/

Yours,

Tom

···

On Mon, 2008-06-02 at 15:12 +0900, Michael T. Richter wrote:

There's a lot that surprises me in the recent gem releases (1.0+).
The fact that there seems to be absolutely no caching of metadata is
the first point of surprise. Even if I tell it NOT to go looking for
the metadata it goes looking for the metadata which indicates to me
that it's not caching locally at all and has to. Then the fact, as
you point out, that it needs to download the whole YAML file instead
of the compressed version is suspicious. I'm seriously unimpressed
with the 1.0+ versions of gem, frankly.

And yet another mailing list to sign up for? God DAMN! How about
just a bug tracker somewhere? Is there an URL for that? I really
don't want to increase my incoming mail any more than I already have
it. :frowning:

16

Thanks, Tom. Lack of sleep made me not even think about looking at
RubyForge. :frowning:

···

On Mon, 2008-06-02 at 22:01 +0900, Tom Copeland wrote:

> And yet another mailing list to sign up for? God DAMN! How about
> just a bug tracker somewhere? Is there an URL for that? I really
> don't want to increase my incoming mail any more than I already have
> it. :frowning:

Yup, there's the usual complement of RubyForge trackers and such here:

http://rubyforge.org/projects/rubygems/

--
Michael T. Richter <ttmrichter@gmail.com> (GoogleTalk:
ttmrichter@gmail.com)
Those who have learned from history are bound to helplessly watch it
repeat itself. (Albert Y. C. Lai)

17

Sorry to repost this, but it should probably have been attached to
this thread....

Wondering why the endless "bulk updating" takes so long I looked that
the source a bit.
There seems that if there are more than 50 gems missing from the quick
list, a bulk update
is done:

source_index.rb:
       use_incremental = missing_gems.size <= INCREMENTAL_THRESHHOLD

(INCREMENTAL_THRESHOLD is 50)

..yet it seems the quick list is always more than 50 gems out of date (for the
bulk update seems to always be done)

When downloading and extracting both lists, it look like there are 13464 gems
listed in the bulk file and 13432 in the quick list.

(For the curious, they are these files, compressed with zlib:
RubyGems.org | your community gem host
RubyGems.org | your community gem host)

This is a difference of 32 gems, but then perhaps some on the quick list are out
of date? Or perhaps I got the numbers wrong?

So to make things faster:

···

On Mon, Jun 2, 2008 at 4:05 PM, Michael T. Richter <ttmrichter@gmail.com> wrote:

On Mon, 2008-06-02 at 22:01 +0900, Tom Copeland wrote:

And yet another mailing list to sign up for? God DAMN! How about
just a bug tracker somewhere? Is there an URL for that? I really
don't want to increase my incoming mail any more than I already have
it. :frowning:

Yup, there's the usual complement of RubyForge trackers and such here:

http://rubyforge.org/projects/rubygems/

Thanks, Tom. Lack of sleep made me not even think about looking at
RubyForge. :frowning:

--------------------------------

1. Since the bulk index is 854k and expands to 20MB, perhaps there's a way to
keep that quick index more up-to-date?

2. Only 3134 of the 13432 gems are unique gems - 10298 are older versions
of these gems. I think that people rarely search or install old gems, so perhaps
the list can be split into a file for latest versions versus old versions.

3. I often search for gems repeatedly, and the bulk index gets pulled down
repeatedly - why not save this file locally for at least a few hours?
(probably try to implement this myself just now)
-- did this by now, but there seems to be an attempt at this already built-in.

4. Perhaps if the server is taking strain, a mirror or two could be
set up? I doubt
many people would care about such relatively small files on their servers -
I'd be willing to ask some people if they'd do a ZA mirror.

Any comments?

Les