Re: gem list -r -d *** generates Error*** on http://gems.github.com

On Jul 31, 2009, at 12:00, John wrote:

> I'm using ruby 1.8.6 & gem 1.3.5  :
>
> I receive and error trying to list the details of
>
> C:\Users\spanton>gem source
> *** CURRENT SOURCES ***
>
> http://gems.github.com
>
> C:\Users\spanton>gem list -r -d
>
> *** REMOTE GEMS ***
>
> ERROR:  While executing gem ... (ArgumentError)
>    comparison of String with 3 failed
>
> Error also occurs, when using    gem list -r -d a
>
> Yet it works ok, when using       gem list -r -d b
>
> Any ideas or workaround?


$ rm -r ~/.gem/specs/gems.github.com%80/

# I did this with just a as the search first and hit this gem:

$ gem list -r -d --source http://gems.github.com -V andreasronge-neo4j

*** REMOTE GEMS ***

GET 200 OK: http://gems.github.com/latest_specs.4.8.gz
GET 200 OK: 
http://gems.github.com/quick/Marshal.4.8/andreasronge-neo4j-0.0.4.gemspec.rz
ERROR:  While executing gem ... (ArgumentError)
    comparison of String with 3 failed

So removing this gem from the index and rebuilding should fix the  
problem.

Anyhow, let's see why it's broken:

$ curl http://gems.github.com/quick/Marshal.4.8/andreasronge-neo4j-0.0.4.gemspec.rz 
 > neo4j.gemspec.rz

Let's dump it out:

$ ruby -rubygems -e 'Marshal.load  
Gem.inflate(File.read("neo4j.gemspec.rz"))'
/Library/Ruby/Site/1.8/rubygems/specification.rb:296:in `>':  
comparison of String with 3 failed (ArgumentError)
        from /Library/Ruby/Site/1.8/rubygems/specification.rb:296:in `_load'
        from -e:1:in `load'
        from -e:1

Yup, still fails, heavy artillery time!  Need to fake out the  
Gem::Specification stuff and dump the spec:

$ ruby -e 'module Gem; class Requirement; def marshal_load(arr) @arr =  
arr; end; def inspect() @arr end; end; Version = Requirement; class  
Specification; def self._load(str) p Marshal.load(str); nil end end;  
end; Marshal.load File.read("neo4j.gemspec")'
["1.2.0", "2", "andreasronge-neo4j", 0.0.4, Wed Oct 22 15:00:00 -0700  
2008, "A graph database for JRuby", >=#<Gem::Requirement:0x34c60>,  
>=#<Gem::Requirement:0x34b98>, "ruby", [], "neo4j", "andreas.ronge@xxxxxxxxx 
", ["Andreas Ronge"], "A graph database for JRuby", "http://github.com/andreasronge/neo4j/tree 
", "true", "ruby", nil]

Ok, so now we have a dump of the gem.  Let's look at where the error  
came from in Gem::Specification:

   285    ##
   286	  # Load custom marshal format, re-initializing defaults as  
needed
   287  
   288    def self._load(str)
   289      array = Marshal.load str
   290  
   291      spec = Gem::Specification.new
   292      spec.instance_variable_set :@specification_version, array[1]
   293  
   294      current_version = CURRENT_SPECIFICATION_VERSION
   295  
-> 296	    field_count = if spec.specification_version >  
current_version then
   297                       
spec.instance_variable_set :@specification_version,
   298                                                 current_version
   299                      MARSHAL_FIELDS[current_version]
   300                    else
   301                      MARSHAL_FIELDS[spec.specification_version]
   302                    end
   303
   304      if array.size < field_count then
   305        raise TypeError, "invalid Gem::Specification format  
#{array.inspec
t}"
   306      end
   307
   308      spec.instance_variable_set :@rubygems_version,           
array[0]
   309      # spec version
   310      spec.instance_variable_set :@name,                       
array[2]
   311      spec.instance_variable_set :@version,                    
array[3]

So, specification_version is what's broken with the gem.  RubyGems  
expects it to be a number, but we can clearly see in the above dump  
that the second item is a String.  The version of the gem is stored in  
the fourth field, 0.0.4

Pulling down this project and looking through the history we find:

$ git log --grep=release
[...]
commit 0e10242e2ef63addf3ac294bb2e932202fa900fc
Author: Andreas Ronge <andreas.ronge@xxxxxxxxx>
Date:   Mon Nov 17 23:05:26 2008 +0100

    release 0.0.5, added CHANGELOG

commit 0b283793ee2c8e0689274d85f049bb8c2cf4250c
Author: andreas <andreas.ronge@xxxxxxxxx>
Date:   Thu Oct 23 16:19:27 2008 +0200

    released to rubyforge gem repository

commit 22bd9de5a5a6874ec4903af60fb5a95606f229ee
Author: andreas <andreas.ronge@xxxxxxxxx>
Date:   Thu Oct 23 10:38:33 2008 +0200

    prepare for rubyforge release of gem. Added license file

commit 715e4384f0cc34a56580c2c52054e2756c5d06ac
Author: andreas <andreas.ronge@xxxxxxxxx>
Date:   Mon Oct 20 12:13:00 2008 +0200

    released as 0.0.3. Updated documentation

So it looks like 0b283793ee2c8e0689274d85f049bb8c2cf4250c is release  
0.0.4.

$ git checkout 0b283793
$ cat neo4j.gemspec
# WARNING : RAKE AUTO-GENERATED FILE. DO NOT MANUALLY EDIT!
# LAST UPDATED : Thu Oct 23 15:05:22 +0200 2008
#
# RUN : 'rake gem:update_gemspec'

Gem::Specification.new do |s|
 s.description = "A graph database for JRuby"
 s.files = [
 # ...
 ]
 s.rubygems_version = "1.2.0"
 s.platform = "ruby"
 s.date = "Thu Oct 23 00:00:00 +0200 2008" # <-- STRING?
 s.homepage = "http://github.com/andreasronge/neo4j/tree";
 s.rubyforge_project = "neo4j"
 s.bindir = "bin"
 s.summary = "A graph database for JRuby"
 s.rdoc_options = [...]
 s.specification_version = "2" # <-- STRING?
 s.email = "andreas.ronge@xxxxxxxxx"
 s.version = "0.0.4"
 s.required_rubygems_version = ">= 0"
 s.require_paths = ["lib"]
 s.required_ruby_version = ">= 1.8.4"
 s.extra_rdoc_files = ["README.rdoc"]
 s.has_rdoc = "true"
 s.name = "neo4j"
 s.authors = ["Andreas Ronge"]
 s.loaded = "false"
end

So the gemspec has a String for specification_version.  Also, why is  
the date a String?  It should be a Time object!  It seems that this  
was hand-built.

There's a bunch of extra fields set here that RubyGems has defaults  
for, like required_rubygems_version, required_ruby_version,  
require_paths.

Hrm, what's that Rake task look like?

$ cat Rakefile
# [...]
  # Thanks to the Merb project for this code.
  desc "Update Github Gemspec"
  task :update_gemspec do
# [...]
    spec.instance_variables.each do |ivar|
      value = spec.instance_variable_get(ivar)
      name = ivar.split("@").last
# [...]
        case value
        when Array
          value = # ...
        when String
          value = # ...
        else
          value = value.to_s.inspect # WHAT?
        end
        result << " s.#{name} = #{value}\n"
      end
    end
    result << "end"
    File.open(File.join(File.dirname(__FILE__),  
"#{spec.name}.gemspec"), "w"){|f
| f << result}
  end

So the author of this gem isn't entirely to blame here.  What's  
especially strange is that this is an especially broken implementation  
of Gem::Specification#to_ruby which had RDoc all the way back to 0.8.11!

Looking in the merb source, it seems this code was added here:

commit 7d14e7e28be657c9a1a6b4f0730dd4f6d2f7031b
Author: Daniel Neighman <has.sox@xxxxxxxxx>
Date:   Fri Apr 25 13:07:49 2008 +1000

    Adds Code for generating a gemspec for github

http://github.com/wycats/merb/commit/7d14e7e28be657c9a1a6b4f0730dd4f6d2f7031b

So it looks like Daniel Neighman is to blame.  (He may not actually  
be, since he doesn't say where he got this code from.)

I'm not sure how this code was written without looking through  
Gem::Specification, nor how the author missed  
Gem::Specification#to_ruby (which is exactly what they wanted).

It appears that there are other projects infected by this code:

http://www.google.com/codesearch?q=skip_fields+new_platform+original_platform

So if you own one of these, please delete that code!

Fortunately, I now have new things to protect gem repositories from!   
(Ryan hit me for not protecting RubyGems enough)

And again, a gemspec is not something that you just dump something on.  
It's not a big truck.