Bug in base64 decoder of xmlrpc?

ruby-talk
1

Hi,

I am currently implementing support for the MetaWeblog/MovableType API in my weblog engine. The API is simply a bunch of XML-RPC methods. One method allows the uploading of files (metaWeblog.newMediaObject). In order to send the binary file via XML-RPC the data gets base64 encoded. The xmlrpc library automatically decodes the data. Writing the file to disk looks something like this:

def newMediaObject(blogid, username, password, data)
   fileData = data["bits"] # Decoding is handled by xmlrpc automatically
   File.open(data["name"], "wb") { |f| f.write(fileData) }
end

The written data, however, is corrupted. Sometimes even some bytes are missing. Uploading plain text files seems to be ok. But uploading arger files such as JPEG files will result in corrupted data. The client I am using is MarsEdit.

Any suggestions?

Regards,
Lars

···

--
"Stil ist die Fähigkeit, komplizierte Dinge einfach zu sagen - nicht umgekehrt." -- Cocteau, Jean

2

Lars Hoss wrote:

Hi,

I am currently implementing support for the MetaWeblog/MovableType

API

in my weblog engine. The API is simply a bunch of XML-RPC methods.

One

method allows the uploading of files (metaWeblog.newMediaObject). In
order to send the binary file via XML-RPC the data gets base64

encoded.

The xmlrpc library automatically decodes the data. Writing the file

to

disk looks something like this:

def newMediaObject(blogid, username, password, data)
   fileData = data["bits"] # Decoding is handled by xmlrpc

automatically

   File.open(data["name"], "wb") { |f| f.write(fileData) }
end

The written data, however, is corrupted. Sometimes even some bytes

are

missing. Uploading plain text files seems to be ok. But uploading

arger

files such as JPEG files will result in corrupted data. The client I

am

using is MarsEdit.

Any suggestions?

Regards,
Lars

--
"Stil ist die Fähigkeit, komplizierte Dinge einfach zu sagen - nicht

umgekehrt." -- Cocteau, Jean

I remember bringing something like this up with the author last year
after reading Randy Ray's use.perl journal entry at
http://use.perl.org/~rjray/journal/9899 and
http://use.perl.org/~rjray/journal/9922\. Randy Ray is the author of
the Perl module RPC::XML, btw.

This was Michael's response at the time (please correct me if things
have changed Michael):

"Implementing [Base64 streaming support] in xmlrpc4r is possible, but
it would require some parts to be redesigned, especially where strings
are returned (e.g. methodResponse).

One problem is that the XML-RPC spec. requires the size of the
XML-message to
be known in the HTTP header (which is send first). Without this
requirement, the whole
XML-RPC message could be streamed out a socket."

Whether anything has been done since then (about a year ago), I don't
know.
If this is unrelated, then I apologize for the noise.

Regards,

Dan

3

Daniel Berger wrote:

Lars Hoss wrote:

Hi,

I am currently implementing support for the MetaWeblog/MovableType

API

in my weblog engine. The API is simply a bunch of XML-RPC methods.

One

method allows the uploading of files (metaWeblog.newMediaObject). In
order to send the binary file via XML-RPC the data gets base64

encoded.

The xmlrpc library automatically decodes the data. Writing the file

to

disk looks something like this:

def newMediaObject(blogid, username, password, data)
  fileData = data["bits"] # Decoding is handled by xmlrpc

automatically

  File.open(data["name"], "wb") { |f| f.write(fileData) }
end

The written data, however, is corrupted. Sometimes even some bytes

are

missing. Uploading plain text files seems to be ok. But uploading

arger

files such as JPEG files will result in corrupted data. The client I

am

using is MarsEdit.

Any suggestions?

Regards,
Lars

--
"Stil ist die Fähigkeit, komplizierte Dinge einfach zu sagen - nicht

umgekehrt." -- Cocteau, Jean

I remember bringing something like this up with the author last year
after reading Randy Ray's use.perl journal entry at
http://use.perl.org/~rjray/journal/9899 and
http://use.perl.org/~rjray/journal/9922\. Randy Ray is the author of
the Perl module RPC::XML, btw.

This was Michael's response at the time (please correct me if things
have changed Michael):

"Implementing [Base64 streaming support] in xmlrpc4r is possible, but
it would require some parts to be redesigned, especially where strings
are returned (e.g. methodResponse).

One problem is that the XML-RPC spec. requires the size of the
XML-message to
be known in the HTTP header (which is send first). Without this
requirement, the whole
XML-RPC message could be streamed out a socket."

Whether anything has been done since then (about a year ago), I don't
know.
If this is unrelated, then I apologize for the noise.

Hm, it should work if you pass a XMLRPC::Base64 object as argument, and not a String. Of course streaming does not work, but that's not related to the corruption. Maybe the best would be to encode every string as base64, but that would probably break many clients or servers (other than xmlrpc4r).

Regards,

  Michael

4

Hi Michael, hi Daniel,

first thanks for your help. After digging in the sources I think I discovered the problem. The problem is that somehow all "+" characters in the base64 encoded string get converted into a " " character. I am still not sure at which point the conversion happens. Probably Rails thinks that the XML-RPC data is a request parameter and therefore automatically converts all "+" characters back into " " characters which is of course ok for regular request parameters. But as I said before, I am still searching ...

Yours,
Lars

···

Am 17.01.2005 um 22:17 schrieb Michael Neumann:

Daniel Berger wrote:

Lars Hoss wrote:

Hi,

I am currently implementing support for the MetaWeblog/MovableType

API

in my weblog engine. The API is simply a bunch of XML-RPC methods.

One

method allows the uploading of files (metaWeblog.newMediaObject). In
order to send the binary file via XML-RPC the data gets base64

encoded.

The xmlrpc library automatically decodes the data. Writing the file

to

disk looks something like this:

def newMediaObject(blogid, username, password, data)
  fileData = data["bits"] # Decoding is handled by xmlrpc

automatically

  File.open(data["name"], "wb") { |f| f.write(fileData) }
end

The written data, however, is corrupted. Sometimes even some bytes

are

missing. Uploading plain text files seems to be ok. But uploading

arger

files such as JPEG files will result in corrupted data. The client I

am

using is MarsEdit.

Any suggestions?

Regards,
Lars

--
"Stil ist die Fähigkeit, komplizierte Dinge einfach zu sagen - nicht
umgekehrt." -- Cocteau, Jean

I remember bringing something like this up with the author last year
after reading Randy Ray's use.perl journal entry at
http://use.perl.org/~rjray/journal/9899 and
http://use.perl.org/~rjray/journal/9922\. Randy Ray is the author of
the Perl module RPC::XML, btw.
This was Michael's response at the time (please correct me if things
have changed Michael):
"Implementing [Base64 streaming support] in xmlrpc4r is possible, but
it would require some parts to be redesigned, especially where strings
are returned (e.g. methodResponse).
One problem is that the XML-RPC spec. requires the size of the
XML-message to
be known in the HTTP header (which is send first). Without this
requirement, the whole
XML-RPC message could be streamed out a socket."
Whether anything has been done since then (about a year ago), I don't
know.
If this is unrelated, then I apologize for the noise.

Hm, it should work if you pass a XMLRPC::Base64 object as argument, and not a String. Of course streaming does not work, but that's not related to the corruption. Maybe the best would be to encode every string as base64, but that would probably break many clients or servers (other than xmlrpc4r).

Regards,

Michael

--
"Stil ist die Fähigkeit, komplizierte Dinge einfach zu sagen - nicht umgekehrt." -- Cocteau, Jean