SQLite-Ruby and "other chrs"

Hi,

a problem with "other chars" hits me... :slight_smile:

I try to read a database of shortwave broadcasters (formatted ascii)
into a SQLIte-Ruby database.

The initialization is ok, but from time to time the population of the
db breaks, cause some broadcasters station names include invalid
chars like

聽聽聽聽聽R.N'Djamena

and the "'" kills everything.

Escaping that character in the textfile, which is read to populate
the db, doesn't work, cause "\'" ist read as "\" + "'", which
again is one "'" to much.

With mysql there was a method called ".Mysql_escape_string" which
does this job. In the SQLite and SQLite-Ruby docs I haven't found
anything similiar...

How can I solve this problem?

Kind regards and thank you very much in advance,
Meino

fields.map!{|f| f.gsub(%r/'/o, "''")}

~ > sqlite db "create table foo(bar); insert into foo values ('''quoted'''); select * from foo;"
'quoted'

-a

路路路

On Sat, 26 Jun 2004, Meino Christian Cramer wrote:

Hi,

a problem with "other chars" hits me... :slight_smile:

I try to read a database of shortwave broadcasters (formatted ascii)
into a SQLIte-Ruby database.

The initialization is ok, but from time to time the population of the
db breaks, cause some broadcasters station names include invalid
chars like

     R.N'Djamena

and the "'" kills everything.

Escaping that character in the textfile, which is read to populate
the db, doesn't work, cause "\'" ist read as "\" + "'", which
again is one "'" to much.

With mysql there was a method called ".Mysql_escape_string" which
does this job. In the SQLite and SQLite-Ruby docs I haven't found
anything similiar...

How can I solve this problem?

Kind regards and thank you very much in advance,
Meino

--

EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
PHONE :: 303.497.6469
A flower falls, even though we love it;
and a weed grows, even though we do not love it. --Dogen

===============================================================================

I require the module 'base64' and then encode64 everything that could have
funky characters before I put it in the DB.

Obviously to get it out you'd want to decode64 everything that was encoded.

This will let you put binary files into sqlite as well, since there's no
'BLOB' column type.

whyTHEluckySTIFF's tutorial got me started...

http://whytheluckystiff.net/articles/aQuickGuideToSQLite.html

-Richard

路路路

----- Original Message -----
From: "Meino Christian Cramer" <Meino.Cramer@gmx.de>
To: "ruby-talk ML" <ruby-talk@ruby-lang.org>
Sent: Saturday, June 26, 2004 6:05 AM
Subject: SQLite-Ruby and "other chrs"

Hi,

a problem with "other chars" hits me... :slight_smile:

I try to read a database of shortwave broadcasters (formatted ascii)
into a SQLIte-Ruby database.

The initialization is ok, but from time to time the population of the
db breaks, cause some broadcasters station names include invalid
chars like

   R.N'Djamena

and the "'" kills everything.

Escaping that character in the textfile, which is read to populate
the db, doesn't work, cause "\'" ist read as "\" + "'", which
again is one "'" to much.

With mysql there was a method called ".Mysql_escape_string" which
does this job. In the SQLite and SQLite-Ruby docs I haven't found
anything similiar...

How can I solve this problem?

Kind regards and thank you very much in advance,
Meino

[...]

  Escaping that character in the textfile, which is read to populate
  the db, doesn't work, cause "\'" ist read as "\" + "'", which
  again is one "'" to much.

Escape single and and double quotes by doubling them, not by preceding
them with a backslash. Note that SQLite strings can handle all
characters except the NUL character.

      Reimer Behrends

路路路

Meino Christian Cramer (Meino.Cramer@gmx.de) wrote:

i'd forgotten i had this - it's old but it should work:

路路路

On Sat, 26 Jun 2004, Meino Christian Cramer wrote:

Hi,

a problem with "other chars" hits me... :slight_smile:

I try to read a database of shortwave broadcasters (formatted ascii)
into a SQLIte-Ruby database.

The initialization is ok, but from time to time the population of the
db breaks, cause some broadcasters station names include invalid
chars like

     R.N'Djamena

and the "'" kills everything.

Escaping that character in the textfile, which is read to populate
the db, doesn't work, cause "\'" ist read as "\" + "'", which
again is one "'" to much.

With mysql there was a method called ".Mysql_escape_string" which
does this job. In the SQLite and SQLite-Ruby docs I haven't found
anything similiar...

How can I solve this problem?

Kind regards and thank you very much in advance,
Meino

===============================================================================
file: quote.rb

#
# a module for quoting sql database entries
#
# eg. #
# tuple = %w( it's hard to quote this )
# values = Quote.q tuple
# sql = "insert into tbl values ( #{ values.join ',' } );"
#
# => insert into tbl values ( 'it''s','hard','to','quote','this' );
#
   module Quote
#{{{
   #
   # escapes (with esc) all occurances of char in s, modifying s in place
   #
     def escape! s, char, esc
#{{{
       re = %r/([#{0x5c.chr << esc}]*)#{char}/
       s.gsub!(re) do
         (($1.size % 2 == 0) ? ($1 << esc) : $1) + char
       end
#}}}
     end
     module_function 'escape!'
     public 'escape!'
   #
   # escapes (with esc) all occurances of char in s, returning a new str
   #
     def escape s, char, esc
#{{{
       ss = "#{ s }"
       escape! ss, char, esc
       ss
#}}}
     end
     module_function 'escape'
     public 'escape'
   #
   # given an object, return a single quoted str of this object, escaping all
   # occurances of ' iff esc is true (default). this method recognizes Array
   # objects as special cases and will recurse over any sub Arrays untill all
   # sub objects are single quoted
   #
   # p (q [42, 'ford']) # >> ["'42'", "'ford'"]
   #
     def q object, esc = true, accum = nil
#{{{
       quote object, esc, "'", accum
#}}}
     end
     module_function 'q'
     public 'q'
   #
   # same a q but with double quotes
   #
     def qq object, esc = true, accum = nil
#{{{
       quote object, esc, '"', accum
#}}}
     end
     module_function 'qq'
     public 'qq'
   #
   # impl of q and qq
   #
     def quote object, esc = true, q = "'", accum = nil
#{{{
       if Array === object
         if accum
           accum <<
           accum = accum[-1]
         else
           accum =
         end
         object.each{|object| quote object, esc, q, accum}
         return accum
       end

       quoted =
         if esc
           "#{ q }#{ escape object, q, q }#{ q }"
         else
           "#{ q }#{ object }#{ q }"
         end

       if accum
         accum << quoted
       else
         quoted
       end
#}}}
     end
     module_function 'quote'
     public 'quote'
#}}}
   end
#
# sample usage
#
if $0 == __FILE__
#{{{

#
# simple
#
    tuple = %w( it's hard to quote this )
    values = Quote.q tuple
    sql = "insert into tbl values ( #{ values.join ',' } );"
    puts sql
    puts

#
# less simple
#

   include Quote

   tests =
     [
       [%Q|a\\\"bc|, '"', '\\'],
       [%Q|a\\\\\"bc|, '"', '\\'],
       [%q|a'b'c|, "'", '\\'],
       [%q|a#b#c|, '#', '\\'],
     ]

   tests.each do |test|
     s, char, esc = test
     e = escape s, char, esc
     puts "s : <#{ s }>"
     puts "e : <#{ e }>"
     puts
   end

   quoted = q 'hi'
   puts quoted

   quoted = q ['hi', 'hello']
   puts(quoted.join(' '))

   quoted = q [['hi', 'hello'],['howdy','hullo']]
   puts(quoted[0].join(' '))
   puts(quoted[1].join(' '))

   quoted = q [[['hi', 'hello'],['howdy','hullo']]]
   puts(quoted[0][0].join(' '))
   puts(quoted[0][1].join(' '))

   quoted = q [[['h\'i', 'he\'llo'],['howdy','hullo']]]
   puts(quoted[0][0].join(' '))
   puts(quoted[0][1].join(' '))

   quoted = qq 'hi'
   puts(quoted)

   quoted = qq ['hi', 'hello']
   puts(quoted.join(' '))

   quoted = qq [['hi', 'hello'],['howdy','hullo']]
   puts(quoted[0].join(' '))
   puts(quoted[1].join(' '))

   quoted = qq [[['hi', 'hello'],['howdy','hullo']]]
   puts(quoted[0][0].join(' '))
   puts(quoted[0][1].join(' '))

   quoted = qq [[['h"i', 'hel"lo'],['howdy','hullo']]]
   puts(quoted[0][0].join(' '))
   puts(quoted[0][1].join(' '))
#}}}
end

cheers.

-a
--

EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
PHONE :: 303.497.6469
A flower falls, even though we love it;
and a weed grows, even though we do not love it. --Dogen

===============================================================================

Meino Christian Cramer wrote:

The initialization is ok, but from time to time the population of the
db breaks, cause some broadcasters station names include invalid
chars like

     R.N'Djamena

and the "'" kills everything.

If you are using the DBI package to access the database, this is handled for you automatically when you use the ? markers in your SQL. E.g.

    db.do("INSERT INTO table(name) VALUES(?);", "R.N'Djamena")

DBI will automatically expand the "?" into the properly escaped string for you. A lot of DBs will do this for you as well, I just know that DBI will handle even if the database does not.

(There is also an escape method in the DBI somewhere that allows you to do it manually, but the ? method is much better).

路路路

--
-- Jim Weirich jim@weirichhouse.org http://onestepback.org
-----------------------------------------------------------------
"Beware of bugs in the above code; I have only proved it correct,
not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas)

I dont want to encode64 everything (nearly 3197507 normal chars) to
avoid problems of a handful "'"s...

There must be another solution.

I require the module 'base64' and then encode64 everything that could have
funky characters before I put it in the DB.

Obviously to get it out you'd want to decode64 everything that was encoded.

This will let you put binary files into sqlite as well, since there's no
'BLOB' column type.

whyTHEluckySTIFF's tutorial got me started...

http://whytheluckystiff.net/articles/aQuickGuideToSQLite.html

  I read this...

路路路

From: Rich <rich@lithinos.com>
Subject: Re: SQLite-Ruby and "other chrs"
Date: Sat, 26 Jun 2004 23:58:13 +0900

-Richard

----- Original Message -----
From: "Meino Christian Cramer" <Meino.Cramer@gmx.de>
To: "ruby-talk ML" <ruby-talk@ruby-lang.org>
Sent: Saturday, June 26, 2004 6:05 AM
Subject: SQLite-Ruby and "other chrs"

> Hi,
>
> a problem with "other chars" hits me... :slight_smile:
>
> I try to read a database of shortwave broadcasters (formatted ascii)
> into a SQLIte-Ruby database.
>
> The initialization is ok, but from time to time the population of the
> db breaks, cause some broadcasters station names include invalid
> chars like
>
> R.N'Djamena
>
> and the "'" kills everything.
>
> Escaping that character in the textfile, which is read to populate
> the db, doesn't work, cause "\'" ist read as "\" + "'", which
> again is one "'" to much.
>
> With mysql there was a method called ".Mysql_escape_string" which
> does this job. In the SQLite and SQLite-Ruby docs I haven't found
> anything similiar...
>
> How can I solve this problem?
>
> Kind regards and thank you very much in advance,
> Meino
>

Other way around:

I have a string containing a backslash.
How can I replace that backslash with nothing?

Or:
How can I store a string in a variable, which looks like this

    "R.N" + "'" + "Djamena"

( start- and end-" not included ).

When trying this:

a="\"R.N\" + \"'\" + \"Djamena\""

I get naturelamente exactly this

  \"R.N\" + \"'\" + \"Djamena\"

as contents of the string. The " weren't recognized as escaped as in
(shell scripting / bash )

     echo "This the value of a \"string\""

This is the value of a "string".

Regexp.escape also seems not to fit this problem...

路路路

From: Rich <rich@lithinos.com>
Subject: Re: SQLite-Ruby and "other chrs"
Date: Sat, 26 Jun 2004 23:58:13 +0900

I require the module 'base64' and then encode64 everything that could have
funky characters before I put it in the DB.

Obviously to get it out you'd want to decode64 everything that was encoded.

This will let you put binary files into sqlite as well, since there's no
'BLOB' column type.

whyTHEluckySTIFF's tutorial got me started...

http://whytheluckystiff.net/articles/aQuickGuideToSQLite.html

-Richard

----- Original Message -----
From: "Meino Christian Cramer" <Meino.Cramer@gmx.de>
To: "ruby-talk ML" <ruby-talk@ruby-lang.org>
Sent: Saturday, June 26, 2004 6:05 AM
Subject: SQLite-Ruby and "other chrs"

> Hi,
>
> a problem with "other chars" hits me... :slight_smile:
>
> I try to read a database of shortwave broadcasters (formatted ascii)
> into a SQLIte-Ruby database.
>
> The initialization is ok, but from time to time the population of the
> db breaks, cause some broadcasters station names include invalid
> chars like
>
> R.N'Djamena
>
> and the "'" kills everything.
>
> Escaping that character in the textfile, which is read to populate
> the db, doesn't work, cause "\'" ist read as "\" + "'", which
> again is one "'" to much.
>
> With mysql there was a method called ".Mysql_escape_string" which
> does this job. In the SQLite and SQLite-Ruby docs I haven't found
> anything similiar...
>
> How can I solve this problem?
>
> Kind regards and thank you very much in advance,
> Meino
>

I'm not sure you can have a generalized quoting method. Different
rdbms may require different escaping, I think, it is wring?
And, IMO it would be nice to make a quote method built-in in any rdbms
library, even if the db does not provide it itself (like it seem
sqlite does).

路路路

il Mon, 28 Jun 2004 21:34:51 -0600, "Ara.T.Howard" <ahoward@noaa.gov> ha scritto::

You could URL encode it...

... I've spent a bit of time in the past working with FORM submitted
information that has to be made 'safe' before it can be stored or even
before it can be brought into a scripted environment... and encoding it the
way it would need to be to be in a URL has always worked.

So...

<START CODE>

require "cgi"
bad_string="a's"+'d"f'
puts CGI::escape(bad_string) # prints out: a%27sd%22f

<END CODE>

That should be a little easier.

-Richard

路路路

----- Original Message -----
From: "Meino Christian Cramer" <Meino.Cramer@gmx.de>
To: <ruby-talk@ruby-lang.org>; <rich@lithinos.com>
Sent: Saturday, June 26, 2004 10:42 AM
Subject: Re: SQLite-Ruby and "other chrs"

From: Rich <rich@lithinos.com>
Subject: Re: SQLite-Ruby and "other chrs"
Date: Sat, 26 Jun 2004 23:58:13 +0900

Other way around:

I have a string containing a backslash.
How can I replace that backslash with nothing?

Or:
How can I store a string in a variable, which looks like this

"R.N" + "'" + "Djamena"

( start- and end-" not included ).

When trying this:

a="\"R.N\" + \"'\" + \"Djamena\""

I get naturelamente exactly this

  \"R.N\" + \"'\" + \"Djamena\"

as contents of the string. The " weren't recognized as escaped as in
(shell scripting / bash )

   echo "This the value of a \"string\""

This is the value of a "string".

Regexp.escape also seems not to fit this problem...

> I require the module 'base64' and then encode64 everything that could

have

> funky characters before I put it in the DB.
>
> Obviously to get it out you'd want to decode64 everything that was

encoded.

>
> This will let you put binary files into sqlite as well, since there's no
> 'BLOB' column type.
>
> whyTHEluckySTIFF's tutorial got me started...
>
> http://whytheluckystiff.net/articles/aQuickGuideToSQLite.html
>
> -Richard
>
> ----- Original Message -----
> From: "Meino Christian Cramer" <Meino.Cramer@gmx.de>
> To: "ruby-talk ML" <ruby-talk@ruby-lang.org>
> Sent: Saturday, June 26, 2004 6:05 AM
> Subject: SQLite-Ruby and "other chrs"
>
>
> > Hi,
> >
> > a problem with "other chars" hits me... :slight_smile:
> >
> > I try to read a database of shortwave broadcasters (formatted ascii)
> > into a SQLIte-Ruby database.
> >
> > The initialization is ok, but from time to time the population of the
> > db breaks, cause some broadcasters station names include invalid
> > chars like
> >
> > R.N'Djamena
> >
> > and the "'" kills everything.
> >
> > Escaping that character in the textfile, which is read to populate
> > the db, doesn't work, cause "\'" ist read as "\" + "'", which
> > again is one "'" to much.
> >
> > With mysql there was a method called ".Mysql_escape_string" which
> > does this job. In the SQLite and SQLite-Ruby docs I haven't found
> > anything similiar...
> >
> > How can I solve this problem?
> >
> > Kind regards and thank you very much in advance,
> > Meino
> >
>
>

I have a string containing a backslash.
How can I replace that backslash with nothing?

str.gsub('\\', '')

Or:
How can I store a string in a variable, which looks like this

                "R.N" + "'" + "Djamena"

str = %{"R.N" + "'" + "Djamena"}

Gavin

路路路

On Sunday, June 27, 2004, 2:42:25 AM, Meino wrote:

I'm not sure you can have a generalized quoting method. Different
rdbms may require different escaping, I think, it is wring?

no, you are right and that's why i've not released this in any real capacity -
nevertheless it's useful now:

   ~/eg/ruby > cat a.rb
   require 'quote.rb'
   include Quote

   tuple = %w( rdbms's quote differently )

   pgsql = q(tuple, '\\')
   sqlite = q(tuple, "'")

   puts(pgsql.join(' , '))
   puts(sqlite.join(' , '))

   ~/eg/ruby > ruby a.rb
   'rdbms\'s' , 'quote' , 'differently'
   'rdbms''s' , 'quote' , 'differently'

(btw. my origninal post had a typo which your comment made me find. thanks!)

And, IMO it would be nice to make a quote method built-in in any rdbms
library, even if the db does not provide it itself (like it seem
sqlite does).

i think it should be an extenstion to String (#escape) and Array (#quote) since
more than just rdbms package would use it. for instance csv, shell commands,
code generation, etc.

cheers.

(see updated quote.rb)

-a

路路路

On Tue, 29 Jun 2004, gabriele renzi wrote:

il Mon, 28 Jun 2004 21:34:51 -0600, "Ara.T.Howard" <ahoward@noaa.gov> > ha scritto::

--

EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
PHONE :: 303.497.6469
A flower falls, even though we love it;
and a weed grows, even though we do not love it. --Dogen

===============================================================================

#
# a module for quoting sql database entries
#
# eg. #
# tuple = %w( it's hard to quote this )
# values = Quote.q tuple
# sql = "insert into tbl values ( #{ values.join ',' } );"
#
# => insert into tbl values ( 'it''s','hard','to','quote','this' );
#
   module Quote
#{{{
   #
   # escapes (with esc) all occurances of char in s, modifying s in place
   #
     def escape! s, char, esc
#{{{
       re = %r/([#{0x5c.chr << esc}]*)#{char}/
       s.gsub!(re) do
         (($1.size % 2 == 0) ? ($1 << esc) : $1) + char
       end
#}}}
     end
     module_function 'escape!'
     public 'escape!'
   #
   # escapes (with esc) all occurances of char in s, returning a new str
   #
     def escape s, char, esc
#{{{
       ss = "#{ s }"
       escape! ss, char, esc
       ss
#}}}
     end
     module_function 'escape'
     public 'escape'
   #
   # given an object, return a single quoted str of this object, escaping all
   # occurances of ' iff esc is true (default). this method recognizes Array
   # objects as special cases and will recurse over any sub Arrays untill all
   # sub objects are single quoted
   #
   # p (q [42, 'ford']) # >> ["'42'", "'ford'"]
   #
     def q object, esc = "'", accum = nil
#{{{
       quote object, esc, "'", accum
#}}}
     end
     module_function 'q'
     public 'q'
   #
   # same a q but with double quotes
   #
     def qq object, esc = '"', accum = nil
#{{{
       quote object, esc, '"', accum
#}}}
     end
     module_function 'qq'
     public 'qq'
   #
   # impl of q and qq
   #
     def quote object, esc = nil, q = nil, accum = nil
#{{{
       if Array === object
         if accum
           accum <<
           accum = accum[-1]
         else
           accum =
         end
         object.each{|object| quote object, esc, q, accum}
         return accum
       end

       quoted =
         if esc
           "#{ q }#{ escape object, q, esc }#{ q }"
         else
           "#{ q }#{ object }#{ q }"
         end

       if accum
         accum << quoted
       else
         quoted
       end
#}}}
     end
     module_function 'quote'
     public 'quote'
#}}}
   end

#
# sample usage
#
if $0 == __FILE__
#{{{

#
# simple
#
    tuple = %w( it's hard to quote this )
    values = Quote.q tuple
    sql = "insert into tbl values ( #{ values.join ',' } );"
    puts sql
    puts

#
# less simple
#

   include Quote

   tests =
     [
       [%Q|a\\\"bc|, '"', '\\'],
       [%Q|a\\\\\"bc|, '"', '\\'],
       [%q|a'b'c|, "'", '\\'],
       [%q|a#b#c|, '#', '\\'],
     ]

   tests.each do |test|
     s, char, esc = test
     e = escape s, char, esc
     puts "s : <#{ s }>"
     puts "e : <#{ e }>"
     puts
   end

   quoted = q 'hi'
   puts quoted

   quoted = q ['hi', 'hello']
   puts(quoted.join(' '))

   quoted = q [['hi', 'hello'],['howdy','hullo']]
   puts(quoted[0].join(' '))
   puts(quoted[1].join(' '))

   quoted = q [[['hi', 'hello'],['howdy','hullo']]]
   puts(quoted[0][0].join(' '))
   puts(quoted[0][1].join(' '))

   quoted = q [[['h\'i', 'he\'llo'],['howdy','hullo']]]
   puts(quoted[0][0].join(' '))
   puts(quoted[0][1].join(' '))

   quoted = qq 'hi'
   puts(quoted)

   quoted = qq ['hi', 'hello']
   puts(quoted.join(' '))

   quoted = qq [['hi', 'hello'],['howdy','hullo']]
   puts(quoted[0].join(' '))
   puts(quoted[1].join(' '))

   quoted = qq [[['hi', 'hello'],['howdy','hullo']]]
   puts(quoted[0][0].join(' '))
   puts(quoted[0][1].join(' '))

   quoted = qq [[['h"i', 'hel"lo'],['howdy','hullo']]]
   puts(quoted[0][0].join(' '))
   puts(quoted[0][1].join(' '))
#}}}
end

Ara.T.Howard said:

   pgsql = q(tuple, '\\')

FYI: PostgreSQL also supports the double '' style escaping too.

路路路

--
-- Jim Weirich jim@weirichhouse.org http://onestepback.org
-----------------------------------------------------------------
"Beware of bugs in the above code; I have only proved it correct,
not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas)