# Converting a string to an array of tokens
**URL:** https://rubytalk.org/t/converting-a-string-to-an-array-of-tokens/9663
**Category:** ruby-talk
**Created:** [13 January 2004 15:52 UTC](https://rubytalk.org/t/converting-a-string-to-an-array-of-tokens/9663 "2004-01-13T15:52:02Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@Weirich\_James](https://rubytalk.org/u/Weirich_James)
#### Post date: [13 January 2004 15:52 UTC](https://rubytalk.org/t/converting-a-string-to-an-array-of-tokens/9663/1 "2004-01-13T15:52:02Z")
> class String
> def tokenize(\*tokens)
> array =
> each\_token(\*tokens){|tk| array \<\< tk}
> array
> end
>
> ```
> def each_token(*tokens)
> regex = Regexp.new(tokens.map { |t| t.kind_of?( Regexp ) ? t :
>
> ```
>
> Regexp::escape(t) }.join(“|”))
> string = self
>
> ```
> while( match = regex.match(string) )
> yield match.pre_match if match.pre_match.length > 0
> yield match[0] if match[0].length > 0
> string = match.post_match
> end
>
> yield string if string.length > 0
> self
> end
>
> ```
>
> end
Question 1:
Except for the pre\_match, isn’t this just doing the same thing as scan. And
if that’s the case, why not just include the expected prematch patterns in
the list of regexps and use scan directly?
Question 2:
Ben Tilly pointed out (a long time ago) that the “string =
string.post\_match” type of statement is enormously inefficient for large
strings because amount of string copying involved. In ruby-talk:89747 Nobu
Nakada indicated that string tails _could_ be shared and use copy-on-write
(COW). In current Ruby, are the strings shared with COW semantics, or was
Nobu just speculating on possible implementations?
> **···**
>
> –
> – Jim Weirich / Compuware
> – FWP Capture Services
> – Phone: 859-386-8855
---
### Author:  [@John\_Long2](https://rubytalk.org/u/John_Long2)
#### Post date: [14 January 2004 04:49 UTC](https://rubytalk.org/t/converting-a-string-to-an-array-of-tokens/9663/2 "2004-01-14T04:49:33Z")
“Weirich, James” wrote:
> Question 1:
>
> Except for the pre\_match, isn’t this just doing the same thing as scan.
> And
> if that’s the case, why not just include the expected prematch patterns in
> the list of regexps and use scan directly?
It’s the unexpected that I’m thinking about. How do you make it so that it
will match anything other than your token? This doesn’t seem to work:
regex = Regexp.new(
“(?m:” \<\< tokens.map { |t|
Regexp::escape(t)
}.join(“|”) \<\< “|.\*)”
)
scan(regex)
> Question 2:
>
> Ben Tilly pointed out (a long time ago) that the “string =
> string.post\_match” type of statement is enormously inefficient for large
> strings because amount of string copying involved. In ruby-talk:89747
> Nobu
> Nakada indicated that string tails _could_ be shared and use copy-on-write
> (COW). In current Ruby, are the strings shared with COW semantics, or was
> Nobu just speculating on possible implementations?
I’m open to other suggestions, but the one presented seems the most elegant
so far.
I’ve attached code and a TestCase for those who want to fool around with it.
[tokens.rb](https://rubytalk.org/uploads/short-url/5xxNiVFdAThEcxhwNS5xEOzvZjy.rb) (2.16 KB)
> **···**
>
> –
> John Long
> [http://wiseheartdesign.com](http://wiseheartdesign.com)
---
### Author:  [@Nobuyoshi\_Nakada](https://rubytalk.org/u/Nobuyoshi_Nakada)
#### Post date: [14 January 2004 06:36 UTC](https://rubytalk.org/t/converting-a-string-to-an-array-of-tokens/9663/3 "2004-01-14T06:36:10Z")
Hi,
> **···**
>
> At Wed, 14 Jan 2004 00:52:02 +0900, Weirich, James wrote:
>
> > Question 2:
> >
> > Ben Tilly pointed out (a long time ago) that the “string =
> > string.post\_match” type of statement is enormously inefficient for large
> > strings because amount of string copying involved. In ruby-talk:89747 Nobu
> > Nakada indicated that string tails _could_ be shared and use copy-on-write
> > (COW). In current Ruby, are the strings shared with COW semantics, or was
> > Nobu just speculating on possible implementations?
>
> It has been implemented.
>
> –
> Nobu Nakada