I try to write a script to fecth some webpages from the internet, parse
the webpages and write them into csv format file.
Here are the formats of webpages:
website....A001
...
website....A101
website....B001
...
website....B401
website....Z001
...
website....Z301
I use two loops to do the job. 1) one loop is to iterate from letter A
to Z,
which I have it done by iterate an array containing letter A to Z.
2) another loop is to iterate the number from 1 to whatever numbers I
prefer (or using a while loop).
If I write the script in a procedural way I have no problem at all but
they look kind of ugly. Now I try to rewrite them with OOP method. My
confusion here is that where should I write the loops: in the main
scripts and define a method for them and call the method in the main
scripts?
[A..Z].each do |letter|
[1...num].each do |number| #Process the website in here
process_website letter number
end
end
As for OOP you could have a Parse class and in and a AppRunner class (which
holds all the websites). the process_website would be in the AppRunner class
and would create a new parser for each website, parse the website and then
either if the websites are indepent (have the parse take care of the writing
to csv), or if dependent in some manner (have the AppRunner write to cvs)
Reuben Doetsch
···
On Thu, Jul 17, 2008 at 9:47 AM, Li Chen <chen_li3@yahoo.com> wrote:
Hi all,
I try to write a script to fecth some webpages from the internet, parse
the webpages and write them into csv format file.
Here are the formats of webpages:
website....A001
...
website....A101
website....B001
...
website....B401
website....Z001
...
website....Z301
I use two loops to do the job. 1) one loop is to iterate from letter A
to Z,
which I have it done by iterate an array containing letter A to Z.
2) another loop is to iterate the number from 1 to whatever numbers I
prefer (or using a while loop).
If I write the script in a procedural way I have no problem at all but
they look kind of ugly. Now I try to rewrite them with OOP method. My
confusion here is that where should I write the loops: in the main
scripts and define a method for them and call the method in the main
scripts?
[A..Z].each do |letter|
[1...num].each do |number| #Process the website in here
process_website letter number
end
end
Hi Reuben,
Thank you very much for the neat script. I have another question:
within the nested loop the maximum num is not fixed, for example, there
are A001-A201 for letter A and B001-B333 for letter B.... And most of
the time I don't know what is the maximum page I can fetch for each
letter. But I know if the page doesn't exit that Ruby will fetch the
same papge again and again until it comes to the upper limit of loop. In
order to break out of the loop what I do is that I compare the website
address of current page and the previous page. If they are the same I am
going to break out the current loop and go to next letter. I wonder how
you implement this idea into the current script.
If I understand what you're asking correctly, it would be something
like this (this is pseudo-code)
[A..Z].each do |letter|
old_website = nil
[1...num].each do |number| #Process the website in here
new_website = fetch_website(letter, number)
if new_website != old_website
process_website(new_website)
else #break out of num-loop
end
old_website = new_website
end
end
This is assuming that the only time you will ever have duplication is
when you have reached the end of the loop for a given letter. Is that
correct?
···
On Jul 21, 9:25 am, Li Chen <chen_...@yahoo.com> wrote:
> [A..Z].each do |letter|
> [1...num].each do |number|
> #Process the website in here
> process_website letter number
> end
> end
In order to break out of the loop what I do is that I compare the website
address of current page and the previous page. If they are the same I am
going to break out the current loop and go to next letter. I wonder how
you implement this idea into the current script.