(noob) need help parsing Apache log file


(Mark Hubbart) #1

Being very new to Ruby, I want to be able to write a small script to
open my Apache access_log file and remove any lines that contain (.exe

.ida) from the source file. I managed to figure out how to get this
information and log it to a new file, but I would really appreciate an
example of how to read and remove from one file. I am the kind of
person who learns best watching others.

This is what I wrote to parse the file. Any revisions, hints, hacks,
etc would also be very much appreciated.

One small change will write out the file. Instead of:

arr.each do |line|
msEvil.push line if line =~ /(.exe| .ida)/
end

try:

reject! deletes matched lines

arr.reject! do |line|
msEvil.push line if line =~ /(.exe| .ida)/
end

truncate the file and write out

File.open(src, “w+”){|f| f.write arr.join }

Array#reject! does the same thing as Array#sort, except it pays
attention to the return value of the passed block. If the block returns
a true value, it deletes the item from the array. There is also the
non-destructive version, Array#reject (no bang), but reject! makes more
sense here.

File.open opens the file (surprise!). In this case, it truncates it and
opens it write only. If you give it a block, it passes the opened file
handle to the block, and automagically closes the file when it’s done
:slight_smile: You might want to use the same trick at the bottom:

log to text file

f = File.new(“codeRed.txt”, “w+”)

msEvil.each do |line|
f.puts line
end

f.close

could become:

log to text file

File.open(“codeRed.txt”, “w+”){|f| f.write msEvil.join }

That’s a handy Ruby idiom. Shortens code considerably :slight_smile:

And finally, here’s the results of running this script on my logs:

mark@imac% sudo mslogstripminer
Password:
Report for: Sat Feb 28 00:06:42 PST 2004

···

On Feb 27, 2004, at 10:19 PM, Koncept wrote:
----------------------------------------

Total lines: 1670
Total Window exploits: 104
mark@imac%

And that’s my box at home!!! absolutely disgusting. :slight_smile:

–Mark


(Mark Hubbart) #2

You need to catch the errors. Enclose the error-generating code in a
begin…rescue…end statement:

begin
# error-prone code here
rescue
warn "You don’t have permission to access the log. Try using sudo."
exit
end

···

On Feb 28, 2004, at 10:24 AM, Koncept wrote:

Mark. One other thing…

If the script is run without sudo permissions, this is the error
thrown. How do I catch something like this and present a nicer
message to the user?

Just replace the comment with the line (or lines) of code. You’ll
probably want to do this with the code where you read in the file, and
also the two places where you write the data out. For much more
detailed information on error catching, refer to the PickAxe Guide1,
“Exceptions, Catch, and Throw”.

–Mark


(Mark Hubbart) #3

We’re all nice here because programming in ruby is fun :slight_smile: as for
Perl… I’d better not say anything.

–Mark

···

On Feb 28, 2004, at 1:19 PM, Koncept wrote:

It is so nice to be a part of a group ( unlike those Perl folks ) where
people are actually nice to each other.


(Eric Schwartz) #4

Mark Hubbart discord@mac.com writes:

It is so nice to be a part of a group ( unlike those Perl folks ) where
people are actually nice to each other.

We’re all nice here because programming in ruby is fun :slight_smile: as for
Perl… I’d better not say anything.

To be fair, unlike Ruby, Perl has tons of documentation distributed
with it that a poster could easily consult before posting a question;
where it’s not fair to tell someone to just “Read the Pickaxe”, since
that’s not definitive or complete, it is fairly reasonable for a Perl
programmer to be told to read the standard documentation on her hard
drive before asking a question it could easily answer.

Hopefully, the hard work being put into the RDoc<->ri integration
effort will result in similarly high-quality documentation being
included with a default distribution of Ruby.

-=Eric

···

On Feb 28, 2004, at 1:19 PM, Koncept wrote:

Come to think of it, there are already a million monkeys on a million
typewriters, and Usenet is NOTHING like Shakespeare.
– Blair Houghton.


(Mac) #5

Eric Schwartz wrote:

Mark Hubbart discord@mac.com writes:

It is so nice to be a part of a group ( unlike those Perl folks ) where
people are actually nice to each other.

We’re all nice here because programming in ruby is fun :slight_smile: as for
Perl… I’d better not say anything.

To be fair, unlike Ruby, Perl has tons of documentation distributed
with it that a poster could easily consult before posting a question;
where it’s not fair to tell someone to just “Read the Pickaxe”, since
that’s not definitive or complete, it is fairly reasonable for a Perl
programmer to be told to read the standard documentation on her hard
drive before asking a question it could easily answer.

Hopefully, the hard work being put into the RDoc<->ri integration
effort will result in similarly high-quality documentation being
included with a default distribution of Ruby.

-=Eric

To be realy fair, as a perl newbie (and even now) I found/find the Perl
docs to be unstructured, disjointed, scattered within the bundle. It
reflects the history of the language, not a concise view of how the
language works and how to get things done.

It also reflects (im very ho) the sometimes hard choices that were made
to keep the language alive in the face of the requirements of it’s users
without completely breaking with it’s history.

As to reponding to us newbies/nubys, Elwood P. Dowd (the movie “Harvey”)
said he recommended " pleasant". Even in the face of documentation.
Hopefully you’re not suffering fools, you’re helping people on an
intellectual journey.

Mac

···

On Feb 28, 2004, at 1:19 PM, Koncept wrote:


(Mark Hubbart) #6

what I meant was:
We’re all nice here because programming in ruby is fun. As for
Perl… I’d better not say anything. :slight_smile:

I think that’s clearer :slight_smile:

Regardless, even if Ruby WAS superbly documented, I still think newbies
should be responded to in the same way: a quick example, and a
reference to more information. When you are new to a programming
language, sometimes you only know what the result of your code should
be, and have no way of knowing how to go about actually getting it.
So even if you did look in the docs, you might not know what to look
for.

Simple questions from these “extreme newbies”, IMHO, should be left for
the more advanced newbies. If they were treated the same way, they will
probably be happy to answer them, sort of giving back to the group.

So everyone should feel it’s okay to jump in; ask a question, give a
response if you think it’s correct, correct someone else if they are
wrong, show another way of doing it. Politely. With this kind of open
communication, people can say things without fear of getting
backhanded. Good temperaments in a community can be a big draw, and it
can be easily evidenced by interactions in the newsgroup :slight_smile:

heh. that sounded really cheezy, but you know what I mean.

Just my 2¢

–Mark

···

On Mar 2, 2004, at 4:49 PM, Eric Schwartz wrote:

Mark Hubbart discord@mac.com writes:

On Feb 28, 2004, at 1:19 PM, Koncept wrote:

It is so nice to be a part of a group ( unlike those Perl folks )
where
people are actually nice to each other.

We’re all nice here because programming in ruby is fun :slight_smile: as for
Perl… I’d better not say anything.

To be fair, unlike Ruby, Perl has tons of documentation distributed
with it that a poster could easily consult before posting a question;
where it’s not fair to tell someone to just “Read the Pickaxe”, since
that’s not definitive or complete, it is fairly reasonable for a Perl
programmer to be told to read the standard documentation on her hard
drive before asking a question it could easily answer.

Hopefully, the hard work being put into the RDoc<->ri integration
effort will result in similarly high-quality documentation being
included with a default distribution of Ruby.