Logfile parsing and sending mail

ruby-talk
1

Hello all,

though I have to admit that I’m a newbie to Ruby I’ll try to devise my
questions intelligently :wink:

I’d like to parse an Apache logfile at regular intervals, say every 15
minutes. If that part of the logfile that has been added since the last
parsing contains a certain String, let’s say “blueice”, I’d like to be
sent an e-mail.

This is my first real Ruby experience, but I thought I should rather
start with something practical instead of something academic.

How do I realize the regular-interval-parsing? How can my application
keep in mind what part of the logfile has already been parsed and which
part is new? How am I able to send an e-mail using Ruby?

Any hints in any form are greatly appreciated!

Regards
Markus

2

Hi Markus,

Welcome to Ruby !

I’d like to parse an Apache logfile at regular intervals, say every 15
minutes. If that part of the logfile that has been added since the last
parsing contains a certain String, let’s say “blueice”, I’d like to be
sent an e-mail.

A brute-force approach might be to keep track of how many lines
of the log file you’ve read, and re-read the logfile each time,
discarding up to the number of lines you’d read previously, and
only considering the new lines.

However, there’s an even easier (and more efficient) way…

If you keep the logfile open, you’ll get a ‘nil’ value back from
gets() when you’ve reached the end of the file… If you keep the
logfile open, though, and wait the 15 minutes, you can just call
gets() again and you’ll get just the new lines that have been
added to the logfile since last time. (Then you’ll get the ‘nil’
again, indicating you’re at the end of the current length of
the logfile… at which point you can wait for 15 min. again…)

You can cause Ruby to pause for 15 minutes, by specifying the
number of seconds to sleep, as: sleep(15 * 60)

You can open the logfile, with, for ex.:

logf = File.open(“/var/log/httpd/access_log”, “r”)

To scan for the string data you’re looking for, regular expressions
might be appropriate… Using your “blueice” example:

while (line = logf.gets)
if line =~ /blueice/ # scan for “blueice” string anywhere on line
send_me_email
end
end

There are likely several ways to send email from Ruby, but here
are a couple I use. If sendmail is available locally, I might
use that; or if I need to talk directly to my ISP’s SMTP server,
that’s easy too…

Here’s a sendmail example:

from_addr = “log-voyeur@somewhere.net”
to_addr = “me@somewhere.net”

email_text = <<“END_EMAIL”

···

From: “Markus Wichmann” wichmann.markus@gmx.net
To: “Me” <#{to_addr}>
From: #{from_addr}
Subject: That thar logfile turned up somethin’

Hi, #{to_addr},

Here’s the email about the logfile thing.
END_EMAIL

now the sendmail part…

IO.popen(“/usr/sbin/sendmail #{to_addr}”, “w”) do |sendmail|
sendmail.print email_text
end

That’s it… Or, using SMTP directly:

require ‘net/smtp’

Net::SMTP.start(“your-isp-smtp-server.com”) do |smtp|
smtp.sendmail(email_text, from_addr, to_addr)
end

That’s it…! Hope this helps, and, apologies in advance if
there are any typos in the above code snippets!! :slight_smile:

Regards,

Bill

3

Hello all,

though I have to admit that I’m a newbie to Ruby I’ll try to devise my
questions intelligently :wink:

i’m not a newbie and my questions are generally arcane and un-intelligent so
i wouldn’t worry about that.

I’d like to parse an Apache logfile at regular intervals, say every 15
minutes. If that part of the logfile that has been added since the last
parsing contains a certain String, let’s say “blueice”, I’d like to be
sent an e-mail.

This is my first real Ruby experience, but I thought I should rather
start with something practical instead of something academic.

How do I realize the regular-interval-parsing? How can my application
keep in mind what part of the logfile has already been parsed and which
part is new? How am I able to send an e-mail using Ruby?

Any hints in any form are greatly appreciated!

i would use ruby’s built database, PStore, to store the date of the last run.
that way, you can simply cron the job - or run it by hand - whenever you feel
like it and the program will know the date of the most recently processed line
and can start from there - here’s a little example. a couple of things

  • the methods are defined in a BEGIN block for clarity/readability only.
    although it also works fine i don’t generally do this in real code.
  • no mailing is going on, but you can read about that in the net/smtp
    section of the pickaxe’s ‘Network and Web Libraries’ section. you can
    download the book from http://www.rubycentral.com, but i’d reccomend
    buying it straight away if you already haven’t - one to support the
    authors, two to let book companies know their is an audience out there for
    ruby books, and three because it is a super book for getting started in
    ruby

----CUT----
#!/usr/bin/env ruby
require ‘pstore’

ACCESS_LOG = ARGV[0] || “/usr/local/apache/logs/access_log”
PSTORE = ARGV[1] || “./access_log.db”

previous = get_previous_date()
date = nil

File.open(ACCESS_LOG) do |access_log|
access_log.each do |line|
date = parsedate(line)
process_line(line) if(date > previous)
end
end

store_current_date(date)

BEGIN {
DATE_RE = %r"\s[(\d+)/(\w+)/(\d+):(\d+):(\d+):(\d+)"o
MONTH_MAP = Hash[
%r/^jan/io => 1,
%r/^feb/io => 2,
%r/^mar/io => 3,
%r/^apr/io => 4,
%r/^may/io => 5,
%r/^jun/io => 6,
%r/^jul/io => 7,
%r/^aug/io => 8,
%r/^sep/io => 9,
%r/^oct/io => 10,
%r/^nov/io => 11,
%r/^dec/io => 12,
]
def parsedate(line, re = DATE_RE)
match = re.match(line) or raise (ArgumentError, “LINE HAS NO DATE”)
dd, mm, yy, h, m, s = match.to_a[1…-1]
n = nil
MONTH_MAP.map{|mp,mn| mp.match(mm) and n = mn}
n or raise (ArgumentError, “COULD NOT MAP MONTH TO MONTH NUMBER”)
return Time.mktime(yy, mm, dd, h, m, s)
end
def get_previous_date()
date = nil
pstore = PStore.new(PSTORE)
pstore.transaction do
date =
begin
pstore[:date]
rescue
Time.at(0)
end
end
return date
end
def process_line(line)
puts(line)
end
def store_current_date(date)
pstore = PStore.new(PSTORE)
pstore.transaction do
pstore[:date] = date
end
end
}
----CUT----

-a

···

On Mon, 21 Apr 2003, Markus Wichmann wrote:

–

Ara Howard
NOAA Forecast Systems Laboratory
Information and Technology Services
Data Systems Group
R/FST 325 Broadway
Boulder, CO 80305-3328
Email: ara.t.howard@fsl.noaa.gov
Phone: 303-497-7238
Fax: 303-497-7259
====================================

4

How do I realize the regular-interval-parsing? How can my
application
keep in mind what part of the logfile has already been parsed and
which
part is new? How am I able to send an e-mail using Ruby?

Another old unix-y way of doing this (not sure if it will work on
Windows, but I think it should) is on the first “iteration”, read the
lines of the file while not eof, looking for what you want, doing
what you need if you find it “seek” to the end of file. This will
put your file pointer exactly at the end, where it already is since
you read to the end, but it will RESET it’s “eof” marker, so
subsequent reads will start from there. Then sleep for your time
delta, and repeat the process.

···

Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo