Mail filter (was RE: announce@ == less email (FAQ item?))

Stefan Schmiedl [mailto:s@xss.de] explains

Julian Fitzell [mailto:julian@beta4.com] enlightens


accept if message.subject =~ /Ruby Weekly News/i
accept if message.subject =~ /ruby-dev summary/i

cool filter.
mind if I ask what mail program are u using?

I guess he’s using ruby to fetch his mail
from his provider and (hopefully) to store it
in a local mailbox.

It seems to be a handmade replacement of the
fetchmail - procmail toolchain on linux.

s.

Thank you.
Yes. I’ve seen procmail scripts, and they don’t come close…

I hope he’d share it though :wink:

-botp

Well, it’s not that difficult … when I upgraded
my box last week, the new version of fetchmail got
confused by something in the first message of the
mailbox, so I took a look at net/smtp.rb (IIRC),
and fumbled something together to take a closer look.

However, a simultaneously performed downgrade of
fetchmail did the job, too, so I stopped tinkering.

It is very easily doable, as long as your output
is easy to manage, like appending to different files.

Go on, do it yourself … or investigate at RAA.

s.

···

On Sat, 5 Oct 2002 16:14:53 +0900, Peña, Botp botp@delmonte-phil.com wrote:

Stefan Schmiedl [mailto:s@xss.de] explains

Julian Fitzell [mailto:julian@beta4.com] enlightens


accept if message.subject =~ /Ruby Weekly News/i
accept if message.subject =~ /ruby-dev summary/i

It seems to be a handmade replacement of the
fetchmail - procmail toolchain on linux.

Yes. I’ve seen procmail scripts, and they don’t come close…

I hope he’d share it though :wink:

In article anmco6$ffo3l$1@ID-57631.news.dfncis.de,

Stefan Schmiedl [mailto:s@xss.de] explains

Julian Fitzell [mailto:julian@beta4.com] enlightens


accept if message.subject =~ /Ruby Weekly News/i
accept if message.subject =~ /ruby-dev summary/i

It seems to be a handmade replacement of the
fetchmail - procmail toolchain on linux.

Yes. I’ve seen procmail scripts, and they don’t come close…

I hope he’d share it though :wink:

Well, it’s not that difficult … when I upgraded
my box last week, the new version of fetchmail got
confused by something in the first message of the
mailbox, so I took a look at net/smtp.rb (IIRC),
and fumbled something together to take a closer look.

However, a simultaneously performed downgrade of
fetchmail did the job, too, so I stopped tinkering.

It is very easily doable, as long as your output
is easy to manage, like appending to different files.

Go on, do it yourself … or investigate at RAA.

s.

    • Here’s mine. It’s got a whole bunch of wierd stuff in it since
      I need to read my email out of NFS and write it into AFS. It
      uses ifile and the rmail and rfilter modules, but it might
      give you some ideas. I’ve found the rmail/rfilter stuff pretty
      useful.
    • Booker C. Bense

#!/var/local/bin/ruby

Get the Mailspool and process it.

···

Stefan Schmiedl s@xss.de wrote:

On Sat, 5 Oct 2002 16:14:53 +0900, >Peña, Botp botp@delmonte-phil.com wrote:

require ‘rmail/parser’
require ‘rmail/parser/mbox’
require ‘tempfile’

require ‘rfilter/deliver’

Manage Ifile interactions

$Id: ifile.rb,v 1.1 2002/09/25 20:02:13 bbense Exp $

Ifile, a class for interacting with ifile program.

Get ifile at http://www.ai.mit.edu/~jrennie/ifile/

Booker C. Bense bbense@slac.stanford.edu

require ‘open3’

module Ifile

This is the wrong name, but I can’t think of anything better.

class Process

Tell me where ifile lives.

def initialize(path=“/var/local/bin/ifile”,args=“–verbosity=0”)
if FileTest.executable?(path) then
@ifile = path
@args = args
else
raise ArgumentError
end
end

Given a message, query folders

def query(msg)
results = Array.new
output = self.run_ifile(msg,"–query ")
i = 0
output.each do |line|
# Format of output is folder score
folder , score = line.split
if ( folder && score ) then
tmp = Hash.new
tmp[‘folder’] = folder
tmp[‘score’] = score.to_f
tmp[‘position’] = i
results << tmp
i = i + 1
end
end
return results
end

Add a message to a folder

def add(msg,folder)
output = self.run_ifile(msg,“–insert=#{folder}”)
end

Delete a message from a folder

def delete(msg,folder)
output = self.run_ifile(msg,“–delete=#{folder}”)
end

Refile

def refile(msg,oldfolder,newfolder)
self.delete(msg,oldfolder)
self.add(msg,newfolder)
end

internal methods

def run_ifile(msg,args)
stdin, stdout, stderr = Open3.popen3(“#{@ifile} #{@args} #{args}”)
#write msg to ifile
msg.each { |line| stdin.puts line }
stdin.close
#Read output
output = stdout.readlines
stdout.close
stderr.close
return output
end

end

end # module Ifile

class MailSpool

Set the spool name

def initialize(user,path=“/var/spool/mail/”)
if ( user == nil ) then
user = Etc.getpwuid(Process.uid).name
end
@spool = File.open(“#{path}#{user}”,“r+”);
@tmpspool = Tempfile.new(“mailspool”)
@parser = RMail::Parser.new
@mreader = RMail::Parser::MBoxReader.new(@tmpspool)
end

Move the spool to tmpspool

def update
begin
if @spool.stat.size > 0 then
# Lock the spool file
@spool.flock(File::LOCK_EX)
# Read it in
@spool.rewind
new_msgs = @spool.read
# Empty it.
@spool.truncate(0)
# Unlock it
@spool.flock(File::LOCK_UN)
@tmpspool.truncate(0)
# Write it out to tmpspool
@tmpspool.write(new_msgs)
@tmpspool.close
return true
else
return nil
end
rescue
return nil
end
end

Return an array of messages.

def messages
@tmpspool.open

messages = []
loop do
  msg = @parser.parse(@mreader.next)
  if msg.header.mbox_from then
messages << msg
  else
break
  end
end
messages

end

end

These are really HeaderFilters.

class Filter

remember the regexp test and the folder it maps to.

def initialize ( test, folder )
@test = test
@folder = folder
@has_proc = nil
end

def test
@test
end

def folder
@folder
end

def has_proc?
@has_proc
end

This proc is for after a successful match to do

dynamic folder setting.

def addProc(proc)
if defined? proc.call then
@proc = proc
@has_proc = true
else
raise ArguementError
end
end

def call(msg)
@proc.call(msg)
end

end

Return a string that indicates a folder to file into.

Add auto month splitting ?

class FilterProcess

def initialize(user, filterdir, default_folder=“incoming” )
@user = user
@default_folder = “#{filterdir}/#{default_folder}”
@filterdir = filterdir

Switch to using date folder for auto month rollover

@folderdir = “#{filterdir}/Now”

tmp = Time.now.asctime
tmpfolder = tmp.split[1] + tmp.split[4]
@folderdir = “#{filterdir}/#{tmpfolder}”
@logfile = File.open(“#{filterdir}/log.#{tmpfolder}”,“a”)

@filters = Hash.new
@filterKeys = Array.new
end

def default_folder
@default_folder
end

def folderdir
@folderdir
end

def addFilter(filter,type=‘envelope’)
my_type = ‘envelope’ unless type
if ( filter.test.respond_to?(:match) ) && ( defined? filter.folder ) then
if @filters.has_key? my_type then
@filters[my_type] << filter
else
@filters[my_type] = Array.new
@filters[my_type] << filter
@filterKeys << my_type
end
else
raise ArgumentError
end
end

def process(msg)
if msg.header then
@filterKeys.each do |key|
if msg.header.mbox_from && ( key == ‘envelope’) then
test_string = msg.header.mbox_from
else
test_string = msg.header[key]
end
if test_string then
activeFilter = @filters[key].detect { |filter| filter.test.match(test_string) }
if activeFilter then
if activeFilter.has_proc? then
folder = activeFilter.call(msg)
if folder then
return “#{@folderdir}/#{folder}”
end
else
# Wrong place to do logging.
# self.log(“Filing #{msg.header[‘message-id’]} in #{activeFilter.folder}”)
return “#{@folderdir}/#{activeFilter.folder}”
end
end
end
end #filterKeys
end #msg.header
return @default_folder
end

def log (string)
print “#{string}\n” if ( string =~ /ERROR/ )
@logfile.print “#{string}\n”
end

def initFilters
filterfile = “#{@filterdir}/filters”
begin
File.open(filterfile) do |file|
while ( line = file.gets ) do
unless ( line =~ /^#/ ) then
flag_string , mbox , type = line.split
if ( flag_string && mbox ) then
self.addFilter(Filter.new(Regexp.new(Regexp.quote(flag_string)),mbox),type)
end
end
end
end
rescue
print “Error in reading #{filterfile}”
exit
end
end

end #FilterProcess

For dealing with file

class IfileProcess

def initialize(filterdir,default_folder)
@filterdir = filterdir
@default_folder = default_folder
@ifile = Ifile::Process.new
@same = Hash.new
@same[“unix-admin”] = “oldmail”
@same[“admin-log”] = “oldmail”
@same[default_folder] = “oldmail”
end

Return a full fledge path

def query(msg)
results = @ifile.query(msg)
best = results[0][“folder”]
# Where did oldmail score
# oldmail = results.detect { |result| result[“folder”] == “oldmail” }
if ( best == “oldmail” ) then
return @default_folder
end
dest = “#{@filterdir}/#{best}”
dest.sub!(/\s/,“”)
return dest
end

def add(msg,destination)
if ( @same[destination] ) then
folder = @same[destination]
else
folder = destination.sub(@filterdir,“”)
folder.sub!(/[/]/,“”)
# Deal with incoming/oldmail problem
if ( @same[folder] )
folder = @same[folder]
end
end

print “Adding message to :#{folder}:\n”

@ifile.add(msg,folder)

end

end #IfileProcess

This is for real.

begin
filterP = FilterProcess.new(“foobar”,“/home/foobar/filtermail”)

begin
# Be more paranoid add a timestamp and pid
backup = “/tmp/foobar.backupspool.#{Time.now.to_i}.#{Process.pid}”
system(“/bin/cp /var/spool/mail/foobar #{backup}”)
rescue
print “Error making backup… /tmp/foobar.backsupspool”
exit
end
emergency = “/tmp/foobar.emergency”

spool = MailSpool.new(“foobar”)
filterP.initFilters

#Ifile interface.
ifile = IfileProcess.new(filterP.folderdir,filterP.default_folder)

rescue => error
print “Error in startup. #{error.inspect}\n”
exit
end

report = Hash.new(0)

if ( spool.update) then
messages = spool.messages

messages.each do |msg|
# Where does it go ?
destination = filterP.process(msg)
# Do some destination dependant checks.
if ( destination == filterP.default_folder ) then
begin
new_destination = ifile.query(msg)
filterP.log(“INFO: Ifile redirect #{destination} => #{new_destination}”)
destination = new_destination
rescue
filterP.log(“ERROR: using ifile.query”)
end
end
report[destination] = report[destination] + 1
# Write to destination
filterP.log(“INFO: Filing #{msg.header[‘message-id’]} in #{destination}”)
# Check for error
begin
RFilter::Deliver.deliver_mbox(destination,msg)
rescue
RFilter::Deliver.deliver_mbox(emergency,msg)
filterP.log(“ERROR: Delivering to #{emergency}”)
end
# Add to ifile database.
begin
ifile.add(msg,destination)
rescue => error
filterP.log(“ERROR: using ifile.add #{error.inspect}”)
end
end
end

print report

report.each do |folder, count|
print “#{folder} : #{count} new messages\n”
end