Here’s an updated version of the Ruby pop3filter that was written. This
checks for the To: address, too, since that could be forged as well.
Note that there’s a bug in the original version that won’t be noticed in
most cases because FROM_RE and SUBJECT_RE are empty. This has been fixed.
-austin
pop3filter.rb (2.6 KB)
–
austin ziegler * austin@halostatue.ca * Toronto, ON, Canada
software designer * pragmatic programmer * 2003.09.19
* 20.16.39
I’ve made more updates. Rather than just putting them here, I’ve created a
page at RubyGarden:
http://www.rubygarden.org/ruby?SoBigPopper
-austin
–
austin ziegler * austin@halostatue.ca * Toronto, ON, Canada
software designer * pragmatic programmer * 2003.09.19
* 21.14.03
Austin Ziegler wrote:
Here’s an updated version of the Ruby pop3filter that was written. This
checks for the To: address, too, since that could be forged as well.
Thanks Austin. This has been a learning experience for me.
Michael
Gavin made a nice improvement that I have incorporated on the main page; I
have also fixed a couple of bugs with the detection code. Thanks, Gavin.
-austin
On Sat, 20 Sep 2003 10:14:39 +0900, Austin Ziegler wrote:
–
austin ziegler * austin@halostatue.ca * Toronto, ON, Canada
software designer * pragmatic programmer * 2003.09.20
* 02.44.54
Thanks so much, Austin (and other contributors)
for putting this together.
daz
(68MB backlog now cleared)
“Austin Ziegler” austin@halostatue.ca wrote:
I’ve made more updates. Rather than just putting them here, I’ve created a
page at RubyGarden:
“Austin Ziegler” austin@halostatue.ca wrote in message
Gavin made a nice improvement that I have incorporated on the main page; I
have also fixed a couple of bugs with the detection code. Thanks, Gavin.
Thank you very Austin and Gavin.
I have been inundated with spam in the last few hours and this nice little
utility has helped me a lot.
I have a question though: the size of email defaults to 120_000. Is there
some significance to this number or is it just an arbitrary number you
picked. I have been getting some spam with attachments which are 106K or in
that range. Of course, I have changed my version to catch them too but was
curious to find out.
Also, would it possible for you to add something like the “kill file”
support so that I can keep adding new patterns to it instead of modifying
the source code.
Thanks a lot.
– shanko
On Sat, 20 Sep 2003 10:14:39 +0900, Austin Ziegler wrote:
I’ve made more updates. Rather than just putting them here, I’ve
created a page at RubyGarden:Thanks so much, Austin (and other contributors)
for putting this together.daz
(68MB backlog now cleared)
A word of warning: don’t set the size threshold too small. Yesterday, I
ran the following script (at home) all day while I was at work:
while [ true ]; do
yes | ruby pop3filter.rb
sleep 300
done
That guarded my inbox just nicely so I could check it at work.
Today, having noticed that quite a few small (1-2K) annoying Microsoft
emails were coming through, I lowered the size threshold to 1500 bytes.
Until I rang home to get it stopped, no mail was getting through to me!
For some reason, every email was matched, and thus killed, by the program!
Until I get home to see what happened, I won’t know what emails I missed
out on.
Cheers,
Gavin
“Austin Ziegler” austin@halostatue.ca wrote:
I just cleaned 27Mb of Swens this morning!!!
Just to close my contribution to this thread, I wanted to comment that
after a few little improvements the original filter was finally
published:
http://freshmeat.net/projects/swendeleter/
– fxn
On Tuesday 23 September 2003 06:32, daz wrote:
“Austin Ziegler” austin@halostatue.ca wrote:
I’ve made more updates. Rather than just putting them here, I’ve
created a page at RubyGarden:Thanks so much, Austin (and other contributors)
for putting this together.
“Austin Ziegler” austin@halostatue.ca wrote in message
Gavin made a nice improvement that I have incorporated on the main page; I
have also fixed a couple of bugs with the detection code. Thanks, Gavin.
Thank you very Austin and Gavin.
I have been inundated with spam in the last few hours and this nice little
utility has helped me a lot.
I have a question though: the size of email defaults to 120_000. Is there
some significance to this number or is it just an arbitrary number you
picked. I have been getting some spam with attachments which are 106K or in
that range. Of course, I have changed my version to catch them too but was
curious to find out.
All of my virus-spam messages have been around 140Kb. One exception
was about 15Kb.
Also, would it possible for you to add something like the “kill file”
support so that I can keep adding new patterns to it instead of modifying
the source code.
No need really, and I don’t have the time. But you can go for it and
change the Wiki code if you like. Or if your version is significantly
different then offer it as a separate implementation.
The easiest and quickest thing to do would be to add filters to the
source code on the Wiki. If you need different filters, then someone
else probably will too.
I think a lot of people have learned about Net::POP3 today, especially
me.
Gavin
On Saturday, September 20, 2003, 9:03:18 PM, Shashank wrote:
On Sat, 20 Sep 2003 10:14:39 +0900, Austin Ziegler wrote:
Thank you very Austin and Gavin.
I have been inundated with spam in the last few hours and this nice
little utility has helped me a lot.I have a question though: the size of email defaults to 120_000. Is
there some significance to this number or is it just an arbitrary
number you picked. I have been getting some spam with attachments which
are 106K or in that range. Of course, I have changed my version to
catch them too but was curious to find out.
All of my virus-spam messages have been around 140Kb. One exception was
about 15Kb.
The Perl version I translated from – and I just modified the WikiPage to
include the historic influences – had 150_000 as the size default. I
figured it was a bit high, so I dropped it to 120_000. In one of my later
changes to the tool, though, I moved it from a constant to the fourth
parameter (defaulted) in the constructor.
Also, would it possible for you to add something like the “kill file”
support so that I can keep adding new patterns to it instead of
modifying the source code.
No need really, and I don’t have the time. But you can go for it and
change the Wiki code if you like. Or if your version is significantly
different then offer it as a separate implementation.
Actually, it’s even easier than that. I just modified the last line so that
it looks like:
if FILE == $0
POP3Filter.new(“server”, “user”, “pass”).process
end
Now, pop3filter is easy to include into a separate program. What you would
do to use this is:
require ‘pop3filter.rb’
POP3Filter.SUBJECT_RE << %r{enlarge}i
POP3Filter.new(“server”, “user”, “pass”, 80_000).process
This will now look for subjects that have the word “enlarge” in them, and it
will look for messages of 80,000 bytes.
The code could be abstracted a bit further – so that we can (via
meta-programming) tell it what headers we want and have multiple categories,
but this is simply a quick hack to get rid of the current infestation.
Another nice change would be to actually have a Tk (or other graphical)
interface so that you could select which messages you want deleted without
having to either (a) delete them all or (b) respond to each message
interactively.
-austin
On Sat, 20 Sep 2003 22:15:40 +0900, Gavin Sinclair wrote:
On Saturday, September 20, 2003, 9:03:18 PM, Shashank wrote:
–
austin ziegler * austin@halostatue.ca * Toronto, ON, Canada
software designer * pragmatic programmer * 2003.09.20
* 10.42.42
Saluton!
All of my virus-spam messages have been around 140Kb. One exception
was about 15Kb.
Concerning 15 KB messages: Notify maintainer of mail server that
sends them about the misconfiguration. A mail server must not
identify an infected message, remove the infected part and deliver
the rest.
Even though most people are not aware of it: Besides the given high
probability that the message was sent unintentionally doing so can be
a crime in some countries (Germany for example).
As long as it is done to correctly deliver the message it is
acceptable that a mail server manipulates e-mails but as soon as
manipulation is done to change the information transmitted you are
possibly comitting a crime because
you may be suppressing information that should be send (e.g.
forwarding some program that happens to result in a false alert)
you are creating a derived work of some copyrighted work without
permission of the copyright holder.
etc. Of course this is only what you officially say. More annoying is
that the removal of the attachment breaks any worm defense that makes
use of the fact that you rarely receive messages of more than 100 KB
that you actually want to recieve.
Note: On 2003-09-24 the European Parliament is going to decide on a
software patent directive. To protest against this my web pages
will offline until (and including) 2003-09-24 (UTC). This
includes but is not limited to the extmath, rcalc, and tldlib
homepage.
Gis,
Josef ‘Jupp’ Schugt
–
Warning to Outlook, Outlook Express and Windows users: In the 9/11
aftermath many countries decided on new anti-terror laws. If you put
web sites or mailboxes under fire this may be seen as a terroristic
act that potentially may buy you a one-way ticket to Guantanamo Bay.
Austin,
Can you post a link to the original code? It would be neat to compare perl
and ruby code for the same task.
THanks,
-Jose
On Sat, 20 Sep 2003 23:52:39 +0900, Austin Ziegler austin@halostatue.ca wrote:
The Perl version I translated from – and I just modified the WikiPage to
include the historic influences – had 150_000 as the size default. I
figured it was a bit high, so I dropped it to 120_000. In one of my later
changes to the tool, though, I moved it from a constant to the fourth
parameter (defaulted) in the constructor.
–
Jose Quesada, PhD.
quesadaj@psych.colorado.edu Research associate
Word Embedding Analysis Institute of Cognitive Science
University of Colorado (Boulder)
Muenzinger psychology building Phone:303 492 1522
office D447A Fax: 303 492 7177
Campus Box 344
University of Colorado at Boulder
Boulder, CO 80309-0344
Well, the original code was the first dirty iteration I wrote in a hurry
because my mailbox was being flooded by minutes. I shared it because of
the thread in case it could be helpful to anyone else.
I have cleaned it up a bit since then, factored it, and added
interactive and nonstop modes:
http://www.hashref.com/prj/pop3filter/pop3filter.pl
I think that’s a more appropiate version for a fair comparison.
– fxn
On Saturday 20 September 2003 18:56, Jose Quesada wrote:
Can you post a link to the original code? It would be neat to compare
perl and ruby code for the same task.
Agreed. I hope no one thinks that I was saying your version wasn’t good
enough. I just didn’t want to have to verify if I had Net::POP3 in Perl (I’m
pretty sure I do, since it’s the ActiveState version) and so I made the Ruby
version based on what you wrote. It was good for what it did, but I found
that it missed a number of the messages (naturally, because it had fewer
regexp), and the main reason for making the change is I didn’t want to
answer yes on EACH message.
At any rate, I think they’re both highly usable. At some point, I think that
I’ll set up a POP3 proxy form of this moving forward and integrate this,
gurgitate, and a bayesian spam checker so that I can look at doing things a
bit differently.
-austin
On Sun, 21 Sep 2003 02:07:44 +0900, Xavier Noria wrote:
On Saturday 20 September 2003 18:56, Jose Quesada wrote:
Can you post a link to the original code? It would be neat to compare
perl and ruby code for the same task.
Well, the original code was the first dirty iteration I wrote in a hurry
because my mailbox was being flooded by minutes. I shared it because of
the thread in case it could be helpful to anyone else.I have cleaned it up a bit since then, factored it, and added interactive
and nonstop modes:http://www.hashref.com/prj/pop3filter/pop3filter.pl
I think that’s a more appropiate version for a fair comparison.
–
austin ziegler * austin@halostatue.ca * Toronto, ON, Canada
software designer * pragmatic programmer * 2003.09.20
* 14.59.18
This whole worm thing brings up a question:
Why is it that those of us who use this as a mailing list, rather than
a newsgroup, have our email addresses passed on to the newsgroups
unchanged? Is this a conscious design decision or a bug?
If it’s a design decision, why???
If it’s a bug, can I get the source of the reflector thingy and fix it?
Ben
I believe that it’s a design decision. The reality, at least for me, is that
I get very little spam directed to me. I get very little spam from usenet.
(I have a separate account I use for that, and while 100% of the mail
received by it is spam, the volume is very low – less than 50 a week.) I
get a lot of spam mail and virus mail because of people who don’t use
properly secure email clients and because I have a domain that I do not
control the MTA on, meaning that my account is also the “catch-all” account.
I can change this, to be certain, but there are many reasons that I do not
at this point.
-austin
On Sun, 21 Sep 2003 04:47:26 +0900, Ben Giddings wrote:
Why is it that those of us who use this as a mailing list, rather than a
newsgroup, have our email addresses passed on to the newsgroups
unchanged? Is this a conscious design decision or a bug?If it’s a design decision, why???
If it’s a bug, can I get the source of the reflector thingy and fix it?
–
austin ziegler * austin@halostatue.ca * Toronto, ON, Canada
software designer * pragmatic programmer * 2003.09.20
* 16.53.52
On Sunday, September 21, 2003, 5:03:51 AM, Austin wrote, in part:
At any rate, I think they’re both highly usable. At some point, I think that
I’ll set up a POP3 proxy form of this moving forward and integrate this,
gurgitate, and a bayesian spam checker so that I can look at doing things a
bit differently.
That would certainly interest me. If you set this up on RubyForge
you’d definitely get my feedback and contributions!
Gavin
It wasn’t a conscious decision: I just reflected the e-mail out to
usenet as it stood. It struck me that as this was a public mailing
list, with no subscription qualifications, then any e-mail address used
was effectively public too.
I’m open to discussion on this
Cheers
Dave
On Saturday, Sep 20, 2003, at 21:47 Europe/Copenhagen, Ben Giddings wrote:
This whole worm thing brings up a question:
Why is it that those of us who use this as a mailing list, rather than
a newsgroup, have our email addresses passed on to the newsgroups
unchanged? Is this a conscious design decision or a bug?If it’s a design decision, why???
For those who were interested, I’ve tweaked my spam filters so almost
nothing is getting through anymore. I’ve posted them on my web log
site in case anybody wants them:
http://infofiend.com/log/index.php/item/223
Ben
It wouldn’t take much effort to change the email address, would it?
email.sub(/@/, " (at) ")
And nobody would complain that their address was being modified.
Gavin
On Monday, September 22, 2003, 8:02:37 PM, Dave wrote:
On Saturday, Sep 20, 2003, at 21:47 Europe/Copenhagen, Ben Giddings > wrote:
This whole worm thing brings up a question:
Why is it that those of us who use this as a mailing list, rather than
a newsgroup, have our email addresses passed on to the newsgroups
unchanged? Is this a conscious design decision or a bug?If it’s a design decision, why???
It wasn’t a conscious decision: I just reflected the e-mail out to
usenet as it stood. It struck me that as this was a public mailing
list, with no subscription qualifications, then any e-mail address used
was effectively public too.
I’m open to discussion on this