ook / isbg

IMAP Spam Begone (isbg): a script that makes it easy to scan an IMAP inbox for spam using SpamAssassin and get your spam moved to another folder.

Home | Edit | New

Home

IMAP Spam Begone

isbg is a script that makes it easy to scan an IMAP inbox for spam using SpamAssassin and get your spam moved to another folder.

Unlike the normal mode of deployments for SpamAssassin, isbg does not need to be involved in mail delivery, and can run on completely different machines to where your mailbox actually is.

  • Current stable version is 0.97 – 26 March 2003
  • Current development version is 0.98RC2 – 27 November 2009

There is a Yahoo group – imapspambegone for discussion and questions.

Table of Contents

Features

  • Works with all common IMAP servers
  • Works on Linux, MacOS X and Windows
  • Can do IMAP over SSL
  • Can remember your password
  • Will work painlessly against multiple IMAP accounts and servers
  • Is not involved in the mail delivery process, and so can run on any machine that can contact your IMAP server
  • Highly configurable
  • Sensible defaults so you don’t have to do any configuring :-)

New in 0.98

  • Special flags to teach Spamassassin Ham and Spam
  • Possibility to skip spam detection to stick only to the teach feature
  • Compatibility with Python 2.4, 2.5, 2.6.

Bugs

No bugs none at the moment. Find one? Please fill a report on the isbg tracker

You can request or discuss feature there.

Does it work?

Yes, very well. It has been filtering my work email, some of which comes from a legacy email address that is from a spam unfiltered system. I run it as a cronjob every 15 minutes, and have had it doing that for 6 months. I get about 500 spams a month. Many other users have been using it for a variety of servers from a variety of platforms.
There are no known outstanding bugs.

Installation

Make sure you have SpamAssassin installed. This is described in spamassassin.org/dist/INSTALL.

SpamAssassin should be on your $PATH (it installs in /usr/bin by default)

Download isbg.py. You can rename it to anything you want, and make it executable (chmod 555 isbg.py). It is written in the Python scripting language. Python is installed by default on most Linux systems. You can can find out more about Python at www.python.org

Simply invoke it by whatever name you called the file.

h1(#firstRun). Your first run

SpamAssassin

If you have never used SpamAssassin before, you’ll probably be quite nervous about it being too good and taking out legitimate email, or not taking out enough spam. It has an easily adustable threshold to change how aggressive it is. Run the following command to create your preferences file.

 $ spamassassin  </dev/null >/dev/null
Created user preferences file: /home/rogerb/.spamassassin/user_prefs

You can then edit $HOME/.spamassassin/user_prefs and change the thresholds.

isbg

The default behaviour of isbg is to not make any changes your Inbox unless you specify specific command line options. Consequently you can experiment without worry at the begining.

Your first step is to create a new folder to receive suspected spam. I use one named ‘spam’.

Run isbg with the —help option to see what options are available:

   $ isbg.py --help
    isbg: IMAP Spam begone 0.97-26Mar03
    All options are optional
   
      --imaphost hostname   IMAP server name [localhost]
      --ssl                 Make an SSL connection to the IMAP server
      --imapuser username   Who you login as [rogerb]
      --imapinbox mbox      Name of your inbox folder [INBOX]
      --spaminbox mbox      Name of your spam folder [INBOX.spam]
      --maxsize numbytes    Messages larger than this will be ignored as they are
                            unlikely to be spam [120000]
      --noreport            Don't include the SpamAssassin report in the message
                            copied to your spam folder
      --flag                The spams will be flagged in your inbox
      --delete              The spams will be marked for deletion from your inbox
      --expunge             Cause marked for deletion messages to also be deleted
                            (only useful if --delete is specified)
      --verbose             Show IMAP stuff happening
      --spamc               Use spamc instead of standalone SpamAssassin binary
      --savepw              Store the password to be used in future runs
      --nostats             Don't print stats
      --exitcodes           Use different exitcodes (see doc)
    (Your inbox will remain untouched unless you specify --flag or --delete)

Do your first run.

 $ isbg.py --imaphost mail.example.com  --savepw
IMAP password for rogerb@mail.example.com:

The amount of time it takes will be proportional to the size of your inbox. You can specify —verbose if you want to see the gory details of what is going on.

You can now examine your spam folder and will see what spam was detected. You can change the SpamAssassin threshold in your user_prefs file it created earlier.

isbg remembers which messages it has already seen, so that it doesn’t process them again every time it is run. If you are testing and do want it to run again, then remove the trackfile (default $HOME/.isbg-track*).

If you specified —savepw then isbg will remember your password the next time you run against the same server with the same username. You should not specify —savepw in future runs unless you want to change the saved password.

Running it

You’ll probably want something to actually be done with the original spams in your inbox. By default nothing happens to them, but you have two options available. If you specify —flag then spams will be flagged.

You can get the messages marked for deletion by specifying —delete. If you never want to see them in your inbox, also specify the —expunge option after —delete and they will be removed when isbg logs out of the IMAP server.

h1(#folderNames). Your folder names

Each IMAP implementation names their folders differently, and most IMAP clients manage to hide most of this from you. If your IMAP server is Courier, then your folders are all below INBOX, and use dots to seperate the components.

The UWash server typically has the folders below Mail and uses slash ‘/’ to seperate components.

Advanced options

If you would like to do something beyond the options listed in the usage message above, isbg actually has several more options that can be used. You will easily be able to figure them out from looking in the isbg.py file.

How does it work?

IMAP assigns each message in a folder a unique id. isbg scans the folder for messages it hasn’t seen before, and for each one, downloads the message and feeds it to SpamAssassin. If SpamAssassin says the message is spam, then the SpamAssassin report is uploaded into your spam folder. Unless you specify the —noreport option, in which case the message is copied from your Inbox to the Spam folder (the copy happens on the IMAP server itself so this option is good if you are on a low bandwidth connection).

Multiple accounts

By default isbg saves the list of seen IMAP message unique ids in a file in your home directory. It is named .isbg-trackXXXX where XXXX is a 16 byte identifier based on the IMAP host, username and port number. Consequently you can just run isbg against different servers/accounts and it will automatically keep the tracked uids seperate. You can override the filename with —trackfile.

Saving your password

If you don’t want isbg to prompt you for your password each time, you can specify the —savepw option. This will save the password in a file in your home directory. The file is named .isbg-XXXX where XXXX is a 16 byte identifier based on the IMAP host, username and port number (the same as for the multiple accounts description above). You can override the filename with —passwordfilename

The password is obfuscated, so anyone just looking at the contents won’t be able to see what it is. However, if they study the code to isbg then they will be able to figure out how to de-obfuscate it, and recover the original password. (isbg needs the original password each time it is run as well).

Consequently you should regard this as providing minimal protection if someone can read the file.

SSL

isbg can do IMAP over SSL if your version of Python has been compiled with SSL support. (Specifically it looks for the function socket.ssl). You can tell if you have ssl support by running isbg.py —help. If —ssl is listed in the options then you have ssl support.

However you should be aware that the SSL support does NOT check the certificate name nor validate the issuer. If an attacker can intercept the connection and modify all the packets flowing by, then they will be able to pose as the IMAP server. Other than that, the connection will have the usual security features of SSL.

Exit Codes

When ISBG exits, it uses the exit code to tell you what happened. In general it is zero if all went well, and non-zero if there was a problem. You can turn on additional reporting by using the —exitcodes command line option.
code —exitcodes needed description
0 All went well
1 yes There was at least one new message, and none of the messages were spam
2 yes There was at least one new message, and all messages were spam
3 yes There were new messages, with at least one spam and one non-spam
10 There were errors in the command line arguments
11 The IMAP server reported an error

Read and Seen flags

There are two flags IMAP uses to mark messages, Recent and Seen. Recent is sent to the first IMAP client that connects after a new message is received. Other clients or subsequent connections won’t see that flag. The Seen flag is used to mark a message as read. IMAP clients explicitly set Seen when a message is being read.

Pine and some other mailers use the Recent flag to mark new mail. Unfortunately this means that if isbg or any other IMAP client has even looked at the Inbox, the messages won’t be shown as new. It really should be using Seen.

The IMAP specification does not permit clients to change the Recent flag.

Upgrading

To 0.95 from earlier versions
The name of the trackfile has changed. If you just run 0.95 it will reprocess your entire INBOX. If you would like to save the existing trackfile, then run isbg with —verbose and don’t specify your password. It will print the name of the trackfile before the password prompt. Press control-C to break out. Rename ~/.isbg to the trackfilename it printed, and then rerun isbg.

Version History

0.97 – 26 March 2003

  • If a message can’t be saved to the spam box, then it is left alone. (Some IMAP servers hand isbg a message that they then refuse to be handed back). Previously isbg would exit.
  • Be careful when inbox has no messages at all
  • Allow using exit codes to see what happened
  • Supply None as last two arguments to socket.ssl (even though api says they aren’t needed). Some platforms implementation doesn’t match the spec.
  • Only attempt to set message flags if the user has specified any. (Some IMAP servers break if you ask them to add no flags to a message)

0.96 – 20 January 2003

  • Actually do NL to CRNL conversion correctly

0.95 – 19 January 2003

  • Removed —stop-at-threshold option as SA no longer has it
  • Changed end of line to since some IMAP servers get upset if the message only has newlines. SpamAssassin helpfully removes all the CRs which the IMAP protocol requires
  • Make /dev/null be nul on Windows
  • Added support for IMAP over SSL.
  • Made the default trackfile be based on host, user and port
  • Some borken IMAP servers didn’t like a space after the opening parenthesis in message flags
  • Provide the ability to store password (filename also based on host, user and port)
  • Added —nostats to prevent printing of final statistics
  • Removed uids from trackfile if they are no longer in the INBOX

0.92 – 28 October 2002

  • Removed —pipe option to SpamAssassin since it is now the default and complains if it is supplied
  • Changed the order of the UID SIZE fetch fields since iPlanet returns them in the order it feels like, not what was requested.

0.9 – 14 July 2002

  • Initial version

Todo list

  • Just delete very high scoring messages and don’t even put them in the spam folder
  • Auto report messages to Razor (high scoring ones that are definitely spam)
  • Seperate out messages that may be false positives (scores close to SpamAssassin thresholds) from the definite spam ones.

Contact and about

This software was written by Roger Binns <rogerb@rogerbinns.com> and is maintained by Thomas Lecavelier <thomas@lecavelier.name> since november 2009.

License

As said by Roger Binns when he hang over isbg to Thomas Lecavelier: " You may use isbg under any OSI approved open source license such as those listed at http://opensource.org/licenses/alphabetical "

There is a Yahoo group – imapspambegone for discussion and questions.

Pages feed

Last edited by ook, Fri Nov 27 06:03:28 -0800 2009
Home | Edit | New
Versions: