skip to: page content | links on this page | site navigation | footer (site information)

Apache Foundation | SpamHaus | Open-SPF | ClamAV | Ubuntu

The Piratefish Open Source Anti-Spam System on Ubuntu Linux.

Piratefish 3.0 Tuning Tips

Please note: Changes listed in this document are for Piratefish 3.0 installations only, and are intended to add features that are not documented in the Piratefish 3.0 Guide, or to correct problems in the documentation.

The software used in the Piratefish system is well documented on the Internet. If you have questions regarding these software packages, please go to their respective home pages and read the online documentation for more detailed help.

Many parts of the Piratefish are well documented within the webmin interface as well. In the MailScanner plug-in for instance, many of the settings are documented in pop-ups accessible just by clinking on the items you're curious about. Don't be afraid to explore!

Install fails with DNS problems

Ubuntu, at version 8.10, has dropped the inclusion of the bind9 DNS server package in the default installation options for Ubuntu Server. This means that any of you building a new setup may run into a conflict between the Piratefish instructions saying not to setup any DNS entries, and the server installation not installing the needed DNS server.

So, for those of you lucky enough to check there first, please do put in DNS entries when installing Ubuntu Server. If you do, skip to the next section under this heading titled Installing Bind.

If you've already installed the base system, but now find that you can't resolve any DNS to reach anything, the following commands will add some DNS entries so things will work until bind is installed:

1) Log into your newly installed server, and become root using the command sudo -s and entering your password when asked.

2) Enter the command pico /etc/resolv.conf

3) Move to the bottom of this file and add the following two lines:

namserver 4.2.2.1
nameserver 4.2.2.2

4) Now exit and save the file. DNS will now resolve to those servers listed - if you wish, add or use your own - those two DNS servers should work for everyone, but might not depending on your location. It's recommended that you use DNS servers that you know will work. No reboot is necessary after this edit.

Installing Bind

The Bind DNS server is needed by the Piratefish because it can cache the blacklist checks that are performed during the course of it's operations. This way, once a spam-source IP address has been resolved as being one, and if it attacks you for an hour or so, trying incessantly to send bad messages in, the server will have cached what it needs to know in order to keep ignoring this bad source of email. This reduces processing overhead, and more importantly, reduces the bandwidth load on the blacklist providers, and anything that helps them is good.

To install bind, use this command when logged into your Piratefish as root:

apt-get install bind9

Once it's done installing it, edit the /etc/resolv.conf file and remove any line that starts with "nameserver". See the previous section on how to access or edit it.

Once the nameserver lines are removed from resolv.conf, the system will rely upon bind for it's DNS resolution. Reboot the system after making this change, then try "ping www.yahoo.com" to make sure the Internet resolves things correctly.

Fixing Winmail.dat problems

For those of you who're not aware, Microsoft handles their attachments just a little bit differently than the rest of us non-Exchange /non-Outlook users. Very common in both setups is the presence of a filetype called "winmail.dat" This file often contains attachments as they're sent from one machine to another, and for those of us who live outside the Microsoft world, the presence of Winmail.dat in emails can be annoying, since thunderbird doesn't know how to unpack them.

Recently Steven Siegel brought to my attention that things weren't working right with these attachements, and I tool a look-see at how things are dealt with. As it turns out, inside the MailScanner config file there's some references to TNEF handling (which is winmail.dat files) and apparently MailScanner has some neat capabilities that I wasn't aware of.

Inside this section of the MailScanner config, we're able to define an external program which can be used to unpack these files, and then MailScanner can also be nice and replace those winmain.dat files with the files contained within it. There are also options for delivering non-decodable winmail.dat files as well - something a Microsoft shop might want turned on.

There's also a reference to a program called /usr/bin/tnef listed in the config file, but that program wasn't present on my fish - so I'm recommending to all that once you've built your fish, it would be wise to install the tnef decoder.

apt-get install tnef

Table of Contents Omission

In the table of contents, Chapter 20 is mis-labeled as "Troubleshooting Tips" - this chapter is actually Chapter 21. Chapter 20 should read "Using Postgrey to add automatic greylisting"

The links at the bottom of the page in Chapter 20 can be used to access the troubleshooting tips chapter.

Chapter 3 - Image Oops

In Chapter 3, Step 5, where an image of updating Webmin should be present, there's a nice picture of Schedule Cron Jobs. The image that should have been present is this:

I would like to extend my most sincere apologies to my end users. The program responsible for this travesty has been sacked.

Thanks David!

Chapter 12 - Type in Logwatch path

In chapter 12, section 3, the book references entering in the path for the Logwatch command - in the book text it's shown as /usr/bin/logwatch but it's not correct - the logwatch path shown in the image is correct - logwatch is at /usr/sbin/logwatch

Thanks Tony!

Chapter 15 - Typo in Relay_Recipient_Maps

In Chapter 15, the section about removing backscatter, I accidently referred to the relay_recipient_maps directive as relay_recipients_maps.

Be sure to correct this in your configuration so that Postfix knows to use the relay_recipient_map file.

Thanks Vincent!

If anyone runs into more troubles or questions regarding this, please contact me and let me know.

Add PDF Scanning to your Piratefish (beta) (updated 10/13/2009)

Recently I was asked about having the Piratefish read PDF files, did a little research, and I found a great way to add that feature in. Now I'll qualify this with "it's Beta" which means it's not that well tested over time yet, and this also requires some console work as well, so be prepared to be root - and it's best to do this from a decent terminal proggy with a substantial scrollback buffer.

This process will install another SpamAssassin plugin, and this plugin will open up PDF files, perform OCR on their contents, and then run the PDF's wording through the regular spamassassin filtering to decide if it's spammy.

1) Perform a round of "apt-get update" and an "apt-get upgrade" and an "apt-get dist-upgrade" to make sure there's no updates hanging around in the wings waiting to be installed. Reboot if apt-get dist-upgrade installs anything.

2) Do an apt-get install xpdf-utils gocr

3) Type mkdir pdf

4) Type cd pdf

5) Type wget http://atmail.com/kb/attach/PDFassassin.tgz

6) Type tar -zxvf PDFassassin.tgz

7) Type cp pdf.cf /etc/spamassassin

8) Type cp Pdf.pm /etc/spamassassin

9) Type spamassassin -p /etc/MailScanner/spam.assassin.prefs.conf -D --lint

This last line will generate pages and pages of output, and can be done again and again. Burried within these lines, there should be a couple of lines that look like this:

[32282] dbg: config: fixed relative path: /etc/spamassassin/Pdf.pm
[32282] dbg: plugin: loading Pdf from /etc/spamassassin/Pdf.pm

The numbers on the start of the line may change, but the message is good - this means that it's using the new plugin.

This useful info was found online at: http://blog.atmail.com/?p=61

I've had this running in my fish now since Thursday, and there seems to be no ill effect, so I think it's ready for prime time.

Pointing Root's Email to an External Address

Russel S. Hall at m33access.com recently threw this my direction, and I realized this was a good idea - he's actually got a few of them and has been helpful - so shouts to him!

Anyway, pointing root's email on your piratefish box will ensure that any messages sent to root locally are sent out to wherever you choose. This is a good habit too, since many *nix boxen can collect tons of messages in their inbox, only to greet you with good news of 10000 messages upon your next root login - so this saves you trouble of discovering mistakes before they become problems.

By editing the text file /etc/aliases you can redirect root's email simply by putting (or editing) the line root: and appending your email address onto the end of that line.

root: johnny@piratefish.org

Works nicely for me. Make sure you put in your email address on your Piratefish though!

Advanced Spam Handling Techniques

Thanks go out to Andy Duplay for his assistance, testing and persistance in developing the updates documented here. www.duplayit.com

A great many folks have asked about ways to deal with spam. The Piratefish default spam handling methods have been classically limited to either forwarding the spam on with a modified subject line, or deleting the spam entirely. Postfix does have provisions for a quarantine, but often times folks find the quarantine to be somewhat limited in it's abilities - and quite frankly, the Webmin GUI is rather limited in it's handing of the quarantine as well.

In order to decide what is done with spam, users must go into the "What to do with SPAM" section of the MailScanner Webmin plugin. The defaults there are been recommended as simply "Delete" for high and low scoring spam - this chapter of the Piratefish guide has always been a little short - well, here's some content to help beef it up a bit.

There are a few extra options available in how MailScanner handles email - the official list from the MailScanner home page is this:

This can be any combination of 1 or more of the following keywords, and these actions are applied to any message which is spam.
"deliver" - the message is delivered to the recipient as normal
"delete" - the message is deleted
"store" - the message is stored in the quarantine
"bounce" - a rejection message is sent back to the sender
"forward" - an email address is supplied, to which the message is forwarded
"striphtml" - convert all in-line HTML content in the message to be stripped to plain text, which removes all images and scripts and so can be used to protect your users from offensive spam. Note that using this action on its own does not imply that the message will be delivered, you will need to specify "deliver" or "forward" to actually deliver the message.

One other option that can be done here is to instruct MailScanner to read a file containing handling rules for the messages - this can be used to perform multiple different actions dependant on recipients and/or domains - depending on how down and dirty you want to get.

Forwarding Spam - If you run your Piratefish as a single company entity, it is possible to use the "forward" action to tell MailScanner to send copies of your spam to a specified "catch-all" address on your mail server. A configuration such as this can be used to collect messages judged as high and low scoring spam. All that needs to happen is that you must create a special mailbox to receive the spam - and you'll then need to remember to check it from time to time as this will fill up quickly.

Fowarding Spam per Domain

If you run your Piratefish for multiple domains, and each domain wants to have their own seperate spam account at that domain, then a custom script will need to be invoked to handle your filtering. Although this is more complex, it is infinitely more flexible as you'll see.

By putting the following changes into your What to do with SPAM section of the MailScanner GUI, you have instructed MailScanner to run the script "spam.action.rules" each time a SPAM or High-Scoring SPAM is received.

The strength of this is that the "spam.action.rules" script you create in the /etc/MailScanner/rules folder can tell the server to handle spam differently for each domain the Piratefish is configured for.

For example:

To: default forward spambox@piratefish.org
To: *@rog.org forward spambox@rog.org
To: *@customer.com forward spambox@customer.com

In this configuration, every message handled by the server is forwarded to the spambox@piratefish.org - if there are other domains not listed, they will follow this default as well.

Any SPAM sent to rog.org will also be forwarded to spambox@rog.org
Any SPAM sent to customer.com will also be forwarded to spambox@customer.com

Forwarding Spam per User

If you have gone through the previous section, then you have seen that per-domain rules can be applied. Per user spam handling is a simple extension of this same technique.

All that's required for per-user spam setting is the creation of specific Spam folders for each user, then placing the appropriate fowarding directives in the spam.actions.rules file.

For example:

To: default forward spambox@piratefish.org
To: johnny@piratefish.org forward johnny-spam@piratefish.org
To: sales@piratefish.org forward sales-spam@piratefish.org

With these rules in operation, any spam's sent to johnny@piratefish.org will be forwarded only top Johnny's spam mailbox. Any spam sent to salesd@piratefish.org will also be forwarded to a mailbox called sales-spam@piratefish.org

For obvious reasons this can get a bit unweildy if your Piratefish installation has too many users, but for certain specific mailboxes, such as human resources, sales, and so forth, this can provide your users a way to ensure that important messages aren't tossed out by your Piratefish.

Spam Attacker Suppression (Beta)

Thanks go out to Angelo Schalley of Linix Webhosting (www.linix.be) for this addition.

This Piratefish update is a little more complex than most, but it's got some nice evil roots to it. Basically put, this update makes the IPTables firewall in your Piratefish keep track of the various connections coming into it, and if the same IP address connects more than 4 times in a single minute, then it's blocked for any further connections until less than 4 connection attempts have occurred in the last minute.

This update is a tad bit controversial as it's possible that it could resulting in a mail delivery slowdown with a popular 3rd party - if that 3rd party is sending too many separate messages anyway. This could result in a delay of delivery if something important is expected, or if you have a lot of folks dealing with large common email domains like Yahoo, MSN, Hotmail, GMail and the like. Controversy aside, this one's still a "beta" addition - only for the most technical users. I have some legwork to make this one ready for prime time still.

Here's the raw IPTables script for consideration:

-A RH-Firewall-1-INPUT -p tcp --dport 25 -i eth 0 -m state --state NEW -m recent --set
-A RH-Firewall-1-INPUT -p tcp --dport 25 -i eth 0 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 -j LOG --log-prefix "BLOCK: "
-A RH-Firewall-1-INPUT -p tcp --dport 25 -i eth 0 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 -j DROP
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 25 -j ACCEPT

I've not converted these commands into something for the Piratefish GUI yet, but I am working on it. Anyone with some input on how best to implement these, please let me know. These rules would have to reside in front of the existing IP Tables rules (if you're using them) and port 25 should be removed from the existing rules as well, as traffic can still get through those - also, the default policy must be set to drop, or the traffic will get through anyway.

ORDB Now Giving False Positives!!! Must Remove

If your Piratefish is still using the ORDB blacklist for identifying spam, you must now remove it or all your email messages will be marked as spam!

The default configuration of the Piratefish does not use the ORDB - in fact it advises against using it entirely, but if you decided it was a good idea, then you should remove it now. Chapter 10 in the Piratefish guide goes into specifics on configuring the various blacklists to use. Remember that blacklists are managed in the Spam Detection and Block Lists section of the MailScanner GUI.

Spam Delivery Control with Microsoft Exchange Server (testing)

Delivering your users spam into their junk folder is possible now that Microsoft has offered features for Intelligent Message Filtering for Microsoft Exchange. Microsoft Exchange offers IMF as an add-on for Exchange 2003, and it's now built into Exchange 2007. If you have an exchange server, you should be using this now to help deliver spam messages into your users junk mail folders. I won't go into the details of installation or setup of this, as Microsoft has already done so and provides ample documentation on their websites - however, only version 2007 has the ability to add in keyword handling without using expensive 3rd party add-in products.

To get around the need for any exchange customization, it is possible to have MailScanner simply mark messages with headers that simulate Microsoft's own headers, and using this it's possible to force Microsoft's handling of the spams identified by the Piratefish.

In the MailScanner "What to do with Spam" control panel, try the following delivery settings:

Spam Actions deliver header "X-MS-Exchange-Organization-SCL: 6.5"
High Scoring Spam Actions deliver header "X-MS-Exchange-Organization-SCL: 8"
Non Spam Actions deliver header "X-MS-Exchange-Organization-SCL: 0"

By placing these headers into your message, hopefully, the IMF will read these SCL (Spam Confidence Levels) and think these messages have already been rated by another Exchange relay and it will then stop delivery of high scoring spam, and store low-scoring spam in the recipients junk mailbox - and deliver non-spam normally.

Things aren't done yet however - one last thing needs to be done, and the details on this are sketchy, but here's the simple version - the Exchange server needs to know it can trust these headers when they're added by the Piratefish. To accomplish this, there is a header firewall - but I don't yet know the details of programming it.

Read here for more detailed information: http://exchangepedia.com/blog/2007/07/header-firewall-why-spammers-cant.html

An Alternative Method

Andrew Bichler (www.infobit.us) recently told me about this utility - its a freeware editer for keywords and weighting used by the IMF in Exchange 2003 sp2. http://www.accendo.co.uk/blog/?p=4

If you combine this editor with your own custom headers, it may be possible to influence Microsoft's SCL (Spam Confidence Level) using customized Piratefish headers without worrying about the Exchange server peeling the headers off as the message comes in. The link above also has links to resources to help you implement your IMF filtering on your Exchange server.

Please test these methods carefully, and report back to Piratefish with your results - if you find different settings or methods work better - please let me know and I'll publish them instead! I'll also post provided screenshots too!

SpamHaus Blacklist Update

It was recently brought to my attention that the long popular SpamHaus SBL+XBL has been replaced with a new server called "Zen" - all Piratefish users should ideally change their blacklist settings to take advantage of this new server.

In order to use this new server, it's necessary to edit one of MailScanner's files. This can be done through the file editor through Webmin, or by logging into the system as root and using the pico editor.

What you need to do is this:

  1. Log in as root (either command line or Webmin)
  2. Edit the file /etc/MailScanner/spam.lists.conf

    Command-line users should use pico to edit the file, or in webmin, use the file editor provided in the file management tool
  3. Within the file, find the SpamHaus section of the file and add this line underneath the SBL+XBL line:

    ZEN zen.spamhaus.org.
  4. Save the file.

    Note: As of this writing, the MailScanner spam.lists.conf file doesn't include an entry for ZEN, but it will - when that happens, this edit won't be necessary, however, the name to reference may need to be changed. For now, we'll use our own.
  5. Finally, follow the instructions in Chapter 10, but replace the SBL+XBL entry with ZEN.

Now save everything, reboot, done!

Webmin Postfix-Regex Error in Chapter 6

This error is about 2-3 months old now, and should hopefully be fixed already - nonetheless, it's documented here just in case.

During the Piratefish build process, early in Chapter 6, you may run into an error that reports the following:

Failed to save options :
Failed to regenerate table /etc/postfix/header_checks:
postmap: fatal: dict_mkmap_func: unsupported dictionary type: regexp:
Is the postfix-regexp package installed?

If you see this error, please realize that this error is being displayed in error. This error is being generated by Webmin, and hopefully Webmin's Postfix module will have this issue fixed sometime soon.

To work around this error, simple ignore it, click on your back button, and continue as though it never happened. Things will work just fine.

Now save everything, reboot, done!

Tesseract Data - no installation candidate

The folks who make the Tesseract OCR package decided to split up their package's language dictionaries in order to make things more standardized. If you ran into problems when trying to install the tesseract-data package, please understand that it has been replaced with the package tesseract-ocr-eng. There are other tesseract packages for other languages as well.

To install the packages you need, log into your piratefish as root, and enter this command:

apt-get install tesseract-ocr-eng

If you want all of the language packages, there's a shortcut that will install everything tesseract:

apt-get install tesseract-ocr*

This should get things right for your OCR needs.

Ubuntu 9 Installation Information

Ubuntu 9 adds in some new features, but also has brought a number of new changes into the mix - so a new Piratefish is coming soon. In the meanwhile, installation headaches can be avoided by removing some things from the mix. All the considerations of this page should be rolled into the installation process - thus the need for a Piratefish 4.0 is apparent.

First up, from the root command line, enter in this command:

apt-get remove apparmor

This will remove an annoying and new security feature that appears to be out of active development - Novell let the AppArmor engineering team go a while back it seems, and the home page for this application hasn't seen a new release in a year.

In Chapter 8 - ClamAV setup

If you see errors in your logs that look like this:

Clamd::ERROR:: UNKNOWN CLAMD RETURN ./lstat() failed. ERROR :: /var/spool/MailScanner/incoming

Manually editing the /etc/MailScanner/MailScanner.conf file is needed to correct the integration of ClamAV with MailScanner.

So, from the root command line, enter these commands:

cp /etc/MailScanner/MailScanner.conf /etc/MailScanner/MailScanner.conf.backup
pico /etc/MailScanner/MailScanner.conf

Find these these lines and change them to read like this:

Incoming Work Group = clamav
Incoming Work Permissions = 0640
Clamd Socket = /tmp/clamd.socket

Adam Osterlund of Wi-Fi Internet Solutions emailed me today with this, and I thought it was already on my list - but to my surprise this ones a new one, so I'm tossing it in. Big thanks Adam!

Changing Root aliases

If you edit the file /etc/aliases to change where root's email is sent, be sure to run the command:

newaliases

Or your changes will not be recognized - even after a reboot.

Terry Hulen from Lifeline Datacenters recently called this one to my attention - thanks Terry.

 

   
   
   
About Us | Privacy Policy | ©2005-2007 Piratefish.org