Dspam & Mail enable?

[Guest]

After some research, I've come across Dspam, a scaleable open-source statistical anti-spam filter with very good accuracy. Accuracy like 99.95% (1 error in 2000) and can sometimes reach peaks as high as 99.991% (2 errors in 22,786, as with one particular user). Far better than most commercial solutions. Here's the website:

http://www.nuclearelephant.com/projects ... ndex.shtml

Just wondering what people think about this product. They offer explanations on how to compile for WIN32 usage, but can it be used with Mailenable?

Right now I'm using MEFilter for spam catching along with reverse-DNS blacklist lookup, which works, but is not terribly accurate.

If anyone has the capacity to complie this & get it running on Win32 platform, I'd be very interested in trying it as a comparison to tradtional filtering.

Post any comments/thoughts you come up with.

[scottkct]

I have extensively researched and tested multiple spam filtering solutions. MEFilter has a lot of potential but by currently only blocking by keywords is well known to not be a very effective solution. Along with any other list or static based blocking.

I have successfully installed the latest release of SpamAssassin but am unable to integrate it into ME. This has the best functionality as it uses bayesian (statistical) analysis to filter spam in combination with a well developed suite of rules based filtering. There are websites that has a pretty nice collection of custom rules people have built for SA, some more rigorous than others. The biggest problem I ran into was a way to implement it into the ME's MTA. The SAPlugin.exe built for SA integration into ME is worthless at this point unless you want to use an old version of SA that you can't even download from their archives. It was really just a hack, not a true integration.

The closest I cam to a great spam solution was ASSP found at http://assp.sourceforge.net/ This is a proxy mail server. It sits in between ME and the public internet. All email entering and leaving your server on port 25 is run through ASSP first. It uses bayesian filtering much like SpamAssassin. I ran it for a full 24 hours and was very impressed. Dspam seems to be the same solution as ASSP. I haven't tried to install dspam. This is from dspam's FAQ:

Q. Does it work with Windows?
A. v3.2 is the first to include a Windows build supplement, which includes the necessary Visual C++ project files and portage to compile the agent and tools under Windows. Check out the win32/ directory in the source tree for more information. Win32 support is still unofficial, but seems to work well. Of course getting it compiled is one thing, getting it integrated is another.

I would be hesitant to install it. One other thing you have to ask yourself is how you want these technologies to function in your environment. I host about 170 domains. ASSP, and Dspam for that matter, won't work in my environment. The bayesian database, blacklists, whitelists, redlists, graylists are all global. Any rule that uses data from these sources are applied to email for all domains. In the majority of cases this is fine but if a client wants for whatever reason to receive porn email and other clients *trained* the data sources to filter this type of email, the client will never receive his (or her) porn email. The other big downside of using a proxy is the kills ME’s ability to Reverse Blacklist. When ME receives email from the proxy it comes from 127.0.0.1 and that’s the only IP ME has to lookup in the blacklist databases.

After all this effort and research I have gone back to just filtering with ME’s built-in reverse blacklisting. I did some research and came up with a good list of databases including the ones that are preconfigured in ME. I built a quick and dirty script that analyzes my activity log to see how effective the blacklisting is working. You can find my real-time stats here:
http://www.oochie.com/log/stats.php

One thing I did change in ME default blacklist settings is on spamhous.org. I changed it from sbl.spamhouse.org to sbl-xbl.spamhous.org. It’s a database that combines their sbl list and their xbl list. You can see right now I am blocking about 70% of my email with the blacklists. When I had ASSP installed it was identifying about 60% of the incoming email as spam so I’m doing a little better right now.

I’ve spent the last week at war with spam. Unless anyone else has suggestion, I think with rigorous blacklisting, I’ll be winning the battle for now. The biggest roadblock I come across is that most spam filtering applications are being built in the *nix environment. Because its opensource the applications have better access to data and it’s easier for the average person to code. Don’t flame me if you differ, its just my observation.

Feel free to email me if you have questions: kaufmannAToochie.com

Scott

[paarlberg]

What are you using to show the stats? That would be nice to use in my environment.. Would be nice to see additional totals such as monthly, YTD, etc...

[scottkct]

the stats are generated using a home grown php script. When you view the page it basicly opens the log file and scans for unique identifiers for each data point. I set up unique responses for each blacklist in ME SMTP settings so that I can scan the log and identify how many times the unique words occur within the log.

It's built to work specificly on my machine. I'm not a coder by trade so I would have to clean it up make it firendly to configure before I gave it out. I will look into that.

I am very interested too in long term stats. This would require additional logging of the logs and I'm not advanced enough to whip it up. I currently trying to impliment http://cacti.net/ into my system. It's very similar to MRTG but its designed for much more than router data. I've been struggling but once I do I'm going to look into feeding the blacklist data into it to generate graphs and longterm trends. If I get it to work I will definately post it here.

Scott

[paarlberg]

I am running cacti for network traffic and server load info. It works great.. I will get my engineer that set it up to see what he can do with the MailEnable logs.

If you can post the identifiers that you used I will see if we can get something up here and share with others.

[Ruiner]

I did something like scott above (thanks for the idea), in php.. just wanted to give out the info in case anyone's interested... seeing as scott hasnt responded in a while. it's fairly simple, i've made it so by default if no file is specified it will show the last file (theorically, this is today's file!)

fyi the lines like "abuseat.org+blacklisted" are configured in the response line for the reverse blacklist lookup. you need to put something in there so the script can find it out.

oh and obviously the IIS guest user needs access to the log folder, so you'll need to change the rights unless you run this with enough credentials.

Code: Select all

<pre>
<?
   /* code by ruiner at konspiracy dot org */
	$abuseat = 0; $spamcop = 0; $ordb = 0; $unknownbox = 0; $mailok = 0; $serverblock = 0; $totalmail = 0;
	if(isset($file)) $fileok = 1;

	$d = dir("C:\\Program Files\\Mail Enable\\Logging\\SMTP");
	echo "select file you wish to see stats for: ";
	echo '<form action="index.php" method="get"><select name="file">';
	do {
		$entry=$d->read();
		if ((substr($entry,-4) == ".log") && (strstr($entry,"ex"))) {
			$i++;
			echo "<option value=\"$entry\">$entry</option>\n";
			if((!isset($fileok)) && ($entry!=false))
				$file = $entry;
		}
	} while ($entry!=false);
	echo '</select><input type="submit" value="Submit"></form>';
	$d->close();

	if($fp = fopen("C:\\Program Files\\Mail Enable\\Logging\\SMTP\\$file","r")) {
		while (!feof ($fp)) {
		    	$line = fgets($fp, 4096);
			if(strstr($line,"RCPT RCPT+TO")) {
				$totalmail++;
				if(strstr($line,"This+mail+server+requires+authentication")) $serverblock++;
				if(strstr($line,"abuseat.org+blacklisted")) $abuseat++;
				if(strstr($line,"spamcop.net+blacklisted")) $spamcop++;
				if(strstr($line,"relays.ordb.org+blacklisted"))	$ordb++;
				if(strstr($line,"mailbox+unavailable+or+not+local")) $unknownbox++;
				if(strstr($line,"Requested+mail+action+okay,+completed")) $mailok++;
			}
		}
		
		fclose($fp);
	}
	
	printf("file ............: $file<br>");
	printf("total emails ....: $totalmail<br>");
	printf("emails allowed ..: $mailok (%.3f)<br>",$mailok / $totalmail * 100);
	printf("email not local .: $unknownbox (%.3f)<br>",$unknownbox / $totalmail * 100);
	printf("server blocked ..: $serverblock (%.3f)<br>",$serverblock / $totalmail * 100);
	printf("abuseat blocked .: $abuseat (%.3f)<br>",$abuseat / $totalmail * 100);
	printf("spamcop blocked .: $spamcop (%.3f)<br>",$spamcop / $totalmail * 100);
	printf("ordb blocked ....: $ordb (%.3f)<br>",$ordb / $totalmail * 100);

?>
</pre>

[Ruiner]

having a bit more fun with the logging.. added virus info, this may be helpful also with people who put on filters and stuff.

Code: Select all

<pre>
<?
   /* code by ruiner at konspiracy dot org */
	$virus = 0; $china = 0; $abuseat = 0; $spamcop = 0; $ordb = 0; $unknownbox = 0; $mailok = 0; $serverblock = 0; $totalmail = 0;
	if(isset($file)) $fileok = 1;

	$d = dir("C:\\Program Files\\Mail Enable\\Logging\\SMTP");
	echo "select file you wish to see stats for: ";
	echo '<form action="index.php" method="get"><select name="file">';
	do {
		$entry=$d->read();
		if ((substr($entry,-4) == ".log") && (strstr($entry,"ex"))) {
			$i++;
			echo "<option value=\"$entry\"";
			if($entry == $file) echo " selected";
			echo ">$entry</option>\n";
			if((!isset($fileok)) && ($entry!=false))
				$file = $entry;
		}
	} while ($entry!=false);
	echo '</select><input type="submit" value="Submit"></form>';
	$d->close();

	if($fp = fopen("C:\\Program Files\\Mail Enable\\Logging\\SMTP\\$file","r")) {
		while (!feof ($fp)) {
		    	$line = fgets($fp, 4096);
			if(strstr($line,"RCPT RCPT+TO")) {
				$totalmail++;
				if(strstr($line,"This+mail+server+requires+authentication")) $serverblock++;
				if(strstr($line,"abuseat.org+blacklisted")) $abuseat++;
				if(strstr($line,"spamcop.net+blacklisted")) $spamcop++;
				if(strstr($line,"cn.countries.nerd.dk+blacklisted")) $china++;
				if(strstr($line,"relays.ordb.org+blacklisted"))	$ordb++;
				if(strstr($line,"mailbox+unavailable+or+not+local")) $unknownbox++;
				if(strstr($line,"Requested+mail+action+okay,+completed")) $mailok++;
			}
		}
		
		fclose($fp);
	}

	$filename = "C:\\Program Files\\Mail Enable\\Logging\\MTA\\MTAFILTER-Report-". substr($file,-10);
	if($fp = fopen($filename,"r")) {
		while (!feof ($fp)) {
		    	$line = fgets($fp, 4096);
			if(strstr($line,"Antivirus")) $virus++;
		}
		
		fclose($fp);
	}
	
	$mailok = $mailok - $virus;

	printf("file ............: $file<br>");
	printf("total emails ....: $totalmail<br>");
	printf("emails allowed ..: $mailok (%.3f)<br>",$mailok / $totalmail * 100);
	printf("email not local .: $unknownbox (%.3f)<br>",$unknownbox / $totalmail * 100);
	printf("server blocked ..: $serverblock (%.3f)<br>",$serverblock / $totalmail * 100);
	printf("abuseat blocked .: $abuseat (%.3f)<br>",$abuseat / $totalmail * 100);
	printf("spamcop blocked .: $spamcop (%.3f)<br>",$spamcop / $totalmail * 100);
	printf("ordb blocked ....: $ordb (%.3f)<br>",$ordb / $totalmail * 100);
	printf("china blocked ...: $china (%.3f)<br>",$china / $totalmail * 100);
	printf("virus blocked ...: $virus (%.3f)<br>",$virus / $totalmail * 100);

?>
</pre>

Enjoy

[Ruiner]

change

Code: Select all

			if(strstr($line,"RCPT RCPT+TO")) {

to

Code: Select all

			if(strstr($line,"RCPT RCPT+T")) {

For some reasons, sometimes it puts TO and sometimes To. hopefully I'm done adding to this thread =)) enjoy.

[DavidPayer]

I have extensively researched and tested multiple spam filtering solutions. MEFilter has a lot of potential but by currently only blocking by keywords is well known to not be a very effective solution. Along with any other list or static based blocking.
Scott

In encourage all here to look at a product called EWall (http://www.sssolutions.net)

They have a reduced cost version for Mail Enable. (called XMail version - only $99, otherwise. . lots more!)

This product acts as either a stand alone proxy or can be integrated onto the same machine as the server. It will handle the antivirus scanning, can integrate with SA or other command line spam scanners. It allows extensive rules based analysis (regex expressions for those so inclined) and rules based on GLOBAL / DOMAIN / USER perspectives.

I use it for part of my mail services and like it alot. For those looking for extensibility and control over incoming email, this is a product that works and plugs into any mail server.

I have no $ interest in this, I just like the product. Alex the developer is very responsive and the newsgroup is at news://news.sssolutions.net

David Payer

[json]

Maybe it's just me, but I can't seem to locate this XMail product you are talking about anywhere on the site!

regards,
Steen

[DavidPayer]

[quote="json"]Maybe it's just me, but I can't seem to locate this XMail product you are talking about anywhere on the site!

regards,
Steen[/quote]

http://sssolutions.net/8-)/register.php

Welcome to our online ordering system. Please select product to purchase.
eWall eWall X 100 Users v2.0.242 $86.95
eWall X Unlimited v2.0.242 $129.95
eWall 10 Users v2.0.242 $99.95
eWall 50 Users v2.0.242 $299.95
eWall 100 Users v2.0.242 $499.95
eWall Unlimited v2.0.242 $799.95
eWall Site v2.0.242 $1999.95

It is not JUST for XMail but the version gets its name from that. eWall X 100 is good for 100 email accounts, eWall X unlimited can be used for multiple domains/users

David P.