MeFilter DB and spam phrases issues

Discussion, support and announcements for third party applications that work with MailEnable.
Post Reply
keever
Posts: 3
Joined: Wed Mar 26, 2008 10:55 pm

MeFilter DB and spam phrases issues

Post by keever » Wed Mar 26, 2008 11:34 pm

I shut down ME's services, compacted the Mefilter DB using the ODBC tool, made sure the compacted database was the same name as what was in Mefilter's Database config tab. When I restarted ME's services, and opened Mfilter's Console, there was nothing in the holding directory and I couldn't rebuild the holding areas due to an error about not being able to flush records from the database. The spam phrase global config list was empty. Where did all the phrases go? What on earth about compacting a database would cause these problems? I rebooted the machine ME and Mefilter run on, and the above problems were fixed. But it would be a pathetic shame to be forced to reboot upon making db changes in Mefilter, especially since rebooting was NEVER mentioned in the MEfilter documentation or forums. Is this the case?

Also, how on earth does MeFilter add phrases to it's dictionary automatically? How does the spam dictionary get updated without manual, tedious, data input? In MailEnable, you can configure Baynesian filtering to compare spam and ham dictionaries, and then generate accurate spam phrases. How does MeFilter do this?

Anyway, to avoid insane tedium in manual spam phrase input, I would like to import a list of spam phrases, but there is ZERO documentation on how you do this. I see the option under the Utils menu, but after clicking on it, there is no way to browse to a file containing the phrases. After clicking "List Imports > Spam Phrases", it responds with a pop up that states "Import List Complete - Please Check the Results" It must expect a text file in a specific path to be put in place ahead of time, but ABSOLUTELY NO WHERE, on Mefilter's pathetic forum or 12 month plus neglected documentation does it say what path to import from, and what txt file format to put the phases into. Nor does it say where or how to "Check the Results".

The next issue is baffling and again is also horribly documented to the point of violent anger. In the Spam Phrase Editor, when manually adding a phrase, (in this case to the global config), what exactly is the difference between the "Add" , "Update" and "Apply" buttons? Why would anyone ever want to add a phrase, but not update it? Or why would anyone want to add a phrase, but not apply it?

Anyway, I'm having an issue where I add a phrase, update it, but upon refreshing the list after clicking "Apply", that phrase disappears from the list. How the $#@! do you add phrases to the spam dictionary when the phrases disappear upon clicking "Apply"?

How do most people manage the spam phrases in MEfilter without the manual phrase addition feature that appears to be broken anyway?

MartynK
Posts: 1326
Joined: Sat Dec 28, 2002 1:12 am
Location: Hong Kong

Post by MartynK » Thu Mar 27, 2008 6:13 am

1st off, if you have a problem with the ODBC tool and compacting the database, you can hardly blame MEFilter for that, its nothing to do with it.

As for 12 months of no documentation, the documentation is up to date as per the latest version v7.0 and has been available in the downloads area for months.

With regards importing phrases, best way is to enter one phrase, then export it. Edit the file that is created in the MEFilter directory, once you have made the required changes, then import the file using MEConsole.

Last but I sure there will be more, if you have an issue, I will help via support@mefilter.com, if not you can always PM me here and I will see what I can do.

Now breath !!!!!

keever
Posts: 3
Joined: Wed Mar 26, 2008 10:55 pm

Post by keever » Thu Mar 27, 2008 7:54 pm

You're right that I should not blame MEFilter for a problem with the ODBC tool. Though, I never had a problem with ODBC, I had a problem with MEFilter using the compacted database upon restarting ME's services. I looked on MEFilter's forum and KB articles for howto's and suggestions regarding db maintenance and found nothing. You'd figure that everyone using MEFilter would have a growing database to deal with, and that maintenance of that database would be covered in either the Installation.doc, forum, or KB articles. But it's not. In the installation.doc that comes with 6.5, page 12 refers to a Database Maintenance tool in MEConsole that doesn't exist in 6.5.

The latest documentation for v7 isn't helpful since I"m not using v7, and v7 is BETA so I wouldn't want to rely on it anyway.

As for the website documentation for 6.5 located here:

http://mefilter.com/Main_Config.asp

There is nothing that addresses the issues I mentioned in the last post. No mention of how to compact the database. No mention of how to get MEfilter to successfully use a db that has been compacted by the ODBC tool. There is a section that addresses MEFilter Spam Phrases, and the spam phrase library:

http://mefilter.com/SpamPh.asp

Here is all the information to be had from the documentation:
Spam Phrase Edit Tab - Spam Phrases can be entered at the "Global Level" as shown in the example above, or at a Domain Level.

The Validate button allows you too check each of the phrases that have been entered into the phrase list against the configured MEFilter rules for spam checking. If anything is found, the window will be updated and you need to press the apply button to save the results. This is a good way to check phrases after doing a manual import.

This level of cutomisation offers you the opportunity to finely tune each Domain individually, or accept the Spam Phrases globally for all Domains on your Mail server.
The above does not answer, "Also, how on earth does MeFilter add phrases to it's dictionary automatically?". It also doesn't explain how to import a list of spam phrases. I haven't found one bit of text that explains how the "Utils > List Imports > Spam Phrases" tool works.

The above text does explain how the "Validate" and "Apply" functions work... poorly, but fails to explain the "Add" "Update" "Delete" buttons. I don't assume that the function of each button is self-evident by the label given to the button. What isn't clear at all is how to manually add a phrase. Should I put a spam phrase in the Spam Phrase text box, and then click "add"? If I do that, should I click "Update" next? Should I then click "Apply"?

The method you describe sounds circuitous and redundant, and also is not mentioned anywhere in the installation.doc, the above configuration html links, the KB articles, or in the forum. Why haven't I read about this before you posted it?:
With regards importing phrases, best way is to enter one phrase, then export it. Edit the file that is created in the MEFilter directory, once you have made the required changes, then import the file using MEConsole.
What is the point of adding a phrase via MEConsole if you have to export the list (a process explained no where in any doc), manually edit the file (when you say edit, you don't say what content I'm editing or why) and then import the file using MEConsole (another process explained exactly no where in documentation) When you write "import the file using MEConsole" you assume a lot of unknown information that isn't available in any provided documentation (.doc, http, or forum). MEConsole doesn't allow you to browse for a specific file, so the file name and location must be known ahead of time. Would you care to let me know what path and file name I have to use to import a list of spam phrases? Shouldn't this information be this howto?:

http://mefilter.com/SpamPh.asp


Back the point of the Spam Phrase Editor within MEConsole being redundant: why bother with the GUI when you have to manually edit the spam phrase file and import it? Why shouldn't I just export the spam phrases, edit the file, and import it, totally avoiding the spam phrase editor? This whole process is a mystery.

Now onto the topic of automated addition of spam phrases into the spam phrase library. First, does it exist? I notice observations that imply there might be a way to add common spam phrases to the library without manually doing so, but no clear statement or description. For instance:
"The Validate button allows you too check each of the phrases that have been entered into the phrase list against the configured MEFilter rules for spam checking. "
"... that have been entered into the phrase list..." implies there are other ways to add phrases other than inputting a phrase into the text box and clicking "add".
"If anything is found, the window will be updated and you need to press the apply button to save the results."
"If anything is found..."????!!!! Why would spam phrases be "found" if the only way was to manually add them? The word "found" implies some other process besides manual data input that adds phrases to the dictionary, and the validate button "finds" them, and then offers the user the option of "applying" them. Can I get some explanation of this process, how it works, and how to configure it properly?
"This is a good way to check phrases after doing a manual import."
Oh but wait, the next sentences say the validate function is a good way to check phrases after a manual import. The hanging question is whether "Validate" is used to check phrases that aren't the result of a manual import.

The last thing I want to get into, is to what "Validate" actually does. The text in the GUI states that it will check if the phrases contain characters that will be removed during the spam filtering "Normalization". There is no explanation of what happens if it does find a phrase that contains characters that will be removed. (Is the phrase not valid? Does one have to "Update" the phrase using the mysterious and enigmatic "update" function that is known only to stone-masons and the Cthulu elders?) Maybe the installation.doc , the KB articles or the html config guide will tell me more about what "Validate" does??

Nope.

Nothing in the .doc. A search for "Validate" in the KB articles return zero results. Here's what we've already seen from the html config doc:


"The Validate button allows you too check each of the phrases that have been entered into the phrase list against the configured MEFilter rules for spam checking."

"MEFilter rules" must refer to the character removal during "Normalization". What exactly is "Validate" checking other than the character removal? "MEFilter rules" is a much broader description than just the character removal process. So what exactly does the "Validate' function find? Once it finds this unknown something, what are to be done with the results? The html config doc says to "apply", but why? What changes am I applying?

As for the forum, here's a look at what I can find regarding db maintenance and spam phrase library maintenance and configuration:

First I'll check in the "MEFilter Documentation" section:

http://mefilter.com/cs/forums/10.aspx

Here is the great, up to date documentation you speak of:

"Oops! There are either no posts or due to a filter there are no posts to display."

Darn.

What about the "General" section under the " MEFilter Database Questions and Answers" Discussion:

http://mefilter.com/cs/forums/20.aspx

"Oops! There are either no posts or due to a filter there are no posts to display."

But surely there will be information in the "MS Access" section:

http://mefilter.com/cs/forums/21.aspx

"Oops! There are either no posts or due to a filter there are no posts to display."

Maybe I'll learn something about the Spam Phrases library from the "Post Your Spam Phrases" section:

http://mefilter.com/cs/forums/26.aspx

"Oops! There are either no posts or due to a filter there are no posts to display."

So now you see why I don't hold the MEFilter forums in high regard. Not even one sticky post covering the most requested information on the topic.

I appreciate your offer for help via email or PM, but I'm not going insane when I notice horribly lacking MEFilter documentation in all of the outlets MEFitler offers to its users. If all my answers are contained within the v7 documentation, then I would suggest that you (or whomever is maintaining MEFilter's docs) let all 6.5 users know that the v7 docs are authoritative. I'm not going to move to v7 until it's a stable release.

keever
Posts: 3
Joined: Wed Mar 26, 2008 10:55 pm

wow

Post by keever » Thu Mar 27, 2008 8:28 pm

I didn't realize I was responding to the creator of MEFilter, and I do feel sheepish that I've been this aggressive in criticism. I would note that I'm actively choosing MEFilter for it's lightweight effectiveness, but I'm having a heck of a time finding explanations for the GUI functions and all features.

MartynK
Posts: 1326
Joined: Sat Dec 28, 2002 1:12 am
Location: Hong Kong

Post by MartynK » Fri Mar 28, 2008 7:17 am

OK, a massive post and I will try and reply over the weekend or early next wee in full, but just a few quick points.

The web pages that you reference are very basic and were setup by a friend to try and help, they were never meant to be the full answer to everything.

V7 is stable, I have been running it now for about 5 months, I just have not renamed the release (which is my fault really as I should).

To be honest, the documentation was something that was not even written until v7, the reason for this is I was to busy working on the actual filter (not a good excuse, but a true one).

The forums are not heavily monitored by myself as I spend the majority of my time working the support system. The forums tend to get answered by other users when they can and I will get there when I can.

To be honest, I have never since writing MEFilter every compacted my database. As long as your not logging to the database, it should grow as needed, but it should not just keep growing. As rows are deleted from tables, the actual database space will get re-used as needed.

Spam phrases have never been added automatically, this cannot be done using the methods that MEFilter uses as one persons spam is another persons work.

thetruth1960
Posts: 4
Joined: Fri Nov 09, 2007 5:33 pm

Post by thetruth1960 » Sun May 11, 2008 4:13 pm

Keever, I just found this post. Excellent post. You saved me the time of writing it myself.

I did download the 7.0 documentation as Martyn suggested, to see if it told me a bit more about importing phrases. Well, there is not much there either. It says you can do it, but it doesn't say anything about the file name, where you put the file. Basically nothing on how you import the phrases.

I did do the export and it created a file called SpamPhraseExport.txt. Great, so should I assume that if I rename the file to SpamPhraseImport.txt then it should work?

MEFilter seems to be a good product. But, the documentation is lacking. I know Martyn has mentioned in other areas that he has a real job and a family and other responsibilities and I definitely understand that b/c I do things on the side too. But for instance, in the example of your post, would it had been too hard to just answer the specific questions you asked? He did take the time to come here and answer you, but he didn't take the time to actually read the entire post, at least that's what it seems, and answer the questions you asked knowing that the documentation is lacking. And he should know that it is lacking since he said he wrote it.

MartynK
Posts: 1326
Joined: Sat Dec 28, 2002 1:12 am
Location: Hong Kong

Post by MartynK » Mon May 12, 2008 5:41 am

And, so what have I missed now.

I think the previous posts were answered directly, hence not on this forum.

Anyway, so what its your issue, how can I help ?

Post Reply