troubleshooting lost messages

Discussion forum for Enterprise Edition.
Post Reply
ShawnKHall
Posts: 113
Joined: Wed Apr 06, 2005 12:03 am
Location: California, USA
Contact:

troubleshooting lost messages

Post by ShawnKHall » Mon Aug 23, 2021 7:53 pm

We're seeing a high number of lost messages today. As far as I can tell the messages are literally being deleted on a whim and there's no record of them anywhere. How are we supposed to locate these lost messages? What logs should we be looking at? What should we be looking for?

Using ME Enterprise 9.84 with Plesk Obsidian 18.0.37 Update #2.

Tracing a lost message results in something like this:

Code: Select all

Trace: Tracing message with Message ID [FB7A8D5805B9403CBD7C84ABFEA255B3.MAI] from the SMTP Inbound Queue

Result: [FB7A8D5805B9403CBD7C84ABFEA255B3.MAI] was not routed from the SMTP inbound message queue by the local MTA Service. 
SMTP Debug log matches:


Virus filter log matches:
Error with reading log file C:\Program Files (x86)\Parallels\Plesk\Mail Servers\Mail Enable\LOGGING\MTA\MEAVGEN-Report-210823.log. Error Could not find file 'C:\Program Files (x86)\Parallels\Plesk\Mail Servers\Mail Enable\LOGGING\MTA\MEAVGEN-Report-210823.log'.

Important: Message ID [FB7A8D5805B9403CBD7C84ABFEA255B3.MAI] has not been routed through MailEnable queues. Check the log details for the reason. If in a cluster you will need to check each cluster server log files.

Status: SMTP Inbound transaction details for Message ID [FB7A8D5805B9403CBD7C84ABFEA255B3.MAI] follow below: 
08/23/21 13:22:36	SMTP-IN	FB7A8D5805B9403CBD7C84ABFEA255B3.MAI	2544	66.163.190.35			220 example.com	23	0		
08/23/21 13:22:36	SMTP-IN	FB7A8D5805B9403CBD7C84ABFEA255B3.MAI	2544	66.163.190.35	EHLO	EHLO sonic307-12.consmr.mail.ne1.yahoo.com	250-example.com [66.163.190.35], this server offers 5 extensions	177	44		
08/23/21 13:22:36	SMTP-IN	FB7A8D5805B9403CBD7C84ABFEA255B3.MAI	2544	66.163.190.35	STARTTLS			24	10		
08/23/21 13:22:36	SMTP-IN	FB7A8D5805B9403CBD7C84ABFEA255B3.MAI	2544	66.163.190.35	STARTTLS	STARTTLS		24	10		
08/23/21 13:22:36	SMTP-IN	FB7A8D5805B9403CBD7C84ABFEA255B3.MAI	2544	66.163.190.35	EHLO	EHLO sonic307-12.consmr.mail.ne1.yahoo.com	250-example.com [66.163.190.35], this server offers 6 extensions	187	44		
08/23/21 13:22:36	SMTP-IN	FB7A8D5805B9403CBD7C84ABFEA255B3.MAI	2544	66.163.190.35	MAIL	MAIL FROM:<example@att.net>	250 Requested mail action okay, completed	43	28		
08/23/21 13:22:37	SMTP-IN	FB7A8D5805B9403CBD7C84ABFEA255B3.MAI	2544	66.163.190.35	RCPT	RCPT TO:<shawn@example.com>	250 Requested mail action okay, completed	43	35		
08/23/21 13:22:37	SMTP-IN	FB7A8D5805B9403CBD7C84ABFEA255B3.MAI	2544	66.163.190.35	DATA	DATA	354 Start mail input; end with <CRLF>.<CRLF>	46	6		
08/23/21 13:22:39	SMTP-IN	FB7A8D5805B9403CBD7C84ABFEA255B3.MAI	2544	66.163.190.35	QUIT	QUIT	221 Service closing TLS SSL transmission session	50	6		Fwd: nnnnn
-Shawn

dcol
Posts: 173
Joined: Fri May 26, 2017 11:25 pm

Re: troubleshooting lost messages

Post by dcol » Mon Aug 23, 2021 8:59 pm

I am seeing the same issue. Outlook and Thunderbird are losing messages. In some cases the message just disappears. All those messages are in webmail, so I suspect the issue is with IMAP. This has been slowly getting worst. I see the messages in the logs, some are from unexpected servers and not the originating domain which is normal so as far as the logs, you are probably just not spotting them.

ShawnKHall
Posts: 113
Joined: Wed Apr 06, 2005 12:03 am
Location: California, USA
Contact:

Re: troubleshooting lost messages - BUG

Post by ShawnKHall » Tue Aug 24, 2021 12:27 am

The destination account isn't using IMAP, and the error suggests that it's actually a bug or issue with the MTA itself, or at least, there's no indication from the actual logs that *are* recording anything that IMAP or even SSL/TLS errors are involved.

It's possible that it's somehow related to a bug in the Plesk MTA - or it could be something else.

In an attempt to duplicate the problem I recreated the messages with attachments that were sent to me and I think the problem is related to message size limits. The attachments were collectively 15.2 MB in size on disk. Attempting to send the same files to myself from Yahoo resulted in the same log text with absolutely zero indication that a message size limit might be the cause:

Code: Select all

Important: Message ID [4674A9D3AB3640708EF7A58C356643E3.MAI] has not been routed through MailEnable queues. Check the log details for the reason. If in a cluster you will need to check each cluster server log files.
After these attachments are RLE'd they end up being 21.26 MB, so are over the message size limitation configured at SMTP, Inbound, Advanced settings, Advertised Maximum Message Size - which we had assigned to 20 MB and were enforcing.

I have increased that limit and tested with the same files again and am now receiving the emails. That appears to have resolved the problem.

I will be contacting the other senders and have them retry as well - but I suspect they're going to be OK.

This is a definitely a BUG in both logging and the SMTP service (and by its dependence, the Message Route Trace Utility). Messages that are rejected for exceeding the message size SHOULD be properly logged as having been rejected somewhere in a log. As it is now, all of the messages that exceeded the 20 MB message size limit simply recorded the exact same logs as every other incoming message in the SMTP log, and do not even appear in the MTA logs:

Code: Select all

08/23/21 19:01:25	SMTP-IN	4674A9D3AB3640708EF7A58C356643E3.MAI	2616	66.163.187.60	DATA	DATA	354 Start mail input; end with <CRLF>.<CRLF>	46	6		
08/23/21 19:01:27	SMTP-IN	4674A9D3AB3640708EF7A58C356643E3.MAI	2616	66.163.187.60	QUIT	QUIT	221 Service closing TLS SSL transmission session	50	6		test with 15mb
-Shawn

MailEnable-Ian
Site Admin
Posts: 9547
Joined: Mon Mar 22, 2004 4:44 am
Location: Melbourne, Victoria, Australia

Re: troubleshooting lost messages

Post by MailEnable-Ian » Tue Aug 31, 2021 11:57 pm

Hi,

You will need to manually search the MTA activity and debug log files for the message ID FB7A8D5805B9403CBD7C84ABFEA255B3.MAI in your SMTP log file to see what happened when the message was to be routed from the SMTP inbound queue. Plesk uses (enables) various MTA com events that we have seen can cause message routing issues.
Regards,

Ian Margarone
MailEnable Support

ShawnKHall
Posts: 113
Joined: Wed Apr 06, 2005 12:03 am
Location: California, USA
Contact:

Re: troubleshooting lost messages

Post by ShawnKHall » Wed Sep 01, 2021 1:26 am

MailEnable-Ian wrote:
Tue Aug 31, 2021 11:57 pm
You will need to manually search the MTA activity and debug log files...
I did. That was literally my first step. NO RECORD of the messages was ever recorded in the MTA logs:
shawnkhall wrote: ...all of the messages that exceeded the 20 MB message size limit simply recorded the exact same logs as every other incoming message in the SMTP log, and do not even appear in the MTA logs...
This isn't a situation where the MTA logs were any help, since the message was never recorded there.

Then I tried the ME message route trace utility, which I hoped would provide some better access or guidance than simply looking at the logs. It didn't. The only thing of value was this juicy tidbit:

Code: Select all

Important: Message ID [4674A9D3AB3640708EF7A58C356643E3.MAI] has not been routed through MailEnable queues. Check the log details for the reason. If in a cluster you will need to check each cluster server log files.
Again, this is saying that the messages NEVER MADE IT TO THE MTA LOGS.

Through trial and error, I was able to figure out that the lack of it being passed to the MTA service was due to the message exceeding the hard message size limit so the message was rejected after the DATA command, but no record of this appears in the logs. Simply increasing that message size limit did resolve the message delivery problem. However, it does not resolve the real problem, in that it took me half a day of trial and error and meeting with a client physically to get a USB stick with the files on it before I was able to determine the cause of the problem which should have been indicated in ANY of the message logs (preferably the SMTP log, since that was where it was dropped).

This is a bug. MailEnable should keep a log whenever it deletes or rejects messages.

Let's assume that it's a Plesk bug instead and that Plesk removed the messages. First, it should be a ridiculously simple test for you to perform in order to find out. Second, even if it's another object handling message events, the ME logs should indicate whether the message was passed to the MTA, removed, or that the MTA could not find the message and that should be logged. ME must be receiving a return value from these com events - why aren't you recording responses in the logs?
-Shawn

MailEnable-Ian
Site Admin
Posts: 9547
Joined: Mon Mar 22, 2004 4:44 am
Location: Melbourne, Victoria, Australia

Re: troubleshooting lost messages

Post by MailEnable-Ian » Wed Sep 01, 2021 2:35 am

Hi Shane,

If the inbound message size is exceeded it will log this in the SMTP Debug log file. I have tested this in 10.35. You will see the following:

Code: Select all

09/01/21 12:31:27	ME-I0101: [824] Local Delivery: Address ([SMTP:test@mailenable.local]) is local.
09/01/21 12:31:27	ME-F0xxx: Inbound message exceeded maximum message size
09/01/21 12:31:30	ME-IXXXX: [824] (Debug) Removing connection
09/01/21 12:31:30	ME-I0074: [824] (Debug) End of conversation
The client will also receive a bounce with the following:

Code: Select all

Your message did not reach some or all of the intended recipients.

      Subject:	test
      Sent:	1/09/2021 12:31 PM

The following recipient(s) cannot be reached:

      'test@mailenable.local' on 1/09/2021 12:31 PM
            552 Message size exceeds fixed maximum message size
Regards,

Ian Margarone
MailEnable Support

ShawnKHall
Posts: 113
Joined: Wed Apr 06, 2005 12:03 am
Location: California, USA
Contact:

Re: troubleshooting lost messages

Post by ShawnKHall » Wed Sep 01, 2021 3:45 am

MailEnable-Ian wrote:
Wed Sep 01, 2021 2:35 am
If the inbound message size is exceeded it will log this in the SMTP Debug log file. I have tested this in 10.35.
It does not include this in the debug file in 9.x
-Shawn

MailEnable-Ian
Site Admin
Posts: 9547
Joined: Mon Mar 22, 2004 4:44 am
Location: Melbourne, Victoria, Australia

Re: troubleshooting lost messages

Post by MailEnable-Ian » Wed Sep 01, 2021 4:23 am

Hi Shawn,

The developers have indicated that this has not changed for quite some time and therefore your version 9 will log this. The issue maybe related to something else and therefore would be best logging a support ticket and providing the full set of log files for the date/time of your tests so I can pass onto the developers.
Regards,

Ian Margarone
MailEnable Support

Post Reply