Page 1 of 1
Extended decorator replacement problem
Posted: Thu Sep 20, 2007 8:41 pm
by adrianbj
OK, so I am using the following code to simply replace the text {EMAIL_ADDRESS} with the recipients email address - it is for the unsubscribe link at the bottom on the email.
I have a testing list in my database of 50 addresses and all addresses are properly composed with the appropriate email address inserted where {EMAIL_ADDRESS} is in the email. However, I have a real list in my database with ~1000 recipients and when I send to this, the replacement works at first, but then seems to get stuck on one address and all emails after that get replaced with this same address. I am not sure what is going on and testing is obviously difficult because I only get to send these when I have something to send out to my real list.
I know the code in the replacement class is a little funky, but as I said, it does work until it gets stuck on this one address. I have looked at the sending process with verbose sending enabled and there is a considerable pause in the sending at the email address before the one that hijacks all subsequent replacements.
I am also using the Swift_Cache_Disk and AntiFlood plugins in case that helps.
I have not tested it with the $address = strtolower($address); line - I only just added that, but there are no email addresses in the vicinity of the one in question that have any uppercase characters, although there are some elsewhere in the database. Mind you, I added an email address with uppercase characters to my testing list and there were no problems.
Thanks for any help - have been going nuts trying to figure this one out.
Code: Select all
//Extend the replacements class
class Replacements extends Swift_Plugin_Decorator_Replacements {
function getReplacementsFor($address) {
$address = strtolower($address);
$result['{EMAIL_ADDRESS}'] = $address;
return $result;
}
}
$query_emails = "SELECT email_address FROM $email_list WHERE seminar_online = '1'";
$emails = mysql_query($query_emails) or die(mysql_error());
while($row_emails = mysql_fetch_assoc($emails)){
$recipients->addTo($row_emails['email_address']);
}
//Load the plugin with the extended replacements class
$swift->attachPlugin(new Swift_Plugin_Decorator(new Replacements()), "decorator");
}
Posted: Fri Sep 21, 2007 9:09 am
by chuckl
What query results do you get when you enter the address select query in a phpmyAdmin or mysqld command line?
What happens if you delete that entry? Does the error vanish or simply move to the next entry?
Does it occur at a 'significant' count value e.g. a multiple of 5 or 10, which might be a server imposed limit/hr or limit/connection?
Posted: Fri Sep 21, 2007 10:31 am
by adrianbj
chuckl,
The query results are fine via PMA - it grabs the required email addresses including the ones around the problem region.
I haven't deleted that entry yet - it is a production list and users are expecting these emails, but also unfortunately I can't simply delete an email address or two and resend to test it. I have another mailout in about 10 days and I will try it without the email address before, and the one that takes over all subsequent replacements.
It is around the 15th email that is sent, but my testing list of 50 emails doesn't pause at this point and it doesn't have any replacement problems.
The email addresses from the query in order look something like this:
- 14 normal email addresses that from what I can tell all get the appropriate email address in the replaced code
- 1 group email address - this address contains a couple of hundred users from our organization. The address has the same domain as the smtp server I am using to send the mailout. Could it be that it is trying to process all the contained addresses in this list causing some sort of timeout problem, causing swift to stop doing the replacements? From memory, this is where the verbose output from swift appears to pause for about 10 secs.
- One normal email address - not sure, but it might be this one that it pauses at
- One email address that gets stuck as the replacement address for all subsequent emails.
- ~1000 emails that from what I can tell all have the replacement address from the previous email
Next time I send i will remove the group email address and the next two and see what happens, but if anyone has any ideas in the meantime, I would love to hear them - thanks.
Posted: Fri Sep 21, 2007 10:57 am
by chuckl
Might well be the group 'bulk list' address that's doing it. Possible to move it to the end? Not sure if people will appreciate being on the 'z list' though.
Posted: Fri Sep 21, 2007 11:23 am
by adrianbj
Yeah, I will remove it for the next mail-out and send to it separately. Hopefully that will work and if it does, I'll figure out how order the query so it is always last.
Posted: Sun Sep 30, 2007 7:20 pm
by adrianbj
Ok, another followup.
I just sent out an email to the entire list. I made sure the group email list address that I mentioned was sent last. It certainly seemed to solve that problem.
However, this time there was another slow down during the send at another address which is not a group list and is not on the smtp domain. When watching them send (via the verbose plugin), it stalled on a particular address for about 15-20 seconds. All addresses up to and including this one received the email with the correct replacement. All those after this address were replaced with the next email address in the list. Interestingly the address the became the replacement for all subsequent addresses was never actually sent to, but it is not listed as a failure in the verbose send, it just doesn't show up at all. So to clarify, it stalled on recipient number 1066. After the stall it sent 1067, but it missed out the address which really should have been 1067 and it became the replacement for all subsequent addresses.
I am using the anti-flood plugin like so:
Code: Select all
$swift->attachPlugin(new Swift_Plugin_AntiFlood(200, 10), "anti-flood");
so it is only 67 messages after a break.
This is really confusing as I can't work out why the email in question would be stalling during send - this one is not a group address.
Anyone have any ideas?
Thanks.
Posted: Mon Oct 01, 2007 7:06 am
by chuckl
The examples you have given are at the 'attach' stage, but the problems are occurring at the send stage.
It sounds to me that at the slowdown stage a 'hidden' error/exception is being tripped and that it is thereafter simply rolling through the loop with the send disabled/disconnected.
The loop count is incrementing, but the message/recipient count is not, and nothing is actually sent?
Posted: Mon Oct 01, 2007 11:39 am
by adrianbj
chuckl - thanks for the thoughts. All the messages are actually being sent, except for the one that becomes the replacement address for all subsequent emails. The verbose sending output shows this, but also I have many email addresses scattered through the list so I can see what is going on. All addresses up to the point of the stall get the correct replacement and all those after the stall get the replacement from that address that didn't send.
I have just set up logging
Straight after my swiftmailer 'require_once' statements
Code: Select all
$log =& Swift_LogContainer::getLog();
$log->setLogLevel(4);
Then at the end after the send operation
Code: Select all
$log =& Swift_LogContainer::getLog();
$log->setMaxSize(0);
echo $log->dump(true);
I sent to my testing list of 50 addresses and got this output
Code: Select all
<< 250 2.1.5 ... Recipient ok >> DATA << 354 Enter mail, end with "." on a line by itself >> >> . << 250 2.0.0 l91G5kC3005053 Message accepted for delivery ++ Message sent to 1/1 recipients >> MAIL FROM: << 250 2.1.0 ... Sender ok >> RCPT TO: << 250 2.1.5 ... Recipient ok >> DATA << 354 Enter mail, end with "." on a line by itself >> >> . << 250 2.0.0 l91G5kC4005053 Message accepted for delivery ++ Message sent to 1/1 recipients >> MAIL FROM: << 250 2.1.0 ... Sender ok >> RCPT TO: << 250 2.1.5 ... Recipient ok >> DATA << 354 Enter mail, end with "." on a line by itself >> >> . << 250 2.0.0 l91G5kC5005053 Message accepted for delivery ++ Message sent to 1/1 recipients >> MAIL FROM: << 250 2.1.0 ... Sender ok >> RCPT TO: << 250 2.1.5 ... Recipient ok >> DATA << 354 Enter mail, end with "." on a line by itself >> >> . << 250 2.0.0 l91G5kC6005053 Message accepted for delivery ++ Message sent to 1/1 recipients >> MAIL FROM: << 250 2.1.0 ... Sender ok >> RCPT TO: << 250 2.1.5 ... Recipient ok >> DATA << 354 Enter mail, end with "." on a line by itself >> >> . << 250 2.0.0 l91G5kC7005053 Message accepted for delivery ++ Message sent to 1/1 recipients >> QUIT << 221 2.0.0 my.smtp.server closing connection ++ Closing down SMTP connection.
Now I know there are no stalls/slowdowns with my testing list, but just wanted to make sure logging is working for when I send to the real list. I don't really understand the output though - the documentation says it will be truncated to a maximum of 50 entries, but this only shows 5 send operations and I know that all 50 addresses received the email. Also, I have set it so there should be no maximum. Obviously I am not understanding something here. Also, the "Enter mail, end with "." on a line by itself " entry is weird - I read about this and was actually having trouble with this when using a different smtp server, but with the one I am using now, I don't get any truncation problems.
Do you think the logging is working properly? Hopefully it might help for my next full send. Should I maybe try the throttler in addition to the AntiFlood, or should I change my anti-flood settings?
Posted: Mon Oct 01, 2007 2:15 pm
by chuckl
Couple of further questions then. Are you
a. Calling 'send in a loop, passing it a single address at a time
b. Passing 'send' a recipient list of the whole shebang
c. Calling 'batchSend' or an instance of BatchMailer
d. Crossing fingers and letting decorator fiddle it.
Next question would be whether you have any try-catch exception handling round the send routine, as transmission errors and network funnies are quire common in large batches. If you're not using it, the batch send also has retries, error handling, automatic reconnect and non delivery logging built in.
To your questions, on the logging side, I must be honest and say I've never used it. The refactoring I've been doing blows up so spectacularly when anything is wrong, that the error messages are enough. That said, I think the log max defaults to 50 lines, which is very different to 50 entries. That list you have there is easily 50 lines,as the lines go
MAIL FROM:
<< 250 2.1.0 ... Sender ok >>
RCPT TO:
<< 250 2.1.5 ... Recipient ok >>
DATA
<< 354 Enter mail, end with "." on a line by itself >> >>
.
The 'Enter mail, end with "." on a' etc is normal SMTP server communication. They can be quite chatty. it simply means - 'I see you have sent a DATA command, and I expect the mail message to follow. I will know it is complete when I receive a single '.' on a line by itself.'
Thats how an end of message is signalled in SMTP.
On the AntiFlood side, the settings depend on your mail server. many servers limit a batch to 50 or a hundred. Or 10. Again, if you're not using it, batchSend may be more robust.
Posted: Mon Oct 01, 2007 2:36 pm
by adrianbj
I am using batchSend. Here is my complete code:
Code: Select all
<?php
set_time_limit(0);
ignore_user_abort();
flush();
ob_flush();
require_once ('Swift.php');
require_once ('Swift/Connection/SMTP.php');
require_once ('Swift/Plugin/Decorator.php');
require_once ('Swift/Plugin/AntiFlood.php');
require_once ('Swift/Plugin/VerboseSending.php');
require_once ('smtp_server.php');
$log =& Swift_LogContainer::getLog();
$log->setLogLevel(4);
//Enable disk caching if we can
if (is_writable("/tmp"))
{
Swift_CacheFactory::setClassName("Swift_Cache_Disk");
Swift_Cache_Disk::setSavePath("/tmp");
}
$swift = new Swift(new Swift_Connection_SMTP($smtp_server));
//Reconnect after 200 emails, but wait for 10 seconds first
$swift->attachPlugin(new Swift_Plugin_AntiFlood(200, 10), "anti-flood");
$view = new Swift_Plugin_VerboseSending_DefaultView();
$swift->attachPlugin(new Swift_Plugin_VerboseSending($view), "verbose");
//Create a message
$message = new Swift_Message(stripslashes ($_POST['email_subject']));
//Add some "parts"
$message->attach(new Swift_Message_Part(stripslashes($_POST['text_only'])));
$message->attach(new Swift_Message_Part(stripslashes($_POST['html_code']), "text/html"));
//Initiate recipient list
$recipients = new Swift_RecipientList();
//Extend the replacements class
class Replacements extends Swift_Plugin_Decorator_Replacements {
function getReplacementsFor($address) {
$result['{EMAIL_ADDRESS}'] = strtolower($address);
return $result;
}
}
$query_emails = "SELECT email_address FROM email_list ORDER BY email_address";
$emails = mysql_query($query_emails) or die(mysql_error());
while($row_emails = mysql_fetch_assoc($emails)){
$recipients->addTo($row_emails['email_address']);
}
//Load the plugin with the extended replacements class
$swift->attachPlugin(new Swift_Plugin_Decorator(new Replacements()), "decorator");
}
//Send messages
$sent = $swift->batchSend($message, $recipients, new Swift_Address("me@myaddress.com","My Name"));
//Disconnect from SMTP, we're done
$swift->disconnect();
if ($sent)
{
print "Success";
}
else
{
print "Failure";
}
$log =& Swift_LogContainer::getLog();
$log->setMaxSize(0);
echo $log->dump(true);
?>
I don't have any try-catch exception handling round the send routine - would you mind giving me an example of how to do this.
Thanks for clarifying the log output, that definitley helps to explain what is going on, but since I set the MaxSize to have no limit, I would expect to see an entry for each of the 50 emails sent, wouldn't I?
I guess I could certainly try sending in batches of 10 next time just to be on the safe side and see what happens. Regardless, I would like to understand why a connection/send glitch would cause the decorator to mess up the replacement - I guess I don't really understand the mechanics of how that is working.
Posted: Mon Oct 01, 2007 4:22 pm
by chuckl
feyd | Please use Code: Select all
and [syntax="..."] tags where appropriate when posting code. Your post has been edited to reflect how we'd like it posted. Please read: [url=http://forums.devnetwork.net/viewtopic.php?t=21171]Posting Code in the Forums[/url] to learn how to do it too.[/color]
Try-catch is very easy, but may well already be implemented inside batchSend. You surround the connect or send operation with a try-catch pair, like this
Code: Select all
try {
//Start Swift
$swift = new Swift(new Swift_Connection_SMTP("smtp.your-host.tld"));
//Create the message
$message = new Swift_Message("My subject", "My body");
//Now check if Swift actually sends it
$swift->send($message, "foo@bar.tld", "me@mydomain.com");
echo "Sent";
} catch (Swift_ConnectionException $e) {
echo "There was a problem communicating with SMTP: " . $e->getMessage();
} catch (Swift_Message_MimeException $e) {
echo "There was an unexpected problem building the email:" . $e->getMessage();
}
you 'try' the operation, and 'catch' any errors you are interested in. They are returned in $e above. I suspect batchSend may already implement it.
However, batchMailer also has a couple more tricks up it's sleeve. It returns the number of mails it thinks it has sent, so you should print out the value of $sent at the endof the run.
And I'm not sure about batchSend, but batchMailer has the ability to print out failed recipients. So if instead of batchSend, you use
Code: Select all
$batch =& new Swift_BatchMailer($swift);
$batch->send($message, $recipients, $from);
you can then add
an echo or
Code: Select all
print_r($batch->getFailedRecipients());
after the send.
And despite the fact that it's unlikely, nay impossible, I can't help feeling that the decorator is somehow getting it's oar in somewhere.
On the log side, don't know, as I say, never used it. I think the syntax changed slightly for version 3.3.1?
On Antiflood batch size, maybe cut down to 50 on the first loop?
feyd | Please use Code: Select all
and [syntax="..."] tags where appropriate when posting code. Your post has been edited to reflect how we'd like it posted. Please read: [url=http://forums.devnetwork.net/viewtopic.php?t=21171]Posting Code in the Forums[/url] to learn how to do it too.[/color]
Posted: Tue Oct 02, 2007 4:18 am
by chuckl
I just checked the source code, batchMailer does indeed have it's own try-catch embedded internally.
In order to track it down, you could try the following:
Go back to straight send, with your own try-catch implemented - in effect force any errors into the open.
Modify the reoutine slightly so you pass it 10 or 20 or 50 addresses at a time, with your own delay between - implement your own anti-flood and batching in other words.
Disable the Decorator for a batch, see if that causes it to send correctly.
Posted: Wed Oct 03, 2007 3:58 pm
by adrianbj
Thanks again for all the new suggestions. I have been been crazy busy the last few days, but I will try some of those options on the next mail out next week. I'll update when I have some more info from the next send.
Thanks.
Posted: Wed Oct 03, 2007 6:31 pm
by Chris Corbyn
As ~chuckl points out, batchSend() does include an internal try/catch. This is needed because 90% of the people using batchSend() are using it as a one-line solution to batch mailing. Swift needs to deal with errors rather than allowing exceptions to be bubbled up to the caller if people are to be able to use it in this way
send() on the other hand is the basic call which batchSend() repeatedly uses. There's nothing magical about batchSend() really other than a big loop with exception handling

You could easily write your own implementation.
Posted: Mon Oct 08, 2007 3:38 pm
by adrianbj
Just wanted to update and say that I took the easy way out with the email I sent out today and simply changed the antiflood to break after every 50 instead of every 200 and it looks like all were sent sucessfully and all replacements were made as they were supposed to be.
I will let you know if there are any further troubles next time I send. Then I will try some of the other options, but for now it looks like everything is working fine.
Thanks again for all your help.