Page 1 of 1

Performance concerns (v3 vs. v4)

Posted: Sun Mar 15, 2009 3:56 am
by Talisto
For the past year or so I've been using Swift Mailer v3 for mailing out approximately 50,000 emails per day to our website's members (all opted in, don't worry :) ). I utilize the connection rotator to spread the load across 4 SMTP servers, with batchSend() batches of 1000 at a time (via cron). The emails are pretty basic HTML/text, with no attachments or embedded images.

With Swift Mailer v3, each batch of 1000 emails would take approximately 30 seconds to send, utilizing about 10% of the server's CPU during the process.

I've since made the required changes to my script to use Swift Mailer v4. Everything about my script has stayed the same, with the exception of using the "LoadBalancedTransport" instead of v3's "Swift_Connection_Rotator", and a few other minor syntax changes. However, with v4, each batch of 1000 emails takes over 10 minutes to send, and uses 100% of the server's CPU (or as much as it can take) during processing. This is on a dual 3ghz xeon. That's quite a performance difference.

I've tried all the various transports to see if it was a problem there, but the result is the same. I'm assuming the delay in processing has to do with the new character encoding system (re: ticket #66?), and it seems the character encoding is done during each individual send regardless of a batch send or not (correct me if I'm wrong?).

Is there anything I can do in my script to help speed up the process? Are there differences in processing requirements for different character encoding types?

For now I've reverted back to v3 as v4 is just putting too much strain on my server for 50,000 emails/day. If there's anything I can do to speed up v4's performance please let me know. Much appreciated!

Matt

Re: Performance concerns (v3 vs. v4)

Posted: Sun Mar 15, 2009 5:29 am
by Chris Corbyn
We're aware of performance differences, and it is related to encoding yes. We can't really roll-back those changes since they are needed, but we'd really like to improve the speed.

I'm working on a full rewrite of this system which will hopefully speed things up (#66 as you say). Also, in v3, if 8BITMIME was advertised by the server we didn't bother QP encoding the data. In v4 we always QP encode the data, and QP encoding is slow. There's a ticket in the system to support 8BITMIME again (#68), thus removing this bottleneck on servers that support 8BITMIME (the majority these days).

My priority right now (since it's quite integral to any feature additions we make) is to optimize the way all the encoding actually works. The slow point is based around the fact that PHP has no concept of "characters" when it comes to anything other than US-ASCII. We've (well, I did at the time, not very well) written character handling code in PHP itself and basically I'll be the first to admit that while it works and solves many of the bugs in version 3, it's not based on any detailed studying and can be greatly improved.

I'm working on it basically, pretty much all of my free time at the moment, so hopefully we can speed this up. But even if QP encoding doesn't get faster (pretty unlikely with a solid rewrite based on Java's Charset classes) 8BITMIME support will be added, thus by-passing this slowness unless it's really needed (i.e. unless the server doesn't support 8-bit transfers).

I'm shocked at the differences you're seeing though... nobody has claimed to go from 30 seconds to 10 minutes before... that sounds kinda crazy.

Re: Performance concerns (v3 vs. v4)

Posted: Sun Mar 15, 2009 5:33 am
by Chris Corbyn
By the way, until we add automated 8BITMIME support, you can force it (I can almost guarantee considerable speed gains) by doing this:

Code: Select all

$message->setEncoder(Swift_Encoding::get8BitEncoding());
NOTE: Caching should be used provided the body has not been reset between sends.

Re: Performance concerns (v3 vs. v4)

Posted: Mon Mar 16, 2009 2:18 am
by Talisto
Chris Corbyn wrote:By the way, until we add automated 8BITMIME support, you can force it (I can almost guarantee considerable speed gains) by doing this:

Code: Select all

$message->setEncoder(Swift_Encoding::get8BitEncoding());
Wow, that definitely worked.. my batch times are back down to less than a minute, with negligible CPU usage. Perfect! Thanks, Chris!!

Re: Performance concerns (v3 vs. v4)

Posted: Mon Mar 16, 2009 3:31 am
by Chris Corbyn
Talisto wrote:
Chris Corbyn wrote:By the way, until we add automated 8BITMIME support, you can force it (I can almost guarantee considerable speed gains) by doing this:

Code: Select all

$message->setEncoder(Swift_Encoding::get8BitEncoding());
Wow, that definitely worked.. my batch times are back down to less than a minute, with negligible CPU usage. Perfect! Thanks, Chris!!
To be RFC compliant with the Quoted-Printable encoding there are so many "buts" in the RFC that lead to expensive sanity checks in the code. The worst part however is the fact that PHP isn't unicode aware... So I has to improvise. That's the bit I'm currently improving (by using unicode encoding algorithms and buffering).

Re: Performance concerns (v3 vs. v4)

Posted: Fri May 22, 2009 6:37 pm
by gmorehoudh
Maybe it'd be faster to write it in C and submit an enhancement to PHP. I hear you can get just about anything put in the function library ;)