Page 1 of 1

300+ e-mails per hour with different content

Posted: Sun Jun 03, 2007 5:43 am
by CRichardson
I run a website which sends out over 300 e-mails per hour each with different user-specific content.

What is the best method of doing this?

Posted: Sun Jun 03, 2007 6:59 am
by Chris Corbyn
Depending upon the amount of variation between these emails you may be interested in looking at the decorator plugin. 300 emails per hour is quite low, but in the interest of scalability I'd probably do it by cron. You throw the emails into a database, marked as "unsent", then the cron script runs every minute or 2 sending all those "unsent" and marking them as sent.

EDIT | Forgot the link:

http://www.swiftmailer.org/wikidocs/v3/ ... /decorator

Posted: Sun Jun 03, 2007 7:33 am
by CRichardson
d11wtq wrote:Depending upon the amount of variation between these emails you may be interested in looking at the decorator plugin. 300 emails per hour is quite low, but in the interest of scalability I'd probably do it by cron. You throw the emails into a database, marked as "unsent", then the cron script runs every minute or 2 sending all those "unsent" and marking them as sent.

EDIT | Forgot the link:

http://www.swiftmailer.org/wikidocs/v3/ ... /decorator
- Each e-mail will be totally different to the one before it
- 300 will go up so it does need to be scalable
- I currently have a cron job run every hour
- The cron runs a script which grabs their e-mail address from a database, creates the messages, then sends the e-mail

I am guessing, from what you have said that I am going about the whole process backwards?

Posted: Sun Jun 03, 2007 8:04 am
by Chris Corbyn
You sound as though you're going about it in a sensible way actually :) I assume you don't just pull out however many emails happen to be there and send them all at once? I'd probably run the script more than once and hour (it wouldn't hurt to run every few minutes if there's no mail to send most of the time). I usually suggest keeping batch sizes to no more than 100, although there are ways around it using the AntiFlood plugin.

Posted: Sun Jun 03, 2007 8:12 am
by CRichardson
There has to be a strict hour between each e-mail sent.
So, User1 should get one e-mail every 60 minutes! The same goes for all 300+ users.

The script I use simply queries the database, finds the required people and within a "while" statement sends the e-mails. So in answer to your question, it does try and send them all at once (stupid, I know!).

I know I should look at batch processing but the way in which the message is created and users grabbed from the database is already complex enough!

Posted: Sun Jun 03, 2007 8:17 am
by Chris Corbyn
CRichardson wrote:There has to be a strict hour between each e-mail sent.
So, User1 should get one e-mail every 60 minutes! The same goes for all 300+ users.

The script I use simply queries the database, finds the required people and within a "while" statement sends the e-mails. So in answer to your question, it does try and send them all at once (stupid, I know!).

I know I should look at batch processing but the way in which the message is created and users grabbed from the database is already complex enough!
I see, I assumed this was just a newsletter or something. Is this some hourly report? Use the AntiFlood plugin to deal with that level of batching then.

http://www.swiftmailer.org/wikidocs/v3/ ... /antiflood

Posted: Sun Jun 03, 2007 8:20 am
by CRichardson
Yea, you could call it that. The content will most likely change every hour! The same number of e-mails per hour will be sent (to a degree).

I've tried the AntiFlood plugin, which works up until 101 e-mails.

I've had so many problems, including:

- mysql connection closing itself
- php timing out
- memory/bandwidth usage
- smtp usage

I've tried to log everything, still no real result!

Posted: Sun Jun 03, 2007 8:24 am
by CRichardson
Note: when I run the script in FireFox after a certain amount of time it wants me to download the file and in IE after the same amount of time it gives me "Page cannot be displayed".

The bandwidth monitor gave me the following going out:
2625
34660
53480
56080
62688
137839
143497
146122
149734

That's just 9 e-mails! I logged the monitors results to a database.

Posted: Sun Jun 03, 2007 8:29 am
by Chris Corbyn
Wanting to download the page is usually a sign of a broken PHP installation. Infinite loops or infinite recursion sometimes cause this behaviour too. You'll need to post the code you're using if I'm to be able to assess that could be causing that.

As for timeouts, you need to set_time_limit().

Posted: Sun Jun 03, 2007 9:01 am
by CRichardson
What is the best way to ensure you don't create an infinite loop?

Code: Select all

while($fetch=mysql_fetch_assoc($result)) {

}
Is that infinite? Surely not because it only runs till there is no more rows left?

Posted: Sun Jun 03, 2007 9:32 am
by Chris Corbyn
CRichardson wrote:The bandwidth monitor gave me the following going out:
2625
34660
53480
56080
62688
137839
143497
146122
149734

That's just 9 e-mails! I logged the monitors results to a database.
FYI, readings are in bytes so this is quite low really. The bandiwdth monitor is logging the total bytes sent, not the bytes for each individual email, this is why it goes up as the number of emails goes up.