Newsletter system with 50,000+ emails

Swift Mailer is a fantastic library for sending email with php. Discuss this library or ask any questions about it here.

Moderators: Chris Corbyn, General Moderators

Post Reply
klang
Forum Newbie
Posts: 10
Joined: Sat May 05, 2007 1:23 pm

Newsletter system with 50,000+ emails

Post by klang »

Hi all,

I've just been asked to develop and host a newsletter system to send out 50,000+ emails twice a month. Each email might contain attached image up to 200k. So, I've searched the internet and like Swift Mailer the most. My main concern is how to send 10Gb emails and still be able to run other websites on my shared server.

1. Is Swift can handle 50,000+ emails?
2. Is it doable on the shared server environment?
3. I don't have another SMTP server, so I guess I need to go with Sendmail connection?
4. I believe Swift run in the background. Can I just send 50,000+ emails with AntiFlood in one batch and let it runs for days?
5. If you were me, what would you charge a hosting fee for this system?

FYI. I have my own server, so I can config and install anything to it.

Thanks,
- Klang
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Re: Newsletter system with 50,000+ emails

Post by Chris Corbyn »

klang wrote:Hi all,

I've just been asked to develop and host a newsletter system to send out 50,000+ emails twice a month. Each email might contain attached image up to 200k. So, I've searched the internet and like Swift Mailer the most. My main concern is how to send 10Gb emails and still be able to run other websites on my shared server.

1. Is Swift can handle 50,000+ emails?
2. Is it doable on the shared server environment?
3. I don't have another SMTP server, so I guess I need to go with Sendmail connection?
4. I believe Swift run in the background. Can I just send 50,000+ emails with AntiFlood in one batch and let it runs for days?
5. If you were me, what would you charge a hosting fee for this system?

FYI. I have my own server, so I can config and install anything to it.

Thanks,
- Klang
1. Yes indeed. The featureset will be better built for intensive batch mailing when 3.2 comes out (soon) but you can do this now anyway. I'd do them in small batches (say 100 at a time) though.

2. Yes, but your shared server host will probably give you a good telling off for sending out so many emails and in all likelihood I'd be suprised if they even allow that many to be sent from your account ;) I'll guess that if someone's asking you to send 100,000 emails per month they are getting money from it so basically you need to tell them they need a server if your host won't deal with it. The problem is that you'll be wasting all the servers resources whilst you're running off batches of over 3000 emails a day (minimum). You may also get a lot of failures and see things running slowly because I'll bet their MX is overused. You can get your own VDS for less than the price of a phone line these days.

3. Either that, or ask your host for SMTP, use NativeMail or use Gmail (although they have now started limiting the rate to 50 at a time).

4. Swift cannot run in the background alone no. PHP can ignore_user_abort(), but for 50,000+ this would be madness ;) Cron is made for things like this so that you can do them in smaller batches. If you try leaving 50,000+ emails sending in one run *on a shared server* even with AntiFlood/Throttling to keep the load down your host will likely kill what appears to them to be a rogue process and you'll be left not knowing how many emails actually got sent (possibly).

5. I can't answer that sorry. I'd probably actually tell the client they need a server organised.

EDIT | I missed the bit about you having your own server. So is this a shared server or is it your own server? I mean, do other people host websites besides yourself on this server? Do you have a root account for it?
klang
Forum Newbie
Posts: 10
Joined: Sat May 05, 2007 1:23 pm

Post by klang »

Hi...

Thanks for a quick response.

I do own and have a root access to this server.

Gmail option is interesting. Can I send a batch of 50 emails through gmail SMTP every 5 minutes? Or even better, can I create multiple gmail accounts (lets say 10 accounts) and send a small batch each account at the same time. I feel like Google will detect my IP and block it.

Thanks,
- Klang
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

klang wrote:Hi...

Thanks for a quick response.

I do own and have a root access to this server.

Gmail option is interesting. Can I send a batch of 50 emails through gmail SMTP every 5 minutes? Or even better, can I create multiple gmail accounts (lets say 10 accounts) and send a small batch each account at the same time. I feel like Google will detect my IP and block it.

Thanks,
- Klang
Gmail actually does appear to track IP addresses because there was a post recently where someone was trying to use AntiFlood to reconnect after the 50 emails but Gmail was having none of it. I'm not sure what delay they will expect but I reckon it it will be more like 50/hour rather than 50/5mins. Relaying on Gmail for sending out newsletters is probably not a good idea anyway. If you have root access to your server you should be able to install exim (or just use sendmail). I'd still avoid trying to send all 50,000 in one go though.

The typical approach to sending out newsletters etc is to create a row in a database for each recipient and flag a column as "unsent" initially. Then (by cron) every 10-15 minutes access the database, select for example 500 recipients where the flag is "unset" then send to those and update the flag to "sent". Eventually the cron job will have no rows to select because you will have sent all emails.

The companies who send out such large lists very quickly are using multiple servers to do it (Swift_Connection_Rotator), in which case you could send out larger batches.

It's not my place to comment, but if whoever is asking you to send 50,000+ emails twice a month for them is not paying you enough to be able to afford the hardware for it something is amiss.

You have to bear in mind that however fast Swift can throw messages at the SMTP server/MTA, the SMTP server/MTA still has to deal with those so you will overload the server if you try throwing 50,000+ emails in one go at it. There are multiple tiers involved with sending email just like there are with delivering web content. The client (the app which uses Swift), and the SMTP server(s).
roscoe
Forum Commoner
Posts: 85
Joined: Tue Aug 05, 2003 10:24 am
Location: essex uk

Post by roscoe »

I am in the process of building a mail system to take future newsletters and attachments, diary them and then (bacause CRON is not available on a shared server and I am limited to 200 mails per blat) it takes the email addresses and batches them into 200s (each one has a name).

I have managed to catalogue them, upload the attachment to the server and I was hoping to drop them onto swift mailer (using sendmail as it doesn't work with smtp, but I think that is an advantage here with the hold on smtp blatting to 200 per day)

The question is how do I attach the emails held on the shared server to the email created?

I have the filename in my sql db and I can get that, but would it require the full path to the file or can it be http://www.mydom.com/myfilestored/thefile.doc ? and what command would I use?

I assume I then point it at swiftmailer and then away we go!
klang
Forum Newbie
Posts: 10
Joined: Sat May 05, 2007 1:23 pm

Post by klang »

Thanks for all your help. I really appreciate it.
alex.barylski
DevNet Evangelist
Posts: 6267
Joined: Tue Dec 21, 2004 5:00 pm
Location: Winnipeg

Post by alex.barylski »

I've just been asked to develop and host a newsletter system to send out 50,000+ emails twice a month.
Typical
My main concern is how to send 10Gb emails
Yikes. Thats alot of bandwidth. :)
and still be able to run other websites on my shared server
Hahahahaha...sorry that gave me a good laugh... :)

Shared servers, wouldn't handle that kind of traffic. For starters I imagine you'd need a daemon to carry out that many as a PHP script would call it quits before it finished sending all the emails. Secondly, you need to usually use your ISP SMTP for sending email for that reason. Your ISP would kick you in the bum if you sent that many duplicate emails. Probably stopping at about 100 as they would detect you as a spammer. Read their policies and TOS.

I would suggest going with an existing newsletter service... :)
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

L

Post by Chris Corbyn »

Yes, if you have shell access you may be worth writing a daemon in PHP. I have this set up. We could use cron, but we chose to go down this route.

Newsletter is written inside our apps.
User groups to target are selected (these are users of our system being informed of changes).
Mail is "spooled" to MySQL database and a row is added to another MySQL table linking each user to that spooled message.
Against the row for each user there's a "active_key" field - I couldn't think of anything better :P
Daemon runs in a permanent while loop, sleeping for 5 seconds after each loop.
Each loop picks up 50 addresses and sends to them, updating the "active_key" to a unique keys for this batch (we do this because we have a server farm and any two servers could potentially send the same batch otherwise).

It works well for our purposes, but cron would be your typical approach to this. Unfortunately I cannot post you the code due to licensing reasons.
Post Reply