Page 1 of 1

Newsletter system with 50,000+ emails

Posted: Sat May 05, 2007 1:58 pm
by klang
Hi all,

I've just been asked to develop and host a newsletter system to send out 50,000+ emails twice a month. Each email might contain attached image up to 200k. So, I've searched the internet and like Swift Mailer the most. My main concern is how to send 10Gb emails and still be able to run other websites on my shared server.

1. Is Swift can handle 50,000+ emails?
2. Is it doable on the shared server environment?
3. I don't have another SMTP server, so I guess I need to go with Sendmail connection?
4. I believe Swift run in the background. Can I just send 50,000+ emails with AntiFlood in one batch and let it runs for days?
5. If you were me, what would you charge a hosting fee for this system?

FYI. I have my own server, so I can config and install anything to it.

Thanks,
- Klang

Re: Newsletter system with 50,000+ emails

Posted: Sat May 05, 2007 2:12 pm
by Chris Corbyn
klang wrote:Hi all,

I've just been asked to develop and host a newsletter system to send out 50,000+ emails twice a month. Each email might contain attached image up to 200k. So, I've searched the internet and like Swift Mailer the most. My main concern is how to send 10Gb emails and still be able to run other websites on my shared server.

1. Is Swift can handle 50,000+ emails?
2. Is it doable on the shared server environment?
3. I don't have another SMTP server, so I guess I need to go with Sendmail connection?
4. I believe Swift run in the background. Can I just send 50,000+ emails with AntiFlood in one batch and let it runs for days?
5. If you were me, what would you charge a hosting fee for this system?

FYI. I have my own server, so I can config and install anything to it.

Thanks,
- Klang
1. Yes indeed. The featureset will be better built for intensive batch mailing when 3.2 comes out (soon) but you can do this now anyway. I'd do them in small batches (say 100 at a time) though.

2. Yes, but your shared server host will probably give you a good telling off for sending out so many emails and in all likelihood I'd be suprised if they even allow that many to be sent from your account ;) I'll guess that if someone's asking you to send 100,000 emails per month they are getting money from it so basically you need to tell them they need a server if your host won't deal with it. The problem is that you'll be wasting all the servers resources whilst you're running off batches of over 3000 emails a day (minimum). You may also get a lot of failures and see things running slowly because I'll bet their MX is overused. You can get your own VDS for less than the price of a phone line these days.

3. Either that, or ask your host for SMTP, use NativeMail or use Gmail (although they have now started limiting the rate to 50 at a time).

4. Swift cannot run in the background alone no. PHP can ignore_user_abort(), but for 50,000+ this would be madness ;) Cron is made for things like this so that you can do them in smaller batches. If you try leaving 50,000+ emails sending in one run *on a shared server* even with AntiFlood/Throttling to keep the load down your host will likely kill what appears to them to be a rogue process and you'll be left not knowing how many emails actually got sent (possibly).

5. I can't answer that sorry. I'd probably actually tell the client they need a server organised.

EDIT | I missed the bit about you having your own server. So is this a shared server or is it your own server? I mean, do other people host websites besides yourself on this server? Do you have a root account for it?

Posted: Sat May 05, 2007 9:27 pm
by klang
Hi...

Thanks for a quick response.

I do own and have a root access to this server.

Gmail option is interesting. Can I send a batch of 50 emails through gmail SMTP every 5 minutes? Or even better, can I create multiple gmail accounts (lets say 10 accounts) and send a small batch each account at the same time. I feel like Google will detect my IP and block it.

Thanks,
- Klang

Posted: Sun May 06, 2007 5:50 am
by Chris Corbyn
klang wrote:Hi...

Thanks for a quick response.

I do own and have a root access to this server.

Gmail option is interesting. Can I send a batch of 50 emails through gmail SMTP every 5 minutes? Or even better, can I create multiple gmail accounts (lets say 10 accounts) and send a small batch each account at the same time. I feel like Google will detect my IP and block it.

Thanks,
- Klang
Gmail actually does appear to track IP addresses because there was a post recently where someone was trying to use AntiFlood to reconnect after the 50 emails but Gmail was having none of it. I'm not sure what delay they will expect but I reckon it it will be more like 50/hour rather than 50/5mins. Relaying on Gmail for sending out newsletters is probably not a good idea anyway. If you have root access to your server you should be able to install exim (or just use sendmail). I'd still avoid trying to send all 50,000 in one go though.

The typical approach to sending out newsletters etc is to create a row in a database for each recipient and flag a column as "unsent" initially. Then (by cron) every 10-15 minutes access the database, select for example 500 recipients where the flag is "unset" then send to those and update the flag to "sent". Eventually the cron job will have no rows to select because you will have sent all emails.

The companies who send out such large lists very quickly are using multiple servers to do it (Swift_Connection_Rotator), in which case you could send out larger batches.

It's not my place to comment, but if whoever is asking you to send 50,000+ emails twice a month for them is not paying you enough to be able to afford the hardware for it something is amiss.

You have to bear in mind that however fast Swift can throw messages at the SMTP server/MTA, the SMTP server/MTA still has to deal with those so you will overload the server if you try throwing 50,000+ emails in one go at it. There are multiple tiers involved with sending email just like there are with delivering web content. The client (the app which uses Swift), and the SMTP server(s).

Posted: Tue May 08, 2007 4:25 am
by roscoe
I am in the process of building a mail system to take future newsletters and attachments, diary them and then (bacause CRON is not available on a shared server and I am limited to 200 mails per blat) it takes the email addresses and batches them into 200s (each one has a name).

I have managed to catalogue them, upload the attachment to the server and I was hoping to drop them onto swift mailer (using sendmail as it doesn't work with smtp, but I think that is an advantage here with the hold on smtp blatting to 200 per day)

The question is how do I attach the emails held on the shared server to the email created?

I have the filename in my sql db and I can get that, but would it require the full path to the file or can it be http://www.mydom.com/myfilestored/thefile.doc ? and what command would I use?

I assume I then point it at swiftmailer and then away we go!

Posted: Tue May 08, 2007 8:10 pm
by klang
Thanks for all your help. I really appreciate it.

Posted: Tue May 08, 2007 10:47 pm
by alex.barylski
I've just been asked to develop and host a newsletter system to send out 50,000+ emails twice a month.
Typical
My main concern is how to send 10Gb emails
Yikes. Thats alot of bandwidth. :)
and still be able to run other websites on my shared server
Hahahahaha...sorry that gave me a good laugh... :)

Shared servers, wouldn't handle that kind of traffic. For starters I imagine you'd need a daemon to carry out that many as a PHP script would call it quits before it finished sending all the emails. Secondly, you need to usually use your ISP SMTP for sending email for that reason. Your ISP would kick you in the bum if you sent that many duplicate emails. Probably stopping at about 100 as they would detect you as a spammer. Read their policies and TOS.

I would suggest going with an existing newsletter service... :)

L

Posted: Sun May 13, 2007 6:37 am
by Chris Corbyn
Yes, if you have shell access you may be worth writing a daemon in PHP. I have this set up. We could use cron, but we chose to go down this route.

Newsletter is written inside our apps.
User groups to target are selected (these are users of our system being informed of changes).
Mail is "spooled" to MySQL database and a row is added to another MySQL table linking each user to that spooled message.
Against the row for each user there's a "active_key" field - I couldn't think of anything better :P
Daemon runs in a permanent while loop, sleeping for 5 seconds after each loop.
Each loop picks up 50 addresses and sends to them, updating the "active_key" to a unique keys for this batch (we do this because we have a server farm and any two servers could potentially send the same batch otherwise).

It works well for our purposes, but cron would be your typical approach to this. Unfortunately I cannot post you the code due to licensing reasons.