[SOLVED] Using Swiftmailer to send bulk messages with CRON

Swift Mailer is a fantastic library for sending email with php. Discuss this library or ask any questions about it here.

Moderators: Chris Corbyn, General Moderators

User avatar
fishnyc22
Forum Commoner
Posts: 38
Joined: Wed May 30, 2007 6:20 pm

[SOLVED] Using Swiftmailer to send bulk messages with CRON

Post by fishnyc22 »

Hi everyone. I'm new to swiftmailer and trying to wrap my head around some things... I was hoping you all can shed some light on some things... Sorry for the long email. I want to give some background then I have some Qs at the end.

My site offers a service that sends out brief text only (opted-in) emails to users many times throughout the day. The number of messages going out with each blast can range anywhere from 1 to 2000+ (currently), but will be growing with time. Right now mail is being sent successfully since we are smaller, but as we grow I want to ensure the server can handle the load and prevent crashes.

I just came across SwfitMailer and am blown away. I wish I had found it earlier. I'm going to be implementing it this weekend.

I should mention the server is a Dedicated Virtual Server w/ 512 MB Ram. The Box has PHP4, Linux (CentOS) and Plesk on it. (and only 1 domain currently running on it)

With all the research I've been doing this week. I've had a few questions I could use help with:

1. I read on this forum that a good option for bulk mailing would be to use SwiftMailer along with CRON to disperse the load. I get the concept but I'm unsure of the proper technique. Would I write all messages to a DB queue and run cron ever X minutes to send X messages out. Or is there another method to track what messages were sent and what needs to still go out?

2. I'm currently processing the php script and sending the mail on the same server (via QMAILs MTA). I'm on a dedicated Virtual Server right now. Seems to be running well, but I have had a few moments of panic when the web server suspended actions. Not sure why exactly... we don't have a huge amount of traffic. I'd like to take advantage of Swiftmailers ability to distrubute the sends out to several servers. Since we only have the one
server now, I'm not exactly sure what other servers I can use. Can anyone recommend any services that we can connect to?

3. I was trying an SMTP test with Swiftmail. I had success sending via the following SMTP test:

Code: Select all

$swift =& new Swift(new Swift_Connection_SMTP("localhost"));
and with

Code: Select all

$swift =& new Swift(new Swift_Connection_SMTP("mail.mydomain.com"));
When sending either way, it went out successfully, though I thought it would fail without authentication. Sending mail fails when I try to send mail from my mail app on my computer without authentication. Is that odd? Is it because the script and server are on the same machine?

4. I tried sending a simple test send via Sendmail using the following code:

Code: Select all

require_once $_SERVER['DOCUMENT_ROOT'] . "/util/lib/swift/Swift.php";
require_once $_SERVER['DOCUMENT_ROOT'] . "/util/lib/swift/Swift/Connection/Sendmail.php";

$swift =& new Swift(new Swift_Connection_Sendmail());
$message =& new Swift_Message("My subject", "My body");
 
if ($swift->send($message, "email@somedomain.com", "me@mydomain.com")) echo "Sent";
else echo "Failed";
I continually get the following error:

Warning: Invalid argument supplied for foreach() in /var/www/vhosts/domain.com/subdomains/dev/httpdocs/util/lib/swift/Swift/Connection/Sendmail.php on line 323t

I also tried both of these lines and received the same error

Code: Select all

$swift =& new Swift(new Swift_Connection_Sendmail(SWIFT_SENDMAIL_AUTO_DETECT));
or

Code: Select all

$swift =& new Swift(new Swift_Connection_Sendmail("/usr/sbin/sendmail -bs"));
Also, side question... should I force it to qmail. Is there a benefit? Is it possible?

5. Would sending w/ Swift_Connection_Rotator via SMTP and sendmail on the same machine as the script help me at all? I assume no.


I realize this is a long email. But i'm about to head into some new stuff I've never done before. Any feedback would really help me out.

Thanks for reading this far :)

Fish
><>
Last edited by fishnyc22 on Fri Jun 01, 2007 8:50 am, edited 1 time in total.
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

1. Use cron every minute or so if you're doing small batches (say 50-100 per batch) yes. It's just a case of storing messages and recipients in a DB with a field flagged "unsent". Then when you send a batch, mark those ones sent.

I actually run a constantly running PHP daemon for this on multiple servers accessing the same database (just an infinite while loop, running in the background with sleep($time) at the end of each loop.

2. You're better off using your own servers.

3. SMTP servers figure out here the connection is being made from. You'll be in a trusted network when running on the web server so you don't need to authenticate.

4. If you know the path to qmail (/usr/sbin/qmail ?) then try that. I've never seen that error before. Ah-ha:

Code: Select all

foreach ($this->pipes as $pipe)
    {
      @fclose($pipe);
    }
I guess disconnect() has already been called. I'll add a check to stop that -- Done :)

Code: Select all

foreach ((array)$this->pipes as $pipe)
    {
      @fclose($pipe);
    }
5. Yes, you can combine the sendmail connection with the SMTP connection, but this would be pointless if it's the same server (i.e. if you're using sendmail + SMTP=localhhost.

Good luck :)

PS: Check for a new release later today if you don't want to add that (array) cast to line 323 in Sendmail.php yourself ;)
User avatar
fishnyc22
Forum Commoner
Posts: 38
Joined: Wed May 30, 2007 6:20 pm

Post by fishnyc22 »

Thanks SO much for all the information. It's really gonna help me out. A few followups:

1. Is there a reason you say 50-100 per batch? So you are using your PHP daemon instead of cron to send your messages? I've never heard of running an infinite loop like that. I assume you have it start it on reboot or something? Why that method over cron? Just curious.

Is there any benefit to removing the sent DB records from the queue vs keeping them in and marking them as flagged

Also, for example, if we are queuing up 5000 emails at say 1pm... and 100 are going out every minute via cron.... My only concern with that is that the emails are a slightly time sensitive (a delay is understandable, but the last batch would go out 50 min later. )

2. Good to know. I figured as much.

3. Do you suggest I go with sendmail/qmail or SMTP?

4. I replaced that piece of code, but I'm getting a blank screen now on test. Here is my PHP code I'm testing:

Code: Select all

require_once $_SERVER['DOCUMENT_ROOT'] . "/util/lib/swift/Swift.php";
require_once $_SERVER['DOCUMENT_ROOT'] . "/util/lib/swift/Swift/Connection/Sendmail.php";

$swift =& new Swift(new Swift_Connection_Sendmail());
$message =& new Swift_Message("My subject", "My body");
 
if ($swift->send($message, "email1@somedomain.com", "email@mydomain.com")) echo "Sent";
else echo "Failed";
I also tried with no luck:

Code: Select all

$swift =& new Swift(new Swift_Connection_Sendmail('/etc/rc.d/init.d/qmail'));
I think thats the right qmail...searching qmail on my system revealed:
/etc/rc.d/init.d/qmail
/var/lock/subsys/qmail
/var/qmail


5. Right, Makes sense. Thanks


Thanks again for everything. Much Much much appreciated
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

1. I don't like to batch more than 100 in a go because I appreciate that's not only my code that takes a strain when you really trash it. You have to remember that you're abusing the SMTP server if you send 1000 emails in one go. Everyone seems to have this really strange misconception that a couple of minutes pause in a hidesously large batch is going to make the difference between make or break in their business. Come on, if you've got that many emails to send a short pause isn't going to hurt. Give the server a break, let it unpool some mail and give it a chance to unpage some memory :)

The only reason I run a daemon constantly is to keep the process IDs down. Simple as that. It's a single process with an open connection to the database. Cron is probably a more suitable option but I'm just a bit unorthodox ;)

5000 emails. Mail hits the server, the server spools it (writes it to a "queue" file). If the mail is not sent right away, or the server is just way overloaded, it sits on the spool in a queue. The server keeps re-running that queue. If you throw 5000 emails at it one shot it's more than likely going to send out 50-100 pretty quickly then memory will start being exhausted and the server gets slower and slower, shoving things on the queue. The queue is usually only re-run every 5-15 minutes by default. If you really want to send out that many emails in an instant, you'll have to use multiple servers. You could whack them all at the mail server and it will spool them in just under 10 minutes if you have a fast connection and a high-spec but they won't all leave the server in that 10 minutes.

There are *much* faster ways to do it but you'll be more likely to get stuck in a spam filter ("undisclosed-recipients:;").

2. If you're sending out this many emails I can only assume you're making a profit. You can get a VDS server for very little money these days. Just get a few VDS servers. Gmail do provide SMTP to their own members but it's limited and they rewrite your message headers to display your gmail address in the Sender: header.

3. I wouldn't go with sendmail. It seems all the other MTAs have been written to replace it. I won't cast judgement either way about which MTA to use because I'm quite opinionated on it. Let's just say I don't like qmail in the same way I don't like Microsoft Windows, but it's a popular choice either way. Exim and Postfix are also popular choices. The best thing to do is to try a fe yourself and see what you think though. SMTP is a bit slower initially because of the network overhead, but sending through qmail, postfix, sendmail or Exim locally you'll still only be spooling mail until the queue is run off. I personally stick to SMTP purely for the fact that I'm balancing the workload across two servers (minimum).

4. '/etc/rc.d/init.d/qmail' is the statrt and stop script for the daemon. /etc/rc.d and /etc/init.d are standard BSD/Unix directories for start-stop scripts. You need the binary, not the start script. It will be in /usr/sbin, /usr/local/sbin or /sbin somewhere most likely. Opening the /etc/rc.d/init.d/qmail file will probably reveal the path. Or this command:

Code: Select all

which qmail
Thinking about it, /var/qmail could be the binary. qmail is a weird one so it wouldn't suprise me. You can pretty easily tell by trying to run it:

Code: Select all

/suspected/path/to/qmail -bs
You should see a line starting with "220". CTRL+C, or type "QUIT" to leave.

Working on getting a release out tonight but I went for beers after work so my concentration is a bit hazey :P
User avatar
fishnyc22
Forum Commoner
Posts: 38
Joined: Wed May 30, 2007 6:20 pm

Post by fishnyc22 »

Awesome stuff. Thanks again...

1. Interesting stuff re: the servers. I think I'll stay away from the daemon option as I'm sure I'll do something I should... I'd love to see an example if you have one.

2. No profit yet. Free service. trying to build up a userbase. Been getting some great feedback and comments from users... I will check out a VDS server. It its worth it, we may go that route at some point. thanks for the tip.

3. I realize that qmail and sendmail are not the same thing but I believe they work together, right? doesnt sendmail just push mail to qmail? I will try to see how smtp works for us. Question: In plesk, I can view our mail queue to see whats left to go out. I would still see mail showing up there if I send via SMTP right? Or is that just the qmail queue?

4. Very weird. I can not find qmail. Its the weirdest thing. /var/qmail is a folder with the following folders:

Code: Select all

alias  bin  boot  control  mailnames  plugins  queue  users
running "which qmail resulted in:

Code: Select all

/usr/bin/which: no qmail in (/usr/kerberos/sbin:/usr/kerberos/bin://sbin://bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/X11R6/bin:/root/bin)
Am I going crazy? I know SMTP works but out of curiosity Id like to get sendmail working too. weird.

Thanks again for all the amazing info. you've been a tremendous help. If we ever need some help getting things buttoned down do you do freelance work?

Fish
><>
User avatar
fishnyc22
Forum Commoner
Posts: 38
Joined: Wed May 30, 2007 6:20 pm

Post by fishnyc22 »

2. If you're sending out this many emails I can only assume you're making a profit. You can get a VDS server for very little money these days. Just get a few VDS servers. Gmail do provide SMTP to their own members but it's limited and they rewrite your message headers to display your gmail address in the Sender: header.
Duh. I just realized after I posted the last message that you were talking about a dedicated virtual server. We are currently on one actually. We're on Media Temple right now ($80/m). Love the service there after dealing with 1and1. If our load gets really big, we'll can always concider getting another one to spread our the load created by sending emails.
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

I'll post a pseudo example of how I run a daemon but I won't be able to post the actual code for licensing reasons. It's fairly generic anyway :)

$80/mo sounds quite high for a VDS unless you have enormous bandwidth limits. I pay £15/mo (roughly $30). I get 15GB disk space, 128MB RAM, 50GB bandwidth. I'm not sure how that compares with your rates:

http://www.blackcatnetworks.co.uk/services/vds

I really can't fault the customer service of these guys, but they're in the UK which is probably not appropriate for you, assuming you're in the US.

I wonder if qmail was installed at some point but isn't now. phpinfo() usually shows the sendmail path, or if you can just look in php.ini look for sendmail_path.

What does this show?

Code: Select all

ls -l /usr/sbin/sendmail
Sendmail and qmail are actually the same thing ;) qmail just has more features than sendmail. It's job is to do the actual delivering of emails. That involves spooling, retrying, freezing, bouncing and all the other aspects of sending mail. qmail should theoretically be a suitable "replacement" for sendmail. It's usual that there will be a symlink pointing /usr/sbin/sendmail to another binary which behaves the same when the same command line flags and data are passed to it. For example, here's my sendmail:

Code: Select all

w3style.co.uk:~# ls -l /usr/sbin/sendmail 
lrwxrwxrwx 1 root root 5 May  7 10:44 /usr/sbin/sendmail -> exim4
It links to /usr/sbin/exim4, but because exim4 takes the same (plus a hell of a lot more) command line arguments it can used as a replacement for sendmail in this way. Postfix and qmail should do the same.

Thanks for the offer of work but I don't do freelance work these days for lack of time after my day job and the personal projects I have on the go. I tried in the past but doing freelance work on top of a full-time job starts to consume your life :)
User avatar
fishnyc22
Forum Commoner
Posts: 38
Joined: Wed May 30, 2007 6:20 pm

Post by fishnyc22 »

Mornin! (or... afternoon, I guess)

I'm surprised that $80 seems high, I did a bit of research before going with MediaTemple. I previously had 1&1 (awful, awful company) and they were $100 and a few others I found were around the same. Maybe the Plesk License is what makes it expensive. (with one domain on the server I really don't need pleask however). Good to know I'm getting ripped off :)

phpinfo def has sendmail at /usr/sbin/sendmail and calling ls -l /usr/sbin/sendmail shows the following

Code: Select all

[root@domain ~]# ls -l /usr/sbin/sendmail
lrwxrwxrwx  1 root root 21 Dec 29 03:31 /usr/sbin/sendmail -> /etc/alternatives/mta
and with that I entered:

Code: Select all

[root@domain ~]# ls -l /etc/alternatives/mta
lrwxrwxrwx  1 root root 23 Mar 19 13:07 /etc/alternatives/mta -> /var/qmail/bin/sendmail
Seems messy? no? I found more on that here I guess that explains the qmail folder but no binary. Wacky...so am I using sendmail or qmail???

as I figured, using

Code: Select all

$swift =& new Swift(new Swift_Connection_Sendmail("/var/qmail/bin/sendmail -bs"));
still results in a blank page.

Thx again...and again...and...
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

Code: Select all

ls -l /var/qmail/bin/sendmail
/var/qmail/bin/sendmail -bs
?

That does look quite messy yes. Is it an Ubuntu server by any chance. It loks very Ubuntu-ish with the /etc/alternatives mess. They do that with the java jdk/jre packages too.
User avatar
fishnyc22
Forum Commoner
Posts: 38
Joined: Wed May 30, 2007 6:20 pm

Post by fishnyc22 »

Its a CentOS server.

Code: Select all

[root@domain ~]# ls -l /var/qmail/bin/sendmail 
-r-xr-xr-x  1 root qmail 9672 Mar 19 13:07 /var/qmail/bin/sendmail

Code: Select all

[root@domain ~]# /var/qmail/bin/sendmail -bs
220 www.domain.com ESMTP
User avatar
Weirdan
Moderator
Posts: 5978
Joined: Mon Nov 03, 2003 6:13 pm
Location: Odessa, Ukraine

Post by Weirdan »

Is it an Ubuntu server by any chance.
He said it was CentOS (weird distro if you ask me).
User avatar
fishnyc22
Forum Commoner
Posts: 38
Joined: Wed May 30, 2007 6:20 pm

Post by fishnyc22 »

Weirdan wrote:
Is it an Ubuntu server by any chance.
He said it was CentOS (weird distro if you ask me).
I think it has to do with Plesk. It wreaks havoc on the system from what I've read.
User avatar
fishnyc22
Forum Commoner
Posts: 38
Joined: Wed May 30, 2007 6:20 pm

Post by fishnyc22 »

Hey guys... so any idea why this isn't working with sendmail. I've had no luck pointing it directly to "/var/qmail/bin/sendmail "
User avatar
fishnyc22
Forum Commoner
Posts: 38
Joined: Wed May 30, 2007 6:20 pm

Post by fishnyc22 »

I've wondered my way back to the subject I started 13 messages ago...
quick question regarding setting up the cron queue.

My outgoing queue contains the following fields

id
toEmail
toName
fromEmail
fromName
message
status
timestamp

When a group of messages is written to my database queue, say 500 messages, they are written individually, though the message is the same for each message in that group.

When cron calls up those messages, do I want to call them one at a time using send(), and track each's success/failure.... or do I want to maybe add a new field in my table called "group_id" that determines if its a group and send that batch using batchSend() and perhaps antiflood. Does antiflood work with send() or only with batchSend(). The reason being, I may have a batch of 500 go out in one minute and a separate batch of 350 with a different message go out 1 minute later. Make sense??

Thanks again for all the great info. Still no luck with sending with sendmail. just SMTP. weird.
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

Seems to me you should be using this:

user_message:

id
toEmail
toName
fromEmail
fromName
message_id
status
timestamp

message:

id
subject
body

Now when you send emails you can pull out messages on a per-batch basis and use batchSend().

You can get any failures at the end of the batch by using this:

Code: Select all

$batch = new Swift_BatchMailer($swift);

$batch->send($message, $recipients, $sender);

$failures = $batch->getFailedRecipients(); //an array of addresses that *didn't* send
Post Reply