Page 1 of 1
Mail queue for the mail queue
Posted: Fri Jan 09, 2009 4:40 pm
by Christopher
I am interested in getting people's experiences and ideas about queuing emails to be sent from a server. The situation I have is an application that provisioned out to many customized sub-websites per server -- say hundreds of sites each with hundreds of users. I need to add the ability for these sub-sites to send various emails based on administrator actions, user actions or daily checks of various status information. My initial thought was that I would implement a single mail queue on the server and all of these sub-sites could add the emails to be sent to the queue. The queue would then continuously process the email in some kind of batches around the clock.
Some of my questions are whether this general scheme or some other scheme is workable? Does something like this already exist? The pros and cons of things like whether using a database table vs a directory of files for the queue? Etc., etc.
Re: Mail queue for the mail queue
Posted: Fri Jan 09, 2009 4:54 pm
by Eran
You check out the Mail Queue package on pear -
http://pear.php.net/package/Mail_Queue
Re: Mail queue for the mail queue
Posted: Fri Jan 09, 2009 6:08 pm
by Christopher
Thanks. Have you ever used it? It looks like you have to use the PEAR DB and Mail classes with it (which is too bad), but it may be workable. It looks like you can create wrappers for any DB. Is there something like this for Swift?
Re: Mail queue for the mail queue
Posted: Fri Jan 09, 2009 6:14 pm
by Eran
I haven't used it. Personally I wouldn't use it as is, just study the things they did and take what I like (the beauty of open-source

)
Re: Mail queue for the mail queue
Posted: Fri Jan 09, 2009 6:17 pm
by Christopher
Yes, I suppose I could just strip out the PEAR/PEAR_Error stuff and make it generic. The code is actually pretty simple and small, which I though a solution would be. It would be nice if it supported file based queue, but that might be easy too. Then I could use it with Zend, Swift, PHPMailer, Skeleton, etc.
What interface to you like for generic Queues? I recall something like put()/get()/clear/flush/delete()/isEmpty() is the standard interface.
Re: Mail queue for the mail queue
Posted: Fri Jan 09, 2009 6:25 pm
by Chris Corbyn
I'd like to have some queuing features in Swift

Feel free to add it to the code if you arrive at a good solution!
I had thought of implementing something along the lines of a SpoolTransport. Let's see...
Code: Select all
$transport = new Swift_SpoolTransport(new Swift_Spool_DatabaseSpool( ... ));
$mailer = new Swift_Mailer($transport);
$mailer->send( ... ); //Transport only writes to the spool
// ... in a cron script
$spool = new Swift_Spool_DatabaseSpool( ... );
$spool->process($smtpTransport);
A message spool needs a few things:
Various possible implementations (disk-based, database, REST/SOAP...)
The ability to write messages into it
The ability to indicate the last time at which the messages was attempted to send
The ability to retry sending at staggered intervals up to X hours
The ability to null dead messages (that haven't been delivered after X hours)
The way I've always done it is just to subclass Swift (in v3) to override the send() method and write the messages to DB. Then have a cron script which processes messages where sent = 0. Really this is a limitation of the v3 Connection API though.
Re: Mail queue for the mail queue
Posted: Fri Jan 09, 2009 6:32 pm
by Christopher
Chris Corbyn wrote:I'd like to have some queuing features in Swift

Feel free to add it to the code if you arrive at a good solution!
I had thought of implementing something along the lines of a SpoolTransport. Let's see...
I did volunteer. This may be my start!
Chris Corbyn wrote:A message spool needs a few things:
Various possible implementations (disk-based, database, REST/SOAP...)
The ability to write messages into it
The ability to indicate the last time at which the messages was attempted to send
The ability to retry sending at staggered intervals up to X hours
The ability to null dead messages (that haven't been delivered after X hours)
For the DB we could define the schema with status fields for resends and timestamp. For file based maybe store that in a .Swift file or in each text file with the rest of the delimited data. Hmmmm...
Chris Corbyn wrote:The way I've always done it is just to subclass Swift (in v3) to override the send() method and write the messages to DB. Then have a cron script which processes messages where sent = 0. Really this is a limitation of the v3 Connection API though.
Do you think this is a better way, or do you want your Spool design above?
Re: Mail queue for the mail queue
Posted: Fri Jan 09, 2009 6:50 pm
by Chris Corbyn
The spool (or call it a queue if it makes more sense) idea is good because it will work just like any of the other Transports. So if you decide that you don't want to spool messages and you'd rather just send them straight out over SMTP you can do so by just swapping the Transport. It just keeps the code consistent
There may be some pre-requisites for this. What would be the best way to save a message object? Serialize it? Saving serialized data in a database scares me

Perhaps we need a way to represent a message object as XML and create one from XML (or JSON, or other format).
Your schema sounds pretty much like I was thinking.
[sql]CREATE TABLE mailspool ( spoolid INT NOT NULL, timecreated INT NOT NULL, STATUS ENUM ( 'queued', 'deferred', 'frozen' ), timelastprocessed INT NOT NULL, messagedata TEXT NOT NULL, PRIMARY KEY ( spoolid ));[/sql]
If this were on disk (I'm probably thinking too much about exim here) then sub-directories for each status and meta-data in the header of each file?
Re: Mail queue for the mail queue
Posted: Fri Jan 09, 2009 6:53 pm
by Chris Corbyn
Actually, I completely over-engineered that

Serialization is already present (one directional) in the form:
Code: Select all
$data = $message->toString(); //or toByteStream($stream)
That produces the contents of a .eml file (the email source). All you'd need then is the sender/recipient info pre-fetched from the message object.