Mail queue for the mail queue
Moderator: General Moderators
- Christopher
- Site Administrator
- Posts: 13596
- Joined: Wed Aug 25, 2004 7:54 pm
- Location: New York, NY, US
Mail queue for the mail queue
I am interested in getting people's experiences and ideas about queuing emails to be sent from a server. The situation I have is an application that provisioned out to many customized sub-websites per server -- say hundreds of sites each with hundreds of users. I need to add the ability for these sub-sites to send various emails based on administrator actions, user actions or daily checks of various status information. My initial thought was that I would implement a single mail queue on the server and all of these sub-sites could add the emails to be sent to the queue. The queue would then continuously process the email in some kind of batches around the clock.
Some of my questions are whether this general scheme or some other scheme is workable? Does something like this already exist? The pros and cons of things like whether using a database table vs a directory of files for the queue? Etc., etc.
Some of my questions are whether this general scheme or some other scheme is workable? Does something like this already exist? The pros and cons of things like whether using a database table vs a directory of files for the queue? Etc., etc.
(#10850)
Re: Mail queue for the mail queue
You check out the Mail Queue package on pear - http://pear.php.net/package/Mail_Queue
- Christopher
- Site Administrator
- Posts: 13596
- Joined: Wed Aug 25, 2004 7:54 pm
- Location: New York, NY, US
Re: Mail queue for the mail queue
Thanks. Have you ever used it? It looks like you have to use the PEAR DB and Mail classes with it (which is too bad), but it may be workable. It looks like you can create wrappers for any DB. Is there something like this for Swift?
(#10850)
Re: Mail queue for the mail queue
I haven't used it. Personally I wouldn't use it as is, just study the things they did and take what I like (the beauty of open-source
)
- Christopher
- Site Administrator
- Posts: 13596
- Joined: Wed Aug 25, 2004 7:54 pm
- Location: New York, NY, US
Re: Mail queue for the mail queue
Yes, I suppose I could just strip out the PEAR/PEAR_Error stuff and make it generic. The code is actually pretty simple and small, which I though a solution would be. It would be nice if it supported file based queue, but that might be easy too. Then I could use it with Zend, Swift, PHPMailer, Skeleton, etc.
What interface to you like for generic Queues? I recall something like put()/get()/clear/flush/delete()/isEmpty() is the standard interface.
What interface to you like for generic Queues? I recall something like put()/get()/clear/flush/delete()/isEmpty() is the standard interface.
(#10850)
- Chris Corbyn
- Breakbeat Nuttzer
- Posts: 13098
- Joined: Wed Mar 24, 2004 7:57 am
- Location: Melbourne, Australia
Re: Mail queue for the mail queue
I'd like to have some queuing features in Swift
Feel free to add it to the code if you arrive at a good solution!
I had thought of implementing something along the lines of a SpoolTransport. Let's see...
A message spool needs a few things:
Various possible implementations (disk-based, database, REST/SOAP...)
The ability to write messages into it
The ability to indicate the last time at which the messages was attempted to send
The ability to retry sending at staggered intervals up to X hours
The ability to null dead messages (that haven't been delivered after X hours)
The way I've always done it is just to subclass Swift (in v3) to override the send() method and write the messages to DB. Then have a cron script which processes messages where sent = 0. Really this is a limitation of the v3 Connection API though.
I had thought of implementing something along the lines of a SpoolTransport. Let's see...
Code: Select all
$transport = new Swift_SpoolTransport(new Swift_Spool_DatabaseSpool( ... ));
$mailer = new Swift_Mailer($transport);
$mailer->send( ... ); //Transport only writes to the spool
// ... in a cron script
$spool = new Swift_Spool_DatabaseSpool( ... );
$spool->process($smtpTransport);
Various possible implementations (disk-based, database, REST/SOAP...)
The ability to write messages into it
The ability to indicate the last time at which the messages was attempted to send
The ability to retry sending at staggered intervals up to X hours
The ability to null dead messages (that haven't been delivered after X hours)
The way I've always done it is just to subclass Swift (in v3) to override the send() method and write the messages to DB. Then have a cron script which processes messages where sent = 0. Really this is a limitation of the v3 Connection API though.
- Christopher
- Site Administrator
- Posts: 13596
- Joined: Wed Aug 25, 2004 7:54 pm
- Location: New York, NY, US
Re: Mail queue for the mail queue
I did volunteer. This may be my start!Chris Corbyn wrote:I'd like to have some queuing features in SwiftFeel free to add it to the code if you arrive at a good solution!
I had thought of implementing something along the lines of a SpoolTransport. Let's see...
For the DB we could define the schema with status fields for resends and timestamp. For file based maybe store that in a .Swift file or in each text file with the rest of the delimited data. Hmmmm...Chris Corbyn wrote:A message spool needs a few things:
Various possible implementations (disk-based, database, REST/SOAP...)
The ability to write messages into it
The ability to indicate the last time at which the messages was attempted to send
The ability to retry sending at staggered intervals up to X hours
The ability to null dead messages (that haven't been delivered after X hours)
Do you think this is a better way, or do you want your Spool design above?Chris Corbyn wrote:The way I've always done it is just to subclass Swift (in v3) to override the send() method and write the messages to DB. Then have a cron script which processes messages where sent = 0. Really this is a limitation of the v3 Connection API though.
(#10850)
- Chris Corbyn
- Breakbeat Nuttzer
- Posts: 13098
- Joined: Wed Mar 24, 2004 7:57 am
- Location: Melbourne, Australia
Re: Mail queue for the mail queue
The spool (or call it a queue if it makes more sense) idea is good because it will work just like any of the other Transports. So if you decide that you don't want to spool messages and you'd rather just send them straight out over SMTP you can do so by just swapping the Transport. It just keeps the code consistent 
There may be some pre-requisites for this. What would be the best way to save a message object? Serialize it? Saving serialized data in a database scares me
Perhaps we need a way to represent a message object as XML and create one from XML (or JSON, or other format).
Your schema sounds pretty much like I was thinking.
[sql]CREATE TABLE mailspool ( spoolid INT NOT NULL, timecreated INT NOT NULL, STATUS ENUM ( 'queued', 'deferred', 'frozen' ), timelastprocessed INT NOT NULL, messagedata TEXT NOT NULL, PRIMARY KEY ( spoolid ));[/sql]
If this were on disk (I'm probably thinking too much about exim here) then sub-directories for each status and meta-data in the header of each file?
There may be some pre-requisites for this. What would be the best way to save a message object? Serialize it? Saving serialized data in a database scares me
Your schema sounds pretty much like I was thinking.
[sql]CREATE TABLE mailspool ( spoolid INT NOT NULL, timecreated INT NOT NULL, STATUS ENUM ( 'queued', 'deferred', 'frozen' ), timelastprocessed INT NOT NULL, messagedata TEXT NOT NULL, PRIMARY KEY ( spoolid ));[/sql]
If this were on disk (I'm probably thinking too much about exim here) then sub-directories for each status and meta-data in the header of each file?
- Chris Corbyn
- Breakbeat Nuttzer
- Posts: 13098
- Joined: Wed Mar 24, 2004 7:57 am
- Location: Melbourne, Australia
Re: Mail queue for the mail queue
Actually, I completely over-engineered that
Serialization is already present (one directional) in the form:
That produces the contents of a .eml file (the email source). All you'd need then is the sender/recipient info pre-fetched from the message object.
Code: Select all
$data = $message->toString(); //or toByteStream($stream)