Page 1 of 1
Batch Mail Processing
Posted: Wed Jan 10, 2007 1:26 am
by Zoram
I am trying to find the optimal solution for processing batchs emails. I have been trying to work with a cron script that would run every minute and periodically check the email with sleep periods, but I was having trouble with the amount of mail that the script could process per minute. It was incredibly low for the amount of mail I want to be able to process. I would like to know your comments on the situation.
Optimally we want to be able to process large batches of emails to pull information for use in a database within a matter of a minute, at most two or three for larger batches (2000ish). Am I thinking on the right lines in having a script run 'continually' or is there a better solution for automatically processing emails as they arrive? Has anyone else had this problem or contemplated the idea?
Posted: Wed Jan 10, 2007 2:17 am
by Chris Corbyn
You're reading mail yes? Not sending?
What will this be for? Perhaps there's already an existing solution to your problem
EDIT | For speed, it would help if you didn't have to go over IMAP or POP and could just read the mail files directly on the server. Obviously, the mail will need to be on the same server as the script for that to work.
Posted: Wed Jan 10, 2007 3:55 am
by jbflam
Why Don't you have a cron running every minute checking if mails have arrived.
If so then start fetching them from mailbox? If you have trouble on how to do that use the the class emailtodb in phpclasses
Posted: Wed Jan 10, 2007 3:16 pm
by Zoram
I am try to just read in the messages as fast as possible. Right now I can read through about 78 messages in 58 seconds. That isn't really that fast compared to the speeds I want the script to achieve. I'm doing testing but I am pretty sure that most of that time that I am wasting is just getting the messages through imap.
I'll have to check on just opening the files directly and see if that would be possible. How would I go about pulling the header and body information from a file once i have access to it?
Posted: Wed Jan 10, 2007 8:50 pm
by jbflam
You want to access the file directly ? You mean on the mail server?
It depends on your mail server type.
I don't know how your mail server works but on linux machines they have those files containing inbox for users (/var/mail/user) , If this is the case you ll need to open the file and parse it... I have no parser for that at hand at the moment... It looks like re-inventing the wheel in a way...
As the other guy post said host your script and mail server on same machine... this will speed up the process.
Posted: Wed Jan 10, 2007 11:55 pm
by Zoram
At the time I am developing on a grid server and am trying to find out if I can gain direct access to the mail files, which might speed things up a lot. I'll post the class that I am using to process the email when I get a little more time and can test it for specific bottlenecking.