Batch Mail Processing

Not for 'how-to' coding questions but PHP theory instead, this forum is here for those of us who wish to learn about design aspects of programming with PHP.

Moderator: General Moderators

Post Reply
User avatar
Zoram
Forum Contributor
Posts: 166
Joined: Sun Aug 18, 2002 3:28 pm
Location: Utah
Contact:

Batch Mail Processing

Post by Zoram »

I am trying to find the optimal solution for processing batchs emails. I have been trying to work with a cron script that would run every minute and periodically check the email with sleep periods, but I was having trouble with the amount of mail that the script could process per minute. It was incredibly low for the amount of mail I want to be able to process. I would like to know your comments on the situation.

Optimally we want to be able to process large batches of emails to pull information for use in a database within a matter of a minute, at most two or three for larger batches (2000ish). Am I thinking on the right lines in having a script run 'continually' or is there a better solution for automatically processing emails as they arrive? Has anyone else had this problem or contemplated the idea?
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

You're reading mail yes? Not sending?

What will this be for? Perhaps there's already an existing solution to your problem ;)

EDIT | For speed, it would help if you didn't have to go over IMAP or POP and could just read the mail files directly on the server. Obviously, the mail will need to be on the same server as the script for that to work.
jbflam
Forum Newbie
Posts: 7
Joined: Wed Jan 10, 2007 3:48 am
Location: Singapore

Post by jbflam »

Why Don't you have a cron running every minute checking if mails have arrived.
If so then start fetching them from mailbox? If you have trouble on how to do that use the the class emailtodb in phpclasses
User avatar
Zoram
Forum Contributor
Posts: 166
Joined: Sun Aug 18, 2002 3:28 pm
Location: Utah
Contact:

Post by Zoram »

I am try to just read in the messages as fast as possible. Right now I can read through about 78 messages in 58 seconds. That isn't really that fast compared to the speeds I want the script to achieve. I'm doing testing but I am pretty sure that most of that time that I am wasting is just getting the messages through imap.

I'll have to check on just opening the files directly and see if that would be possible. How would I go about pulling the header and body information from a file once i have access to it?
jbflam
Forum Newbie
Posts: 7
Joined: Wed Jan 10, 2007 3:48 am
Location: Singapore

Post by jbflam »

You want to access the file directly ? You mean on the mail server?
It depends on your mail server type.
I don't know how your mail server works but on linux machines they have those files containing inbox for users (/var/mail/user) , If this is the case you ll need to open the file and parse it... I have no parser for that at hand at the moment... It looks like re-inventing the wheel in a way...

As the other guy post said host your script and mail server on same machine... this will speed up the process.
User avatar
Zoram
Forum Contributor
Posts: 166
Joined: Sun Aug 18, 2002 3:28 pm
Location: Utah
Contact:

Post by Zoram »

At the time I am developing on a grid server and am trying to find out if I can gain direct access to the mail files, which might speed things up a lot. I'll post the class that I am using to process the email when I get a little more time and can test it for specific bottlenecking.
Post Reply