I am trying to find the optimal solution for processing batchs emails. I have been trying to work with a cron script that would run every minute and periodically check the email with sleep periods, but I was having trouble with the amount of mail that the script could process per minute. It was incredibly low for the amount of mail I want to be able to process. I would like to know your comments on the situation.
Optimally we want to be able to process large batches of emails to pull information for use in a database within a matter of a minute, at most two or three for larger batches (2000ish). Am I thinking on the right lines in having a script run 'continually' or is there a better solution for automatically processing emails as they arrive? Has anyone else had this problem or contemplated the idea?
Batch Mail Processing
Moderator: General Moderators
- Chris Corbyn
- Breakbeat Nuttzer
- Posts: 13098
- Joined: Wed Mar 24, 2004 7:57 am
- Location: Melbourne, Australia
You're reading mail yes? Not sending?
What will this be for? Perhaps there's already an existing solution to your problem
EDIT | For speed, it would help if you didn't have to go over IMAP or POP and could just read the mail files directly on the server. Obviously, the mail will need to be on the same server as the script for that to work.
What will this be for? Perhaps there's already an existing solution to your problem
EDIT | For speed, it would help if you didn't have to go over IMAP or POP and could just read the mail files directly on the server. Obviously, the mail will need to be on the same server as the script for that to work.
I am try to just read in the messages as fast as possible. Right now I can read through about 78 messages in 58 seconds. That isn't really that fast compared to the speeds I want the script to achieve. I'm doing testing but I am pretty sure that most of that time that I am wasting is just getting the messages through imap.
I'll have to check on just opening the files directly and see if that would be possible. How would I go about pulling the header and body information from a file once i have access to it?
I'll have to check on just opening the files directly and see if that would be possible. How would I go about pulling the header and body information from a file once i have access to it?
You want to access the file directly ? You mean on the mail server?
It depends on your mail server type.
I don't know how your mail server works but on linux machines they have those files containing inbox for users (/var/mail/user) , If this is the case you ll need to open the file and parse it... I have no parser for that at hand at the moment... It looks like re-inventing the wheel in a way...
As the other guy post said host your script and mail server on same machine... this will speed up the process.
It depends on your mail server type.
I don't know how your mail server works but on linux machines they have those files containing inbox for users (/var/mail/user) , If this is the case you ll need to open the file and parse it... I have no parser for that at hand at the moment... It looks like re-inventing the wheel in a way...
As the other guy post said host your script and mail server on same machine... this will speed up the process.