Managing *nix (Email) Server Logs
Posted: Tue Jan 03, 2012 10:00 pm
Hey Everyone,
I have been trying to find a scale-able solution for managing *nix server logs and not just your typical '.log' files either, I'm talking about automatically generated reports from scripts which send emails upon completion. Ideally the solution would easily run on a *nix platform.
Most of these scripts run from cron-jobs or triggered by specific events occurring on the systems and is it stands there are approximately 100+ emails that are recorded daily in this form of logging.
I have been trying to find a solution to this problem where if more logs are to be generated, how do you manage them? Because no-body is going to read through over 100 logs every single day.
There have been a couple of applications which i have noted which seem to focus on the specific .log files of a server in order to keep track of what's going on in the server. My thoughts to this problem could actually be to custom build a small application which directly hooks into a singular email account which we redirect all of these logs to. So rather than recieving these 100+ logs over several users, we combine all log emails into a single account and then create a usable front-end which essentially just acts as an IMAP client to read the logs and track the subjects of each etc.
Then once a keyword has been picked up in one of these log emails, it could then send an alert message to an RSS feed/email to specific users which would deal with those logs.
The cons of using this approach obviously is the number of emails being recorded, the size of the mailbox and the ever-increasing slowdown of the system as more emails are logged and needing to be filtered/searched. (Unless the application converts the emails into database records and then deletes the email from the IMAP account)
These are my inital thoughts, but im very interested in what other options might be.
regards,
Weiry.
I have been trying to find a scale-able solution for managing *nix server logs and not just your typical '.log' files either, I'm talking about automatically generated reports from scripts which send emails upon completion. Ideally the solution would easily run on a *nix platform.
Most of these scripts run from cron-jobs or triggered by specific events occurring on the systems and is it stands there are approximately 100+ emails that are recorded daily in this form of logging.
I have been trying to find a solution to this problem where if more logs are to be generated, how do you manage them? Because no-body is going to read through over 100 logs every single day.
There have been a couple of applications which i have noted which seem to focus on the specific .log files of a server in order to keep track of what's going on in the server. My thoughts to this problem could actually be to custom build a small application which directly hooks into a singular email account which we redirect all of these logs to. So rather than recieving these 100+ logs over several users, we combine all log emails into a single account and then create a usable front-end which essentially just acts as an IMAP client to read the logs and track the subjects of each etc.
Then once a keyword has been picked up in one of these log emails, it could then send an alert message to an RSS feed/email to specific users which would deal with those logs.
The cons of using this approach obviously is the number of emails being recorded, the size of the mailbox and the ever-increasing slowdown of the system as more emails are logged and needing to be filtered/searched. (Unless the application converts the emails into database records and then deletes the email from the IMAP account)
These are my inital thoughts, but im very interested in what other options might be.
regards,
Weiry.