Page 1 of 1

Memory issue.

Posted: Fri Feb 02, 2007 9:40 am
by brunodp
Hello.

I have to develop a platform that will be used to send a company's newsletter. Each user that request the newsletter can modify the info that they want to get. I'm having memory issues when sending the newsletter.
Since each newsletter is personalized I'm not using batch sending... the code is pretty simple, I have the emails of the people I have to send the newsletter to and I cycle through them reformatting the html that has to be sent and sending the newsletter. Everything works great except that the script is using a huge amount of memory, I've assigned 250MB of memory to the script and it dies after sending 9500 emails or so, due to lack of memory.
I know that there are workarounds; I could make the script send X emails, reload after some time and send the next X emails and so on... but this script is running in background in console mode, and the amount of memory being used just amaze me... any comments on this?

EDIT: I've realized I should have given more info on how I'm making this... I'm making an smtp connection to localhost's exim, turning the logsize to 1 and I'm not even sending a huge amount of data for now (I'm making tests...the newsletter will be pretty big)... just some html that loads the images from the server and the unsubscribe link.

Posted: Fri Feb 02, 2007 11:01 am
by Chris Corbyn
I need to see your code. Swift doesn't sotre anything other than the log, and if you truncate that to 1 element then something oustside of Swift is constantly storing data. I'm going to take a stab and guess you place recipients/messages into an array as you go... either that, or there's some recursion going on where you could use iteration. But rather than keeping me in suspense, post your code ;)

Posted: Fri Feb 02, 2007 12:26 pm
by brunodp
Edit: I used instead of and mistakenly submitted twice ... check below for the corrected post...

Posted: Fri Feb 02, 2007 12:27 pm
by brunodp
Okay, here goes the code... originally I was querying the database for the necessary data but attemping to sort out this memory problem I've made a dump of the data to a file, and proceed to read from it.

Code: Select all

<?php
	require_once(dirname(__FILE__) . '/inc/config.php');
	require_once(dirname(__FILE__) . '/SwiftMailer/Swift.php');
	require_once(dirname(__FILE__) . '/SwiftMailer/Swift/Connection/SMTP.php');

  function getQueueCount() {
	  $cmd = 'exim -bpr | grep "<" | wc -l';
	  $s = exec($cmd);
	  return (integer)$s;
  }
  
  set_time_limit(0);
  
  @ignore_user_abort(true);
		
	$conexion = new Swift_Connection_SMTP(SMTP_HOST);
	$mailer = new Swift($conexion);
	
	$staticHtml = 'Personalized HTML'; /* Static for now */
			
	$file = dirname(__FILE__) . "/news.dat";
	$fp = fopen($file,"r");
	
	$counter = 0;
	$counterTotal = 0;

  $mailer->setMaxLogSize(1);
			
	if($mailer->isConnected()) {
			while(!feof($fp)) {
				$data = fgets($fp);
				$data = explode("|",$data);
				$data[1] = rtrim($data[1]); /* Here I obtain the email address and the id, which is used for unsubscibe purpose */
				
			  /* HTML para la baja */
			  $unsubscribeHTML = 'HTML for newsletter unsubscription';
  		  			              
				$mailer->addPart($staticHtml .  $unsubscribeHTML, 'text/html');
		   
				$mailer->send($data[0],'bruno@rcnet.com.ar','Title');
			
			  echo $counterTotal++ . " ";
			  
			  if($counter == 100) {
			    /* Log de cantidad de emails enviados */
			    $fd = fopen("quantity.log","w");
  		    fwrite($fd, $counterTotal);
  		    fclose($fd);
  		    /* If we've hogged exim's queue, give it a rest */
			    $qty = getQueueCount();
			    if($qty > 499) {
			      $mailer->close();
			      sleep(1200);
			      $mailer->connect();
			    }
			    $counter = 0;   
			  }
			  else {
			    $counter++;
			    sleep(2);
			 	}
			 	
			 	if($mailer->hasFailed()) {
			 	  echo "**ERROR**. There was a problem. Errors: ".print_r($mailer->errors, 1).". Log: ".print_r($mailer->transactions, 1);
	        die();
			 	}
			}
			
			fclose($fp);
	    $mailer->close();
	}
	else {
	  echo "Swift couldn“t connect. Errors: ".print_r($mailer->errors, 1).". Log: ".print_r($mailer->transactions, 1);
	  die();
  }
?>
I've stripped every comment (they where all in spanish) and changed the variables to english. Hope I didn't missed anything.
The config.php that is included are just some defines and the connection to the database.... which I should get rid of since I'm no longer using it. SMTP_HOST is defined in config.php as 'localhost'. Also, config.php has the 250MB memory allocation for the script
There's nothing too weird in this script... it's pretty straightforward. I keep a log in a file every 100 emails sent and check if exim's queue is under 500 (and sleep during 20 minutes if it is not).

Posted: Fri Feb 02, 2007 4:33 pm
by Chris Corbyn
Hmm... looks fine to me :? Unless I'm missing something. Earlier tests showed that Swift doesn't use more memory the more messages get sent. Also, I'm aware of other projects which send tens of thousands using Swift without any issues (*cough* ActiveCollab *cough*).

Do you have XDebug installed by any chance? It could be worth running a profile of your code.

I'll find the code for these benchmarks tomorrow (I'm off to bed now!) so I can re-run them although I have a feeling it's been lost -- either way it was only a few lines and easy to recreate :)

Posted: Sat Feb 03, 2007 2:58 pm
by Chris Corbyn
By the way, have you looked at the AntiFlood plugin? It's not quite the same as what you're doing by reading the Exim queue, but it is made for the same reasons.

I have no problems sending 10,000 emails with a 8MB memory limit, using Swift 2.1.17. Nothing looks logically wrong in this excerpt of code neither.

Posted: Mon Feb 05, 2007 7:26 am
by brunodp
It's strange, when I changed database reading to file reading the test I made died at about 10000 emails again. But since I've asked for help here it seems that it has started to run properly without any change :S (don't know why this has happened, but I'm gladly accepting it trying not to think too much about it)
I've made a test of 15000 emails and there was no problem, and now I'm making a bigger test and so far so good. I'm still setting 250MB of memory for the script, but if everything keeps OK, I'll test it with less memory assigned.
Thanks for the tip on antiflood plugin, I'll check it out.
Well, thanks for your time and Swiftmailer, any news on this matter I'll post it here so that it can helps anyone with a similar problem-

Bye!