Page 1 of 1
Sending an array to a background process..?
Posted: Sat Mar 05, 2005 11:47 am
by ron_j_m
I am working on a program that sends an array from an uploaded text file to a script that processes that array, and stores the results in a database. The problem is the text file is sometimes verry large and can take an hour or more to process. What I would like to do is send the array to process.php and have process.php run in the background, and ultimatly dispatch an email when the process is complete. Ive looked all over and either cant find anything that I can get working, or just plain dont understand how to get it to work. If anyone has any sugestions please let me know..
Here is the upload.php code that I use to send to process.php
In this example I am using a form to send the contents of the text file to process.php. In process.php I use preg_split to seperate the data into variables for processing.
Code: Select all
<?php
//===UPLOAD INFORMATION
$uploaddir = '/home/cybersu/public_html/2/upload/';
$filename=$_FILESї'files']ї'name'];
$domainfile = ("/home/cybersu/public_html/2/upload/$filename");
if (move_uploaded_file($_FILESї'files']ї'tmp_name'], $uploaddir."$filename"))
{
$fp=fopen("$textfile", "r");
$content=file_get_contents("$textfile");
$lines = file ("$textfile");
$num_lines = count ($lines);
echo ("<center><p><form action="process.php" method="POST"> <p><textarea rows="20" name="data" cols="40" class="input">$content</textarea></p>");
echo ("<input type="submit" value="Process">");
echo ("<input type="hidden" value="0" name="single"></p>");
echo ("</form>");
echo ("The file ");
echo ("$filename ");
echo ("was successfully uploaded and is ");
echo strlen($content);
echo (" bytes.<br>");
echo ("There are a total number of <b>" . $num_lines);
echo ("</b> names in your list.<br></center>");
}
else {
echo "Possible file upload attack!\n";}
?>
Thanks for looking..
Ron
Posted: Sat Mar 05, 2005 11:50 am
by Supremacy
trying find out if you can run cronjobs, it might be your solution.
a cronjob is designed to run at a certain time.
so you might not get the job done right away, but it will get done, on the server when the cronjob kicks in.

Posted: Sat Mar 05, 2005 11:57 am
by feyd
I'd couple the cron idea with adding the file to a queue of what files to process when the cron runs. This queue would be stored in the database somewhere, and would contain all information needed to process the file, who to email when it completes, status flags for the cron script..
Your upload script here, as it stands, loads the file into memory twice. Once with file_get_contents, and once with file().. there are ways of counting lines once a file is loaded with file_get_contents().. preg_match_all() for instance.
If you post process.php, we may be able to help speed up the script significantly. You may also consider using exec() or it's siblings, as they can be run in the background if called right.
Posted: Sat Mar 05, 2005 12:00 pm
by Supremacy
the only problem with the cronjob is it's predefined run time.
if it could be called and started when needed, it might be awesome.
and cronjobs is typecally used to do boring standard stuff, like cleaning up a forum, or taking backup of a database and stuff like that.
Posted: Sat Mar 05, 2005 12:22 pm
by ron_j_m
Thanks for the quick reply's
I have thought about using cron but the problem is the data to be processed is not static. It would change on a daily basis. Something more like exec() would be more what Im looking for. Could popoen be another option? Ive looked at both of these and just dont seem to get how to get them to work.
Here is a simplified process.php
Code: Select all
<?
$total=0;
mysql_connect ("localhost", "username", "password") or die ('I cannot connect to the database because: ' . mysql_error());
@mysql_select_db ("database") or die( "Unable to select database");
$data = preg_split ("/\s+/", $data);
foreach ($data as $domain) {
// check Yahoo!
$path ="http://search.yahoo.com/search?p=linkdomain%3A".$domain."&ei=UTF-8&fr=fp-tab-web-t&cop=mss&tab=";
{
$data = strtolower(file_get_contents("$path"));
$data = substr($data, strpos($data, "of about")+9, strlen($data));
$data = strip_tags(substr($data, 0, strpos($data, " ")));
if(preg_match('#їa-z]#i', $data)) {
$resultsї'yahoo'] = array('0', $path);
} else {
$data = preg_replace('/ї^0-9]/','',$data);
$resultsї'yahoo'] = array($data, $path);
}
}
//ENGINE TOTALS
$yahoototal = $resultsї'yahoo']ї0];
//INSERT INTO DATABASE
$query = "INSERT INTO linkpop VALUES ('','$domain','$yahoototal')";
mysql_query($query);
mysql_close();
?>
Thanks again.
Ron
Posted: Mon Mar 07, 2005 6:15 pm
by ron_j_m
I still havent got this figured out and Im hoping someone out there can point me in the right direction. I am using exec() to start the background script, but It doesnt seem to either send the array along with it or it just plain doesnt work..
Heres what Im working with
Code: Select all
upload.php
<?
$uploaddir = '/home/cybersu/public_html/2/upload/';
$filename=$_FILESї'files']ї'name'];
$domainfile = ("/home/cybersu/public_html/2/upload/$filename");
if (move_uploaded_file($_FILESї'files']ї'tmp_name'], $uploaddir."$filename"))
{
$content=file_get_contents("$domainfile");
$data = preg_split ("/\s+/", $content);
exec('/usr/bin/php /home/cybersu/public_html/2/process.php $data > /dev/null 2>&1 &');
}
else {
echo "Possible file upload attack!\n";}
?>
Code: Select all
process.php
<?
mysql_connect ("localhost", "username", "password") or die ('I cannot connect to the database because: ' . mysql_error());
@mysql_select_db ("database") or die( "Unable to select database");
foreach ($data as $value)
{
$query = "INSERT INTO linkpop VALUES ('','$value','')";
mysql_query($query);
}
mysql_close();
?>
Basicaly with this code Im just trying to upload a text file with names on it.
Then turn the file into an array, then send the array to process.php for processing in the background.
No luck as of yet...
Can any one help?
Thanks,, Ron
Posted: Mon Mar 07, 2005 6:53 pm
by feyd
you have a php variable inside a single quote string. Single quote strings do not parse variables. Adding $data to a string will only insert 'Array' into the string. You need to compact the data back together into a usable form for the string use.
Posted: Mon Mar 07, 2005 7:52 pm
by ron_j_m
I dont understand what you mean. Should I not use preg_split to turn contents into an array before using exec()?
If I add preg_split() to process.php instead, I still get the same thing, one blank entry into the database.
Could you explain a little more about what you mean? Sorry Im prety new to php and get lost quite easily.
Would it be possible to save the array using sessions?
Thanks again..