Page 1 of 1
PHP Job Queue?
Posted: Fri Sep 03, 2010 2:36 pm
by Garett_Haight
Hello all,
I'm working on a script that allows users to upload photos to this project's website that automatically
resizes, crops and generates galleries for the photos. I'm using the GD library to manipulate the images
and since they're usually large jpeg files straight from the user's camera, the file processing takes several
minutes.
What I would like to do, is to have a setup which allows the users to upload the photos, then photos would
be added to a queue and processed on the server, allowing the user to navigate away from the page. I would
also like to avoid having more than one image processed at a time since the large images take up a significant
amount of server memory.
So.. I suppose my question would be: Is there a way to have a php file execute on the server whenever there are
items listed in the queue table?
Thanks!
Re: PHP Job Queue?
Posted: Fri Sep 03, 2010 3:38 pm
by cpetercarter
A partial answer to your problem would be to use a pop-up window for image upload and processing. The user could then navigate to other parts of the site from the main widow while the image processing was going on in the popup.
Re: PHP Job Queue?
Posted: Fri Sep 03, 2010 4:52 pm
by Jonah Bron
I don't know for sure, but I think you could run a PHP script in the background by running
Code: Select all
shell_exec('php "somephpscript.php" &' . PHP_EOL . PHP_EOL);
Assuming you're server runs Linux, this will start PHP in the background running somephpscript.php. I'm not sure how the relative url would be set up, so it would be best to use an absolute url.
Re: PHP Job Queue?
Posted: Fri Sep 03, 2010 7:42 pm
by Weirdan
The easiest way would be accept the file, store a record in the database and then have a script running from cron pick it up and convert the file. You can also look into Gearman - it's kind of distributed message bus with support for both async and sync communications between client and worker(s).
However, the more interesting and scalable solution would be to make user's browser to do all the grunt work for you - it's certainly possible to perform most of the graphic operations with flash and upload ready to use images. Using javascript + canvas could also be an option - however I'm not sure if you can load local image into canvas (if not it should be possible to make flash read the file and yield data to javascript).
Re: PHP Job Queue?
Posted: Fri Sep 03, 2010 8:25 pm
by McInfo
I second Weirdan's motion to enlist the user's browser to handle the image scaling. Maybe Java applets would be worth investigating.
Re: PHP Job Queue?
Posted: Sat Sep 04, 2010 7:54 am
by yacahuma
I am using gearman + supervisord. It worked great so far.
Re: PHP Job Queue?
Posted: Sat Sep 04, 2010 1:25 pm
by eFishy
Personally I would save the uploaded file on the server and keep a record in mySQL (or what ever database that your using).
Then run a cron every minute that looks for images in the que and process them as required.
Alternatively you could use folders and list their contents for the queuing system if you don't want to use a database.
Re: PHP Job Queue?
Posted: Sat Sep 04, 2010 2:10 pm
by yacahuma
Thecool thing about gearman is that you dont have to wait that minute for the cron. It is instant
Re: PHP Job Queue?
Posted: Tue Sep 07, 2010 11:49 am
by pickle
Also, look into imagemagick instead of GD. I've heard GD is quite a bit slower & uses more memory. Imagemagick you can invoke right from the command line, so you could use just a shell script rather than a full PHP script - should save you some clock cycles & memory usage.
Re: PHP Job Queue?
Posted: Fri Sep 10, 2010 7:22 pm
by Garett_Haight
Thanks for the suggestions everyone! I have some research to do now
I'd love to use imagemagick, but I'm working with this Synology server that
doesn't come with it and it seems that I'd have to do a lot to get it installed.
I've had to do a lot of working around the limitations of this server...
Re: PHP Job Queue?
Posted: Fri Sep 10, 2010 7:30 pm
by Garett_Haight
Actually... it looks like there's something about imagemagick on here....
it's not listed in the php info, but I found an application on the server
that has imagemagick functions....