Build Amazon S3-style interface to a folder
Posted: Tue May 12, 2009 6:34 am
Hi everyone
I'm building a CMS onto our work intranet, to enable us to post news onto our website easier.
I'm also adding a file upload/deletion tool so that users of the CMS (2-4 users, pretty knowledgable at HTML and web stuff) don't have to get involved with uploading files manually through FTP.
At the moment I've got it running with Amazon's S3 service, using this PHP class... http://undesigned.org.za/2007/10/22/amazon-s3-php-class
It works great, however Amazon's charging structure is killing us a bit. The fact you get charged per 10,000 GET requests means that hosting loads of small files on there isn't that economical. Especially when we sent out an e-mail newsletter to 10,000 subscribers those requests soon add up. It would be better to host the images with our hosting provider.
However the thing that makes Amazon S3 so great is you can return bucket contents, limiting it by a certain prefix.
All the images that users upload using our intranet CMS have to be named properly (checked by regex), starting with the ID number and an underscore. So on the CMS edit page, it's easy to quickly list the images associated with that article.
If we were hosting it on our own FTP server, listing the images associated with that article would be more time-consuming... as you'd need to do an FTP list operation (could take a while for 20,000+ files) and use PHP to loop through discarding 99% of those returned.
So my idea was to have a PHP script on that FTP server, set by htaccess DirectoryIndex, and make a request to that script. Secured by a couple of secret passwords sent in GET.
The script then accesses the local file system using scandir() and loops through, discarding 99% of the files and returns the matches, XML or something.
That should be far quicker than making an FTP list request and discarding.
What does everyone think to that as a solution?
Is there a better way to get PHP to loop through a local file system other than scandir?
Or is there a way more like Amazon's S3 "prefix" when listing a bucket contents?
I mean we're only talking $7-$10 a month of extra Amazon charges here, but it seems stupid to be paying that when we already have capacity on another hosting provider. Unless we downsize that hosting provider and move over to Amazon, which could actually be a future option
Hope all that makes sense, any comments/suggestions?
Thanks, B
I'm building a CMS onto our work intranet, to enable us to post news onto our website easier.
I'm also adding a file upload/deletion tool so that users of the CMS (2-4 users, pretty knowledgable at HTML and web stuff) don't have to get involved with uploading files manually through FTP.
At the moment I've got it running with Amazon's S3 service, using this PHP class... http://undesigned.org.za/2007/10/22/amazon-s3-php-class
It works great, however Amazon's charging structure is killing us a bit. The fact you get charged per 10,000 GET requests means that hosting loads of small files on there isn't that economical. Especially when we sent out an e-mail newsletter to 10,000 subscribers those requests soon add up. It would be better to host the images with our hosting provider.
However the thing that makes Amazon S3 so great is you can return bucket contents, limiting it by a certain prefix.
All the images that users upload using our intranet CMS have to be named properly (checked by regex), starting with the ID number and an underscore. So on the CMS edit page, it's easy to quickly list the images associated with that article.
If we were hosting it on our own FTP server, listing the images associated with that article would be more time-consuming... as you'd need to do an FTP list operation (could take a while for 20,000+ files) and use PHP to loop through discarding 99% of those returned.
So my idea was to have a PHP script on that FTP server, set by htaccess DirectoryIndex, and make a request to that script. Secured by a couple of secret passwords sent in GET.
The script then accesses the local file system using scandir() and loops through, discarding 99% of the files and returns the matches, XML or something.
That should be far quicker than making an FTP list request and discarding.
What does everyone think to that as a solution?
Is there a better way to get PHP to loop through a local file system other than scandir?
Or is there a way more like Amazon's S3 "prefix" when listing a bucket contents?
I mean we're only talking $7-$10 a month of extra Amazon charges here, but it seems stupid to be paying that when we already have capacity on another hosting provider. Unless we downsize that hosting provider and move over to Amazon, which could actually be a future option
Hope all that makes sense, any comments/suggestions?
Thanks, B