Mass file management file handling (and FTP List limit?)

Not for 'how-to' coding questions but PHP theory instead, this forum is here for those of us who wish to learn about design aspects of programming with PHP.

Moderator: General Moderators

Post Reply
User avatar
JAB Creations
DevNet Resident
Posts: 2341
Joined: Thu Jan 13, 2005 6:44 pm
Location: Sarasota Florida
Contact:

Mass file management file handling (and FTP List limit?)

Post by JAB Creations »

I'm currently working on my first generation of PHP/MySQL membership based (practice) scripts though keeping long term goals in mind (such as managing very large numbers of users) a few things have come to mind.

Earlier I was dealing with order issues (1, 10, 11, 2, 20, 21, 22, etc) and I resolved that with a small though useful function. Image/avatar format looks like this now: 0000001_jabcreations_avatar.gif ($id.'_'.$user.'_'avatar'.$extension) with only the file extension having to be saved in the database versus a uselessly random file name. Looking at the files in the FTP I obviously know what the id and username is per image. This empowers me to act much quicker to remove undesirable content (especially since I won't have an automated system until I get über good. :P

However a few months ago or so I discovered that my server was filling up with emails (700MB / 30,000 or so IMAP files) and through that I discovered that FTP list command has some sort of limitation for the number of listed files before it cuts off. While I could obviously create a script that allows me to delete an image automatically in my honest opinion I think retaining a way to quickly make manual management of the file system (most likely through an FTP client) would empower me to enforce rules should they be broken. If some dingus uploads a TOS violating image (something unspeakably violent in example) and I encounter some sort of error in the site (making it not immediately possible to remove the image or prevent access to it) having a well thought out file system should make it much easier to remove offending content.

So my main question is this: what is the limit of return queries (files) in the FTP list command?

Based off of that I would want to have uploaded images stored in a series of folders.

For example users 1~999 (user 1) stored in /uploads/000/000/0000001_jabcreations_avatar.gif

For user 1,567,234 stored in /uploads/001/567/1567234_mrdingus_avatar.gif

I would also have to consider a numbering system where the number of folders did not exceed the maximum return for the FTP list command though I'd be happy with "even" a 1337 number of folders for users. :wink:

I'm vaguely aware of being able to store files inside of a database though that makes absolutely no sense and would from my current point of view dramatically increase down time when having to restore or back up a databases (and the insanity of having to wait hours for a tech because your DB is over a dramatically insanely huge size of OMG 2 megabytes!) Having a reasonably sized database is from all that I can tell much easier to manage in the long term.

Any way I'm interested in other people's thoughts on such a setup, the benefits, and the potential pitfalls.
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: Mass file management file handling (and FTP List limit?)

Post by josh »

Divide by the size of the chunks of files you want in each folder and then use ceil() or floor(), this will give you a "divisor factor" as I call it ( dont know if theres a proper name ) to use as the folder names, then when you need an image, just call the same function on the ID of the image you need to retrieve its path and read it back out

I'd make sure to write an object to make an API that handles the file operations automatically so Im not littering my application with spaghetti calls to these functions
Post Reply