Mass file management file handling (and FTP List limit?)
Posted: Thu Aug 21, 2008 3:15 pm
I'm currently working on my first generation of PHP/MySQL membership based (practice) scripts though keeping long term goals in mind (such as managing very large numbers of users) a few things have come to mind.
Earlier I was dealing with order issues (1, 10, 11, 2, 20, 21, 22, etc) and I resolved that with a small though useful function. Image/avatar format looks like this now: 0000001_jabcreations_avatar.gif ($id.'_'.$user.'_'avatar'.$extension) with only the file extension having to be saved in the database versus a uselessly random file name. Looking at the files in the FTP I obviously know what the id and username is per image. This empowers me to act much quicker to remove undesirable content (especially since I won't have an automated system until I get über good.
However a few months ago or so I discovered that my server was filling up with emails (700MB / 30,000 or so IMAP files) and through that I discovered that FTP list command has some sort of limitation for the number of listed files before it cuts off. While I could obviously create a script that allows me to delete an image automatically in my honest opinion I think retaining a way to quickly make manual management of the file system (most likely through an FTP client) would empower me to enforce rules should they be broken. If some dingus uploads a TOS violating image (something unspeakably violent in example) and I encounter some sort of error in the site (making it not immediately possible to remove the image or prevent access to it) having a well thought out file system should make it much easier to remove offending content.
So my main question is this: what is the limit of return queries (files) in the FTP list command?
Based off of that I would want to have uploaded images stored in a series of folders.
For example users 1~999 (user 1) stored in /uploads/000/000/0000001_jabcreations_avatar.gif
For user 1,567,234 stored in /uploads/001/567/1567234_mrdingus_avatar.gif
I would also have to consider a numbering system where the number of folders did not exceed the maximum return for the FTP list command though I'd be happy with "even" a 1337 number of folders for users.
I'm vaguely aware of being able to store files inside of a database though that makes absolutely no sense and would from my current point of view dramatically increase down time when having to restore or back up a databases (and the insanity of having to wait hours for a tech because your DB is over a dramatically insanely huge size of OMG 2 megabytes!) Having a reasonably sized database is from all that I can tell much easier to manage in the long term.
Any way I'm interested in other people's thoughts on such a setup, the benefits, and the potential pitfalls.
Earlier I was dealing with order issues (1, 10, 11, 2, 20, 21, 22, etc) and I resolved that with a small though useful function. Image/avatar format looks like this now: 0000001_jabcreations_avatar.gif ($id.'_'.$user.'_'avatar'.$extension) with only the file extension having to be saved in the database versus a uselessly random file name. Looking at the files in the FTP I obviously know what the id and username is per image. This empowers me to act much quicker to remove undesirable content (especially since I won't have an automated system until I get über good.
However a few months ago or so I discovered that my server was filling up with emails (700MB / 30,000 or so IMAP files) and through that I discovered that FTP list command has some sort of limitation for the number of listed files before it cuts off. While I could obviously create a script that allows me to delete an image automatically in my honest opinion I think retaining a way to quickly make manual management of the file system (most likely through an FTP client) would empower me to enforce rules should they be broken. If some dingus uploads a TOS violating image (something unspeakably violent in example) and I encounter some sort of error in the site (making it not immediately possible to remove the image or prevent access to it) having a well thought out file system should make it much easier to remove offending content.
So my main question is this: what is the limit of return queries (files) in the FTP list command?
Based off of that I would want to have uploaded images stored in a series of folders.
For example users 1~999 (user 1) stored in /uploads/000/000/0000001_jabcreations_avatar.gif
For user 1,567,234 stored in /uploads/001/567/1567234_mrdingus_avatar.gif
I would also have to consider a numbering system where the number of folders did not exceed the maximum return for the FTP list command though I'd be happy with "even" a 1337 number of folders for users.
I'm vaguely aware of being able to store files inside of a database though that makes absolutely no sense and would from my current point of view dramatically increase down time when having to restore or back up a databases (and the insanity of having to wait hours for a tech because your DB is over a dramatically insanely huge size of OMG 2 megabytes!) Having a reasonably sized database is from all that I can tell much easier to manage in the long term.
Any way I'm interested in other people's thoughts on such a setup, the benefits, and the potential pitfalls.