Page 1 of 1

Which file system is best

Posted: Wed Apr 18, 2007 10:09 pm
by alex.barylski
Here are my requirements:

1) Fastest in terms of read/write
2) Store the largest number of files in single directory (100+ million)

Error checking, rollbacks, etc are not nessecary as the data is backed up daily to another external drive with all those bells and whistles. This is purely required for performance and ease of use (hence the single directory requirements).

I recall reading an article once that said Linux cops out at about 10000 files per folder. This kind sucks, is there a work around short of re-writing the FS driver?

Cheers :)

Posted: Wed Apr 18, 2007 11:45 pm
by Kieran Huggins
afaik there aren't any filesystems without a soft folder limit

I would use multiple folders: i.e. "fahadjfhkahfa" would be stored at "/f/a/h/a/fahadjfhkahfa"

Posted: Thu Apr 19, 2007 1:00 am
by volka
Hockey wrote:I recall reading an article once that said Linux cops out at about 10000 files per folder.
Check the date of that article. Once upon a time there was a 2/4 color limit and computers were called c64 ;)


Hockey wrote:files in single directory (100+ million)
8O
http://namesys.com/faq.html#reiserfsspecs wrote:max number files a dir can have

3.5
518701895,
but in practice this value is limited by hash function. r5 hash allows about 1 200 000 file names without collisions

3.6
232 - 4 => 4 Gi - 4
but in practice this value is limited by hash function. r5 hash allows about 1 200 000 file names without collisions
http://www-128.ibm.com/developerworks/library/l-jfs.html wrote:JFS dynamically allocates space for disk inodes as required, freeing the space when it is no longer required. This support avoids the traditional approach of reserving a fixed amount of space for disk inodes at the file system creation time, thus eliminating the need for users to estimate the maximum number of files and directories that a file system will contain.
But still
Hockey wrote:files in single directory (100+ million)
8O smells like bad design.

Posted: Thu Apr 19, 2007 5:35 am
by Jenk

Posted: Thu Apr 19, 2007 8:23 am
by jayshields
Suprised no one's mentioned Sun's ZFS! http://www.sun.com/2004-0914/feature/

Is XFS a modified ZFS?

Posted: Thu Apr 19, 2007 8:55 am
by volka
jayshields wrote:Is XFS a modified ZFS?
not really ;)
http://en.wikipedia.org/wiki/Xfs wrote:XFS is the oldest journaling file system available for UNIX systems, and has a mature, stable and well-debugged codebase.
[...]
Introduced 1994 (IRIX v5.3)
http://en.wikipedia.org/wiki/Zfs wrote:Introduced November 2005 (OpenSolaris)