Creating Large ZIP Files
Posted: Mon Oct 02, 2006 7:15 pm
Gday Guys, i'm currently using a brilliant class by Eric Mueller (modified) http://www.zend.com/codex.php?id=696&single=1 to create zip files 'on the fly'. Basically, i retrieve a file from a read only location, dump it into a string add it to the class, create the zip then send it to the client with the appropriate headers.
Simple stuff and it works great, when using files of a smaller nature. The problem that I have is that I'm using highly compressible, yet very large files (Autocad Drawings) and I run into memory & script time constraints. When I say large, some files that i'm using could be upwards of 300mb which will obviously use a large memory footprint, and will take some time (>60seconds).
At present, i've dropped a script timeout when creating these zip's, and have allowed the server to use up to around 600mb of memory to 'temporarily get around this problem'. Obviously this is not going to be feasible when the server goes live.
What I want, is a 'chunkable' zip file creator where I could 'feed' arbitrary amounts of the file (say <50mb) at a time, to create the zip file using a smaller memory footprint. Assuming that for the time being, time is no major constraint (if the client has to wait 3-4minutes for the file to compress, it's far better than the additional 2 hours to download the extra 100mb), has anyone enountered a class, or ZIP method that would allow me to do this? Failing that, does anyone know of a zip algorithm/format that would allow me to begin encoding a file without requiring the entire file loaded into memory?
Cheers
Simple stuff and it works great, when using files of a smaller nature. The problem that I have is that I'm using highly compressible, yet very large files (Autocad Drawings) and I run into memory & script time constraints. When I say large, some files that i'm using could be upwards of 300mb which will obviously use a large memory footprint, and will take some time (>60seconds).
At present, i've dropped a script timeout when creating these zip's, and have allowed the server to use up to around 600mb of memory to 'temporarily get around this problem'. Obviously this is not going to be feasible when the server goes live.
What I want, is a 'chunkable' zip file creator where I could 'feed' arbitrary amounts of the file (say <50mb) at a time, to create the zip file using a smaller memory footprint. Assuming that for the time being, time is no major constraint (if the client has to wait 3-4minutes for the file to compress, it's far better than the additional 2 hours to download the extra 100mb), has anyone enountered a class, or ZIP method that would allow me to do this? Failing that, does anyone know of a zip algorithm/format that would allow me to begin encoding a file without requiring the entire file loaded into memory?
Cheers