PharData Error: Seek Failed, on Large Files

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
User avatar
Skara
Forum Regular
Posts: 703
Joined: Sat Mar 12, 2005 7:13 pm
Location: US

PharData Error: Seek Failed, on Large Files

Post by Skara »

I'm compressing very large files (between 25MB on the low end, 20GB on the high end). Using PharData to create a .tar.gz. The compression works for smaller files (totalling roughly 2GB or less), but bums out on larger files. It throws the following PharException:

Code: Select all

exception 'PharException' with message 'tar-based phar "C:/Apache24/htdocs/_frames/storage/glacier/archive_cache/339.tar" cannot be created, contents of file "DOM_9483.MOV" could not be written, seek failed' in C:/Apache24/htdocs/_frames/storage/glacier/cron/upload_archives.php:250 Stack trace: #0 C:/Apache24/htdocs/_frames/storage/glacier/cron/upload_archives.php(250): PharData->buildFromDirectory('C:/Apache24/htd...') #1 {main}
Where DOM_9483.MOV is 4.62GB, a fairly typical size for what I need to compress.

Code: Select all

set_time_limit(0);
ini_set('memory_limit','2048M');

//...

    $archive_filename_tar = GLACIER_CACHE.$archive['id'].'.tar';
    $archive_filename = GLACIER_CACHE.$archive['id'].'.tar.gz';
    $tmp_location = GLACIER_CACHE.$archive['id'].'.tmp';
    $src_location = rtrim($archive['folder'],'/');

    //wait, exceptions; throws ErrorException
    set_error_handler('exception_error');

    try {
        // Copy directory to temporary location to divorce it from file sharing
        rcopy($src_location,$tmp_location);

        // Create archive
        $archive_class = new PharData($archive_filename_tar);
        $archive_class->buildFromDirectory($tmp_location);

        $archive_class->compress(Phar::GZ);
    }
    catch (ErrorException $e) {
        _log(1,'Uploading Archives [Copy/Compression Error #1]',(string)$e);
        continue;
    }
    catch (Exception $e) {
        _log(1,'Uploading Archives [Copy/Compression Error #2]',(string)$e);
        continue;
    }
    finally {
        // Clean Up Memory
        unset($archive_class);
    }
Running on Windows Server 2012 R2 local SAN.

Thoughts?
User avatar
requinix
Spammer :|
Posts: 6617
Joined: Wed Oct 15, 2008 2:35 am
Location: WA, USA

Re: PharData Error: Seek Failed, on Large Files

Post by requinix »

32-bit PHP can't handle files above 2GB. 64-bit is experimental on Windows so you wouldn't want to use it in production, and I'm not 100% sure it supports large files either.

1. Why are you using Phars for what looks like non-PHP files?
2. Can you use tar+gzs instead?
User avatar
Skara
Forum Regular
Posts: 703
Joined: Sat Mar 12, 2005 7:13 pm
Location: US

Re: PharData Error: Seek Failed, on Large Files

Post by Skara »

Not more than 2GB? That's a huge issue... The AWS Glacier SDK I'm using is in PHP, so this needs to work. And it has to work for files > 20GB.
It's php 5.6.8, but now I don't remember if I installed the 64-bit version or not. Is there a way I can check that important bit of information?

PharData seems to be the easiest way to create .tar.gz in PHP. By default PharData creates tar files, not phar. $archive_class->compress(Phar::GZ) compresses the tar file and adds the .gz extension.
User avatar
Skara
Forum Regular
Posts: 703
Joined: Sat Mar 12, 2005 7:13 pm
Location: US

Re: PharData Error: Seek Failed, on Large Files

Post by Skara »

What exactly is the 2GB limitation? I'm experimenting with the PEAR Archive_Tar package, and it's... sometimes working. Mostly. It managed to create a 4.8GB .tar.gz file. I would have to comb through data to find out if that was composed of files smaller than 2GB. Chances are it was.

The AWS Glacier SDK, for example, uploads large files section by section until it's complete. The limitation for Glacier is 40TB per file and 4GB per part. Is there no way around the 32-bit integer issue of parsing large files..? There has to be. :(
User avatar
requinix
Spammer :|
Posts: 6617
Joined: Wed Oct 15, 2008 2:35 am
Location: WA, USA

Re: PharData Error: Seek Failed, on Large Files

Post by requinix »

Skara wrote:Not more than 2GB? That's a huge issue... The AWS Glacier SDK I'm using is in PHP, so this needs to work. And it has to work for files > 20GB.
Saying it has to work doesn't mean it's going to work.
Skara wrote:It's php 5.6.8, but now I don't remember if I installed the 64-bit version or not. Is there a way I can check that important bit of information?
phpinfo() will probably say. Otherwise check the value of PHP_INT_SIZE: 4 for 32-bit and 8 for 64-bit.
Skara wrote:PharData seems to be the easiest way to create .tar.gz in PHP. By default PharData creates tar files, not phar.
tar is merely an archive format - sticking multiple files into one file. Phar archives are implemented using tar.
Skara wrote:What exactly is the 2GB limitation?
I believe the exact limitation is trying to seek to a point >2GB into the archive. Theoretically one could create an archive of arbitrary size, or append to an existing archive, but trying to extract a specific file beyond the 2GB limit would fail. Seems PharData is also trying to seek while writing.
Skara wrote:I'm experimenting with the PEAR Archive_Tar package, and it's... sometimes working. Mostly. It managed to create a 4.8GB .tar.gz file. I would have to comb through data to find out if that was composed of files smaller than 2GB. Chances are it was.
Are there any files in it starting past the 2GB point? (Another way, are there 2GBs of files starting at the beginning followed by another file?)

A simple test would be trying to create an archive of two or more 2GB+ files.
Skara wrote:The AWS Glacier SDK, for example, uploads large files section by section until it's complete. The limitation for Glacier is 40TB per file and 4GB per part. Is there no way around the 32-bit integer issue of parsing large files..? There has to be. :(
Not from within PHP. It would be easy to get a Window ports of tar and use exec functions to call it in your PHP code.
User avatar
Skara
Forum Regular
Posts: 703
Joined: Sat Mar 12, 2005 7:13 pm
Location: US

Re: PharData Error: Seek Failed, on Large Files

Post by Skara »

Damn.

Ok.

I've tried exec()ing, but with issues more complex than this. The Glacier SDK still has to point within the file, so it's going to fail on large files too, nevermind just trying to get the filesize of anything >2GB.

Thanks anyway. Looking for another solution.
Post Reply