Best Way to Compress Hundreds of GB

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

User avatar
requinix
Spammer :|
Posts: 6617
Joined: Wed Oct 15, 2008 2:35 am
Location: WA, USA

Re: Best Way to Compress Hundreds of GB

Post by requinix »

Skara wrote:To upload to AWS, the entire payload as well as each part uploaded has to be treehashed. Way more intensive than a simple md5.
If you're talking about how MD5-ing a large file would suck, PHP has md5_file (which works the way it should).
User avatar
Vegan
Forum Regular
Posts: 574
Joined: Fri Sep 05, 2008 3:34 pm
Location: Victoria, BC
Contact:

Re: Best Way to Compress Hundreds of GB

Post by Vegan »

I am used to using the Linux and Windows MD5summer that I use to make checksums with

I have always considered the PHP5 md5 etc to be more for messaging etc rather than for file integrity

I guess a PHP shell script would be as capable as md5summer for a given folder
Hardcore Games™ Legendary is the Only Way to Play™
My site is powered by LAMP
User avatar
requinix
Spammer :|
Posts: 6617
Joined: Wed Oct 15, 2008 2:35 am
Location: WA, USA

Re: Best Way to Compress Hundreds of GB

Post by requinix »

MD5 has always been for verifying data "integrity" - it's when people use it for cryptography is where things go downhill. Message hashes, file hashes, it's good for that.
User avatar
Vegan
Forum Regular
Posts: 574
Joined: Fri Sep 05, 2008 3:34 pm
Location: Victoria, BC
Contact:

Re: Best Way to Compress Hundreds of GB

Post by Vegan »

AES256 seems to be reasonable strong, I could find no flaws with it, and I use that when I need to encrypt something

I used MD5 which seems to be OK with files even as large as 2-10 TB or even more, NTFS can cope with larger files with x64, ext4 is still a problem

BD video can be 40-50TB after editing etc
Hardcore Games™ Legendary is the Only Way to Play™
My site is powered by LAMP
Post Reply