Page 2 of 2

Re: Best Way to Compress Hundreds of GB

Posted: Mon Aug 31, 2015 8:45 pm
by requinix
Skara wrote:To upload to AWS, the entire payload as well as each part uploaded has to be treehashed. Way more intensive than a simple md5.
If you're talking about how MD5-ing a large file would suck, PHP has md5_file (which works the way it should).

Re: Best Way to Compress Hundreds of GB

Posted: Sun Nov 01, 2015 2:19 pm
by Vegan
I am used to using the Linux and Windows MD5summer that I use to make checksums with

I have always considered the PHP5 md5 etc to be more for messaging etc rather than for file integrity

I guess a PHP shell script would be as capable as md5summer for a given folder

Re: Best Way to Compress Hundreds of GB

Posted: Sun Nov 01, 2015 7:17 pm
by requinix
MD5 has always been for verifying data "integrity" - it's when people use it for cryptography is where things go downhill. Message hashes, file hashes, it's good for that.

Re: Best Way to Compress Hundreds of GB

Posted: Sun Nov 01, 2015 7:29 pm
by Vegan
AES256 seems to be reasonable strong, I could find no flaws with it, and I use that when I need to encrypt something

I used MD5 which seems to be OK with files even as large as 2-10 TB or even more, NTFS can cope with larger files with x64, ext4 is still a problem

BD video can be 40-50TB after editing etc