Page 1 of 1

File Security

Posted: Tue Jun 09, 2009 11:24 pm
by William
Hey all,

Quick summary, I'm building a system that has files that are user submitted and I need them to be secure. All transactions will be done through SSL, files will be encrypted and the filenames will be regenerated. You would need access to the PHP code, MySQL database, and storage servers to access any of the files.

Main questions
What encryption algorithm should I use to encrypt the files (speed isn't really an issue, but be reasonable).
What do you think of what I have so far in terms of security?

Note
This isn't really a question about application security, SQL injection, blah blah blah, more of a question about how I'm working with the files and keeping them secure. If there is a few things I missed or a few things that I left out that you'd like to ask, ask away.

Long version
I'm currently working on a project that is going to require relatively high security. I was hoping to get a few peoples opinions on what I've thought of so far, mainly from kaisellgren.

This is what I've thought of so far. A user uploads a file, I generate a transaction ID and generate two different hashes. These hashes share the same salt and transaction ID, but a different pepper to create two different hashes. One hash is used for the filename, the other for the encryption key (each file is encrypted). All transactions are done through SSL (including when files are uploaded to the site and transferred to our storage servers and back.)

I originally thought about asking a user for a unique password for each file, adding a salt & pepper and setting the encryption key & filename from that. This way the only way for us to know the filename and encryption key would be to have that users password. This would make it hard for even someone with full file access to find the file & decrypt it. The user would have to go through each file (of millions), crack the encryption to find the correct file since he wouldn't know which file to find. This isn't an option since we need to be able to remove the file on a users request.

I hope I described it well it was actually like 3 times as long but I cut it down so hopefully I didn't miss anything.

Thanks again for anyone taking the time out of their day to read this. It will help a lot.

Re: File Security

Posted: Wed Jun 10, 2009 7:02 am
by kaisellgren
William wrote:What encryption algorithm should I use to encrypt the files (speed isn't really an issue, but be reasonable).
Definitely go for an AES candidate. I would say either RC6, TwoFish, or Rijndael. Personally, I prefer Rijndael as it is widely implemented (helps if you need to port your encrypted data into other systems, languages, etc) and provides a strong encryption with good performance.
William wrote:What do you think of what I have so far in terms of security?
The idea sounds fine. Now you just need to implement it properly in your code, which is the harder part.

One thing I would like to know is the purpose of your system/site. Could you elaborate? I understand you let users upload files, but what are these files? Patient charts? Mecical history? Secret business documents? Knowing the exact purpose may help me to give you ideas/tips.
William wrote:I generate a transaction ID and generate two different hashes. These hashes share the same salt and transaction ID, but a different pepper to create two different hashes. One hash is used for the filename, the other for the encryption key (each file is encrypted).
So, your hashes are constructed like Hash(Salt + ID + Pepper)? Where do you store this encryption key? Do you store the filename somewhere? Do you have a special purpose for this hashing, or did you do this to make unguessable filenames and strong encryptions?
William wrote:All transactions are done through SSL (including when files are uploaded to the site and transferred to our storage servers and back.)
Make sure that your pages deny to load in HTTP -mode. You can check $_SERVER['HTTPS'] to know whether the site was not loaded in HTTPS (and redirect to HTTPS).
William wrote:I originally thought about asking a user for a unique password for each file,
That has two problems: users do not provide unique passwords and they do not understand security concerns.
William wrote:This would make it hard for even someone with full file access to find the file & decrypt it.
You are right about decrypting a file. It does make it harder. However, you can still require one global password from each user that is part of the encryption (and possibly the file naming) process. This would be less complex for the user, because he only needs to use one password and is therefore easier for him to use a more complex password. So, to decrypt a user's files, one need to access the files, access the keys, know the user's password and maybe access the database depending on how you have spread your secrets. So, to read users' files, the whole system has to be compromised and the intruder would need to exploit your system to its fullest extent. Do note that in case a user forgets his password, there is no way to recover his files anymore (except with brute force). I am not sure whether you want that or not.

As what comes to finding a file. It becomes harder if there are enough files and you have modified the file access, modified, etc date time values. So, if you want to make sure no one can determine the file location without knowing the keys, you need to alter the file access, last modified and such dates. As far as I can tell you, PHP does not support this, but you can have a third party application on your server that PHP calls with an exec() call. The creation date value is mainly the worst. It is a big hint. You also would need to make sure your web server (possibly Apache) does not keep logs of file uploads.
William wrote:This isn't an option since we need to be able to remove the file on a users request.
When he requests file removal, ask the password again (which may sound reasonable depending on the purpose of your site). Therefore, you could have the necessary information to find the correct file.

Re: File Security

Posted: Wed Jun 10, 2009 8:59 am
by William
kaisellgren wrote: Definitely go for an AES candidate. I would say either RC6, TwoFish, or Rijndael. Personally, I prefer Rijndael as it is widely implemented (helps if you need to port your encrypted data into other systems, languages, etc) and provides a strong encryption with good performance.
Thank you very much.
kaisellgren wrote:One thing I would like to know is the purpose of your system/site. Could you elaborate? I understand you let users upload files, but what are these files? Patient charts? Mecical history? Secret business documents? Knowing the exact purpose may help me to give you ideas/tips.
The best answer I can give that relates would be something like business documents.
kaisellgren wrote: So, your hashes are constructed like Hash(Salt + ID + Pepper)? Where do you store this encryption key? Do you store the filename somewhere? Do you have a special purpose for this hashing, or did you do this to make unguessable filenames and strong encryptions?
When a file is viewed we get a transaction ID. I'll then look it up, get the shared salt for that transaction and regenerate the two hashes. The hashes will not be stored in the database to force the hacker to have access to the web server to figure out how I create the hashes (pepper and format). Yes, this is for filenames and encryption security only.

kaisellgren wrote: Make sure that your pages deny to load in HTTP -mode. You can check $_SERVER['HTTPS'] to know whether the site was not loaded in HTTPS (and redirect to HTTPS).
Will do, thanks.
kaisellgren wrote: That has two problems: users do not provide unique passwords and they do not understand security concerns.
Although I was I don't plan on doing this my initial thought would be that sense I'm using a hash it wouldn't matter the users password in terms of trying to crack the encryption key via brute force. Although you have a point, I said that even if they had full access and if they did they'd just need to brute force the users password not the hash which would make it easier. Thanks for pointing that out. Although it would add a bit of security since now if a user has full access to the system they will have access to the file no matter what. While this method would require someone to crack their password and ontop of that they'd have to get the filename first which they also can't get without the password. It would make it a lot harder but it would also make it harder for us too.
kaisellgren wrote: You are right about decrypting a file. It does make it harder. However, you can still require one global password from each user that is part of the encryption (and possibly the file naming) process. This would be less complex for the user, because he only needs to use one password and is therefore easier for him to use a more complex password. So, to decrypt a user's files, one need to access the files, access the keys, know the user's password and maybe access the database depending on how you have spread your secrets. So, to read users' files, the whole system has to be compromised and the intruder would need to exploit your system to its fullest extent. Do note that in case a user forgets his password, there is no way to recover his files anymore (except with brute force). I am not sure whether you want that or not.
Although asking for a user password seems correct enough, with the type of files we're dealing with a user needs to be able to get the file deleted if a legal issue comes up. If they don't remember their password and sue us for keeping their file and we have no possible way of knowing where it is, that could be a problem.
kaisellgren wrote:As what comes to finding a file. It becomes harder if there are enough files and you have modified the file access, modified, etc date time values. So, if you want to make sure no one can determine the file location without knowing the keys, you need to alter the file access, last modified and such dates. As far as I can tell you, PHP does not support this, but you can have a third party application on your server that PHP calls with an exec() call. The creation date value is mainly the worst. It is a big hint. You also would need to make sure your web server (possibly Apache) does not keep logs of file uploads.
Very good points. Thank you for this.
kaisellgren wrote: When he requests file removal, ask the password again (which may sound reasonable depending on the purpose of your site). Therefore, you could have the necessary information to find the correct file.
Hehe I think I answered this question above, oops.

Sorry it's hard to read I'm on a netbook driving through the city right now (as a passenger) and it makes it harder write.

Thanks a lot Kai for your help, you've been great.

Re: File Security

Posted: Wed Jun 10, 2009 10:23 am
by kaisellgren
What platform does this operate on? Linux?

Encrypting files is not too great if the encryption key is on the server. An intruder has to just find the key (probably by looking at your code, files) and then decrypt the files he wants. So, you should either distribute the key in different places (storage server, db server, etc) and/or use the user-supplied password as part of it.

This comes to another problem. If an intruder has an access to the files, does he has an access to your application? If he does, he can rewrite your application so that it writes down user submitted passwords. Therefore, instead of brute forcing, the intruder only sniffs the passwords. This is one thing that may render passwords in the encryption process more or less useless. When adding the fact that users may forget their passwords, the approach for using user passwords in the encryption process (or the filename generation) does not seem that good after all. Do also note that an intruder does not need to decrypt the files (or even read your code) if he can access the user's control panel area.

As you probably noticed, if the server is compromised - so are the files regardless of the encrypting applied to it (unless the keys are separated wisely). You need to make sure no one ever accesses your system. That is the primary objective. Due to the fact that your PHP application must be able to decrypt & encrypt, if an intruder has an access to the system where the application (PHP files) are kept, he can just modify those files to log encryption keys, user passwords, etc.

This is usually what I do with uploaded files:
1) I generate pseudo random data from /dev/urandom of 16-32 bytes.
2) I base64 encode it and use it as the name for the uploaded filename. I While() loop this as long as there are no filename clashes (aka if the filename exists, recreate a new filename).
3) I insert the generated filename into the database (which has an identifier column with AUTO_INCREMENT - I use MySQL mostly)
4) When the files are viewed, e.g. view.php?file=123 then I query the database and look for filenames, which ID has the value 123. Then I use the filename to fetch the file.

I am still not entirely sure about the purpose of your system, but I am starting to believe encrypting files does not help you too much.

Re: File Security

Posted: Wed Jun 10, 2009 10:44 am
by William
kaisellgren wrote:What platform does this operate on? Linux?

Encrypting files is not too great if the encryption key is on the server. An intruder has to just find the key (probably by looking at your code, files) and then decrypt the files he wants. So, you should either distribute the key in different places (storage server, db server, etc) and/or use the user-supplied password as part of it.
There will be a minimum of 3 servers. Web server, db server, and storage server. The web server will have my PHP code that will contain the pepper for the filename hash, and a pepper for the encryption key hash. The DB server will have an entry, there will be an AUTO_INCREMENT identifier column but there will also be a UNIQUE index on a transaction ID, think AHSF98H802FAF that will be given to the public. Now each transaction in the database (aka file) will have a salt in it's entry, this salt will be global to that file for both the encryption key AND filename. So think of it this way...

Transaction ID a1b2c3d4e5f6g7h8j9, so filename is something like... PEPPER + transaction_id + SALT. The encryption key will be PEPPER2 + transaction_id + SALT. Now all thats stored in the DB is the transaction ID, and the salt, whats stored in the web server is the two different peppers. The filename & encryption key won't be stored in the database, they'll have to be generated by taking data from the web server, and the PHP code. But the more I think about this I'm getting your next point belo
kaisellgren wrote:This comes to another problem. If an intruder has an access to the files, does he has an access to your application? If he does, he can rewrite your application so that it writes down user submitted passwords. Therefore, instead of brute forcing, the intruder only sniffs the passwords. This is one thing that may render passwords in the encryption process more or less useless. When adding the fact that users may forget their passwords, the approach for using user passwords in the encryption process (or the filename generation) does not seem that good after all. Do also note that an intruder does not need to decrypt the files (or even read your code) if he can access the user's control panel area.
As I said above I understand your point, all a user needs to have is access to the web server and then he/she can access our file storage and have access to the code to generate the hash along with all the data from the DB to create it. So obviously application security is a must but I can't think of any other way to stop a hacker from accessing this if he gains access tot he web server since the web server has to process it all. The only solution would be to have a password the user enters but like I said that is not really possible UNLESS we generate a pass code for the user for each document that we then store somewhere else ourselves so if they lose it we still have it. The problem with this is if the web server generates the pass code and "stores" it somewhere else, it obviously has access to it. Unless we insert it into a DB with only INSERT access and not select access that would require gain to that DB server to access the data, the web server wouldn't have access to any data to read.

Basically something like... User uploads a file, we generate them a transaction ID and a unique code that they need to store, this is how they verify who they are (It might be smart to make this pin/code global to all their files, less secure but still easier for the user), we then INSERT this code into the mysql db with write only access, no read. Obviously if a hacker gets into our web server he can make the script send him the new pass codes but thats only once he gets in and after, nothing prior to that. The problem is though if the user brute forces the code he can access a whole users files but only one at a time. The biggest problem with this whole thing is that it can't only be amazingly secure, but it also has to be user friendly. Obviously the user needs to risk some parts of it being easy for security, but it can't be so it's unusable.
kaisellgren wrote:As you probably noticed, if the server is compromised - so are the files regardless of the encrypting applied to it (unless the keys are separated wisely). You need to make sure no one ever accesses your system. That is the primary objective. Due to the fact that your PHP application must be able to decrypt & encrypt, if an intruder has an access to the system where the application (PHP files) are kept, he can just modify those files to log encryption keys, user passwords, etc.
I think I kind of touched this subject above, overall good points.
kaisellgren wrote: This is usually what I do with uploaded files:
1) I generate pseudo random data from /dev/urandom of 16-32 bytes.
2) I base64 encode it and use it as the name for the uploaded filename. I While() loop this as long as there are no filename clashes (aka if the filename exists, recreate a new filename).
3) I insert the generated filename into the database (which has an identifier column with AUTO_INCREMENT - I use MySQL mostly)
4) When the files are viewed, e.g. view.php?file=123 then I query the database and look for filenames, which ID has the value 123. Then I use the filename to fetch the file.
This is pretty much the same as anything else I did, I guess technically it doesn't matter if I try to over complicate things by what I said above or do this method, either way if they have access to the web server I'm screwed. Obviously I have really big issues if the web server gets accessed and I don't expect it, but I'm not trying to think about "what probably will happen", have to secure with worst case in scenario in my mind.
kaisellgren wrote:I am still not entirely sure about the purpose of your system, but I am starting to believe encrypting files does not help you too much.
To give you an idea users upload files that are meant to be kept secured. Having someone else download these files isn't a good thing :).

Re: File Security

Posted: Wed Jun 10, 2009 11:29 am
by kaisellgren
William wrote:Obviously if a hacker gets into our web server he can make the script send him the new pass codes but thats only once he gets in and after, nothing prior to that.
Indeed. The encryption scheme only stops the reading of files that have not yet been read by the users and it might prevent some stupid script kiddies, who accidentally found a hole in your application.
William wrote:The problem is though if the user brute forces the code he can access a whole users files but only one at a time.
The brute force process will take long. Very long. It is basically infeasible and not just because it takes long to iteratively crack it, but also because it is hard to know when it was decrypted "successfully".

You have two options what I can think of. The first is simpler: build a secure site so that no one ever gets in. The second, however, is a bit different and the basic idea is this:

Write a Java/Flash application that takes a file in a form and when the user has chosen the file, the application (=Java/Flash) uses a strong key (will talk about it later) to encrypt the file with Rijndael. The encrypted file is uploaded to your server. Your servers keeps the files secure. Now, when a user wants to download/view a file, the same Flash/Java application will launch and it asks for the encrypted file that your server sends. All this happens through SSL/TLS, of course. Now, the Flash/Java application will decrypt it after it has got the file.

So, instead of giving keys to your web server, the client keeps his key. This way, anyone accessing your server cannot decrypt files. Not even by altering your application. To make things better, the client can use the Flash/Java application on any computer and still be able to access the files. The thing that requires thinking from your part is how you are going to implement this. Flash or Java? I do not know. I am not even sure if Flash can easily encrypt data... but with Java it should be "pretty simple". The files would be completely secure as long as the client's computer is secure. An intruder cannot obviously target custom user computers just like that.

In addition to this, the client would need to enter his own password, which is kept on the server and never kept on the clients computer. Therefore, one would either need to crack into both client's computer and the web server or setup a keylogger on the client's computer and trick him to view/upload files. This becomes extremely difficult for attackers. The keys are used for encryption while the password is used for authenticating the user.

For encryption keys, you can't use weak passwords like what users typically use at websites. Rijndael, for instance, requires 128-bit or 256-bit keys. There might be other key sizes, I do not remember. Anyway, a secure key like 256-bit is 32 bytes and it must be completely random. So, you should probably generate key files for your users. You could, in fact, make a simple web interface to generate these keys:

Code: Select all

if (($handle = @fopen('/dev/urandom','rb')))
{
 $rand = fread($handle,32); // 32 bytes = 256 bits
 fclose($handle);
}
header('Content-Type: application/octet-stream');
header('Content-Disposition: attachment; filename="keyfile.bin"');
header('Content-Length: '.strlen($rand));
header('X-Content-Type-Options: nosniff');
echo $rand;
 
By running the code it starts to download a keyfile.bin containing very strong random binary data of 32 bytes for the Rijndael cipher. Basically, all users would need to visit this page throught HTTPS once to make a keyfile for themselves. Then the Flash/Java application could ask for the keyfile to encrypt/decrypt files.

I have never done this myself, but I hope this gives you some ideas.
William wrote:I guess technically it doesn't matter if I try to over complicate things by what I said above or do this method, either way if they have access to the web server I'm screwed.
I honestly just think the hash construct is... hacky. Like you noticed, the intruder could find out the filename either way, but the filename would be a lot more random with my method and creates less clashes (and makes more sense in overall, cleaner too).

Re: File Security

Posted: Wed Jun 10, 2009 11:45 am
by William
I really should of given a bit more information. Obviously I don't want to give out too much information of the system but there is something that I should have brought up at first that is going to change a lot. We need to process these files. We do something to these files which is part of our whole point, it's not just a file storage service. I know I REALLY should of brought this up I was naive in thinking that it needed to be said. That said I have written an application in Java that takes the files they upload and outputs a new file.

So what if we do something like this... A user uploads a file which goes through SSL, we take the file run it through the app I said before, the original file and the new file I create will both be saved. The filename will be 100% random like you said. Now here comes the issue, encrypting the file. Well the more I think about it the more I realize that it's fine if the user doesn't remember his code because we won't be using that as the filename, so instead we'll accept a code from the user that they'll type in, that we'll then take and modify it to make it a bit longer and more random and use that as the key. We won't store the key or the users pass code into the site but we will hash their pass code (with a pepper & salt) so we can verify the pass code the type in next time they want to access the file. After that it will download the file from our storage servers (through SSL of course) and output it to the browser.

The only thing left is to figure out how to take their code they give us and make it more random / secure.

Also, thanks again for taking your time to help out!

Re: File Security

Posted: Wed Jun 10, 2009 12:00 pm
by kaisellgren
William wrote:A user uploads a file which goes through SSL, we take the file run it through the app I said before,
And this application runs on the server? Does it have to be so?
William wrote:so we'll accept a code from the user that they'll type in, that we'll then take and modify it to make it a bit longer and more random and use that as the key.
An intruder will be able to get this key.
William wrote:We won't store the key or the users pass code into the site
but an intruder would. So, like we earlier stated, an intruder can read all encrypted files that users access after he has got into the server.

I think everything else is pretty much okay, but the encryption of files is a little bit useless. It does help in some cases, but it does not prevent anything. With that kind of system, you can only hope no one gets in.

Can I ask you what does your application do with the files? Your project is mysterious to me. If you like, we could talk in private.

Re: File Security

Posted: Wed Jun 10, 2009 12:12 pm
by William
Talking is private sounds good, is AIM good? I'm going to continue this here so other people can learn from everything we're talking about.

Encryption isn't useless since if a user did hack into the site they would have to modify the sites code and have it log the users passwords, and it would only log passwords that users type in AFTER they gotten into our system. Although this doesn't keep EVERY user secure it will keep a large percentage, especially if we catch the intruder right away.

I do see your point though. What we do to the files need to be done on the web server, having it done outside the web server isn't really an option. Honestly I don't think there is a better option then asking the user for a pass code for each document they upload and using that to help create the encryption key since it has to be done server side. I guess we're just going to have to hope the best for our application. No frameworks will be used so the issue of having known software bugs won't be an issue, although since it's in-house there is a chance that we won't catch a bug that a bigger application would know about.

All in all this is a tiny application so it shouldn't be that difficult to secure, I would love to talk more privately, although I won't be able to give out too many details in terms of who this application is for, etc. I can give more information on the product it's self.

Re: File Security

Posted: Wed Jun 10, 2009 12:35 pm
by kaisellgren
William wrote:Encryption isn't useless since if a user did hack into the site they would have to modify the sites code and have it log the users passwords, and it would only log passwords that users type in AFTER they gotten into our system. Although this doesn't keep EVERY user secure it will keep a large percentage, especially if we catch the intruder right away.
That's why I said "a little bit" useless.
William wrote:What we do to the files need to be done on the web server, having it done outside the web server isn't really an option.
Okay, then we must store the files on the server and let the server handle the encryption.
William wrote:Honestly I don't think there is a better option then asking the user for a pass code for each document they upload and using that to help create the encryption key since it has to be done server side.
If you can ask them to use individual passwords for invidual files, then you should go for it, but if it gets too complex to remember passwords, then you should just use one password per user's files.
William wrote:I guess we're just going to have to hope the best for our application. No frameworks will be used so the issue of having known software bugs won't be an issue, although since it's in-house there is a chance that we won't catch a bug that a bigger application would know about.
If you write your application well, no one may never get into your server. There are hundreds of complex sites that has never got cracked. Maybe they will, maybe they wont.
William wrote:All in all this is a tiny application so it shouldn't be that difficult to secure, I would love to talk more privately, although I won't be able to give out too many details in terms of who this application is for, etc. I can give more information on the product it's self.
Well I guess there is no magic you could use, so, better write everything in your application as securely as possible and ask here for help if needed. You should actually ask someone to evaluate your application.

Re: File Security

Posted: Wed Jun 10, 2009 12:42 pm
by William
Sounds good, I plan on having multiple people review the application to tests it's security. It's kind of an on going thing with the kind of application I'm building. I should be good in terms of the application security, I was just a little unsure about the best way to go about things with files. It's not that I don't think I could figure out a good way to do it, it's just that other people have most likely had to do something similar so I wanted to see what other peoples opinions are.

Thanks a lot for the help. If you're still interested in talking in private about the application let me know.