Page 1 of 1

fetching file via curl.. filesize issues..

Posted: Wed Mar 11, 2009 2:13 am
by estarapapax
I'm building a website which fetches text files which path is appended by users in the query parameter. For example:
domain.com/read.php?path=http://www.webhost.com/username/file.txt

Now, my website will fetch that file and process its contents. I will not go into details on what my website is for since my question is only about the file size of the text file.

What I want is that whenever a user attempts to have a very large text file to be processed by my website, it will abort processing it. I'm implementing something like this: http://snipplr.com/view/29/get-remote-filesize/ , which uses cURL to check the filesize.

However, I'm afraid someone will attempt to modify the header information of his text file. Say,
webhost.com/username/file.txt is actually generated dynamically and along that is the modification of the header information. He will modify the filesize header information to be low but in fact his file is very large.

So, is there a way such that while I'm fetching the file (using cURL) I can set a maximum number of bytes in the stream to be transferred?

Re: fetching file via curl.. filesize issues..

Posted: Wed Mar 11, 2009 2:19 am
by Benjamin
Well looking through the curl options I see CURLOPT_RESUME_FROM. You could open a test connection with a resume from setting and see if anything is returned. If nothing is returned or if there is an error then you know the file is smaller than the CURLOPT_RESUME_FROM size. I'm not sure how that would work with dynamic files though.

Re: fetching file via curl.. filesize issues..

Posted: Wed Mar 11, 2009 2:24 am
by estarapapax
hey thanks.. i'll try this one.

Re: fetching file via curl.. filesize issues..

Posted: Wed Mar 11, 2009 2:27 am
by Benjamin
Also, I believe browsers close connections once they receive the byte amount indicated in the header. I'm not sure if cURL does the same thing but you may be able to rely on that value.