Page 1 of 1
curl request to upload a minimum of 100MB file
Posted: Mon Oct 16, 2006 11:13 am
by raghavan20
i have a CURL script to upload to a site a listing file which is in a range of 100MB on an average. When I tried to upload as POST request and tried to get the file as file_get_contents(), it managed to get around 2MB which is well below the expected limit so I need some solution to upload larger files. The reason I am using CURL is because the target server is using HTTP authentication only.
current dynamic settings
Code: Select all
set_time_limit(0);
ini_set( "max_input_time", 0 );
memory reported by PHP
php.ini
Code: Select all
memory_limit = 32M ; Maximum amount of memory a script may consume (8MB)
1. I need your opinion on how to optimise PHP and HTTPD settings. Our server is running Fedora Core 5.
2. Are there are any better ways to upload large files like 100 MB ones?
I really appreciate any help.
Posted: Mon Oct 16, 2006 11:19 am
by feyd
If you have access, FTP it. I have no idea how much memory cURL will need to upload a file off hand.
Posted: Tue Oct 17, 2006 3:11 am
by raghavan20
feyd, the system supports HTTP authentication and HTTP POST upload so I am afraid I cannot use FTP.
Is there is anyway like instead of storing file contents in a string variable is it possible to send N bytes by N bytes until the end of the file.
Posted: Tue Oct 17, 2006 8:07 am
by feyd
You could build an alterate using fsockopen() and send the same headers cURL would have, then do a piece-meal upload of the file into that stream.
Posted: Tue Oct 17, 2006 9:27 am
by raghavan20
that is the same i am thinking about..i will give it a try
it is a pity that cURL cannot do it!
Posted: Tue Oct 17, 2006 9:33 am
by raghavan20
i have a doubt, normally most of the target system expect content-length, should it be in characters or in bytes, because if we use strlen() then we have to get content in string variable which the system cannot handle.
Posted: Tue Oct 17, 2006 9:44 am
by feyd
Posted: Tue Oct 17, 2006 10:27 am
by raghavan20
thanks feyd!
any idea why this is not working...in the sense, POST data is empty?
sending data by POST
Code: Select all
<?php
$fileName = "replication.txt";
$fileContents = file_get_contents( $fileName );
##echo "\n\n\nfile contents: ".$fileContents."\n\n\n";
$host = "***";
$header = "Host: $host\r\n";
$header .= "User-Agent: PHP Script\r\n";
$header .= "Content-Type: text/xml\r\n";
$header .= "Content-Length: ".filesize( $fileName )."\r\n";
$header .= "Connection: close\r\n\r\n";
$fp = fsockopen( $host, 80, $errno, $errstr );
$service_uri = "/curlTest/readsock.php";
fputs($fp, "POST $service_uri HTTP/1.1\r\n");
fputs($fp, $header);
$newFP = fopen( $fileName, "r" );
while( !feof( $newFP ) ){
$temp = fread( $newFP, 100 );
echo "\n\ntempData:\n".$temp;
fwrite( $fp, $temp );
}
echo "\nError number: ".$errno;
echo "\nError string: ".$errstr;
echo "\nReturn response: ".fread( $fp, 5000 )."\n";
?>
target script
Code: Select all
<?php
print_r( $_GET );
echo "\n";
print_R( $_POST );
echo "\n";
file_put_contents( "post", var_export( $_POST, TRUE).var_export( get_headers($url), TRUE ) );
?>
response
Code: Select all
Suse:/var/www # /opt/lampp/bin/php fsock.php
tempData:
replication:
involves one master and many slaves
master chooses which databases to use for gener
tempData:
ate binary logs
slave has settings about master name, ip, user, password
and the database whic
tempData:
h needs replicated
and also specify the tables to be ignored in each database
also can specify ret
tempData:
ry time
slave has two threads
1. gets data from binary log of server into relay log
2. reads th
tempData:
e relay log and executes it
the slave thread communicates with server asking about some database
tempData:
log with the last read position
the other read also has a marker to know which position it last rea
tempData:
d from the relay log
Error number: 0
Error string:
Return response: HTTP/1.1 200 OK
Date: Tue, 17 Oct 2006 15:22:52 GMT
Server: Apache/2.0.54 (Fedora)
X-Powered-By: PHP/5.0.4
Content-Length: 22
Connection: close
Content-Type: text/html
Array
(
)
Array
(
)
contents written to file named 'post'
Posted: Tue Oct 17, 2006 10:34 am
by feyd
File submissions end up in $_FILES and are multipart messages. That I remember, posted field data comes in similarly to how the query string looks:
Code: Select all
fieldName1=value2&fieldname2=value2
Posted: Tue Oct 17, 2006 10:42 am
by raghavan20
I understand what you are saying.
I am working with implementing an API, the target company obtain all the input as just POST data not like
postField=postValue
or
HTTP upload
so I guess there should be a way to receive the POST data even if it not an array.
Posted: Tue Oct 17, 2006 12:12 pm
by choppsta
If I understand you correctly you want to get at the raw post data, not the usual urlencoded data that get's sent from an HTML form.
In your target script, try the following to output what the request you're sending looks like:
Code: Select all
ini_set('always_populate_raw_post_data', 1);
$data = apache_request_headers();
$data['POST data'] = file_get_contents('php://input');
echo '<pre>'.htmlspecialchars(print_r($data, true)).'</pre>';
Posted: Wed Oct 18, 2006 5:07 am
by raghavan20
great choppsta it worked, i am now trying to see how big files can fsock upload?
Posted: Wed Oct 18, 2006 5:36 am
by raghavan20
The target script..
Code: Select all
<?php
ini_set('always_populate_raw_post_data', 1);
file_put_contents( "post", file_get_contents('php://input') );
?>
Appears like this target script loads all data from POST into memory and finally writes the file with the content stored in memory but I want it to create a file as soon as it gets POST data and keep transferring to the file until all the POST data is received.
Edit:
it timesout like this..
Code: Select all
Notice: fwrite(): send of 100 bytes failed with errno=32 Broken pipe in /var/www/fsock.php on line 28
Notice: fwrite(): send of 100 bytes failed with errno=32 Broken pipe in /var/www/fsock.php on line 28
Notice: fwrite(): send of 100 bytes failed with errno=32 Broken pipe in /var/www/fsock.php on line 28
Notice: fwrite(): send of 100 bytes failed with errno=32 Broken pipe in /var/www/fsock.php on line 28
Notice: fwrite(): send of 100 bytes failed with errno=32 Broken pipe in /var/www/fsock.php on line 28
Notice: fwrite(): send of 100 bytes failed with errno=32 Broken pipe in /var/www/fsock.php on line 28
Notice: fwrite(): send of 28 bytes failed with errno=32 Broken pipe in /var/www/fsock.php on line 28
Any ideas?