curl request to upload a minimum of 100MB file

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
User avatar
raghavan20
DevNet Resident
Posts: 1451
Joined: Sat Jun 11, 2005 6:57 am
Location: London, UK
Contact:

curl request to upload a minimum of 100MB file

Post by raghavan20 »

i have a CURL script to upload to a site a listing file which is in a range of 100MB on an average. When I tried to upload as POST request and tried to get the file as file_get_contents(), it managed to get around 2MB which is well below the expected limit so I need some solution to upload larger files. The reason I am using CURL is because the target server is using HTTP authentication only.

current dynamic settings

Code: Select all

set_time_limit(0);
ini_set( "max_input_time", 0 );

memory reported by PHP

Code: Select all

memory_get_usage()
82648

php.ini

Code: Select all

memory_limit = 32M      ; Maximum amount of memory a script may consume (8MB)

1. I need your opinion on how to optimise PHP and HTTPD settings. Our server is running Fedora Core 5.
2. Are there are any better ways to upload large files like 100 MB ones?

I really appreciate any help.
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

If you have access, FTP it. I have no idea how much memory cURL will need to upload a file off hand.
User avatar
raghavan20
DevNet Resident
Posts: 1451
Joined: Sat Jun 11, 2005 6:57 am
Location: London, UK
Contact:

Post by raghavan20 »

feyd, the system supports HTTP authentication and HTTP POST upload so I am afraid I cannot use FTP.
Is there is anyway like instead of storing file contents in a string variable is it possible to send N bytes by N bytes until the end of the file.
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

You could build an alterate using fsockopen() and send the same headers cURL would have, then do a piece-meal upload of the file into that stream.
User avatar
raghavan20
DevNet Resident
Posts: 1451
Joined: Sat Jun 11, 2005 6:57 am
Location: London, UK
Contact:

Post by raghavan20 »

that is the same i am thinking about..i will give it a try
it is a pity that cURL cannot do it!
User avatar
raghavan20
DevNet Resident
Posts: 1451
Joined: Sat Jun 11, 2005 6:57 am
Location: London, UK
Contact:

Post by raghavan20 »

i have a doubt, normally most of the target system expect content-length, should it be in characters or in bytes, because if we use strlen() then we have to get content in string variable which the system cannot handle.
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

Bytes. use filesize().
User avatar
raghavan20
DevNet Resident
Posts: 1451
Joined: Sat Jun 11, 2005 6:57 am
Location: London, UK
Contact:

Post by raghavan20 »

thanks feyd!

any idea why this is not working...in the sense, POST data is empty?


sending data by POST

Code: Select all

<?php

	$fileName = "replication.txt";
	$fileContents = file_get_contents( $fileName );
	##echo "\n\n\nfile contents: ".$fileContents."\n\n\n";
	$host = "***";
	
	
	$header = "Host: $host\r\n";
	$header .= "User-Agent: PHP Script\r\n";
	$header .= "Content-Type: text/xml\r\n";
	$header .= "Content-Length: ".filesize( $fileName )."\r\n";
	$header .= "Connection: close\r\n\r\n";
	
	
	$fp = fsockopen( $host, 80, $errno, $errstr );
	
	$service_uri = "/curlTest/readsock.php";
	fputs($fp, "POST $service_uri  HTTP/1.1\r\n");
   	fputs($fp, $header);	
   	
   	$newFP = fopen( $fileName, "r" );
   	
   	while( !feof( $newFP ) ){
   		
   		$temp = fread( $newFP, 100 );
   		echo "\n\ntempData:\n".$temp;
		fwrite( $fp, $temp );
		
	}
	
	echo "\nError number: ".$errno;
	echo "\nError string: ".$errstr;
   	
	echo "\nReturn response: ".fread( $fp, 5000 )."\n";
	
	
	
	
?>


target script

Code: Select all

<?php



print_r( $_GET );
echo "\n";

print_R( $_POST );
echo "\n";

file_put_contents( "post", var_export( $_POST, TRUE).var_export( get_headers($url), TRUE ) );

?>

response

Code: Select all

Suse:/var/www # /opt/lampp/bin/php fsock.php


tempData:
replication:
involves one master and many slaves

master chooses which databases to use for gener

tempData:
ate binary logs


slave has settings about master name, ip, user, password
and the database whic

tempData:
h needs replicated
and also specify the tables to be ignored in each database
also can specify ret

tempData:
ry time

slave has two threads
1. gets data from binary log of server into relay log
2. reads th

tempData:
e relay log and executes it

the slave thread communicates with server asking about some database

tempData:
log with the last read position
the other read also has a marker to know which position it last rea

tempData:
d from the relay log
Error number: 0
Error string:
Return response: HTTP/1.1 200 OK
Date: Tue, 17 Oct 2006 15:22:52 GMT
Server: Apache/2.0.54 (Fedora)
X-Powered-By: PHP/5.0.4
Content-Length: 22
Connection: close
Content-Type: text/html

Array
(
)

Array
(
)
contents written to file named 'post'

Code: Select all

array (
)false
Last edited by raghavan20 on Tue Oct 17, 2006 10:54 am, edited 1 time in total.
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

File submissions end up in $_FILES and are multipart messages. That I remember, posted field data comes in similarly to how the query string looks:

Code: Select all

fieldName1=value2&fieldname2=value2
User avatar
raghavan20
DevNet Resident
Posts: 1451
Joined: Sat Jun 11, 2005 6:57 am
Location: London, UK
Contact:

Post by raghavan20 »

I understand what you are saying.
I am working with implementing an API, the target company obtain all the input as just POST data not like

postField=postValue
or
HTTP upload

so I guess there should be a way to receive the POST data even if it not an array.
choppsta
Forum Contributor
Posts: 114
Joined: Thu Jul 03, 2003 11:11 am

Post by choppsta »

If I understand you correctly you want to get at the raw post data, not the usual urlencoded data that get's sent from an HTML form.

In your target script, try the following to output what the request you're sending looks like:

Code: Select all

ini_set('always_populate_raw_post_data', 1);

$data = apache_request_headers();
$data['POST data'] = file_get_contents('php://input');

echo '<pre>'.htmlspecialchars(print_r($data, true)).'</pre>';
User avatar
raghavan20
DevNet Resident
Posts: 1451
Joined: Sat Jun 11, 2005 6:57 am
Location: London, UK
Contact:

Post by raghavan20 »

great choppsta it worked, i am now trying to see how big files can fsock upload?
User avatar
raghavan20
DevNet Resident
Posts: 1451
Joined: Sat Jun 11, 2005 6:57 am
Location: London, UK
Contact:

Post by raghavan20 »

The target script..

Code: Select all

<?php

ini_set('always_populate_raw_post_data', 1); 

file_put_contents( "post", file_get_contents('php://input') ); 

?>
Appears like this target script loads all data from POST into memory and finally writes the file with the content stored in memory but I want it to create a file as soon as it gets POST data and keep transferring to the file until all the POST data is received.

Edit:
it timesout like this..

Code: Select all

Notice: fwrite(): send of 100 bytes failed with errno=32 Broken pipe in /var/www/fsock.php on line 28

Notice: fwrite(): send of 100 bytes failed with errno=32 Broken pipe in /var/www/fsock.php on line 28

Notice: fwrite(): send of 100 bytes failed with errno=32 Broken pipe in /var/www/fsock.php on line 28

Notice: fwrite(): send of 100 bytes failed with errno=32 Broken pipe in /var/www/fsock.php on line 28

Notice: fwrite(): send of 100 bytes failed with errno=32 Broken pipe in /var/www/fsock.php on line 28

Notice: fwrite(): send of 100 bytes failed with errno=32 Broken pipe in /var/www/fsock.php on line 28

Notice: fwrite(): send of 28 bytes failed with errno=32 Broken pipe in /var/www/fsock.php on line 28
Any ideas?
Post Reply