Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » U++ Library support » U++ MT-multithreading and servers » [SOLVED][FeatureRequests]Use HttpRequest to upload large file
[SOLVED][FeatureRequests]Use HttpRequest to upload large file [message #40788] Tue, 17 September 2013 06:06 Go to next message
kasome is currently offline  kasome
Messages: 78
Registered: July 2008
Location: Taiwan
Member
Hello,

class HttpRequest can be used to upload file by using Post member function, e.g.

HttpRequest httpRequest;
Upp::int64 offset = 0;
Upp::String uploadFile = "test.mkv";
httpRequest.SSL(true).Url( someUrl ).ClearPost().ClearHeaders().Post( Upp::LoadFile(uploadFile).Mid(offset) ).Execute();


But when the file is large, larger than 1GB, for example, than HttpRequest will need large memory to upload by using Upp::LoadFile

Is there possible to add feature to HttpRequest, something like:
HttpRequest httpRequest;
Upp::int64 offset = 0;
Upp::String uploadFile = "test.mkv";
Upp::FileIn in( uploadFile );
in.Seek(offset);

httpRequest.SSL(true).Url( someUrl ).ClearPost().ClearHeaders().Post( in ).Execute();

Thanks.

[Updated on: Mon, 28 July 2014 13:46]

Report message to a moderator

Re: [FeatureRequests]Use HttpRequest to upload large file [message #40824 is a reply to message #40788] Sat, 21 September 2013 18:56 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
Definitely a good idea, added to RM. Might take a while to implement though... (contributions welcome).

Mirek
Re: [FeatureRequests]Use HttpRequest to upload large file [message #40834 is a reply to message #40824] Sun, 22 September 2013 03:37 Go to previous messageGo to next message
kasome is currently offline  kasome
Messages: 78
Registered: July 2008
Location: Taiwan
Member
I see. and Thanks. Smile
Re: [FeatureRequests]Use HttpRequest to upload large file [message #43373 is a reply to message #40834] Mon, 14 July 2014 10:08 Go to previous messageGo to next message
kasome is currently offline  kasome
Messages: 78
Registered: July 2008
Location: Taiwan
Member
hi,

Inorder to upload large file, I make some modification to HttpRequest to do so.

Here is the usage:

Upp::int64 uploadFileSourceStartPosition = 1000;
Upp::String uploadFileSourcePath = "TestVideo.mkv";
_httpRequest.ClearPost().PUT().SSL( true ).Url( hostName ).Path( hostPath ).ClearHeaders().Header( "Content-Length", contentLength ).KeepAlive( true ).ContentType( contentType ).PostDataStream( uploadFileSourcePath, uploadFileSourceStartPosition ).Execute();
//_httpRequest.ClearPost().PUT().SSL( true ).Url( hostName ).Path( hostPath ).ClearHeaders().Header( "Content-Length", contentLength ).KeepAlive( true ).ContentType( contentType ).PostData( Upp::LoadFile(uploadFileSourcePath).Mid(uploadFileSourceStar tPosition) ).Execute();


The total modification is as follows:

1. upp\uppsrc\Core\Inet.h

Original (version:7373)
class HttpRequest : public TcpSocket {
public:
	....................................
	HttpRequest&  PostData(const String& pd)              { postdata = pd; return *this; }
	....................................
	HttpRequest&  ClearPost()                             { PostData(Null); multipart.Clear(); GET(); return *this; }
	....................................
};


Modified
class HttpRequest : public TcpSocket {
	....................................
	int64       postdataPos;		// Added
	Upp::String postdataName;		// Added
	
	....................................
	bool		 SendingStream();	// Added
public:
	....................................
	HttpRequest&  PostData(const String& pd)              						{ postdata = pd; return *this; }
	HttpRequest&  PostDataStream(const String& pdn, const int64 pos = 0)        { postdataName = pdn; postdataPos = pos; return *this; }	// Added
	
	....................................
//	HttpRequest&  ClearPost()                             { PostData(Null); multipart.Clear(); GET(); return *this; }	// Delete
	HttpRequest&  ClearPost()                             { PostData(Null); PostDataStream(Null); multipart.Clear(); GET(); return *this; }	// Added

	....................................
};



2. upp\uppsrc\Core\Http.cpp

Original (version:7373)
bool HttpRequest::Do()
{
	....................................
	case REQUEST:
		if(SendingData())
			break;
		StartPhase(HEADER);
		break;
	....................................
}

void HttpRequest::StartRequest()
{
	....................................
	if(method == METHOD_GET || method == METHOD_HEAD)
		pd.Clear();
	....................................
			if(pd.GetCount() || method == METHOD_POST || method == METHOD_PUT)
			data << "Content-Length: " << pd.GetCount() << "\r\n";
	....................................
}

bool HttpRequest::SendingData()
{
	for(;;) {
		int n = min(2048, data.GetLength() - (int)count);
		n = TcpSocket::Put(~data + count, n);
		if(n == 0)
			break;
		count += n;
	}
	return count < data.GetLength();
}



Modified
bool HttpRequest::Do()
{
	....................................
	case REQUEST:
		if( IsNull(postdataName) ) {
			if(SendingData())
				break;
		}
		else {
			if(SendingStream())
				break;
		}
		StartPhase(HEADER);
		break;
	....................................
}

void HttpRequest::StartRequest()
{
	....................................
	if(method == METHOD_GET || method == METHOD_HEAD){
		pd.Clear();
		postdataName.Clear();
	}
	....................................
			if((IsNull(postdataName)? pd.GetCount() : Upp::GetFileLength(postdataName)-postdataPos) || method == METHOD_POST || method == METHOD_PUT)
				data << "Content-Length: " << (IsNull(postdataName)? pd.GetCount() : Upp::GetFileLength(postdataName)-postdataPos) << "\r\n";
	....................................
}

bool HttpRequest::SendingData()
{
	for(;;) {
		int n = min(2048, data.GetLength() - (int)count);
		n = TcpSocket::Put(~data + count, n);
		if(n == 0)
			break;
		count += n;
	}
	return count < data.GetLength();
}

bool HttpRequest::SendingStream() {
	Upp::FileIn in( postdataName );
	in.Seek( postdataPos );	

	int bufferSize = 2048 - (data.GetLength() & 2047);	

	Upp::StringBuffer buffer( bufferSize );
	int readingSize = in.Get( buffer, bufferSize );
	int64 postdataSize = in.GetSize() - postdataPos - readingSize;

	data << Upp::String( ~buffer, readingSize );

	for(;;) {
		int n = min(2048, data.GetLength() - (int)count);
		if( data.GetLength() - (int)count < 0 ) {
			int kk = 0;
		}
		n = TcpSocket::Put(~data + count, n);
		if(n == 0)
			break;
		count += n;
	}

	if( data.GetLength() == count ) {
		Upp::StringBuffer buffer( 2048 );
		for(;;) {
			int n = min((int64)2048, (int64)data.GetLength() + postdataSize - (int64)count);
			int readingSize = in.Get( buffer, n );
			n = TcpSocket::Put(~buffer, readingSize);
			if(n == 0)
				break;
			count += n;
		}
	}
	return count < data.GetLength() + postdataSize;
}



Hope that helps.
Re: [FeatureRequests]Use HttpRequest to upload large file [message #43381 is a reply to message #43373] Wed, 16 July 2014 07:39 Go to previous messageGo to next message
kasome is currently offline  kasome
Messages: 78
Registered: July 2008
Location: Taiwan
Member
Add some error handling to prevent the program from crashing.

bool HttpRequest::SendingStream() {
	Upp::FileIn in( postdataName );
	in.Seek( postdataPos );

	int readingSize = 0;
	Upp::String dataPadding = data;
	if( dataPadding.GetLength() & 2047 ) {
		int bufferSize = 2048 - (dataPadding.GetLength() & 2047);

		Upp::StringBuffer buffer( bufferSize );
		readingSize = in.Get( buffer, bufferSize );
		dataPadding << Upp::String( ~buffer, readingSize );
	}
	int64 postdataSize = in.GetSize() - postdataPos - readingSize;

	if( count == 0 ) {
		for(;;) {
			int n = min(2048, dataPadding.GetLength() - (int)count);
			n = TcpSocket::Put(~dataPadding + count, n);
			if(n == 0)
				break;
			count += n;
		}

		if( (count == dataPadding.GetLength()) && postdataSize ) {
			Upp::StringBuffer buffer( 2048 );
			for(;;) {
				int n = min((int64)2048, (int64)dataPadding.GetLength() + postdataSize - (int64)count);
				int readingSize = in.Get(buffer, n);
				n = TcpSocket::Put(~buffer, readingSize);
				if(n == 0)
					break;
				count += n;
			}
		}
	}
	else {
		if( count < dataPadding.GetLength() ) {
			for(;;) {
				int n = min(2048, dataPadding.GetLength() - (int)count);
				n = TcpSocket::Put(~dataPadding + count, n);
				if(n == 0)
					break;
				count += n;
			}
		}
		if( (count >= dataPadding.GetLength()) && postdataSize ) {
			Upp::StringBuffer buffer( 2048 );
			in.Seek( postdataPos + readingSize + (count - dataPadding.GetLength()) );
			for(;;) {
				int n = min((int64)2048, (int64)dataPadding.GetLength() + postdataSize - (int64)count);
				int readingSize = in.Get(buffer, n);
				n = TcpSocket::Put(~buffer, readingSize);
				if(n == 0)
					break;
				count += n;
			}
		}
	}
	return count < dataPadding.GetLength() + postdataSize;
}
Re: [FeatureRequests]Use HttpRequest to upload large file [message #43383 is a reply to message #43381] Wed, 16 July 2014 08:39 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
Would it be possible to post the whole file here as attachment, please? Much easier to compare it with existing one...

Mirek
Re: [FeatureRequests]Use HttpRequest to upload large file [message #43384 is a reply to message #43383] Wed, 16 July 2014 09:01 Go to previous messageGo to next message
kasome is currently offline  kasome
Messages: 78
Registered: July 2008
Location: Taiwan
Member
Sorry. The whole file is post as attachment now. (modified from the original upp version 7373)

  • Attachment: Inet.h
    (Size: 22.78KB, Downloaded 242 times)
  • Attachment: Http.cpp
    (Size: 20.18KB, Downloaded 298 times)
Re: [FeatureRequests]Use HttpRequest to upload large file [message #43419 is a reply to message #43384] Mon, 28 July 2014 12:27 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
I think this should be more flexible (not coupled to filesystem), something like

HttpRequest& PostStream(Stream& s, int64 len = Null);

I would also like to ask what is the deal with all that padding code?

Mirek
Re: [FeatureRequests]Use HttpRequest to upload large file [message #43420 is a reply to message #43419] Mon, 28 July 2014 13:17 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
I have tried that altered variant, please check it that is OK.

Mirek
Re: [FeatureRequests]Use HttpRequest to upload large file [message #43421 is a reply to message #43419] Mon, 28 July 2014 13:45 Go to previous message
kasome is currently offline  kasome
Messages: 78
Registered: July 2008
Location: Taiwan
Member
Hi, Mirek

Quote:

I think this should be more flexible (not coupled to filesystem), something like

HttpRequest& PostStream(Stream& s, int64 len = Null);

Great! That is better than mine. Cool


Quote:
I would also like to ask what is the deal with all that padding code?

I am just try to make the two different part of the data combined as if they are the same continuous data, but i am really not sure if it is necessary. Embarassed


Quote:
I have tried that altered variant, please check it that is OK.

I will check it, and thanks a lot. Razz
Previous Topic: MT and variables simple question
Next Topic: [SOLVED]Use multiple HttpRequest instance (>32) to upload files will stuck
Goto Forum:
  


Current Time: Thu Mar 28 10:51:32 CET 2024

Total time taken to generate the page: 0.01225 seconds