Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » U++ Library support » U++ MT-multithreading and servers » [SOLVED]Use multiple HttpRequest instance (>32) to upload files will stuck
[SOLVED]Use multiple HttpRequest instance (>32) to upload files will stuck [message #43425] Tue, 29 July 2014 05:21 Go to next message
kasome is currently offline  kasome
Messages: 78
Registered: July 2008
Location: Taiwan
Member
Hi,

I found that when using multiple thread (>32) and each thread has one HttpRequest instance to upload files will lead to stuck in its member function HttpRequest::Dns,

the reason is as follows:

In upp\uppsrc\Core\Inet.h

class IpAddrInfo {
	enum { COUNT = 32 };
	struct Entry {
		const char *host;
		const char *port;
		int         family;
		int         status;
		addrinfo   *addr;
	};
	static Entry     pool[COUNT];
	
	..............................
};


The size of variable IpAddrInfo::Entry::pool is COUNT (= 32),
and when each HttpRequest instance upload one file,
it will call IpAddrInfo::Start to retrive one entry from the IpAddrInfo::Entry::pool,
but it did not release the entry back to IpAddrInfo::Entry::pool when the uploading file operation was done or fail,
then another HttpRequest instance will stuck in its member function HttpRequest::Dns,
tring to retrive one available entry from the IpAddrInfo::Entry::pool,

void HttpRequest::Dns()
{
	for(int i = 0; i <= Nvl(GetTimeout(), INT_MAX); i++) {
		if(!addrinfo.InProgress()) {
			StartConnect();
			return;
		}
		Sleep(1);
	}
}


So, i made some modification, and the whole file is uploaded as attechment, hope that helps.

Modify from upp r7550
bool HttpRequest::Do()
{
	int c1, c2;
	switch(phase) {
	case BEGIN:
		retry_count = 0;
		redirect_count = 0;
		start_time = msecs();
		GlobalTimeout(timeout);
	case START:
		Start();
		break;
	case DNS:
		Dns();
		break;
	case SSLPROXYREQUEST:
		if(SendingData())
			break;
		StartPhase(SSLPROXYRESPONSE);
		break;
	case SSLPROXYRESPONSE:
		if(ReadingHeader())
			break;
		ProcessSSLProxyResponse();
		break;
	case SSLHANDSHAKE:
		if(SSLHandshake())
			break;
		StartRequest();
		break;
	case REQUEST:
		if(SendingData())
			break;
		StartPhase(HEADER);
		break;
	case HEADER:
		if(ReadingHeader())
			break;
		StartBody();
		break;
	case BODY:
		if(ReadingBody())
			break;
		Finish();
		break;
	case CHUNK_HEADER:
		ReadingChunkHeader();
		break;
	case CHUNK_BODY:
		if(ReadingBody())
			break;
		c1 = TcpSocket::Get();
		c2 = TcpSocket::Get();
		if(c1 != '\r' || c2 != '\n')
			HttpError("missing ending CRLF in chunked transfer");
		StartPhase(CHUNK_HEADER);
		break;
	case TRAILER:
		if(ReadingHeader())
			break;
		header.ParseAdd(data);
		Finish();
		break;
	case FINISHED:
	case FAILED:
		WhenDo();
		return false;
	default:
		NEVER();
	}

	if(phase != FAILED) {
		if(IsSocketError() || IsError())
			phase = FAILED;
		else
		if(msecs(start_time) >= timeout) {
			HttpError("connection timed out");
			phase = FAILED;
		}
		else
		if(IsAbort()) {
			HttpError("connection was aborted");
			phase = FAILED;
		}
	}
	
	if(phase == FAILED) {
		if(retry_count++ < max_retries) {
			LLOGS("HTTP retry on error " << GetErrorDesc());
			start_time = msecs();
			GlobalTimeout(timeout);
			StartPhase(START);
		}
	}
	WhenDo();
	return phase != FINISHED && phase != FAILED;
}


to
bool HttpRequest::Do()
{
	int c1, c2;
	switch(phase) {
	case BEGIN:
		retry_count = 0;
		redirect_count = 0;
		start_time = msecs();
		GlobalTimeout(timeout);
	case START:
		Start();
		break;
	case DNS:
		Dns();
		break;
	case SSLPROXYREQUEST:
		if(SendingData())
			break;
		StartPhase(SSLPROXYRESPONSE);
		break;
	case SSLPROXYRESPONSE:
		if(ReadingHeader())
			break;
		ProcessSSLProxyResponse();
		break;
	case SSLHANDSHAKE:
		if(SSLHandshake())
			break;
		StartRequest();
		break;
	case REQUEST:
		if(SendingData())
			break;
		StartPhase(HEADER);
		break;
	case HEADER:
		if(ReadingHeader())
			break;
		StartBody();
		break;
	case BODY:
		if(ReadingBody())
			break;
		Finish();
		break;
	case CHUNK_HEADER:
		ReadingChunkHeader();
		break;
	case CHUNK_BODY:
		if(ReadingBody())
			break;
		c1 = TcpSocket::Get();
		c2 = TcpSocket::Get();
		if(c1 != '\r' || c2 != '\n')
			HttpError("missing ending CRLF in chunked transfer");
		StartPhase(CHUNK_HEADER);
		break;
	case TRAILER:
		if(ReadingHeader())
			break;
		header.ParseAdd(data);
		Finish();
		break;
	case FINISHED:
	case FAILED:
		WhenDo();
		return false;
	default:
		NEVER();
	}

	if(phase != FAILED) {
		if(IsSocketError() || IsError())
			phase = FAILED;
		else
		if(msecs(start_time) >= timeout) {
			HttpError("connection timed out");
			phase = FAILED;
		}
		else
		if(IsAbort()) {
			HttpError("connection was aborted");
			phase = FAILED;
		}
	}
	
	if(phase == FAILED) {
		if(retry_count++ < max_retries) {
			LLOGS("HTTP retry on error " << GetErrorDesc());
			start_time = msecs();
			GlobalTimeout(timeout);
			StartPhase(START);
		}
	}

if( phase == FINISHED || phase == FAILED ) {
addrinfo.Clear();
}
WhenDo();
	return phase != FINISHED && phase != FAILED;
}
  • Attachment: Http.cpp
    (Size: 19.01KB, Downloaded 322 times)

[Updated on: Mon, 11 August 2014 14:04]

Report message to a moderator

Re: Use multiple HttpRequest instance (>32) to upload files will stuck [message #43459 is a reply to message #43425] Wed, 06 August 2014 19:48 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 14255
Registered: November 2005
Ultimate Member
Perhaps it would rather be possible to release the slot by ::Clear call after connection is established in HttpRequest::StartConnect() ?

Report message to a moderator

Re: Use multiple HttpRequest instance (>32) to upload files will stuck [message #43463 is a reply to message #43459] Thu, 07 August 2014 14:02 Go to previous messageGo to next message
kasome is currently offline  kasome
Messages: 78
Registered: July 2008
Location: Taiwan
Member
That's better way, thanks. Very Happy

Report message to a moderator

Re: Use multiple HttpRequest instance (>32) to upload files will stuck [message #43477 is a reply to message #43463] Sat, 09 August 2014 10:00 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 14255
Registered: November 2005
Ultimate Member
kasome wrote on Thu, 07 August 2014 14:02
That's better way, thanks. Very Happy


Change committed into trunk, can you please provide the final confirmation?

Mirek

Report message to a moderator

Re: Use multiple HttpRequest instance (>32) to upload files will stuck [message #43481 is a reply to message #43477] Sat, 09 August 2014 19:22 Go to previous message
kasome is currently offline  kasome
Messages: 78
Registered: July 2008
Location: Taiwan
Member
Hello Mirek,

I try to upload 500 files by using 64 threads, and each thread has one Upp::HttpRequest instance, each file has size range from 1 MB to 150MB,

I have to say that it work perfectly. Laughing


PS:
Here is the 2 web storage i try to upload files and succeed. (by using Upp::HttpRequest)

Amazon S3
https://aws.amazon.com/s3/

ASUS WebStorage
https://www.asuswebstorage.com/navigate/

[Updated on: Mon, 11 August 2014 14:32]

Report message to a moderator

Previous Topic: [SOLVED][FeatureRequests]Use HttpRequest to upload large file
Next Topic: Implementing a crash handler
Goto Forum:
  

[ Syndicate this forum (XML) ] [ RSS ] [ PDF ]

Current Time: Tue Apr 29 11:13:09 CEST 2025

Total time taken to generate the page: 0.03915 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.5.
Copyright ©2001-2013 FUDforum Bulletin Board Software