Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » U++ Library support » U++ Core » Upp::CopyStream reports wrong size.
Upp::CopyStream reports wrong size. [message #50201] Sat, 25 August 2018 21:39 Go to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello,

While I've tried to use the CopyStream variant in SSH/SFtp, I found out that it is not reporting the final size.

This:
int64 CopyStream(Stream& dest, Stream& src, int64 count, Gate<int64, int64> progress)
{
	int block = (int)min<int64>(count, 32768);
	Buffer<byte> temp(block);
	int loaded;
	int64 done = 0;
	int64 total = count;
	while(count > 0 && (loaded = src.Get(~temp, (int)min<int64>(count, block))) > 0) {
		if(progress(done, total)) // <-- This doesn't report the final size.
			return -1;        //
		dest.Put(~temp, loaded); 
		count -= loaded;
		done += loaded;
	}
	return done;
}


Should be (?):

int64 CopyStream(Stream& dest, Stream& src, int64 count, Gate<int64, int64> progress)
{
	int block = (int)min<int64>(count, 32768);
	Buffer<byte> temp(block);
	int loaded;
	int64 done = 0;
	int64 total = count;
	while(count > 0 && (loaded = src.Get(~temp, (int)min<int64>(count, block))) > 0) {
	        dest.Put(~temp, loaded);
		count -= loaded;
		done += loaded;
	        if(progress(done, total)) // <-- This works.
			return -1;
	
	}
	return done;
}


Best regards,
Oblivion


Re: Upp::CopyStream reports wrong size. [message #50214 is a reply to message #50201] Mon, 27 August 2018 18:08 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
Good point.

I think moving Put after progress might be a bit better:

int64 CopyStream(Stream& dest, Stream& src, int64 count, Gate<int64, int64> progress)
{
	int block = (int)min<int64>(count, 32768);
	Buffer<byte> temp(block);
	int loaded;
	int64 done = 0;
	int64 total = count;
	while(count > 0 && (loaded = src.Get(~temp, (int)min<int64>(count, block))) > 0) {
		count -= loaded;
		done += loaded;
		if(progress(done, total))
			return -1;
		dest.Put(~temp, loaded);
	}
	return done;
}


what do you think? (it will reach 100% sooner, but in situation where you close progress immediately after finish it will be at 100% for some time).
Re: Upp::CopyStream reports wrong size. [message #50215 is a reply to message #50214] Mon, 27 August 2018 18:58 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Edit: On further thinking, I remove the error checking line since the CopyStream returns the bytes actually read.

Quote:
what do you think? (it will reach 100% sooner, but in situation where you close progress immediately after finish it will be at 100% for some time).


Well, I think it might not be a big problem if the "dest" stream is on local machine, but for example, in SFtp::SaveFile(), where the destination stream is a remote file system object, the call might block for a long time or indefinitely (or fail/timeout), and this may be confusing for the client. Also, errors on destination stream should better be checked too. Not to mention the existing two CopyStreams have, for some reason, different block sizes (32 and 64 k). So here is my proposal:

int64 CopyStream(Stream& dest, Stream& src, int64 count, Gate<int64, int64> progress, int chunk_size)
{
	int block = (int)min<int64>(count, chunk_size);
	Buffer<byte> temp(block);
	int loaded;
	int64 done = 0;
	int64 total = count;
	while(count > 0 && (loaded = src.Get(~temp, (int)min<int64>(count, block))) > 0) {
		count -= loaded;
		done += loaded;
		if(progress(done, total))
			return -1;
		dest.Put(~temp, loaded);
	}
	return done;
}

int64 CopyStream(Stream& dest, Stream& src, int64 count)
{
	return CopyStream(dest, src, count, Null, 65536);
}

int64 CopyStream(Stream& dest, Stream& src, int64 count, Gate<int64, int64> progress)
{
	return CopyStream(dest, src, count, pick(progress), 65536);
}


Would this be ok?
In this way I can re-use this code in SFtp too.


Best regards,
Oblivion.


[Updated on: Mon, 27 August 2018 20:35]

Report message to a moderator

Re: Upp::CopyStream reports wrong size. [message #50217 is a reply to message #50214] Mon, 27 August 2018 20:36 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
By the way, why not just use GetPos()? Doesn't it work for all streams? If it works the error checking I suggested (previously) can be added.

E.g.
int64 CopyStream(Stream& dest, Stream& src, int64 count, Gate<int64, int64> progress, int chunk_size)
{
	int block = (int)min<int64>(count, chunk_size);
	Buffer<byte> temp(block);
	int loaded;
	int64 done = 0;
	int64 total = count;
	while(count > 0 && (loaded = src.Get(~temp, (int)min<int64>(count, block))) > 0) {
		dest.Put(~temp, loaded);
		int64 pos = dest.GetPos();
		if(pos > done) { // at least one byte is written to dest.
			count -= pos;
			done  =  pos;
			if(progress(done, total))
				return -1;
		}
		if(dest.IsError())
			break;
	}
	return done;
}




[Updated on: Mon, 27 August 2018 21:31]

Report message to a moderator

Re: Upp::CopyStream reports wrong size. [message #50219 is a reply to message #50215] Tue, 28 August 2018 09:46 Go to previous message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
OK, sounds good.

GetPos does not need to work for all streams.
Previous Topic: Folder for configuration file
Next Topic: DEPRECATED flag
Goto Forum:
  


Current Time: Fri Mar 29 16:54:38 CET 2024

Total time taken to generate the page: 0.01014 seconds