Home » U++ Library support » U++ Core » Upp::CopyStream reports wrong size.
Upp::CopyStream reports wrong size. [message #50201] |
Sat, 25 August 2018 21:39 |
Oblivion
Messages: 1094 Registered: August 2007
|
Senior Contributor |
|
|
Hello,
While I've tried to use the CopyStream variant in SSH/SFtp, I found out that it is not reporting the final size.
This:
int64 CopyStream(Stream& dest, Stream& src, int64 count, Gate<int64, int64> progress)
{
int block = (int)min<int64>(count, 32768);
Buffer<byte> temp(block);
int loaded;
int64 done = 0;
int64 total = count;
while(count > 0 && (loaded = src.Get(~temp, (int)min<int64>(count, block))) > 0) {
if(progress(done, total)) // <-- This doesn't report the final size.
return -1; //
dest.Put(~temp, loaded);
count -= loaded;
done += loaded;
}
return done;
}
Should be (?):
int64 CopyStream(Stream& dest, Stream& src, int64 count, Gate<int64, int64> progress)
{
int block = (int)min<int64>(count, 32768);
Buffer<byte> temp(block);
int loaded;
int64 done = 0;
int64 total = count;
while(count > 0 && (loaded = src.Get(~temp, (int)min<int64>(count, block))) > 0) {
dest.Put(~temp, loaded);
count -= loaded;
done += loaded;
if(progress(done, total)) // <-- This works.
return -1;
}
return done;
}
Best regards,
Oblivion
Github page: https://github.com/ismail-yilmaz
upp-components: https://github.com/ismail-yilmaz/upp-components
Bobcat the terminal emulator: https://github.com/ismail-yilmaz/Bobcat
|
|
|
Re: Upp::CopyStream reports wrong size. [message #50214 is a reply to message #50201] |
Mon, 27 August 2018 18:08 |
|
mirek
Messages: 13984 Registered: November 2005
|
Ultimate Member |
|
|
Good point.
I think moving Put after progress might be a bit better:
int64 CopyStream(Stream& dest, Stream& src, int64 count, Gate<int64, int64> progress)
{
int block = (int)min<int64>(count, 32768);
Buffer<byte> temp(block);
int loaded;
int64 done = 0;
int64 total = count;
while(count > 0 && (loaded = src.Get(~temp, (int)min<int64>(count, block))) > 0) {
count -= loaded;
done += loaded;
if(progress(done, total))
return -1;
dest.Put(~temp, loaded);
}
return done;
}
what do you think? (it will reach 100% sooner, but in situation where you close progress immediately after finish it will be at 100% for some time).
|
|
|
Re: Upp::CopyStream reports wrong size. [message #50215 is a reply to message #50214] |
Mon, 27 August 2018 18:58 |
Oblivion
Messages: 1094 Registered: August 2007
|
Senior Contributor |
|
|
Edit: On further thinking, I remove the error checking line since the CopyStream returns the bytes actually read.
Quote:what do you think? (it will reach 100% sooner, but in situation where you close progress immediately after finish it will be at 100% for some time).
Well, I think it might not be a big problem if the "dest" stream is on local machine, but for example, in SFtp::SaveFile(), where the destination stream is a remote file system object, the call might block for a long time or indefinitely (or fail/timeout), and this may be confusing for the client. Also, errors on destination stream should better be checked too. Not to mention the existing two CopyStreams have, for some reason, different block sizes (32 and 64 k). So here is my proposal:
int64 CopyStream(Stream& dest, Stream& src, int64 count, Gate<int64, int64> progress, int chunk_size)
{
int block = (int)min<int64>(count, chunk_size);
Buffer<byte> temp(block);
int loaded;
int64 done = 0;
int64 total = count;
while(count > 0 && (loaded = src.Get(~temp, (int)min<int64>(count, block))) > 0) {
count -= loaded;
done += loaded;
if(progress(done, total))
return -1;
dest.Put(~temp, loaded);
}
return done;
}
int64 CopyStream(Stream& dest, Stream& src, int64 count)
{
return CopyStream(dest, src, count, Null, 65536);
}
int64 CopyStream(Stream& dest, Stream& src, int64 count, Gate<int64, int64> progress)
{
return CopyStream(dest, src, count, pick(progress), 65536);
}
Would this be ok?
In this way I can re-use this code in SFtp too.
Best regards,
Oblivion.
Github page: https://github.com/ismail-yilmaz
upp-components: https://github.com/ismail-yilmaz/upp-components
Bobcat the terminal emulator: https://github.com/ismail-yilmaz/Bobcat
[Updated on: Mon, 27 August 2018 20:35] Report message to a moderator
|
|
|
Re: Upp::CopyStream reports wrong size. [message #50217 is a reply to message #50214] |
Mon, 27 August 2018 20:36 |
Oblivion
Messages: 1094 Registered: August 2007
|
Senior Contributor |
|
|
By the way, why not just use GetPos()? Doesn't it work for all streams? If it works the error checking I suggested (previously) can be added.
E.g.
int64 CopyStream(Stream& dest, Stream& src, int64 count, Gate<int64, int64> progress, int chunk_size)
{
int block = (int)min<int64>(count, chunk_size);
Buffer<byte> temp(block);
int loaded;
int64 done = 0;
int64 total = count;
while(count > 0 && (loaded = src.Get(~temp, (int)min<int64>(count, block))) > 0) {
dest.Put(~temp, loaded);
int64 pos = dest.GetPos();
if(pos > done) { // at least one byte is written to dest.
count -= pos;
done = pos;
if(progress(done, total))
return -1;
}
if(dest.IsError())
break;
}
return done;
}
Github page: https://github.com/ismail-yilmaz
upp-components: https://github.com/ismail-yilmaz/upp-components
Bobcat the terminal emulator: https://github.com/ismail-yilmaz/Bobcat
[Updated on: Mon, 27 August 2018 21:31] Report message to a moderator
|
|
|
|
Goto Forum:
Current Time: Sat Jun 08 07:33:27 CEST 2024
Total time taken to generate the page: 0.01901 seconds
|