Home » U++ Library support » U++ Library : Other (not classified elsewhere) » Filtering streams for bz2
Filtering streams for bz2 [message #49312] |
Wed, 24 January 2018 20:28 |
Novo
Messages: 1358 Registered: December 2006
|
Ultimate Contributor |
|
|
I've implemented bz2::CompressStream and bz2::DecompressStream. Source code is attached.
Could you please add them to the Upp code?
UPD: My DecompressStream can read multistream bz2 files like this one
By default it will read all streams, but it would be nice to be able to allow DecompressStream/InFilterStream to read just one logical stream. The problem is that Filter is an Event (unlike More) and it doesn't allow to pass info about "end of stream" from Filter to InFilterStream the way More does that.
-
Attachment: bz2.zip
(Size: 3.31KB, Downloaded 255 times)
Regards,
Novo
[Updated on: Thu, 25 January 2018 05:29] Report message to a moderator
|
|
|
|
|
|
Re: Filtering streams for bz2 [message #49350 is a reply to message #49348] |
Tue, 30 January 2018 14:24 |
Novo
Messages: 1358 Registered: December 2006
|
Ultimate Contributor |
|
|
mirek wrote on Tue, 30 January 2018 05:46
However, I am completely puzzled why have you changed OutFilterStream. AFAIK, all those new bool's there are ignored?
Code wouldn't compile otherwise. I'm getting an error "void function 'Execute' should not return a value" (Clang) in the method OutFilterStream::Set "Filter = callback<F, F, const void *, int>(&filter, &F::Put);"
"bool Put(const void *ptr, int size)" is symmetric (used by both OutFilterStream and InFilterStream).
So, either interface of filter object should be redesigned (separate methods Put and Get), or OutFilterStream::Filter should be also be Gate.
Regards,
Novo
[Updated on: Tue, 30 January 2018 17:13] Report message to a moderator
|
|
|
Re: Filtering streams for bz2 [message #49428 is a reply to message #49350] |
Fri, 09 February 2018 09:54 |
|
mirek
Messages: 13975 Registered: November 2005
|
Ultimate Member |
|
|
Novo wrote on Tue, 30 January 2018 14:24mirek wrote on Tue, 30 January 2018 05:46
However, I am completely puzzled why have you changed OutFilterStream. AFAIK, all those new bool's there are ignored?
Code wouldn't compile otherwise. I'm getting an error "void function 'Execute' should not return a value" (Clang) in the method OutFilterStream::Set "Filter = callback<F, F, const void *, int>(&filter, &F::Put);"
"bool Put(const void *ptr, int size)" is symmetric (used by both OutFilterStream and InFilterStream).
So, either interface of filter object should be redesigned (separate methods Put and Get), or OutFilterStream::Filter should be also be Gate.
Well, I really dislike changing so many things because of this.
I have tried to resolve the issue with Gate<> FilterEof in InFilterStream. That should work. Please check trunk.
BUT, after further thinking, I believe even that is overkill. In fact, I now believe that it should work without adding anything: After encountering BZ_STREAM_END, no more data are decompressed, so no more output is produced and we get nice eof via:
int InFilterStream::_Term()
{
while(ptr == rdlim && !eof)
Fetch();
return ptr == rdlim ? -1 : *ptr;
}
(because after Fetch, ptr == rdlime)
What do you think?
Mirek
|
|
|
Re: Filtering streams for bz2 [message #49448 is a reply to message #49428] |
Sun, 11 February 2018 21:22 |
Novo
Messages: 1358 Registered: December 2006
|
Ultimate Contributor |
|
|
mirek wrote on Fri, 09 February 2018 03:54
I have tried to resolve the issue with Gate<> FilterEof in InFilterStream. That should work. Please check trunk.
Solution with FilterEof works. Thank you.
mirek wrote on Fri, 09 February 2018 03:54
BUT, after further thinking, I believe even that is overkill. In fact, I now believe that it should work without adding anything: After encountering BZ_STREAM_END, no more data are decompressed, so no more output is produced and we get nice eof via:
int InFilterStream::_Term()
{
while(ptr == rdlim && !eof)
Fetch();
return ptr == rdlim ? -1 : *ptr;
}
(because after Fetch, ptr == rdlime)
What do you think?
Mirek
Setting up of FilterEof cannot be skipped because in Fetch we have this:
Stream::buffer = ptr = buffer.begin();
rdlim = buffer.end();
And ptr != rdlim.
Below is my test.
#include <Core/Core.h>
#include <plugin/bz2/bz2.h>
using namespace Upp;
CONSOLE_APP_MAIN
{
enum { BUF_SIZE = 65536 };
const Vector<String>& cmdline = CommandLine();
if (cmdline.GetCount() < 2)
return;
FileIn in(cmdline[0]);
FileOut out(cmdline[1]);
#if 0
bz2::DecompressStream bz2(in);
char buff[BUF_SIZE];
while (!bz2.IsEof()) {
const int n = bz2.Get(buff, BUF_SIZE);
out.Put(buff, n);
}
#endif
#if 1
// Read only a first stream.
bz2::DecompressStream bz2(in, false);
char buff[BUF_SIZE];
while (!bz2.IsEof()) {
const int n = bz2.Get(buff, BUF_SIZE);
out.Put(buff, n);
}
#endif
#if 0
bz2::CompressStream bz2(out);
char buff[BUF_SIZE];
while (!in.IsEof()) {
const int n = in.Get(buff, BUF_SIZE);
bz2.Put(buff, n);
}
#endif
}
Current version of code in svn/git works fine.
Thank you again!
Regards,
Novo
[Updated on: Sun, 11 February 2018 21:27] Report message to a moderator
|
|
|
|
|
|
Goto Forum:
Current Time: Fri Apr 26 00:31:08 CEST 2024
Total time taken to generate the page: 0.03725 seconds
|