Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » U++ Library support » U++ Library : Other (not classified elsewhere) » Filtering streams for bz2
icon4.gif  Filtering streams for bz2 [message #49312] Wed, 24 January 2018 20:28 Go to next message
Novo is currently offline  Novo
Messages: 890
Registered: December 2006
Experienced Contributor
I've implemented bz2::CompressStream and bz2::DecompressStream. Source code is attached.
Could you please add them to the Upp code?

UPD: My DecompressStream can read multistream bz2 files like this one

By default it will read all streams, but it would be nice to be able to allow DecompressStream/InFilterStream to read just one logical stream. The problem is that Filter is an Event (unlike More) and it doesn't allow to pass info about "end of stream" from Filter to InFilterStream the way More does that.
  • Attachment: bz2.zip
    (Size: 3.31KB, Downloaded 51 times)


Regards,
Novo

[Updated on: Thu, 25 January 2018 05:29]

Report message to a moderator

Re: Filtering streams for bz2 [message #49333 is a reply to message #49312] Sun, 28 January 2018 10:54 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 12022
Registered: November 2005
Ultimate Member
Thanks, in trunk now.
Re: Filtering streams for bz2 [message #49344 is a reply to message #49312] Tue, 30 January 2018 02:54 Go to previous messageGo to next message
Novo is currently offline  Novo
Messages: 890
Registered: December 2006
Experienced Contributor
Thank you!

I've attached another patch which slightly changes InFilterStream and replaces type of InFilterStream::Filter (as I wrote in my previous message).

Could you please apply it as well?
  • Attachment: bz2.02.zip
    (Size: 22.15KB, Downloaded 53 times)


Regards,
Novo
Re: Filtering streams for bz2 [message #49348 is a reply to message #49344] Tue, 30 January 2018 11:46 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 12022
Registered: November 2005
Ultimate Member
I am not particulary happy about changing the interface, but I can see the reason to signal eof in InFilterStream.

However, I am completely puzzled why have you changed OutFilterStream. AFAIK, all those new bool's there are ignored?

Mirek
Re: Filtering streams for bz2 [message #49350 is a reply to message #49348] Tue, 30 January 2018 14:24 Go to previous messageGo to next message
Novo is currently offline  Novo
Messages: 890
Registered: December 2006
Experienced Contributor
mirek wrote on Tue, 30 January 2018 05:46

However, I am completely puzzled why have you changed OutFilterStream. AFAIK, all those new bool's there are ignored?

Code wouldn't compile otherwise. I'm getting an error "void function 'Execute' should not return a value" (Clang) in the method OutFilterStream::Set "Filter = callback<F, F, const void *, int>(&filter, &F::Put);"

"bool Put(const void *ptr, int size)" is symmetric (used by both OutFilterStream and InFilterStream).
So, either interface of filter object should be redesigned (separate methods Put and Get), or OutFilterStream::Filter should be also be Gate.



Regards,
Novo

[Updated on: Tue, 30 January 2018 17:13]

Report message to a moderator

Re: Filtering streams for bz2 [message #49428 is a reply to message #49350] Fri, 09 February 2018 09:54 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 12022
Registered: November 2005
Ultimate Member
Novo wrote on Tue, 30 January 2018 14:24
mirek wrote on Tue, 30 January 2018 05:46

However, I am completely puzzled why have you changed OutFilterStream. AFAIK, all those new bool's there are ignored?

Code wouldn't compile otherwise. I'm getting an error "void function 'Execute' should not return a value" (Clang) in the method OutFilterStream::Set "Filter = callback<F, F, const void *, int>(&filter, &F::Put);"

"bool Put(const void *ptr, int size)" is symmetric (used by both OutFilterStream and InFilterStream).
So, either interface of filter object should be redesigned (separate methods Put and Get), or OutFilterStream::Filter should be also be Gate.



Well, I really dislike changing so many things because of this.

I have tried to resolve the issue with Gate<> FilterEof in InFilterStream. That should work. Please check trunk.

BUT, after further thinking, I believe even that is overkill. In fact, I now believe that it should work without adding anything: After encountering BZ_STREAM_END, no more data are decompressed, so no more output is produced and we get nice eof via:

int InFilterStream::_Term()
{
	while(ptr == rdlim && !eof)
		Fetch();
	return ptr == rdlim ? -1 : *ptr;
}



(because after Fetch, ptr == rdlime)

What do you think?

Mirek

Re: Filtering streams for bz2 [message #49448 is a reply to message #49428] Sun, 11 February 2018 21:22 Go to previous messageGo to next message
Novo is currently offline  Novo
Messages: 890
Registered: December 2006
Experienced Contributor
mirek wrote on Fri, 09 February 2018 03:54


I have tried to resolve the issue with Gate<> FilterEof in InFilterStream. That should work. Please check trunk.

Solution with FilterEof works. Thank you.

mirek wrote on Fri, 09 February 2018 03:54

BUT, after further thinking, I believe even that is overkill. In fact, I now believe that it should work without adding anything: After encountering BZ_STREAM_END, no more data are decompressed, so no more output is produced and we get nice eof via:

int InFilterStream::_Term()
{
	while(ptr == rdlim && !eof)
		Fetch();
	return ptr == rdlim ? -1 : *ptr;
}



(because after Fetch, ptr == rdlime)

What do you think?

Mirek


Setting up of FilterEof cannot be skipped because in Fetch we have this:
	Stream::buffer = ptr = buffer.begin();
	rdlim = buffer.end();

And ptr != rdlim.

Below is my test.
#include <Core/Core.h>
#include <plugin/bz2/bz2.h>

using namespace Upp;

CONSOLE_APP_MAIN
{
	enum { BUF_SIZE = 65536 };
	const Vector<String>& cmdline = CommandLine();
	if (cmdline.GetCount() < 2)
		return;
	FileIn in(cmdline[0]);
	FileOut out(cmdline[1]);
#if 0
	bz2::DecompressStream bz2(in);
	char buff[BUF_SIZE];
	while (!bz2.IsEof()) {
		const int n = bz2.Get(buff, BUF_SIZE);
		out.Put(buff, n);
	}
#endif
#if 1
	// Read only a first stream.
	bz2::DecompressStream bz2(in, false);
	char buff[BUF_SIZE];
	while (!bz2.IsEof()) {
		const int n = bz2.Get(buff, BUF_SIZE);
		out.Put(buff, n);
	}
#endif
#if 0
	bz2::CompressStream bz2(out);
	char buff[BUF_SIZE];
	while (!in.IsEof()) {
		const int n = in.Get(buff, BUF_SIZE);
		bz2.Put(buff, n);
	}
#endif
}

Current version of code in svn/git works fine.
Thank you again!


Regards,
Novo

[Updated on: Sun, 11 February 2018 21:27]

Report message to a moderator

Re: Filtering streams for bz2 [message #49449 is a reply to message #49448] Mon, 12 February 2018 02:10 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 12022
Registered: November 2005
Ultimate Member
I believe: If Filter does not produce any output, then buffer.SetCount(0); holds and begin == end... Actually, this also needs to work with partially filled buffer.

Anyway, can you post me an example .bz2 to test it with?
Re: Filtering streams for bz2 [message #49452 is a reply to message #49449] Mon, 12 February 2018 14:26 Go to previous messageGo to next message
Novo is currently offline  Novo
Messages: 890
Registered: December 2006
Experienced Contributor
mirek wrote on Sun, 11 February 2018 20:10

Anyway, can you post me an example .bz2 to test it with?

I'm using this file. It is quite big (14.5G). First stream has size 2.8K.
There is another file, which contains offsets of streams.


Regards,
Novo

[Updated on: Tue, 13 February 2018 16:53]

Report message to a moderator

Re: Filtering streams for bz2 [message #49474 is a reply to message #49452] Thu, 15 February 2018 20:40 Go to previous message
mirek is currently offline  mirek
Messages: 12022
Registered: November 2005
Ultimate Member
OK, I have indentified the issue, but I am not sure that fixing that would work in all cases so I am staying with current solution for now.

(post edited to remove incorrect claims)

Mirek

[Updated on: Fri, 16 February 2018 08:36]

Report message to a moderator

Previous Topic: libpng on linux
Next Topic: RegExp: newlines in replacement string
Goto Forum:
  


Current Time: Sun Sep 22 20:23:21 CEST 2019

Total time taken to generate the page: 0.01304 seconds