U++ forum: Welcome to the forum

Re: Large data ahead [message #51127 is a reply to message #51126]

Wed, 30 January 2019 15:07

Tom1
Messages: 1301
Registered: March 2007

Ultimate Contributor

mirek wrote on Wed, 30 January 2019 13:09

Tom1 wrote on Wed, 30 January 2019 11:50
OK, I see. I agree it is a stupid thing to waste RAM for storing a whole lot of zeros. Efficiency and speed must be considered in every step along the way and this is one of those steps.

As you pointed out, this is really a "99.999% of 32 bit and 0.001 % of 64 bit" -type of situation, so maybe I should first consider using Buffer<T> instead of Vector<T>. This may be beneficial from multiple points of view after all...

Thanks for your insight.

BR, Tom

Well, all that said, I am 100% OK with HugeVector (and then most of other containers) with int64 GetCount()...

OK, Thanks Mirek. I will keep this in mind. The other points of view above include the fact that when loading huge data sets to RAM (near physical memory limits), re-allocating the buffer when Vector grows is a problem. Therefore, an accurately pre-calculated memory block size and a single allocation of that block really pays off -- both in processing time and in memory efficiency. Then Buffer<T> is just about the right choice.

Thanks and best regards,

Tom

Report message to a moderator