U++ forum: Welcome to the forum

Search on this site

Search in forums

Home » U++ Library support » U++ Core » How to read file into Memory

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

How to read file into Memory [message #12113]

Thu, 11 October 2007 14:05

spidertp
Messages: 16
Registered: October 2007
Location: Poland

Promising Member

Hello everybody,
I need to work with big files (> 100MB) and I like to load part of them to memory in a thread, process data in main app and read next part if needed.

I have 2 tables.
At first I load both of them with 10MB of binary data (so it gives me 20MB in memory). First I proccess data from table1 and when index goes to table2, I'm starting a thread to load new data to table1 and so on. Basically, in main app I want to process data without interrupts due to harddisk access.

The problem is that I have a lot of exceptions because of defining tables with new operator and deleting them.

I'm reading contents of file with code:

object * table1 = new object[10*1024*1024]; // 10MB
FileIn in(filename);
in.Get(table1, sizeof(table1));

After processing I make:

delete table1;

And... here I have a lot of problems with heap.

Can I load a file to memory in a different method than new/delete to some UPP container?

Please help.
Best regards
Tomek

Report message to a moderator

Re: How to read file into Memory [message #12124 is a reply to message #12113]

Fri, 12 October 2007 00:02

mirek
Messages: 14271
Registered: November 2005

Ultimate Member

spidertp wrote on Thu, 11 October 2007 08:05

object * table1 = new object[10*1024*1024]; // 10MB
FileIn in(filename);
in.Get(table1, sizeof(table1));

After processing I make:

delete table1;

And... here I have a lot of problems with heap.

Well, using new / delete is not wise, anyway it should work, what problems exactly do you have?

Quote:

Can I load a file to memory in a different method than new/delete to some UPP container?

Well, I would use Buffer:

Buffer<object> table1(10*1024*1024);

BTW, are you aware that the allocation will most likely be sizeof(object) * 10MB (not only 10MB, if sizeof(object) > 1)?

Mirek

Report message to a moderator

Re: How to read file into Memory [message #12129 is a reply to message #12124]

Fri, 12 October 2007 08:08

spidertp
Messages: 16
Registered: October 2007
Location: Poland

Promising Member

Thank you Mirek for your answer.

Quote:

Well, I would use Buffer:

Buffer<object> table1(10*1024*1024);

I read Container Tutorial but not "Overview of U++ containers - NTL" and I was not aware of Buffer container.

Quote:

BTW, are you aware that the allocation will most likely be sizeof(object) * 10MB (not only 10MB, if sizeof(object) > 1)?

Well, I'm dividing 10MB by sizeof(object) and then creating table with number of objects which can fit inside.

Quote:

Well, using new / delete is not wise, anyway it should work, what problems exactly do you have?

I think I had problems with delete operator because I used

delete

and not

delete[]

operator.
I had exceptions in FreeDebug function.

Thank you or your help and keep doing good work on UPP!
Best regards
Tomasz

Ah, I have one more question about debugging - how can I see specific object in a table or vector?

For example:

struct MyStruct
{
float table[100];
String text;
}

main
{
...
MyStruct myTable[100];
Vector<MyStruct> myVector[100];

myVector[99].table[1] = 15.5;
...
}

In debugging mode, in Explorer I write: