Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » U++ Library support » ArrayCtrl, HeaderCtrl & GridCtrl » Excessive memory use
Excessive memory use [message #35636] Fri, 09 March 2012 05:10 Go to next message
nixnixnix is currently offline  nixnixnix
Messages: 415
Registered: February 2007
Location: Kelowna, British Columbia
Senior Member
Hi Daniel,

I love GridCtrl and use it in many many places in my code. I am currently writing a data processing app in which I want the users to be able to look at multiple data sets and then process these datasets and be able to step back and forth through the steps being able to see each data set at each step. The data sets might take up 20MB each in ASCII format and I might have 12 or 20 loaded at once and then want to take them through 10 or more steps.

My concern is that when I load just three 20MB ASCII data files and display them on GridCtrl, my memory usage grows to 2.9GB of which I am pretty sure the majority is GridCtrl.

My question is: are there obvious things that I might be doing wrong or any tips that you have for conserving memory in GridCtrl when displaying large datasets?

[One of my 20MB datasets might be 4 years of 10 minute data with 30 fields per record. I need to write my software to be able to cope with much much more though.]

Cheers,

Nick




Re: Excessive memory use [message #35638 is a reply to message #35636] Fri, 09 March 2012 05:51 Go to previous messageGo to next message
Novo is currently offline  Novo
Messages: 1358
Registered: December 2006
Ultimate Contributor
ArrayCtrl can behave as a virtual array (it doesn't store data in the control itself in this case). GridCtrl doesn't seem to have this feature. So, you need to sacrifice some nice features of GridCtrl to save memory with ArrayCtrl. Smile

It is a common design pattern of UPP to store data within controls. And it work very well if you do not have a lot of data.


Regards,
Novo
Re: Excessive memory use [message #35643 is a reply to message #35638] Fri, 09 March 2012 14:22 Go to previous messageGo to next message
unodgs is currently offline  unodgs
Messages: 1366
Registered: November 2005
Location: Poland
Ultimate Contributor

Novo is right. GridCtrl doesn't support virtual rows. They are relatively easy to add but I have no time to do this. All I can do right now is to add it on my todo list Smile
Besides I think that displaying large data sets is useless. In most cases user is interested in a few first rows (instead of scrolling down user will sort or filter columns). Large sets of data have also another disadvantage. When you move a scroll thumb content of grid is scrolling very fast and you very often skip some data (I assume that scrolling is used to browse data) In my applications I simply use pagination. This is not built in feature of gridctrl but it's quite easy to implement.
It doesn't mean I'm against adding virtual rows/columns. Certainly there are cases when it fits in. Right now I have lot of other work to finish.

[Updated on: Fri, 09 March 2012 14:24]

Report message to a moderator

Re: Excessive memory use [message #35657 is a reply to message #35638] Sun, 11 March 2012 05:34 Go to previous messageGo to next message
nixnixnix is currently offline  nixnixnix
Messages: 415
Registered: February 2007
Location: Kelowna, British Columbia
Senior Member
Thanks. ArrayCtrl isn't as pretty but it will do what I need.

Daniel, you're right about the awkwardness of scrolling through tens of thousands of lines of data. One of the other things I need to code is an interface that has various ways of navigating the data. The users need to be able to see the data and edit it and so it does need to be in a grid. It's a very niche application. Up till now they have been using a mixture of Excel spreadsheets and macros - the entire industry does this.

ArrayCtrl question: even though I SetCount() it still takes a minute or more to allocate the memory for 150,0000 rows on a i7 980X with 24GB of 1600Mhz DDR3. Is there a way to disable update until I have finished adding the data?

Cheers,

Nick








Re: Excessive memory use [message #35660 is a reply to message #35657] Sun, 11 March 2012 15:27 Go to previous messageGo to next message
Novo is currently offline  Novo
Messages: 1358
Registered: December 2006
Ultimate Contributor
IMHO, you need to call SetVirtualCount(). My app WordNet 3.0 browser is an example of using this technique. It is basically a database browser. I'm not sure that a table, I'm trying to scroll, has 150K rows, but it should be close to that. ArrayCtrl initialization code snippet is below.

	warray.AddRowNumColumn("lemma", 80).SetConvert(*Number2Lexeme);
	// Set count.
	{
		String sql;
		Sql stmt(db_session);
		
		sql  = " SELECT max(wordid)";
		sql << " FROM words";

		stmt.SetStatement(sql);
		
		stmt.RunX();
		if (stmt.Fetch())
			warray.SetVirtualCount(stmt[0]);
	}


Regards,
Novo
Re: Excessive memory use [message #35682 is a reply to message #35660] Mon, 12 March 2012 22:44 Go to previous messageGo to next message
nixnixnix is currently offline  nixnixnix
Messages: 415
Registered: February 2007
Location: Kelowna, British Columbia
Senior Member
Hi Novo,

SetVirtualCount doesn't appear to change anything. It still takes a minute or more to load up the data from my data structures into the ArrayCtrl one row at a time using ArrayCtrl::Set(int row, Vector<Value> vals); Is there a quicker way please? I can't see anything in the popup list of functions.

Cheers,

Nick
Re: Excessive memory use [message #35683 is a reply to message #35682] Mon, 12 March 2012 22:52 Go to previous messageGo to next message
Novo is currently offline  Novo
Messages: 1358
Registered: December 2006
Ultimate Contributor
nixnixnix wrote on Mon, 12 March 2012 17:44

Hi Novo,

SetVirtualCount doesn't appear to change anything. It still takes a minute or more to load up the data from my data structures into the ArrayCtrl one row at a time using ArrayCtrl::Set(int row, Vector<Value> vals); Is there a quicker way please? I can't see anything in the popup list of functions.

Cheers,

Nick



Smile The secret of virtual array is that you do not need to load data into it. Smile

You need to do two things:
1) Set number of virtual rows by calling SetVirtualCount().
2) Set converters for your columns, which will convert row number into data you want to display.

Take a closer look at
	warray.AddRowNumColumn("lemma", 80).SetConvert(*Number2Lexeme);


There is an app VirtualArray in the "reference" assembly.


Regards,
Novo

[Updated on: Mon, 12 March 2012 22:53]

Report message to a moderator

Re: Excessive memory use [message #35684 is a reply to message #35683] Mon, 12 March 2012 23:08 Go to previous messageGo to next message
nixnixnix is currently offline  nixnixnix
Messages: 415
Registered: February 2007
Location: Kelowna, British Columbia
Senior Member
You just literally blew my mind Shocked

4 seconds to load 170k+ records Cool


However, when I go to edit a value, all the values in the row show as the row number and not the value that should be there. Is there a good way to deal with this? I noticed that your app doesn't do in place editing and of course the reference app doesn't either.

Nick


[Updated on: Tue, 13 March 2012 00:44]

Report message to a moderator

Re: Excessive memory use [message #35685 is a reply to message #35684] Tue, 13 March 2012 00:56 Go to previous messageGo to next message
nixnixnix is currently offline  nixnixnix
Messages: 415
Registered: February 2007
Location: Kelowna, British Columbia
Senior Member
index.php?t=getfile&id=3671&private=0Hey Daniel,

My client is mostly interested in being able to examine flags in the data so I colour the flags and then use the panel at the right hand side to display the entire data set and whilst mousemove over that a popup window shows the data above and below the point that the mouse is over in the entire dataset along with the central section of the popup showing what will be on displayed on the main grid if you currently click in the right hand pane.

Just thought I'd share in case these ideas can be useful to others here.

Nick

[Updated on: Tue, 13 March 2012 00:59]

Report message to a moderator

Re: Excessive memory use [message #35687 is a reply to message #35684] Tue, 13 March 2012 04:49 Go to previous messageGo to next message
Novo is currently offline  Novo
Messages: 1358
Registered: December 2006
Ultimate Contributor
nixnixnix wrote on Mon, 12 March 2012 18:08


However, when I go to edit a value, all the values in the row show as the row number and not the value that should be there. Is there a good way to deal with this? I noticed that your app doesn't do in place editing and of course the reference app doesn't either.

Nick



Well, I made a small experiment with EditField and Convert.
I tried to overwrite Format() and Scan() in Convert and set this new converter on EditField. Now EditField can display correct text instead of a number. Scan() seems to be a right place to store edited value.


Regards,
Novo

[Updated on: Tue, 13 March 2012 04:50]

Report message to a moderator

icon14.gif  Re: Excessive memory use [message #35688 is a reply to message #35687] Tue, 13 March 2012 08:04 Go to previous message
nixnixnix is currently offline  nixnixnix
Messages: 415
Registered: February 2007
Location: Kelowna, British Columbia
Senior Member
Thanks Novo,

Worked a treat. My app can now load up 70MB of ASCII and display it using only 145MB of memory and the files load in 3 seconds and can be edited in place and I don't even have to retrieve the data from the Array Ctrl. Fantastic! Thanks for all your help.

Cheers,

Nick
Previous Topic: Customising GridCtrl Search
Next Topic: GridCtrl: how to set label of an Option
Goto Forum:
  


Current Time: Thu Mar 28 20:44:51 CET 2024

Total time taken to generate the page: 0.01039 seconds