Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » Community » Newbie corner » Problem loading very large data files (The system hangs up when loading data from large files.)
Problem loading very large data files [message #61445] Thu, 13 February 2025 22:45 Go to next message
DMStevenson is currently offline  DMStevenson
Messages: 6
Registered: January 2025
Location: Wisconsin
Promising Member
I am still new, and am having problems loading data from very large files (>35MB) into Vectors. I know this works in other languages, and being new I might be missing something obvious. With smaller data files the load time into Vectors is quite fast, but with very large files the system hangs up, and will simply not proceed (no error message). Has anyone had any experience with this?
Re: Problem loading very large data files [message #61448 is a reply to message #61445] Fri, 14 February 2025 09:45 Go to previous messageGo to next message
koldo is currently offline  koldo
Messages: 3437
Registered: August 2008
Senior Veteran
Dear DMStevenson

What you are talking about is unusual. I use vectors of very large sizes on a daily basis with great efficiency.
Please include a small example code and tell us the basic characteristics of your computer and operating system.
This will make it easier for us to find the cause of the problem.


Best regards
Iñaki
Re: Problem loading very large data files [message #61452 is a reply to message #61448] Fri, 14 February 2025 16:46 Go to previous messageGo to next message
DMStevenson is currently offline  DMStevenson
Messages: 6
Registered: January 2025
Location: Wisconsin
Promising Member
Thanks for the reply Koldo. As a Newbie I continue to learn quickly, and have found that, over time, my large files do load, but they simply do not appear in a DocEdit for a while. For the code below I expected the contents of a file to appear line-by-line. Instead it seems to completely fill a buffer and *then* is posted to the DocEdit. Am I correct in this? My aim is *NOT* to fill a DocEdit with a huge amount of text, but I was doing this just as a programming step... Thanks, DMStevenson

#include <CtrlLib/CtrlLib.h>

using namespace Upp;

struct WindowType : TopWindow {
Button button;
String FileName;
DocEdit memo;
Progress progress1;
Vector<String> DataLines;

void ReadFile(String file_name) {
String line;
FileIn infile;
if (infile.Open(file_name)) {
while(!infile.IsEof()) {
line = infile.GetLine();
memo.Append(line);
memo.Append("\n");
DataLines.Add(line);
}
}
infile.Close();
}

WindowType()
{
Title("File Read");
button << [=] {
FileName = SelectFileOpen("All Files\t*.*");
ReadFile(FileName); };
button.SetLabel("Get File");
Add(button.LeftPos(10, 60).TopPos(10, 30));
Add(memo.LeftPos(10, 680).TopPos(50, 430));
}
};

GUI_APP_MAIN
{
WindowType app;
app.SetRect(0,0,700,500);
app.Run();
}
Re: Problem loading very large data files [message #61458 is a reply to message #61452] Sat, 15 February 2025 16:14 Go to previous messageGo to next message
koldo is currently offline  koldo
Messages: 3437
Registered: August 2008
Senior Veteran
Hi DMStevenson

You haven't done badly, but I'm including your modified code to be more similar to how I would have done it.
(remember to use the Forum {...} option to include code)

#include <CtrlLib/CtrlLib.h>

using namespace Upp;

struct WindowType : TopWindow {
	Button button;
	String FileName;
	DocEdit memo;
	//Progress progress1;
	//Vector<String> DataLines;
	
	void ReadFile(String file_name) {
		String line;
		FileIn infile;
		if (infile.Open(file_name)) //{
			memo.Load(infile);
			//while(!infile.IsEof()) {
			//	line = infile.GetLine();
			//	memo.Append(line);
			//	memo.Append("\n");
			//	DataLines.Add(line);
			//}
		//}
		//infile.Close();
	}
	
	WindowType()
	{
		Title("File Read");
		button << [=] {
			FileName = SelectFileOpen("All Files\t*.*");
			ReadFile(FileName); };
		button.SetLabel("Get File");
		Add(button.LeftPos(10, 60).TopPos(10, 30));
		Add(memo.LeftPos(10, 680).TopPos(50, 430));
	}
};

GUI_APP_MAIN
{
	WindowType app;
	app.SetRect(0,0,700,500);
	app.Run();
}


Best regards
Iñaki
Re: Problem loading very large data files [message #61463 is a reply to message #61458] Mon, 17 February 2025 20:45 Go to previous messageGo to next message
DMStevenson is currently offline  DMStevenson
Messages: 6
Registered: January 2025
Location: Wisconsin
Promising Member
Thanks for the reply! You modified code is helpful. I have a follow-up question, mostly out of academic interest. Is there a way to simply modify the above code to 'post' each line individually? That is, post the line to the DocEdit before retrieving another line (As I am used to in Fortran or Pascal). Of course I could use command line, or rewrite the code, but is there a simple modification that could be added to the existing code?
Re: Problem loading very large data files [message #61464 is a reply to message #61463] Tue, 18 February 2025 08:14 Go to previous messageGo to next message
koldo is currently offline  koldo
Messages: 3437
Registered: August 2008
Senior Veteran
Hi DMStevenson

In that case you can simply use Insert() or Append(). This is slower than doing a Load().


Best regards
Iñaki
Re: Problem loading very large data files [message #61465 is a reply to message #61463] Tue, 18 February 2025 08:14 Go to previous message
Oblivion is currently offline  Oblivion
Messages: 1211
Registered: August 2007
Senior Contributor
Hello DMStevenson,

Quote:
That is, post the line to the DocEdit before retrieving another line




		if (infile.Open(file_name)) {
			while(!infile.IsEof()) {
				line = infile.GetLine();
                                memo.Append(line + "\n");
                                // Do some stuff...
			}
		}


This posts (appends) the line to DocEdit before retrieving the next line. Then you can do some other stuff in the loop if you need to.

Best regards,
Oblivion


[Updated on: Tue, 18 February 2025 08:15]

Report message to a moderator

Previous Topic: Getting Started U++ Framework examples?
Next Topic: Just a simple hello u++
Goto Forum:
  


Current Time: Fri Jun 13 22:35:39 CEST 2025

Total time taken to generate the page: 0.04048 seconds