Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » Developing U++ » U++ Developers corner » theide with libclang
Re: theide with libclang [message #58909 is a reply to message #58579] Sat, 24 September 2022 08:44 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
OK, so one week in, I guess I owe the community a bit of explanation about libclang and the way theide uses it...

So, obviously what libclang does is parsing the files and giving info about them. Additionally, it can provide autocomplete lists at specified point in the source files, but NOT HEADERS (and also not inside templates, but that is store for another time).

The important thing to know is that parsing the file with all headers is long: on my not so slow machine, it can take up to several seconds to parse the file. To remedy this, libclang has a concept of "preamble" and "reparsing". Preamble is basically automated precompiled header - if file starts with multiple headers, preamble is creating and when after the first parse the file is reparsed or autocomplete requested, preamble is used to greatly speedup the process. With preamble, autocomplete (after the first parse) is almost immediate. That said, premable creation is sort of fragile. Also, normally it gets created in system temp directory, which would be fine unless sometimes libclang does not delete it. But it can be solved be setting env variable which we do now.

So what we do now is the when you start editing some file, in the background thread the first parse of the file is started with a request to create preamble. In theide, you can know that this is happening by left bar going "sort of orange" for a couple of seconds - it does that each time file is parsed or reparsed. If you then change the file, it gets (after some time of inactivity) reparsed again. Parsing current file gets it ready for autocomplete, but also provides information about file for file annotations, current file navigation and "jumps" (Alt-J / Ctrl clicks).

Originally, there was just one parsed file, which meant each time after switching to another file it had to have that initial a couple of seconds parsing period. Later I have added a cache, there are now several files with parsed info. Downside is that each "parsing unit" consumes a lot of memory, about 300MB so with 12 entries in the cache, you can do the math....

Include files that clang is unable to process: What we do is that we find "master file" - the well defined file that includes the header and we pretend that we are parsing that one. It is not so simple: we still want to use preamble to make things fast and if we just parsed master file and hoped to catch the info include file from it, it would not work with preambles as our include file would actually be in preamble. I have spent a lot of time trying to solve this: What we actually do now is that we create a syntetic file content that actually manually does including so that it all works (it is hard to explain in detail; in assist diagnostic mode you can see the content of the syntetic file with Assist/Current parsed file content).

So that is about all I can say about "current file" without going into gruesome details; if you want to study the code, libclang/CurrentFile.cpp and ide/AssistTrick.cpp are good starting points.

Next troublesome part of code is "Indexer" - that is something that scans the whole sources and creates a program database of, well, everything everywhere. That is needed for Navigator and Alt+J command. We are using blitz(like) process there and we are running that in multiple threads. Now there is "Scheduler" (Indexer::SchedulerThread()) thread that scans for changed files and basically prepares jobs for "Indexing" (Indexer::IndexerThread) threads. Once again, each of indexer threads can eat up to 300MB of memory, so finetuning default values will take some time. More indexer threads mean faster reindexing but perhaps too much memory...

In terms of slowdown/freezing during reindexing, well, the plan was to give indexer threads really low priority. It should be possible with

clang_CXIndex_setGlobalOptions(clang.index, CXGlobalOpt_ThreadBackgroundPriorityForIndexing);

but maybe this does not work as expected in Linux. In Windows, it seems to work fine.

Now for testing and troubleshooting, please note that there is "Assist diagnostics" option in Assist/Debug. When active, it will start reporting in Console what it is doing and also there are new entries at the end of assist menu, showing a dump of actual index, errors libclang reported during parsing and what was given to clang to parse as current file.

So if e.g. there is no autocomplete for include files, please check "Current file parse errors" and "Current parsed file content".

I know this is a big change (and one I was extremly hesitant to do) and not everything works smoothly at this point, but hopefully all will be finetuned in time. I guess we fixed a lot already in the first week....

Mirek

[Updated on: Sat, 24 September 2022 08:47]

Report message to a moderator

Re: theide with libclang [message #58910 is a reply to message #58909] Sat, 24 September 2022 09:53 Go to previous messageGo to next message
jjacksonRIAB is currently offline  jjacksonRIAB
Messages: 220
Registered: June 2011
Experienced Member
Nah, Mirek, I don't feel I'm owed an explanation Very Happy

Windows users seem to be OK, like you said it's linux that needs some love. In my case I had an ancient version of libclang 7.0 sitting in /usr/local/lib. Once I zapped that the excessive memory consumption disappeared so I'm pretty sure that was libclang. I updated to 15 and things are still looking good. I did find valgrind complain about some branches on uninitialized variables in theide itself:

==562946== Conditional jump or move depends on uninitialised value(s)
==562946==    at 0x56DB7A: Upp::CodeEditor::DirtyFrom(int) (CodeEditor.cpp:74)
==562946==    by 0x5A2DEF: Upp::TextCtrl::Insert0(int, Upp::WString const&) (Text.cpp:799)
==562946==    by 0x5A3C2E: Upp::TextCtrl::InsertU(int, Upp::WString const&, bool) (Text.cpp:935)
==562946==    by 0x5A3FB7: Upp::TextCtrl::Insert(int, Upp::WString const&, bool) (Text.cpp:980)
==562946==    by 0x315C96: Upp::TextCtrl::Insert(int, Upp::WString const&) (TextEdit.h:218)
==562946==    by 0x5A4E6B: Upp::TextCtrl::Paste(Upp::WString const&) (Text.cpp:1153)
==562946==    by 0x3075CD: Console::Append(Upp::String const&) (Console.cpp:77)
==562946==    by 0x326EDB: Console::operator<<(Upp::String const&) (ide.h:127)
==562946==    by 0x3CA52A: Ide::PutConsole(char const*) (idewin.cpp:50)
==562946==    by 0x46B05A: PutConsole(char const*) (Ide.cpp:9)
==562946==    by 0x34EC3E: AssistEditor::SyncMaster() (IncludeTrick.cpp:18)
==562946==    by 0x3536C7: AssistEditor::NewFile(bool) (Assist.cpp:438)
==562946== 
==562946== Conditional jump or move depends on uninitialised value(s)
==562946==    at 0x56D63F: Upp::CodeEditor::GetSyntax(int) (CodeEditor.cpp:29)
==562946==    by 0x56DC5B: Upp::CodeEditor::GetRefreshInfo(int) (CodeEditor.cpp:94)
==562946==    by 0x56DEFA: Upp::CodeEditor::PostInsert(int, Upp::WString const&) (CodeEditor.cpp:111)
==562946==    by 0x5A335E: Upp::TextCtrl::Insert0(int, Upp::WString const&) (Text.cpp:851)
==562946==    by 0x5A3C2E: Upp::TextCtrl::InsertU(int, Upp::WString const&, bool) (Text.cpp:935)
==562946==    by 0x5A3FB7: Upp::TextCtrl::Insert(int, Upp::WString const&, bool) (Text.cpp:980)
==562946==    by 0x315C96: Upp::TextCtrl::Insert(int, Upp::WString const&) (TextEdit.h:218)
==562946==    by 0x5A4E6B: Upp::TextCtrl::Paste(Upp::WString const&) (Text.cpp:1153)
==562946==    by 0x3075CD: Console::Append(Upp::String const&) (Console.cpp:77)
==562946==    by 0x326EDB: Console::operator<<(Upp::String const&) (ide.h:127)
==562946==    by 0x3CA52A: Ide::PutConsole(char const*) (idewin.cpp:50)
==562946==    by 0x46B05A: PutConsole(char const*) (Ide.cpp:9)


I was also able to trigger a crash of TheIDE by just holding down Ctrl+Space for a minute or so to make it continually attempt to autocomplete, put some pressure on it. Unfortunately it only happens in release mode, I cannot reproduce it in debug mode.

Re: theide with libclang [message #58911 is a reply to message #58910] Sat, 24 September 2022 10:03 Go to previous messageGo to next message
peterh is currently offline  peterh
Messages: 108
Registered: November 2018
Location: Germany
Experienced Member
Hi,

I have reported 3 GB memory in windows.
I must add, I had the IDE open for probably 2 days.
Normally I do not shut down my computer but put it to sleep at night or when I do not use it.
I have now rebooted and after opening TheIDE with the same project and having tried "Homebudget" also, it consumes only 300MB.

[Updated on: Sat, 24 September 2022 10:05]

Report message to a moderator

Re: theide with libclang [message #58912 is a reply to message #58910] Sat, 24 September 2022 10:16 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
jjacksonRIAB wrote on Sat, 24 September 2022 09:53
Nah, Mirek, I don't feel I'm owed an explanation Very Happy


It was not meant specifically for you...

I felt that general explanation of at least some inner workings will help us fix things.

In your case I remember complaining about headers. Do they work now? (If not, there are some hints in my long post where to look for troubles).

As for old libclang... I can put limit for minimal version, e.g. libclang 10 (I am quite sure that one works), what do you think?

Mirek
Re: theide with libclang [message #58915 is a reply to message #58912] Sat, 24 September 2022 10:30 Go to previous messageGo to next message
jjacksonRIAB is currently offline  jjacksonRIAB
Messages: 220
Registered: June 2011
Experienced Member
mirek wrote on Sat, 24 September 2022 10:16

It was not meant specifically for you...


Laughing Serves me right for attempting to be conciliatory.

Quote:

In your case I remember complaining about headers. Do they work now?


I'll get latest and check. But the list of outstanding issues I have:

1. Includes don't autocomplete
2. Parser gets stuck in a reparse loop if I cycle through files in a project using alt+down and hit a file that is not code (like a .lay or .iml file)
3. Alt+C, which you are already working on

Quote:

As for old libclang... I can put limit for minimal version, e.g. libclang 10 (I am quite sure that one works), what do you think?


Sounds good.

[Updated on: Sat, 24 September 2022 10:34]

Report message to a moderator

Re: theide with libclang [message #58916 is a reply to message #58912] Sat, 24 September 2022 10:45 Go to previous messageGo to next message
jjacksonRIAB is currently offline  jjacksonRIAB
Messages: 220
Registered: June 2011
Experienced Member
Still no joy on include autocomplete.
Re: theide with libclang [message #58917 is a reply to message #58912] Sat, 24 September 2022 11:10 Go to previous messageGo to next message
jjacksonRIAB is currently offline  jjacksonRIAB
Messages: 220
Registered: June 2011
Experienced Member
Mirek,

The problem appears to manifest with assist_cursor in Assist.cpp line 502.

It gathers all of the potential include files appropriately but assist_cursor never matches current pos so it returns before performing the completion.

void AssistEditor::PopUpAssist(bool auto_insert)
{
	LTIMING("PopUpAssist");
	int pos = GetCursor();
	ReadIdBackPos(pos, false);
	if(pos != assist_cursor)
		return;


EDIT It's uninitialized. The only place I can find it being set is around line 463 but there's an early return

include_assist = false;
if(IncludeAssist())
    return;

int pos = GetCursor();
ReadIdBackPos(pos, false); // libclang does not work well if file is not truncated for autocomplete (?)
assist_cursor = pos;


before assist_cursor is set. If I move that block of code after the pos set it shows the files but it has strange index numbers next to the icons. <- never mind, that appears to be for debugging purposes and doesn't appear in release mode.

So the fix is:

int pos = GetCursor();
ReadIdBackPos(pos, false); // libclang does not work well if file is not truncated for autocomplete (?)
assist_cursor = pos;

include_assist = false;
if(IncludeAssist())
    return;

[Updated on: Sat, 24 September 2022 19:43]

Report message to a moderator

Re: theide with libclang [message #58918 is a reply to message #58908] Sat, 24 September 2022 15:00 Go to previous messageGo to next message
Lance is currently offline  Lance
Messages: 526
Registered: March 2007
Contributor
mirek wrote on Sat, 24 September 2022 02:08

Is autocomplete working?


No.It is not. Is it because libclang not present in my system?


PS:
I already have clang installed. After I installed libclang-dev, key event in theide will immediately crash theide with error message
Quote:

Fatal Error!
Invalid memory access!


The libclang.so shipped with most recent(?) libclang-dev(1:14.0-55~exp2) doesn't seem to work with theide. It's most likely to be some kind of chaos in my system.


ps2: Now even after I remove'd libclang-dev, theide in upp version 16443 won't work. You don't even need to type in anything, it will crash itself with invalid memory access in just a few seconds(when background parsing threads start). It looks like a libclang version problem. For now, I will go back to upp version 16303 on which autocomplete works fine.

[Updated on: Sat, 24 September 2022 15:54]

Report message to a moderator

Re: theide with libclang [message #58920 is a reply to message #58918] Sat, 24 September 2022 19:55 Go to previous messageGo to next message
jjacksonRIAB is currently offline  jjacksonRIAB
Messages: 220
Registered: June 2011
Experienced Member
Lance wrote on Sat, 24 September 2022 15:00

ps2: Now even after I remove'd libclang-dev, theide in upp version 16443 won't work. You don't even need to type in anything, it will crash itself with invalid memory access in just a few seconds(when background parsing threads start). It looks like a libclang version problem. For now, I will go back to upp version 16303 on which autocomplete works fine.


I didn't try dev packages, I just uninstalled clang through package manager and ran

sudo bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"


to bring everything to clang-15, including libclang. Should work with debian/apt systems.

[Updated on: Sat, 24 September 2022 19:57]

Report message to a moderator

Re: theide with libclang [message #58921 is a reply to message #58917] Sat, 24 September 2022 21:44 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
jjacksonRIAB wrote on Sat, 24 September 2022 11:10
before assist_cursor is set. If I move that block of code after the pos set it shows the files but it has strange index numbers next to the icons. <- never mind, that appears to be for debugging purposes and doesn't appear in release mode.


That is actually the feature of "assist diagnostics". (Those numbers are libclang defined constants for that type of object, listing it helps in case the icon is wrong etc...)

Otherwise, thanks for spotting the bug. Does include assist work now?

Mirek
Re: theide with libclang [message #58922 is a reply to message #58921] Sat, 24 September 2022 21:45 Go to previous messageGo to next message
jjacksonRIAB is currently offline  jjacksonRIAB
Messages: 220
Registered: June 2011
Experienced Member
mirek wrote on Sat, 24 September 2022 21:44

Otherwise, thanks for spotting the bug. Does include assist work now?

Mirek


Yes, it works now.
Re: theide with libclang [message #58923 is a reply to message #58920] Sat, 24 September 2022 23:09 Go to previous messageGo to next message
jjacksonRIAB is currently offline  jjacksonRIAB
Messages: 220
Registered: June 2011
Experienced Member
This is the problem I'm currently trying to work out but not getting anywhere yet:

https://files.catbox.moe/t4kcpd.mp4

It will get stuck in a reparse loop. Same thing also happens occasionally when I'm exiting TheIDE. Note that it happens when I move rapidly to a resource-type file. I didn't capture it in the video but it behaves normally and exits the loop if I go back to a parseable .cpp or .h file, so it's not stuck that way permanently - it is just attempting to redo the same old job repeatedly until it gets a new job. When it happens on exit it becomes permanently stuck.

[Updated on: Sun, 25 September 2022 03:06]

Report message to a moderator

Re: theide with libclang [message #58924 is a reply to message #58920] Sun, 25 September 2022 02:13 Go to previous messageGo to next message
Lance is currently offline  Lance
Messages: 526
Registered: March 2007
Contributor
jjacksonRIAB wrote on Sat, 24 September 2022 13:55
Lance wrote on Sat, 24 September 2022 15:00

ps2: Now even after I remove'd libclang-dev, theide in upp version 16443 won't work. You don't even need to type in anything, it will crash itself with invalid memory access in just a few seconds(when background parsing threads start). It looks like a libclang version problem. For now, I will go back to upp version 16303 on which autocomplete works fine.


I didn't try dev packages, I just uninstalled clang through package manager and ran

sudo bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"


to bring everything to clang-15, including libclang. Should work with debian/apt systems.


Thank you, jjacksonRIAB! Your command worked fine and I was able to upgrade my libclang. Unfortunately the problem remains with UPP version 16443: theide will crash in a few seconds after a package is open with "Invalid Memory Access" error.

[Updated on: Sun, 25 September 2022 02:14]

Report message to a moderator

Re: theide with libclang [message #58925 is a reply to message #58924] Sun, 25 September 2022 13:48 Go to previous messageGo to next message
Klugier is currently offline  Klugier
Messages: 1076
Registered: September 2012
Location: Poland, Kraków
Senior Contributor
Hello Lance,

Please remove cahce and cofnig directories:
rm -rf ${upp_dir}/.cache
rm -rf ${upp_dir}/.config

It should help.

I know that in the past we eliminated some migration issues related to new clang implementation, but it seems that there is something from the old installation that is causing crash.

Klugier


U++ - one framework to rule them all.
Re: theide with libclang [message #58926 is a reply to message #58925] Sun, 25 September 2022 14:48 Go to previous messageGo to next message
Lance is currently offline  Lance
Messages: 526
Registered: March 2007
Contributor
Klugier wrote on Sun, 25 September 2022 07:48
Hello Lance,

Please remove cahce and cofnig directories:
rm -rf ${upp_dir}/.cache
rm -rf ${upp_dir}/.config

It should help.

I know that in the past we eliminated some migration issues related to new clang implementation, but it seems that there is something from the old installation that is causing crash.

Klugier


Thank you, Klugier. It helped but only a bit. When I opened the examples/Days package after deleting cache&config, I observed right panel starting from blank and some class/function etc are populated and then it crashed again with the same error (invalid memory access).
index.php?t=getfile&id=6663&private=0
  • Attachment: libclang.png
    (Size: 134.24KB, Downloaded 243 times)
Re: theide with libclang [message #58927 is a reply to message #58926] Sun, 25 September 2022 15:28 Go to previous messageGo to next message
Klugier is currently offline  Klugier
Messages: 1076
Registered: September 2012
Location: Poland, Kraków
Senior Contributor
Hello Lance,

Could you open ide project and build it in debug mode. After that in TheIDE in debug mode (under debugger) open examples/Days package. Thanks to that we should be able to obtain callstack of the crash.

If you have trouble with opening uppsrc/ide package due to crash in TheIDE, disable clang parsing in Settings (Assist tab and first option from the left). Screenshot below:
index.php?t=getfile&id=6664&private=0

Klugier
  • Attachment: Assist.png
    (Size: 12.34KB, Downloaded 250 times)


U++ - one framework to rule them all.
Re: theide with libclang [message #58928 is a reply to message #58926] Sun, 25 September 2022 15:28 Go to previous messageGo to next message
Lance is currently offline  Lance
Messages: 526
Registered: March 2007
Contributor
Now I tested upp.16443 on a more recently installed machine that can be regarded as clean.

After installed and launched theide and opened Examples/Days in theide, I notice the right panel is blank and not changing. Presumably theide could not find libclang.so and no parsing were taking place.

Then I ran the jjacksonRIAB script
sudo bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"

to update/make sure the presence of libclang. After that I launched theide and examples/Days again, right panel remained blank. Then I delete ${upp_src}/.{config,cache}, launched theide & examples/Days, same blank. Now on this supposedly cleaner system, theide.16443 could no even locate libclang.so.

Re: theide with libclang [message #58929 is a reply to message #58927] Sun, 25 September 2022 15:30 Go to previous messageGo to next message
Lance is currently offline  Lance
Messages: 526
Registered: March 2007
Contributor
Klugier wrote on Sun, 25 September 2022 09:28
Hello Lance,

Could you open ide project and build it in debug mode. After that in TheIDE in debug mode (under debugger) open examples/Days package. Thanks to that we should be able to obtain callstack of the crash.

If you have trouble with opening uppsrc/ide package due to crash in TheIDE, disable clang parsing in Settings (Assist tab and first option from the left). Screenshot below:
index.php?t=getfile&id=6664&private=0

Klugier



libclang.so obviously is successfully detected. In the generated theide, I opened examples/AddressBook, I can observe the right panel populated with parsed results, gradually, starting from blank.

This time it took much longer for the generated theide to crash. Unfortunately nothing useful could be obtained from that/after that.
index.php?t=getfile&id=6665&private=0

As you can see, I can either Wait(and it eventually bring me to the same screen) or Force Quit the generated debug-mode theide which end the generated debug-mode theide completely,leaving me no chance to check the call stack.

In the host theide(which creates the debug-mode theide), I have something like this...oh, I can not update more than one picture, anyways, it bears no information and should not be of any relevance anyways.

[Updated on: Sun, 25 September 2022 16:13]

Report message to a moderator

Re: theide with libclang [message #58930 is a reply to message #58929] Sun, 25 September 2022 15:47 Go to previous messageGo to next message
Klugier is currently offline  Klugier
Messages: 1076
Registered: September 2012
Location: Poland, Kraków
Senior Contributor
Hello Lance,

OK, I see where the problem is. We are dynamically loading libclang without notify the user about the actions of this operation. It must change and in situation when we do not detect it we should show error and siabled parsing...

Here is the code in TheIDE (ide/main.cpp):
#ifdef PLATFORM_POSIX
void TryLoadLibClang() // Should return bool...
{
	String libdir = TrimBoth(Sys("llvm-config --libdir"));
	if(LoadLibClang(libdir + "/libclang.so"))
		return;
	if(LoadLibClang("/usr/lib/libclang.so"))
		return;
	for(int i = 20; i >= 10; i--)
		if(LoadLibClang("/usr/lib/llvm-" + AsString(i) + "/lib/libclang.so"))
			return;
}
#endif

#ifdef flagMAIN
GUI_APP_MAIN
#else
void AppMain___()
#endif
{
#ifdef PLATFORM_POSIX
	TryLoadLibClang(); // If it fails... just notrmal execution without any notification or error. However user will now if libclang is conffigured correctly?
#endif
        // Above operation fails and we are starting to use libclang, it can not end good...


The above code must be fixed and then you can try to run TheIDE. If you do not have correctly set libclang then you should see the error.
-------------------
Edit: In general I assume that for your configuration there is a problem with finding libclang.so and lack of this library is causing crash.

Klugier


U++ - one framework to rule them all.

[Updated on: Sun, 25 September 2022 15:55]

Report message to a moderator

Re: theide with libclang [message #58931 is a reply to message #58930] Sun, 25 September 2022 16:05 Go to previous messageGo to previous message
Lance is currently offline  Lance
Messages: 526
Registered: March 2007
Contributor
Klugier wrote on Sun, 25 September 2022 09:47
Hello Lance,

OK, I see where the problem is. We are dynamically loading libclang without notify the user about the actions of this operation. It must change and in situation when we do not detect it we should show error and siabled parsing...

Here is the code in TheIDE (ide/main.cpp):
#ifdef PLATFORM_POSIX
void TryLoadLibClang() // Should return bool...
{
	String libdir = TrimBoth(Sys("llvm-config --libdir"));
	if(LoadLibClang(libdir + "/libclang.so"))
		return;
	if(LoadLibClang("/usr/lib/libclang.so"))
		return;
	for(int i = 20; i >= 10; i--)
		if(LoadLibClang("/usr/lib/llvm-" + AsString(i) + "/lib/libclang.so"))
			return;
}
#endif

#ifdef flagMAIN
GUI_APP_MAIN
#else
void AppMain___()
#endif
{
#ifdef PLATFORM_POSIX
	TryLoadLibClang(); // If it fails... just notrmal execution without any notification or error. However user will now if libclang is conffigured correctly?
#endif
        // Above operation fails and we are starting to use libclang, it can not end good...


The above code must be fixed and then you can try to run TheIDE. If you do not have correctly set libclang then you should see the error.
-------------------
Edit: In general I assume that for your configuration there is a problem with finding libclang.so and lack of this library is causing crash.

Klugier


Hi Klugier,

I have updated my previous reply. Please let me know if it helps for me to try your revised code. Thanks!
Previous Topic: Impressive improvement in stl::vector when dealing with raw memory.
Next Topic: about unicode supporting
Goto Forum:
  


Current Time: Fri Apr 19 07:52:47 CEST 2024

Total time taken to generate the page: 0.02750 seconds