|
|
Home » Developing U++ » U++ Developers corner » theide with libclang
Re: theide with libclang [message #58909 is a reply to message #58579] |
Sat, 24 September 2022 08:44   |
 |
mirek
Messages: 14255 Registered: November 2005
|
Ultimate Member |
|
|
OK, so one week in, I guess I owe the community a bit of explanation about libclang and the way theide uses it...
So, obviously what libclang does is parsing the files and giving info about them. Additionally, it can provide autocomplete lists at specified point in the source files, but NOT HEADERS (and also not inside templates, but that is store for another time).
The important thing to know is that parsing the file with all headers is long: on my not so slow machine, it can take up to several seconds to parse the file. To remedy this, libclang has a concept of "preamble" and "reparsing". Preamble is basically automated precompiled header - if file starts with multiple headers, preamble is creating and when after the first parse the file is reparsed or autocomplete requested, preamble is used to greatly speedup the process. With preamble, autocomplete (after the first parse) is almost immediate. That said, premable creation is sort of fragile. Also, normally it gets created in system temp directory, which would be fine unless sometimes libclang does not delete it. But it can be solved be setting env variable which we do now.
So what we do now is the when you start editing some file, in the background thread the first parse of the file is started with a request to create preamble. In theide, you can know that this is happening by left bar going "sort of orange" for a couple of seconds - it does that each time file is parsed or reparsed. If you then change the file, it gets (after some time of inactivity) reparsed again. Parsing current file gets it ready for autocomplete, but also provides information about file for file annotations, current file navigation and "jumps" (Alt-J / Ctrl clicks).
Originally, there was just one parsed file, which meant each time after switching to another file it had to have that initial a couple of seconds parsing period. Later I have added a cache, there are now several files with parsed info. Downside is that each "parsing unit" consumes a lot of memory, about 300MB so with 12 entries in the cache, you can do the math....
Include files that clang is unable to process: What we do is that we find "master file" - the well defined file that includes the header and we pretend that we are parsing that one. It is not so simple: we still want to use preamble to make things fast and if we just parsed master file and hoped to catch the info include file from it, it would not work with preambles as our include file would actually be in preamble. I have spent a lot of time trying to solve this: What we actually do now is that we create a syntetic file content that actually manually does including so that it all works (it is hard to explain in detail; in assist diagnostic mode you can see the content of the syntetic file with Assist/Current parsed file content).
So that is about all I can say about "current file" without going into gruesome details; if you want to study the code, libclang/CurrentFile.cpp and ide/AssistTrick.cpp are good starting points.
Next troublesome part of code is "Indexer" - that is something that scans the whole sources and creates a program database of, well, everything everywhere. That is needed for Navigator and Alt+J command. We are using blitz(like) process there and we are running that in multiple threads. Now there is "Scheduler" (Indexer::SchedulerThread()) thread that scans for changed files and basically prepares jobs for "Indexing" (Indexer::IndexerThread) threads. Once again, each of indexer threads can eat up to 300MB of memory, so finetuning default values will take some time. More indexer threads mean faster reindexing but perhaps too much memory...
In terms of slowdown/freezing during reindexing, well, the plan was to give indexer threads really low priority. It should be possible with
clang_CXIndex_setGlobalOptions(clang.index, CXGlobalOpt_ThreadBackgroundPriorityForIndexing);
but maybe this does not work as expected in Linux. In Windows, it seems to work fine.
Now for testing and troubleshooting, please note that there is "Assist diagnostics" option in Assist/Debug. When active, it will start reporting in Console what it is doing and also there are new entries at the end of assist menu, showing a dump of actual index, errors libclang reported during parsing and what was given to clang to parse as current file.
So if e.g. there is no autocomplete for include files, please check "Current file parse errors" and "Current parsed file content".
I know this is a big change (and one I was extremly hesitant to do) and not everything works smoothly at this point, but hopefully all will be finetuned in time. I guess we fixed a lot already in the first week....
Mirek
[Updated on: Sat, 24 September 2022 08:47] Report message to a moderator
|
|
|
Re: theide with libclang [message #58910 is a reply to message #58909] |
Sat, 24 September 2022 09:53   |
jjacksonRIAB
Messages: 227 Registered: June 2011
|
Experienced Member |
|
|
Nah, Mirek, I don't feel I'm owed an explanation 
Windows users seem to be OK, like you said it's linux that needs some love. In my case I had an ancient version of libclang 7.0 sitting in /usr/local/lib. Once I zapped that the excessive memory consumption disappeared so I'm pretty sure that was libclang. I updated to 15 and things are still looking good. I did find valgrind complain about some branches on uninitialized variables in theide itself:
==562946== Conditional jump or move depends on uninitialised value(s)
==562946== at 0x56DB7A: Upp::CodeEditor::DirtyFrom(int) (CodeEditor.cpp:74)
==562946== by 0x5A2DEF: Upp::TextCtrl::Insert0(int, Upp::WString const&) (Text.cpp:799)
==562946== by 0x5A3C2E: Upp::TextCtrl::InsertU(int, Upp::WString const&, bool) (Text.cpp:935)
==562946== by 0x5A3FB7: Upp::TextCtrl::Insert(int, Upp::WString const&, bool) (Text.cpp:980)
==562946== by 0x315C96: Upp::TextCtrl::Insert(int, Upp::WString const&) (TextEdit.h:218)
==562946== by 0x5A4E6B: Upp::TextCtrl::Paste(Upp::WString const&) (Text.cpp:1153)
==562946== by 0x3075CD: Console::Append(Upp::String const&) (Console.cpp:77)
==562946== by 0x326EDB: Console::operator<<(Upp::String const&) (ide.h:127)
==562946== by 0x3CA52A: Ide::PutConsole(char const*) (idewin.cpp:50)
==562946== by 0x46B05A: PutConsole(char const*) (Ide.cpp:9)
==562946== by 0x34EC3E: AssistEditor::SyncMaster() (IncludeTrick.cpp:18)
==562946== by 0x3536C7: AssistEditor::NewFile(bool) (Assist.cpp:438)
==562946==
==562946== Conditional jump or move depends on uninitialised value(s)
==562946== at 0x56D63F: Upp::CodeEditor::GetSyntax(int) (CodeEditor.cpp:29)
==562946== by 0x56DC5B: Upp::CodeEditor::GetRefreshInfo(int) (CodeEditor.cpp:94)
==562946== by 0x56DEFA: Upp::CodeEditor::PostInsert(int, Upp::WString const&) (CodeEditor.cpp:111)
==562946== by 0x5A335E: Upp::TextCtrl::Insert0(int, Upp::WString const&) (Text.cpp:851)
==562946== by 0x5A3C2E: Upp::TextCtrl::InsertU(int, Upp::WString const&, bool) (Text.cpp:935)
==562946== by 0x5A3FB7: Upp::TextCtrl::Insert(int, Upp::WString const&, bool) (Text.cpp:980)
==562946== by 0x315C96: Upp::TextCtrl::Insert(int, Upp::WString const&) (TextEdit.h:218)
==562946== by 0x5A4E6B: Upp::TextCtrl::Paste(Upp::WString const&) (Text.cpp:1153)
==562946== by 0x3075CD: Console::Append(Upp::String const&) (Console.cpp:77)
==562946== by 0x326EDB: Console::operator<<(Upp::String const&) (ide.h:127)
==562946== by 0x3CA52A: Ide::PutConsole(char const*) (idewin.cpp:50)
==562946== by 0x46B05A: PutConsole(char const*) (Ide.cpp:9)
I was also able to trigger a crash of TheIDE by just holding down Ctrl+Space for a minute or so to make it continually attempt to autocomplete, put some pressure on it. Unfortunately it only happens in release mode, I cannot reproduce it in debug mode.
|
|
|
Re: theide with libclang [message #58911 is a reply to message #58910] |
Sat, 24 September 2022 10:03   |
 |
peterh
Messages: 108 Registered: November 2018 Location: Germany
|
Experienced Member |
|
|
Hi,
I have reported 3 GB memory in windows.
I must add, I had the IDE open for probably 2 days.
Normally I do not shut down my computer but put it to sleep at night or when I do not use it.
I have now rebooted and after opening TheIDE with the same project and having tried "Homebudget" also, it consumes only 300MB.
[Updated on: Sat, 24 September 2022 10:05] Report message to a moderator
|
|
|
|
Re: theide with libclang [message #58915 is a reply to message #58912] |
Sat, 24 September 2022 10:30   |
jjacksonRIAB
Messages: 227 Registered: June 2011
|
Experienced Member |
|
|
mirek wrote on Sat, 24 September 2022 10:16
It was not meant specifically for you...
Serves me right for attempting to be conciliatory.
Quote:
In your case I remember complaining about headers. Do they work now?
I'll get latest and check. But the list of outstanding issues I have:
1. Includes don't autocomplete
2. Parser gets stuck in a reparse loop if I cycle through files in a project using alt+down and hit a file that is not code (like a .lay or .iml file)
3. Alt+C, which you are already working on
Quote:
As for old libclang... I can put limit for minimal version, e.g. libclang 10 (I am quite sure that one works), what do you think?
Sounds good.
[Updated on: Sat, 24 September 2022 10:34] Report message to a moderator
|
|
|
|
Re: theide with libclang [message #58917 is a reply to message #58912] |
Sat, 24 September 2022 11:10   |
jjacksonRIAB
Messages: 227 Registered: June 2011
|
Experienced Member |
|
|
Mirek,
The problem appears to manifest with assist_cursor in Assist.cpp line 502.
It gathers all of the potential include files appropriately but assist_cursor never matches current pos so it returns before performing the completion.
void AssistEditor::PopUpAssist(bool auto_insert)
{
LTIMING("PopUpAssist");
int pos = GetCursor();
ReadIdBackPos(pos, false);
if(pos != assist_cursor)
return;
EDIT It's uninitialized. The only place I can find it being set is around line 463 but there's an early return
include_assist = false;
if(IncludeAssist())
return;
int pos = GetCursor();
ReadIdBackPos(pos, false); // libclang does not work well if file is not truncated for autocomplete (?)
assist_cursor = pos;
before assist_cursor is set. If I move that block of code after the pos set it shows the files but it has strange index numbers next to the icons. <- never mind, that appears to be for debugging purposes and doesn't appear in release mode.
So the fix is:
int pos = GetCursor();
ReadIdBackPos(pos, false); // libclang does not work well if file is not truncated for autocomplete (?)
assist_cursor = pos;
include_assist = false;
if(IncludeAssist())
return;
[Updated on: Sat, 24 September 2022 19:43] Report message to a moderator
|
|
|
Re: theide with libclang [message #58918 is a reply to message #58908] |
Sat, 24 September 2022 15:00   |
Lance
Messages: 656 Registered: March 2007
|
Contributor |
|
|
mirek wrote on Sat, 24 September 2022 02:08
Is autocomplete working?
No.It is not. Is it because libclang not present in my system?
PS:
I already have clang installed. After I installed libclang-dev, key event in theide will immediately crash theide with error message
Quote:
Fatal Error!
Invalid memory access!
The libclang.so shipped with most recent(?) libclang-dev(1:14.0-55~exp2) doesn't seem to work with theide. It's most likely to be some kind of chaos in my system.
ps2: Now even after I remove'd libclang-dev, theide in upp version 16443 won't work. You don't even need to type in anything, it will crash itself with invalid memory access in just a few seconds(when background parsing threads start). It looks like a libclang version problem. For now, I will go back to upp version 16303 on which autocomplete works fine.
[Updated on: Sat, 24 September 2022 15:54] Report message to a moderator
|
|
|
Re: theide with libclang [message #58920 is a reply to message #58918] |
Sat, 24 September 2022 19:55   |
jjacksonRIAB
Messages: 227 Registered: June 2011
|
Experienced Member |
|
|
Lance wrote on Sat, 24 September 2022 15:00
ps2: Now even after I remove'd libclang-dev, theide in upp version 16443 won't work. You don't even need to type in anything, it will crash itself with invalid memory access in just a few seconds(when background parsing threads start). It looks like a libclang version problem. For now, I will go back to upp version 16303 on which autocomplete works fine.
I didn't try dev packages, I just uninstalled clang through package manager and ran
sudo bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"
to bring everything to clang-15, including libclang. Should work with debian/apt systems.
[Updated on: Sat, 24 September 2022 19:57] Report message to a moderator
|
|
|
|
|
Re: theide with libclang [message #58923 is a reply to message #58920] |
Sat, 24 September 2022 23:09   |
jjacksonRIAB
Messages: 227 Registered: June 2011
|
Experienced Member |
|
|
This is the problem I'm currently trying to work out but not getting anywhere yet:
https://files.catbox.moe/t4kcpd.mp4
It will get stuck in a reparse loop. Same thing also happens occasionally when I'm exiting TheIDE. Note that it happens when I move rapidly to a resource-type file. I didn't capture it in the video but it behaves normally and exits the loop if I go back to a parseable .cpp or .h file, so it's not stuck that way permanently - it is just attempting to redo the same old job repeatedly until it gets a new job. When it happens on exit it becomes permanently stuck.
[Updated on: Sun, 25 September 2022 03:06] Report message to a moderator
|
|
|
Re: theide with libclang [message #58924 is a reply to message #58920] |
Sun, 25 September 2022 02:13   |
Lance
Messages: 656 Registered: March 2007
|
Contributor |
|
|
jjacksonRIAB wrote on Sat, 24 September 2022 13:55Lance wrote on Sat, 24 September 2022 15:00
ps2: Now even after I remove'd libclang-dev, theide in upp version 16443 won't work. You don't even need to type in anything, it will crash itself with invalid memory access in just a few seconds(when background parsing threads start). It looks like a libclang version problem. For now, I will go back to upp version 16303 on which autocomplete works fine.
I didn't try dev packages, I just uninstalled clang through package manager and ran
sudo bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"
to bring everything to clang-15, including libclang. Should work with debian/apt systems.
Thank you, jjacksonRIAB! Your command worked fine and I was able to upgrade my libclang. Unfortunately the problem remains with UPP version 16443: theide will crash in a few seconds after a package is open with "Invalid Memory Access" error.
[Updated on: Sun, 25 September 2022 02:14] Report message to a moderator
|
|
|
|
|
|
|
Re: theide with libclang [message #58929 is a reply to message #58927] |
Sun, 25 September 2022 15:30   |
Lance
Messages: 656 Registered: March 2007
|
Contributor |
|
|
Klugier wrote on Sun, 25 September 2022 09:28Hello Lance,
Could you open ide project and build it in debug mode. After that in TheIDE in debug mode (under debugger) open examples/Days package. Thanks to that we should be able to obtain callstack of the crash.
If you have trouble with opening uppsrc/ide package due to crash in TheIDE, disable clang parsing in Settings (Assist tab and first option from the left). Screenshot below:

Klugier
libclang.so obviously is successfully detected. In the generated theide, I opened examples/AddressBook, I can observe the right panel populated with parsed results, gradually, starting from blank.
This time it took much longer for the generated theide to crash. Unfortunately nothing useful could be obtained from that/after that.

As you can see, I can either Wait(and it eventually bring me to the same screen) or Force Quit the generated debug-mode theide which end the generated debug-mode theide completely,leaving me no chance to check the call stack.
In the host theide(which creates the debug-mode theide), I have something like this...oh, I can not update more than one picture, anyways, it bears no information and should not be of any relevance anyways.
[Updated on: Sun, 25 September 2022 16:13] Report message to a moderator
|
|
|
Re: theide with libclang [message #58930 is a reply to message #58929] |
Sun, 25 September 2022 15:47   |
 |
Klugier
Messages: 1099 Registered: September 2012 Location: Poland, Kraków
|
Senior Contributor |
|
|
Hello Lance,
OK, I see where the problem is. We are dynamically loading libclang without notify the user about the actions of this operation. It must change and in situation when we do not detect it we should show error and siabled parsing...
Here is the code in TheIDE (ide/main.cpp):
#ifdef PLATFORM_POSIX
void TryLoadLibClang() // Should return bool...
{
String libdir = TrimBoth(Sys("llvm-config --libdir"));
if(LoadLibClang(libdir + "/libclang.so"))
return;
if(LoadLibClang("/usr/lib/libclang.so"))
return;
for(int i = 20; i >= 10; i--)
if(LoadLibClang("/usr/lib/llvm-" + AsString(i) + "/lib/libclang.so"))
return;
}
#endif
#ifdef flagMAIN
GUI_APP_MAIN
#else
void AppMain___()
#endif
{
#ifdef PLATFORM_POSIX
TryLoadLibClang(); // If it fails... just notrmal execution without any notification or error. However user will now if libclang is conffigured correctly?
#endif
// Above operation fails and we are starting to use libclang, it can not end good...
The above code must be fixed and then you can try to run TheIDE. If you do not have correctly set libclang then you should see the error.
-------------------
Edit: In general I assume that for your configuration there is a problem with finding libclang.so and lack of this library is causing crash.
Klugier
U++ - one framework to rule them all.
[Updated on: Sun, 25 September 2022 15:55] Report message to a moderator
|
|
|
|
Goto Forum:
Current Time: Fri Apr 25 20:31:38 CEST 2025
Total time taken to generate the page: 0.01913 seconds
|
|
|