Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » Developing U++ » U++ Developers corner » theide with libclang
Re: theide with libclang [message #58860 is a reply to message #58859] Sun, 18 September 2022 21:23 Go to previous messageGo to next message
Novo is currently offline  Novo
Messages: 1358
Registered: December 2006
Ultimate Contributor
valgrind explains everything.
  • Attachment: memchk.zip
    (Size: 9.01KB, Downloaded 68 times)


Regards,
Novo
Re: theide with libclang [message #58864 is a reply to message #58860] Tue, 20 September 2022 13:09 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
Novo wrote on Sun, 18 September 2022 21:23
valgrind explains everything.


While it is definitely a good idea, only one of these errors seems libclang related and fixable and it happens on theide exit. Anyway, now fixed.

That said, everything valigrind says is worth investigation. I was able to fix many of these issues (in master now), unfortunately it looks like majority of remaining problems is cairo related (these happen when we are reading images from gtk).

Any help identifying the cause(s) is welcome.

Mirek

[Updated on: Tue, 20 September 2022 13:11]

Report message to a moderator

Re: theide with libclang [message #58873 is a reply to message #58864] Tue, 20 September 2022 19:16 Go to previous messageGo to next message
Novo is currently offline  Novo
Messages: 1358
Registered: December 2006
Ultimate Contributor
mirek wrote on Tue, 20 September 2022 07:09
Novo wrote on Sun, 18 September 2022 21:23
valgrind explains everything.


While it is definitely a good idea, only one of these errors seems libclang related and fixable and it happens on theide exit. Anyway, now fixed.

That said, everything valigrind says is worth investigation. I was able to fix many of these issues (in master now), unfortunately it looks like majority of remaining problems is cairo related (these happen when we are reading images from gtk).

Any help identifying the cause(s) is welcome.

Mirek

Thanks a lot! TheIde doesn't crash anymore.
I personally do not see any cairo problems. I guess Void Linux has a better valgrind exceptions list.
The only problem I do see is an uninitialized value in Ide::Serialize.
Interestingly, this problem depends on when you close TheIde.
I guess this data is uninitialized during libclang indexing/reindexing.
My command line:
valgrind --track-origins=yes --log-file=memchk.txt ~/dvlp/cpp/code/upp/out/CLANG.Debug.Debug_Full.Gui.Shared/ide


Regards,
Novo
Re: theide with libclang [message #58874 is a reply to message #58873] Tue, 20 September 2022 19:21 Go to previous messageGo to next message
jjacksonRIAB is currently offline  jjacksonRIAB
Messages: 219
Registered: June 2011
Experienced Member
Quote:

Thanks a lot! TheIde doesn't crash anymore.


Really? I got the latest updates and it was completely broken for me. My assemblies/nests were all empty except some example code and then trying to open even one of those crashed the ide. I must be having a run of bad luck lately, I'll try to clone the whole thing again.
Re: theide with libclang [message #58875 is a reply to message #58873] Tue, 20 September 2022 19:37 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
Novo wrote on Tue, 20 September 2022 19:16
mirek wrote on Tue, 20 September 2022 07:09
Novo wrote on Sun, 18 September 2022 21:23
valgrind explains everything.


While it is definitely a good idea, only one of these errors seems libclang related and fixable and it happens on theide exit. Anyway, now fixed.

That said, everything valigrind says is worth investigation. I was able to fix many of these issues (in master now), unfortunately it looks like majority of remaining problems is cairo related (these happen when we are reading images from gtk).

Any help identifying the cause(s) is welcome.

Mirek

Thanks a lot! TheIde doesn't crash anymore.
I personally do not see any cairo problems. I guess Void Linux has a better valgrind exceptions list.
The only problem I do see is an uninitialized value in Ide::Serialize.
Interestingly, this problem depends on when you close TheIde.
I guess this data is uninitialized during libclang indexing/reindexing.


I am getting that one (in Ide::Serialize) too; my guts feeling is that it is not related to libclang, just some old serialisation bug like forgotten unitialised variable. Want to hunt that down, but it is time consuming....

What about those "_dl_catch_exception " things? Does not look like U++ fault, right?

[Updated on: Tue, 20 September 2022 19:38]

Report message to a moderator

Re: theide with libclang [message #58876 is a reply to message #58875] Tue, 20 September 2022 19:46 Go to previous messageGo to next message
jjacksonRIAB is currently offline  jjacksonRIAB
Messages: 219
Registered: June 2011
Experienced Member
Quote:

I am getting that one (in Ide::Serialize) too; my guts feeling is that it is not related to libclang


Possible but I was getting crashes on close on the previous version and it was dumping a libclang exception to the terminal because it was indexing at the time, so it was reporting an indexing exception. I don't have the text for it anymore unfortunately.
Re: theide with libclang [message #58877 is a reply to message #58579] Tue, 20 September 2022 20:13 Go to previous messageGo to next message
jjacksonRIAB is currently offline  jjacksonRIAB
Messages: 219
Registered: June 2011
Experienced Member
OK looks like I had a bad repo. I rebuilt and I'm back in. I'm still having excessive memory usage though and another curious thing I noticed is that it reindexes the file I have open in the editor regardless of whether it's changed. If I click away to another app and back to theide it will reindex. It never frees memory, even if I switch to another package entirely.

I'm not getting the clang error in the console anymore though, so that appears to be fixed. I am, however, getting freezes if I cancel a build.

[Updated on: Tue, 20 September 2022 20:40]

Report message to a moderator

Re: theide with libclang [message #58878 is a reply to message #58875] Tue, 20 September 2022 21:32 Go to previous messageGo to next message
jjacksonRIAB is currently offline  jjacksonRIAB
Messages: 219
Registered: June 2011
Experienced Member
OK so with address sanitizer and USEMALLOC enabled, I'm getting a bunch of reported heap overflows all on memcmp.

lib/clang Signature.cpp line 171, for example:

if(memcmp(s, " = {", 4) == 0)


what happens if s is shorter than the sequence it's being compared against? Does it overrun?
Re: theide with libclang [message #58880 is a reply to message #58579] Wed, 21 September 2022 00:33 Go to previous messageGo to next message
jjacksonRIAB is currently offline  jjacksonRIAB
Messages: 219
Registered: June 2011
Experienced Member
Once I replaced those memcmp instructions with strnlen it shut up address sanitizer. They're probably fine but at least I was able to get further and... there was nothing worthwhile except the usual font stuff.

Found another errata to potentially test. If I go into any sufficiently complex package (like CtrlCore or CtrlLib) and hold down alt + down arrow to cycle through/open as many files as quickly as possible, libclang gets stuck in an infinite reparsing loop on one file until I click off to another file. I'm assuming this is because I'm leaving the last parsable .cpp/.h file and moving into other non-parseable file types rapidly.

Edit also if you left ctrl click a symbol with indexing turned off it will still try to index but pin a core instead.

[Updated on: Wed, 21 September 2022 00:49]

Report message to a moderator

Re: theide with libclang [message #58881 is a reply to message #58875] Wed, 21 September 2022 01:33 Go to previous messageGo to next message
Novo is currently offline  Novo
Messages: 1358
Registered: December 2006
Ultimate Contributor
mirek wrote on Tue, 20 September 2022 13:37

I am getting that one (in Ide::Serialize) too; my guts feeling is that it is not related to libclang, just some old serialisation bug like forgotten unitialised variable. Want to hunt that down, but it is time consuming....

One time I ran valgring I saw ZERO problems. I didn't believe my eyes. I reran it and got this Ide::Serialize problem. This is why I think it depends on something else.

mirek wrote on Tue, 20 September 2022 13:37

What about those "_dl_catch_exception " things? Does not look like U++ fault, right?

It is hard to tell. In my latest rerun I didn't get this "_dl_catch_exception " thing.
I believe it happens when I close TheIde before it finishes loading of libclang data.
I'm closing it via unorthodox window manager called herbstluft, which is sending a message to TheIde, I believe.
So, something gets interrupted in the middle, I guess.
Theoretically, a loader should automatically unload everything correctly ...


Regards,
Novo
Re: theide with libclang [message #58882 is a reply to message #58881] Wed, 21 September 2022 01:56 Go to previous messageGo to next message
Novo is currently offline  Novo
Messages: 1358
Registered: December 2006
Ultimate Contributor
I've got a different call stack with Ide::Serialize.
Maybe it will help to locate a problem.


Regards,
Novo
Re: theide with libclang [message #58884 is a reply to message #58882] Wed, 21 September 2022 10:52 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
Novo wrote on Wed, 21 September 2022 01:56
I've got a different call stack with Ide::Serialize.
Maybe it will help to locate a problem.


Thanks, fixed. (Unfortunately it was complete non-bug that could not have influenced anything real, just unitialized variable that is never used again).
Re: theide with libclang [message #58890 is a reply to message #58884] Wed, 21 September 2022 22:09 Go to previous messageGo to next message
jjacksonRIAB is currently offline  jjacksonRIAB
Messages: 219
Registered: June 2011
Experienced Member
Thanks for doing this stuff Mirek. I was looking through the code trying to figure it out and learn about libclang ASTs but I simply don't know enough about TheIDE's internal operations (or libclang) to make heads or tails of it. I also swear I find you everywhere. Very Happy

I came across you on Stack Overflow asking about libclang, then a couple of weeks ago I was reading some posts from Andrei Alexandrescu and I saw you in there too.

As for libclang one of the strange problems I had that shouldn't happen is that clang_parseTranslationUnit was returning a translation unit of 0 and as a consequence Dispose() was never getting called. I wasn't able to fix it because it mysteriously stopped happening and I don't know if it's anything I did. All I did was put in the hook for clang_parseTranslationUnit2 to see if it would give me a more meaningful error... but that turned out to be useless. It stopped happening even after I switched back.

The only other change I made was I hooked clang_defaultReparseOptions and passed that to clang_reparseTranslationUnit. Supposedly it provides some default options that improve performance, don't really know if it did... so I flailed around uselessly for a few hours. The other behavior I had and am still having is that it's not simply parsing and then calling reparse after the initial parse, it's calling both every time. Once I got into CurrentFileThread it got too deep into TheIDE internals to understand what was going on.

[Updated on: Wed, 21 September 2022 22:14]

Report message to a moderator

Re: theide with libclang [message #58900 is a reply to message #58884] Fri, 23 September 2022 05:41 Go to previous messageGo to next message
Novo is currently offline  Novo
Messages: 1358
Registered: December 2006
Ultimate Contributor
mirek wrote on Wed, 21 September 2022 04:52

Thanks, fixed.

Thanks a lot!
Could you also fix a problem below, please?
./umk reference SMTP CLANG -busr
/home/buildbot/worker/l-upp/build/uppsrc/Core/SMTP/Smtp.cpp:457:2: error: expected expression
        DDUMP(GetMessage(true));
        ^
/home/buildbot/worker/l-upp/build/uppsrc/Core/Diag.h:159:26: note: expanded from macro 'DDUMP'
#define DDUMP(x)         @ // To clean logs after debugging, this produces error in release mode
                         ^
1 error generated.


Regards,
Novo
Re: theide with libclang [message #58901 is a reply to message #58884] Fri, 23 September 2022 05:52 Go to previous messageGo to next message
Novo is currently offline  Novo
Messages: 1358
Registered: December 2006
Ultimate Contributor
mirek wrote on Wed, 21 September 2022 04:52
Novo wrote on Wed, 21 September 2022 01:56
I've got a different call stack with Ide::Serialize.
Maybe it will help to locate a problem.


Thanks, fixed. (Unfortunately it was complete non-bug that could not have influenced anything real, just unitialized variable that is never used again).

I do see two problems with valgrind messages:

1) Tons of "uninitialized value" messages hide real problems.
2) Nobody will use Upp with commercial apps for a very simple reason: who knows what kind of problems can be caused by these "uninitialized values".

IMHO, Upp should be checked with all available valgrind tools and all available sanitizers. At least before release.


Regards,
Novo
Re: theide with libclang [message #58903 is a reply to message #58579] Fri, 23 September 2022 07:09 Go to previous messageGo to next message
jjacksonRIAB is currently offline  jjacksonRIAB
Messages: 219
Registered: June 2011
Experienced Member
Mirek,

Runaway memory allocations seem not to be an issue using it all day. The one thing I can report today is libclang getting stuck in a reparse loop on app exit
and refusing to let the ide close. It happens rarely but I can reliably force it to happen by attempting to exit the ide during a build.
Re: theide with libclang [message #58904 is a reply to message #58901] Fri, 23 September 2022 10:48 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
Novo wrote on Fri, 23 September 2022 05:52
mirek wrote on Wed, 21 September 2022 04:52
Novo wrote on Wed, 21 September 2022 01:56
I've got a different call stack with Ide::Serialize.
Maybe it will help to locate a problem.


Thanks, fixed. (Unfortunately it was complete non-bug that could not have influenced anything real, just unitialized variable that is never used again).

I do see two problems with valgrind messages:

1) Tons of "uninitialized value" messages hide real problems.
2) Nobody will use Upp with commercial apps for a very simple reason: who knows what kind of problems can be caused by these "uninitialized values".

IMHO, Upp should be checked with all available valgrind tools and all available sanitizers. At least before release.


I do not argue with that (as you have seen, I have fixed what I could).

The comment was not meant as an apology for the bug, just a note that it cannot explain any weird behaviour with libclang.
Re: theide with libclang [message #58906 is a reply to message #58904] Sat, 24 September 2022 04:41 Go to previous messageGo to next message
Lance is currently offline  Lance
Messages: 526
Registered: March 2007
Contributor
I don't know how much improvement we can expect in, say half an year's time. At the moment, I Ctrl+Left click on a member function in a moderately complex project, the parsing starts and goes on for ever. At least 30 minutes past and no idea how much longer it's going to take. In contrast, it used to work so smoothly.

Theoretically moving to libclang is the right thing to do, but there is a huge chance it may render theide completely useless.

Some system info
Quote:

Upp version: 16443

OS: Ubuntu 22.04.1 LTS x86_64
CPU: Intel i7-8650U (8) @ 4.200GHz
GPU: Intel UHD Graphics 620
Memory: 3757MiB / 15875MiB


PS: I tested with Examples/HomeBudget. In HomeBudget.cpp, Ctrl+left click some member function, eg LoadCategories(...) to start parsing. Now roughly 10 minutes past, still parsing. Something might not be right.

[Updated on: Sat, 24 September 2022 05:08]

Report message to a moderator

Re: theide with libclang [message #58907 is a reply to message #58906] Sat, 24 September 2022 07:26 Go to previous messageGo to next message
peterh is currently offline  peterh
Messages: 108
Registered: November 2018
Location: Germany
Experienced Member
Fortunately I can say, on my computer it works almost immediately.
I am on Windows 10 64 bit.
The computer is a Ryzen 7, 32 GByte Memory.

I had a look into taskmanager. Total memory used was 12 GBytes.
I had some other programs open, Browser, email, Fotoviewer Faststone, Fotosoftware.
TheIde used 2 GBytes and had the biggest memory chunk in use.

I can imagine, if the computer had less memory, if the drive is a harddrive and not a SSD, and if TheIDE temporarily wants more memory, then the computer would start swapping and it could terribly slow down.
(If it had an SSD it could be faster, but would lead to unneccessary SSD wear) (I have a SSD but I hope it will not be used for swapping)

Fortunately swapping does not happen here.
It might be useful to monitor and report this machine data and properties.

Possibly reboot and do not start other programs than TheIDE and compare the behavior and memory consumption.
So far I know, libclangd can be disabled in the setup. This could be a solution as a last ressort if memory is low.

[Updated on: Sat, 24 September 2022 07:47]

Report message to a moderator

Re: theide with libclang [message #58908 is a reply to message #58906] Sat, 24 September 2022 08:08 Go to previous messageGo to previous message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
Lance wrote on Sat, 24 September 2022 04:41
I don't know how much improvement we can expect in, say half an year's time. At the moment, I Ctrl+Left click on a member function in a moderately complex project, the parsing starts and goes on for ever. At least 30 minutes past and no idea how much longer it's going to take. In contrast, it used to work so smoothly.

Theoretically moving to libclang is the right thing to do, but there is a huge chance it may render theide completely useless.


It is just first week after the merge...

Quote:
Something might not be right.


The is the most likely explanation. There is no reason for it to take so long, it is probably stuck somewhere.

Is autocomplete working?

Mirek
Previous Topic: Impressive improvement in stl::vector when dealing with raw memory.
Next Topic: about unicode supporting
Goto Forum:
  


Current Time: Fri Mar 29 00:40:36 CET 2024

Total time taken to generate the page: 0.01516 seconds