U++ Forum - RDF feed
type: channel
description: Main place for discussions about U++ cross platform development framework.
items: theide with libclang
items: Re: theide with libclang, first milestone finished
items: Re: theide with libclang

theide with libclang
type: item
description: In https://github.com/ultimatepp/ultimatepp/tree/llbm_ide branch there lives my recent development effort of integrating theide with libclang. Now I can proudly announce that I have reached the first milestone: libclang autocomplete is now integrated well enough for me to actually start using this version of theide. Did not tested/fixed outside of windows yet though... (that said, windows seems the most complicated as we have to provide libclang.dll to the release). Anyway, I would appreciate some more testers for this. In order to compile/use this version in windows, you need libclang.dll from official llvm release and you will also need to add include/lib paths to build method (in future, this will be coming with theide). Mirek
creator: mirek
date: 2022-06-26T07:32:38-00:00

Re: theide with libclang, first milestone finished
type: item
description: Hello Mirek, I've succesfully compiled the branch on Linux, however, it doesn't seem to work at all. (No assist/code completion, etc...) (Linux 5.18, GNOME 42.1 (Wayland), LLVM/CLANG 13.0.1 (x64)) Keep in mind that this was just a quick test, Later I'll do some more... Best regards, Oblivion
creator: Oblivion
date: 2022-06-26T10:53:49-00:00

Re: theide with libclang, first milestone finished
type: item
description: Oblivion wrote on Sun, 26 June 2022 12:53 Hello Mirek, I've succesfully compiled the branch on Linux, however, it doesn't seem to work at all. (No assist/code completion, etc...) (Linux 5.18, GNOME 42.1 (Wayland), LLVM/CLANG 13.0.1 (x64)) Keep in mind that this was just a quick test, Later I'll do some more... Best regards, Oblivion Probably RedefineMacros() function problems - in linux, I believe it should not be there at all.
creator: mirek
date: 2022-06-26T12:18:55-00:00

Re: theide with libclang, first milestone finished
type: item
description: Hello Mirek, Good news! I will test when it will be compatible with Linux. Klugier
creator: Klugier
date: 2022-06-26T14:37:24-00:00

Re: theide with libclang, first milestone finished
type: item
description: OK, good news and bad news (for now): The problem is caused by CXTranslationUnit_PrecompiledPreamble flag for clang parser. For some reason, when I specify this in Linux, libclang ignores #includes. Without this flag, it now works. Downside is that it is much slower than in Win32 where CXTranslationUnit_PrecompiledPreamble works... Now I am pretty sure this must be some problem of my code, as you are actually supposed to use this and I bet other ides are using this flag. But so far I was not able to crack it... Anyway, you should now be able to test it, hopefully, for initial impressions.
creator: mirek
date: 2022-06-26T16:07:43-00:00

Re: theide with libclang, first milestone finished
type: item
description: Hi, I must be missing something here... Compilation fails with: C:\upp-git\upp.src\uppsrc\ide/clang/clang.h (5): fatal error C1083: Cannot open include file: 'clang-c/Index.h': No such file or directory Best regards, Tom EDIT: This is on Windows using MSBT22x64 and the same happens with CLANGx64.
creator: Tom1
date: 2022-06-27T08:58:12-00:00

Re: theide with libclang, first milestone finished
type: item
description: Tom1 wrote on Mon, 27 June 2022 10:58 Hi, I must be missing something here... Compilation fails with: C:\upp-git\upp.src\uppsrc\ide/clang/clang.h (5): fatal error C1083: Cannot open include file: 'clang-c/Index.h': No such file or directory Best regards, Tom EDIT: This is on Windows using MSBT22x64 and the same happens with CLANGx64. It is using LLVM, so you need to have LLVM installed and paths properly set (in build method), libclang.dll has to be in the same dir as the new theide.exe. Alternatively, you can wait before it becomes part of nightly (in the beginning, I plan to just add .dll and part of llvm sdk so that you can build llbm_ide out of box).
creator: mirek
date: 2022-06-27T09:16:50-00:00

Re: theide with libclang, first milestone finished
type: item
description: OK, I see. Maybe I'll wait a while, until it becomes part of nightly with those dependencies and settings included... Thanks and best regards, Tom
creator: Tom1
date: 2022-06-27T09:25:07-00:00

Re: theide with libclang, first milestone finished
type: item
description: Tom1 wrote on Mon, 27 June 2022 11:25 OK, I see. Maybe I'll wait a while, until it becomes part of nightly with those dependencies and settings included... Thanks and best regards, Tom That might take a while, I do not think I will merge before everything else is changed to use libclang (like navigator, Alt+J etc...) Mirek
creator: mirek
date: 2022-06-27T12:35:35-00:00

Re: theide with libclang, first milestone finished
type: item
description: Linux: ultimatepp/uppsrc/ide/clang/clang.cpp:105:2: error: expected expression DDUMP(cmdline); ^
creator: Novo
date: 2022-07-03T03:28:14-00:00

Re: theide with libclang, first milestone finished
type: item
description: I checked a clang-enabled version of TheIDE in Linux (debug version). It is using 100% of CPU all the time. Initially, I thought that this is Clang indexing UPP files, but after waiting for 50 minutes I figured out that this must be a bug with TheIDE.
creator: Novo
date: 2022-07-03T14:19:52-00:00

Re: theide with libclang, first milestone finished
type: item
description: Novo wrote on Sun, 03 July 2022 16:19 I checked a clang-enabled version of TheIDE in Linux (debug version). It is using 100% of CPU all the time. Initially, I thought that this is Clang indexing UPP files, but after waiting for 50 minutes I figured out that this must be a bug with TheIDE. Yeah, that is temporary issue, it is fixed in latest version (hopefully). Mirek
creator: mirek
date: 2022-07-03T22:08:08-00:00

Re: theide with libclang, first milestone finished
type: item
description: So after developing this in Windows for a while (the hardest part now is to make clang obtained code references compatible with existing topic++ documentation) I have returned to try to fix linux issue with preamble and to my surprise it worked immediately, so probably something I have accidentally fixed in Windows. Can please somebody retest in Linux? I believe autocomplete works now great (I mean, fast enough). BTW, one hard job I would be happy to delegate is changing installation scripts to accound for libclang - or maybe even lack of it. Mirek
creator: mirek
date: 2022-07-07T11:23:12-00:00

Re: theide with libclang
type: item
description: Second milestone finished: File annotations and Topic++ (I mean, those documentation boxes in the left bar) are now clang based. Moving on to third milestone: Alt+C, hopefully this one will be easier...
creator: mirek
date: 2022-07-07T13:32:15-00:00

Re: theide with libclang, first milestone finished
type: item
description: mirek wrote on Thu, 07 July 2022 07:23 Can please somebody retest in Linux? I believe autocomplete works now great (I mean, fast enough). "make -f umkMakefile" - works. Regular "make" (build TheIDE) didn't work (problem with linking). Problem with CPU is fixed. Autocomplete doesn't work for me. For example: WStringBuffer txt; txt. Dot won't be autocompled. Ctrl+Space doesn't help either. I also tried autocomplete with TAB. To be fair, I never used autocomplete with TheIDE before, but I use it with VIM all the time.
creator: Novo
date: 2022-07-07T14:33:45-00:00

Re: theide with libclang, first milestone finished
type: item
description: Novo wrote on Thu, 07 July 2022 16:33 mirek wrote on Thu, 07 July 2022 07:23 Can please somebody retest in Linux? I believe autocomplete works now great (I mean, fast enough). "make -f umkMakefile" - works. Regular "make" (build TheIDE) didn't work (problem with linking). Problem with CPU is fixed. Autocomplete doesn't work for me. For example: WStringBuffer txt; txt. Dot won't be autocompled. Ctrl+Space doesn't help either. I also tried autocomplete with TAB. To be fair, I never used autocomplete with TheIDE before, but I use it with VIM all the time. That is weird. Looks like the whole thing is somewhat fragile in Linux Are you testing latest llbm_ide branch, right? (make reference is confusing me a bit) Mirek
creator: mirek
date: 2022-07-07T15:01:25-00:00

Re: theide with libclang, first milestone finished
type: item
description: Can you test removing "CXTranslationUnit_PrecompiledPreamble" from sources (you will easily find where)?
creator: mirek
date: 2022-07-07T15:02:14-00:00

Re: theide with libclang, first milestone finished
type: item
description: mirek wrote on Thu, 07 July 2022 11:01 Are you testing latest llbm_ide branch, right? (make reference is confusing me a bit) Yes, I'm using the llbm_ide branch. Last time I tested against rev. 14dad3e. The only possible problem I can imagine is that I copied .config (all of it) from the previous version of TheIDE.
creator: Novo
date: 2022-07-07T16:00:56-00:00

Re: theide with libclang, first milestone finished
type: item
description: mirek wrote on Thu, 07 July 2022 11:02 Can you test removing "CXTranslationUnit_PrecompiledPreamble" from sources (you will easily find where)? I checked that. Removing "CXTranslationUnit_PrecompiledPreamble" doesn't make any difference. Ctrl+Space uses string on the left as a prefix for search. Dot doesn't trigger method autocomplition. Interestingly, umk doesn't check dependencies anymore. I need to "rebuild all" each time.
creator: Novo
date: 2022-07-07T16:41:15-00:00

Re: theide with libclang, first milestone finished
type: item
description: So Ctrl+Space does show something? You can also try something selfcontained (no headers) like struct Foo { int x; }; void foo() { Foo(). }; Mirek
creator: mirek
date: 2022-07-07T16:49:11-00:00

Re: theide with libclang, first milestone finished
type: item
description: mirek wrote on Thu, 07 July 2022 12:49 So Ctrl+Space does show something? You can also try something selfcontained (no headers) like struct Foo { int x; }; void foo() { Foo(). }; Mirek With or without CXTranslationUnit_PrecompiledPreamble?
creator: Novo
date: 2022-07-07T16:55:06-00:00

Re: theide with libclang, first milestone finished
type: item
description: Novo wrote on Thu, 07 July 2022 12:41 Interestingly, umk doesn't check dependencies anymore. I need to "rebuild all" each time. This happens after "git restore". Maybe, git is getting time of last change and setting it on a file ... It is hard to tell what is going on here.
creator: Novo
date: 2022-07-07T16:58:53-00:00

Re: theide with libclang, first milestone finished
type: item
description: mirek wrote on Thu, 07 July 2022 12:49 So Ctrl+Space does show something? You can also try something selfcontained (no headers) like struct Foo { int x; }; void foo() { Foo(). }; Mirek In this case it works (with CXTranslationUnit_PrecompiledPreamble and no headers at all).
creator: Novo
date: 2022-07-07T17:05:37-00:00

Re: theide with libclang, first milestone finished
type: item
description: Novo wrote on Thu, 07 July 2022 19:05 mirek wrote on Thu, 07 July 2022 12:49 So Ctrl+Space does show something? You can also try something selfcontained (no headers) like struct Foo { int x; }; void foo() { Foo(). }; Mirek In this case it works (with CXTranslationUnit_PrecompiledPreamble and no headers at all). Now probably stupid question, I am just asking because I was there before: Are you sure your test code is syntactically correct? Does it compile? Is not it e.g. in #if 0 block? Also: What happens if you actually add some header? Does it break? Mirek
creator: mirek
date: 2022-07-07T17:10:24-00:00

Re: theide with libclang, first milestone finished
type: item
description: mirek wrote on Thu, 07 July 2022 13:10 Now probably stupid question, I am just asking because I was there before: Are you sure your test code is syntactically correct? Does it compile? Is not it e.g. in #if 0 block? I tested with this app. It is a real app. And autocomplition doesn't work at any place. I checked with another app. Dot works there ... Another problem: Ctrl+Space ignores locally defined symbols.
creator: Novo
date: 2022-07-07T17:39:09-00:00

Re: theide with libclang, first milestone finished
type: item
description: mirek wrote on Thu, 07 July 2022 13:10 Also: What happens if you actually add some header? Does it break? Adding "#include <iostream>" doesn't break it.
creator: Novo
date: 2022-07-07T17:45:23-00:00

Re: theide with libclang, first milestone finished
type: item
description: I'm testing TheIDE on Void Linux (rolling distro), which is using clang version 12.0.1 at this time.
creator: Novo
date: 2022-07-07T18:15:47-00:00

Re: theide with libclang, first milestone finished
type: item
description: Novo wrote on Thu, 07 July 2022 20:15 I'm testing TheIDE on Void Linux (rolling distro), which is using clang version 12.0.1 at this time. Well, in Windows I am at clang 14, in Linux clang 10 so that is unlikely the issue. That said, you still have not told me what Ctrl+Space does. Obviously it shows something based on your posts. Does it show U++ functions e.g.? Or, opposite, does autocomplete work with std::string? (with proper includes, of course) (in that case there would be a problem with include paths) Mirek
creator: mirek
date: 2022-07-07T18:44:02-00:00

Re: theide with libclang, first milestone finished
type: item
description: mirek wrote on Thu, 07 July 2022 14:44 That said, you still have not told me what Ctrl+Space does. Obviously it shows something based on your posts. Does it show U++ functions e.g.? App, where autocompletion works: bool amap::AnalyzeMap(String fn) { String map = LoadFile(fn); char nm[1024]; if (IsNull(map)) return false; MapFileParser::MSVC MSVC(map); MapFileParser::CodeWarriorW CWW(map); MapFileParser::CodeWarrior CW(map); MapFileParser::PS3 PS3(map); MapFileParser::GCC3 GCC3(map, mcr); int addr; int sect_len; int prev_size = 0; uint64 pref_addr; int size; int modulen; int filen; String sect_name; String sect_class; String funct_name; String module_name; String file_name; String app_name; String time_stamp; One<MFile> cur_mfile(new MFile); if (map.StartsWith(elf::ElfMagic, 4)) { // Elf file. is32bit = map[4] == elf::ELFCLASS32; isLittleEndian = map[5] == elf::ELFDATA2LSB; const uint8 osABI = map[6]; const uint8 abiVersion = map[7]; const elf::Elf32_Ehdr* hdr = reinterpret_cast<const elf::Elf32_Ehdr*>(map.Begin()); If I enter "file_" and press Ctrl+Space I get "file_handle". file_name, which is defined locally, is ignored. I do not know what Ctrl+Space is supposed to do, but it definitely not doing anything useful to me.
creator: Novo
date: 2022-07-07T19:35:01-00:00

Re: theide with libclang, first milestone finished
type: item
description: mirek wrote on Thu, 07 July 2022 14:44 Or, opposite, does autocomplete work with std::string? (with proper includes, of course) (in that case there would be a problem with include paths) std::string works. Below is my test case. #include <string> struct Foo { int x; }; void foo() { Foo(); }; int main(int argc, const char *argv[]) { std::string str; str. return 0; }
creator: Novo
date: 2022-07-07T19:42:06-00:00

Re: theide with libclang, first milestone finished
type: item
description: mirek wrote on Thu, 07 July 2022 14:44 That said, you still have not told me what Ctrl+Space does. Obviously it shows something based on your posts. Does it show U++ functions e.g.? In an app, where "dot" doesn't work Ctrl+Space shows functions from the top level (U++ functions). That works.
creator: Novo
date: 2022-07-07T20:04:43-00:00

Re: theide with libclang, first milestone finished
type: item
description: Novo wrote on Thu, 07 July 2022 13:39 I tested with this app. It is a real app. And autocompletion doesn't work at any place. Actually, it works for locally defined symbols. TheIDE is unable to resolve and autocomplete members of a class. The app is just one cpp file less that 2K lines of code. I believe this is a problem with libclang because I'm getting the same behavior with VIM and ccls Language Server. I believe you need to use LibTooling instead of LibClang with TheIDE.
creator: Novo
date: 2022-07-08T22:32:13-00:00

Re: theide with libclang, first milestone finished
type: item
description: Novo wrote on Sat, 09 July 2022 00:32 Novo wrote on Thu, 07 July 2022 13:39 I tested with this app. It is a real app. And autocompletion doesn't work at any place. Actually, it works for locally defined symbols. TheIDE is unable to resolve and autocomplete members of a class. The app is just one cpp file less that 2K lines of code. I believe this is a problem with libclang because I'm getting the same behavior with VIM and ccls Language Server. I believe you need to use LibTooling instead of LibClang with TheIDE. After playing with it for a while I think it is that preamble issue again. I have returned #ifdef to HEAD, after that your code snippet started showing correct autocomplete. Now it is definitely a temporary solution as it is quite slow that way, hopefully we will find a fix soon. Mirek
creator: mirek
date: 2022-07-09T11:11:48-00:00

Re: theide with libclang, first milestone finished
type: item
description: mirek wrote on Sat, 09 July 2022 07:11 Now it is definitely a temporary solution as it is quite slow that way, hopefully we will find a fix soon. I tested latest source code. I do not see any changes. Basically, source code where autocompletion doesn't work: bool App::Parse(const String& file_name) { using namespace conllu; FileMapping s; if (!s.Open(file_name)) return false; if (!s.Map(0, static_cast<dword>(s.GetFileSize()))) return false; arr.Clear(); sent.Clear(); conlluTree.Clear(); sentv.Clear(); Parser p(reinterpret_cast<const char*>(s.Begin()), reinterpret_cast<const char*>(s.End()), sentv); while (p.Sentence()) ; for (const drtree::Sentence& v: sentv) sent.Add(AsString(v.GetWordV().GetCount()), v.GetText()); return true; } In case of "arr.", "sent.", "conlluTree.", e.t.c. autocompletion doesn't work. In case of "s." and "p." it does. The only reason LibClang is needed is "name resolution". It is much harder to develop a name resolution algorithm than to make a C++ parser itself. If name resolution doesn't work for some reason, then there is no reason to use LibClang. There are better solutions for parsing.
creator: Novo
date: 2022-07-09T17:44:48-00:00

Re: theide with libclang, first milestone finished
type: item
description: Novo wrote on Sat, 09 July 2022 19:44 mirek wrote on Sat, 09 July 2022 07:11 Now it is definitely a temporary solution as it is quite slow that way, hopefully we will find a fix soon. I tested latest source code. I do not see any changes. Basically, source code where autocompletion doesn't work: bool App::Parse(const String& file_name) { using namespace conllu; FileMapping s; if (!s.Open(file_name)) return false; if (!s.Map(0, static_cast<dword>(s.GetFileSize()))) return false; arr.Clear(); sent.Clear(); conlluTree.Clear(); sentv.Clear(); Parser p(reinterpret_cast<const char*>(s.Begin()), reinterpret_cast<const char*>(s.End()), sentv); while (p.Sentence()) ; for (const drtree::Sentence& v: sentv) sent.Add(AsString(v.GetWordV().GetCount()), v.GetText()); return true; } In case of "arr.", "sent.", "conlluTree.", e.t.c. autocompletion doesn't work. In case of "s." and "p." it does. The only reason LibClang is needed is "name resolution". It is much harder to develop a name resolution algorithm than to make a C++ parser itself. If name resolution doesn't work for some reason, then there is no reason to use LibClang. It works 100% with clang 14 in Windows, hence I think the problem is somewhere else than in clang. It works without preamble in clang 10 for me too. Too soon to try something else at this point. Now this is one of troubles I expected before going clang - we are now victim of external incompatibilities. But what is done is done. Quote: There are better solutions for parsing. And these are? Mirek
creator: mirek
date: 2022-07-09T22:49:30-00:00

Re: theide with libclang, first milestone finished
type: item
description: mirek wrote on Sat, 09 July 2022 18:49 And these are? Tree-sitter - it has been a big thing for several years.
creator: Novo
date: 2022-07-09T23:49:46-00:00

Re: theide with libclang, first milestone finished
type: item
description: mirek wrote on Sat, 09 July 2022 18:49 It works 100% with clang 14 in Windows, hence I think the problem is somewhere else than in clang. It works without preamble in clang 10 for me too. Too soon to try something else at this point. IMHO, you just haven't tested it very well yet. I was just "lucky" to hit a bug with my first test. My code is not that complicated. It is mostly C++98, but I use a lot of simple templates, but nothing fancy like SFINAE. mirek wrote on Sat, 09 July 2022 18:49 Now this is one of troubles I expected before going clang - we are now victim of external incompatibilities. But what is done is done. I do not think this is a Windows/Linux (or version of clang) incompatibility problem. I'm getting the same behavior with VIM and ccls (which is similar to clangd), and with Kakoune and kak-lsp + ccls. I use this technology for several years and I never rely on clang autocompletion, but "goto definition" works very well for me. Even in this broken app "goto definition" of sentv work fine, but for some reason language server is unable to figure out a type of sentv. This is weird. This shouldn't be a problem. So, I personally feel fine with using LibClang + VIM because VIM has at least half a dozen of different builtin autocompletion tools. I do not care even when LibClang-related tools crash from time to time because they are out of process tools. I just do not know which part of the system (libclang or ccls) is responsible for all these problems. But in case of TheIDE this can be a bigger problem, IMHO.
creator: Novo
date: 2022-07-10T00:53:55-00:00

Re: theide with libclang, first milestone finished
type: item
description: Novo wrote on Sun, 10 July 2022 02:53 mirek wrote on Sat, 09 July 2022 18:49 It works 100% with clang 14 in Windows, hence I think the problem is somewhere else than in clang. It works without preamble in clang 10 for me too. Too soon to try something else at this point. IMHO, you just haven't tested it very well yet. I was just "lucky" to hit a bug with my first test. My code is not that complicated. It is mostly C++98, but I use a lot of simple templates, but nothing fancy like SFINAE. In Windows, I am using autocomplete for real work (really complex C++17 rich project) for about 14 days. Works flawlessly (of course, gradually maturing some things, but I have not encountered moment when it would fail to provide correct list of methods/variables). I could try with your code if I could download it somewhere. If I delete "unknown" stuff from short snippets you are posting, it works fine in Windows/libclang 14. Mirek
creator: mirek
date: 2022-07-10T06:46:59-00:00

Re: theide with libclang, first milestone finished
type: item
description: Hello Mirek, Few remarks from me while testing new implementation with uppsrc/ide project on Linux. It doesn't assist with member variables. For example in idewin.cpp (Ide::ConsolePaste()) with following code: void Ide::ConsolePaste() { String s = ReadClipboardText(); if(!IsNull(s)) { s.Insert(0, '\n'); int len = console.GetLength(); console.Insert(len, s.ToWString()); console.SetCursor(len + 1); console. // <- No help at all } } The other problem within the same method is that CTRL+Space work with s. It doesn't show functionality with AString, which is essential for working with String: void Ide::ConsolePaste() { String s = ReadClipboardText(); if(!IsNull(s)) { s.Insert(0, '\n'); int len = console.GetLength(); console.Insert(len, s.ToWString()); console.SetCursor(len + 1); s. // <- No AString method, only derived method from String } } In the same place when you "include <string>" from standard library, assist will do not show anything: void Ide::ConsolePaste() { String s = ReadClipboardText(); if(!IsNull(s)) { s.Insert(0, '\n'); int len = console.GetLength(); console.Insert(len, s.ToWString()); console.SetCursor(len + 1); std::string str2; str2. // <- No assist... } } Also, I just noticed that CTRL+Space for String is slow for example in release mode. It is not immediate and it takes some time until dialog appear on the screen. The lag is perceptible. -------------------------- The code navigation works correct and it is fast (CTRL+Left Mouse Button) and for me it looks like improvement. Klugier
creator: Klugier
date: 2022-07-10T09:06:38-00:00

Re: theide with libclang, first milestone finished
type: item
description: So there is definitely some issue with my libclang usage that demonstrates as a problem in Linux, something to figure out, but I do not think this is time to capitulate yet. As for code navigation, that is still my old code, that is the next milestone to change that to libclang. Things being slow are worked on as well. Mirek
creator: mirek
date: 2022-07-10T14:56:30-00:00

Re: theide with libclang, first milestone finished
type: item
description: OK, so I have tried to install llvm-14 in my linux distro and linked theide with its libclang - and things started to work even with preamble, totally fine, fast, just as in Windows (except some fixing of include paths to get autocomplete of std:: stuff, but nothing serious; that part is under development anyway). (Note - if you wonder why I am speaking about preamble so much - preamble is a flag that you give to libclang that makes it create implicit precompiled header in the first file parse. This is very important for autocomplete because we do that parse in background thread and when user needs the autocomplete, all is ready and fast. So it is really important for comfort - using preamble, autocomplete is almost instant (~400ms). Without it can take 1-5 seconds to popup). So my conclusion: I will only use preamble only when I detect libclang 14, I do not care about outdated distros. Autocomplete will be slow if libclang 14 is not available. Latest ubuntu has 14, so... Maybe one last thing to try is whether libclang 13 works, if somebody has time to test. This will be hell in the installation script (to get the most fresh libclang for given distro or even disable libclang completely), but what can we do... I really would appreciate help with installation script BTW! Mirek
creator: mirek
date: 2022-07-11T16:40:24-00:00

Re: theide with libclang, first milestone finished
type: item
description: Hello Mirek, libclang-14 requierments explains a lot. On Manjaro current available version is libclang-13, so maybe the issues mentioned by me in my previous message are caused by insufficient libclang library version. Manjaro is rolling release, it means that it has relatively new packages. So, the general adoption of libclang-14 in the Linux word is low. Quote: So it is really important for comfort - using preamble, autocomplete is almost instant (~400ms). Without it can take 1-5 seconds to popup). I can confirm that CTRL+Space take some time to appear on the screen. Is there a way to optimize it for example by pre generating auto complete or using caching? Also, I think user should be somehow inform that something is generating under the hood. For me it looks like TheIDE is unresponsive after using Ctrl+Space. Moreover, ~400ms might be too much. You need to take into consideration that it calculates that time on your CPU. With weaker CPU's this value might be bigger. So, maybe we could also optimize this using similar techniques I proposed. Could you also tell me from where on Windows I can download libclang-14? I would like to have comparison. Klugier
creator: Klugier
date: 2022-07-11T19:57:30-00:00

Re: theide with libclang, first milestone finished
type: item
description: Klugier wrote on Mon, 11 July 2022 21:57 Hello Mirek, libclang-14 requierments explains a lot. OK, some good news. I have finally figured it out and it now works fine with libclang 10 as well (commit 046f816b5a12aae4813cf4a7a83e4b6f11c13ca9) The problem was actually in those headers, sort of as expected. In windows, I need to add compiler options for header paths even for standard headers. I left that feature in linux as well and it turns out that this is the primary cause of the problem. I guess the real issue is there is too many header paths or maybe duplicating standard paths is confusing clang. All in all, it seems to work great in linux now. Mirek
creator: mirek
date: 2022-07-11T20:27:10-00:00

Re: theide with libclang, first milestone finished
type: item
description: Hello Mirek, I tested on the latest commit and from my perspective the performance is better, however my feelings are that Assist++ is not supper responsive and you can fell that it takes some time to appear. Could you add a release log file that will measure the Assist++ opening time? From request to appear. For the rest of the issues I have mentioned in my previous message, everything is the same. Here is screenshot from String without AString methods: The console.${Assist++} doesn't open assist. Klugier
creator: Klugier
date: 2022-07-11T20:49:17-00:00

Re: theide with libclang, first milestone finished
type: item
description: Hello Mirek, I think it would be nice to add "No suggestions" information when you explicitly hit CTRL+Space. Right now it is difficult to distinguish whenever Assist++ can not find any symbols or it is working under the hood. Here is the feature from VSCode ("Brak sugesti." = "No sugesttions."): Klugier
creator: Klugier
date: 2022-07-11T21:00:46-00:00

Re: theide with libclang, first milestone finished
type: item
description: So I guess I have reached another (and sort of final) milestone - for more than week I am now working with theide with clang based navigator (that means, it is running 'indexer' in background - the code that continually maintains the database of the whole project). BTW, it takes 20 seconds to reindex theide sources on my machine - BLITZ and 16 cores in action.... Not everything is happy valley though, theide requires far more powerful HW than before. Reindexing consumes a lot of memory (around 6GB) - that is understandable considering we are running 32 C++ compilers in parallel to do it. Also autocomplete is sort of lazy in large projects as compared to my in-house parser, but that is the price to pay I guess... There is a lot of things to fix and mature yet, my defect list has around 70 entries... Simultaneously, I will be adding nice new shiny features; new theide already got "Search for usages" that seems to be pretty useful. I have also changed win32 nightly to contain all .dll and headers to be able to compile this llbm_ide branch directly, all you need to make sure is that new ide.exe is placed into the same directory as installed one (you can actually even use "Setup/Upgrade theide.." I think). The remaining major issue is how to install this in linux....
creator: mirek
date: 2022-08-15T08:44:33-00:00

Re: theide with libclang, first milestone finished
type: item
description: Hello Mirek, Thanks for the update, I will play with llbm_ide branch. Quote: Not everything is happy valley though, theide requires far more powerful HW than before. Reindexing consumes a lot of memory (around 6GB) - that is understandable considering we are running 32 C++ compilers in parallel to do it. Also autocomplete is sort of lazy in large projects as compared to my in-house parser, but that is the price to pay I guess... As, I understand the number of C++ compiler jobs depends on HYDRA threads parameter. Could it be limited to one thread only if HYDRA threads are set to one? It would be good if TheIDE will still work on all these ARM devices that has single CPU. With only one thread running, the memory requirements should also be lower. We have a lot of users from countries where there are difficulties to have access to modern hardware. In such cases TheIDE should work fine, even if indexing will take longer. Scanning operation takes some time, it would be good to let user know that something is happening under the hood. We should avoid situation when user fells confusion. I think we can introduce following things: - scanning progress is present in status bar (A lot of ide's are using this option...). Status bar is visible by default. - CTRL+SPACE should indicate that the scanning is in progress and symbols are not ready yet. The information can be displayed in tooltip window. - CTRL+SPACE should indicate that there are no symbols after scan (please back to my previous message). The same situation it can be displayed in tooltip window. This is exactly what I have presented in my previous message. Klugier
creator: Klugier
date: 2022-08-15T16:44:42-00:00

Re: theide with libclang, first milestone finished
type: item
description: Klugier wrote on Mon, 15 August 2022 18:44 - CTRL+SPACE should indicate that there are no symbols after scan (please back to my previous message). The same situation it can be displayed in tooltip window. This is exactly what I have presented in my previous message. Klugier Well, there are always some symbols to display after CTRL+SPACE (unless you are in the completely empty file)... Better indication of autocomplete in progress is coming.. Mirek
creator: mirek
date: 2022-08-16T06:04:31-00:00

Re: theide with libclang
type: item
description: Linux. Branch: llbm_ide Command: make -f umkMakefile In file included from ./uppsrc/CtrlCore/CtrlCore.h:49, from ./uppsrc/CtrlLib/CtrlLib.h:4, from ./uppsrc/RichEdit/RichEdit.h:4, from ./uppsrc/ide/ide.h:4, from ./uppsrc/ide/Core/Ide.cpp:1: ./uppsrc/CtrlCore/Gtk.h:12:10: fatal error: gtk/gtk.h: No such file or directory 12 | #include <gtk/gtk.h> | ^~~~~~~~~~~ umk is a console app ...
creator: Novo
date: 2022-08-16T17:18:36-00:00

Re: theide with libclang
type: item
description: Linux. Branch: llbm_ide Command: ./umk uppsrc ide CLANG -arbus Problem: Fatal error Invalid memory access! (on opening a project)
creator: Novo
date: 2022-08-16T17:25:21-00:00

Re: theide with libclang
type: item
description: Novo wrote on Tue, 16 August 2022 19:18 Linux. Branch: llbm_ide Command: make -f umkMakefile In file included from ./uppsrc/CtrlCore/CtrlCore.h:49, from ./uppsrc/CtrlLib/CtrlLib.h:4, from ./uppsrc/RichEdit/RichEdit.h:4, from ./uppsrc/ide/ide.h:4, from ./uppsrc/ide/Core/Ide.cpp:1: ./uppsrc/CtrlCore/Gtk.h:12:10: fatal error: gtk/gtk.h: No such file or directory 12 | #include <gtk/gtk.h> | ^~~~~~~~~~~ umk is a console app ... Fixed.
creator: mirek
date: 2022-08-19T11:03:03-00:00

Re: theide with libclang, first milestone finished
type: item
description: Klugier wrote on Mon, 15 August 2022 18:44 - scanning progress is present in status bar (A lot of ide's are using this option...). Status bar is visible by default. Working on it all, however the idea is that scanning will be represented by left bar changing color (which makes sense as there are annotation that get updated).
creator: mirek
date: 2022-08-19T11:04:58-00:00

Re: theide with libclang, first milestone finished
type: item
description: Hopefully fixed Linux crashing in release mode.
creator: mirek
date: 2022-08-19T11:05:14-00:00

Re: theide with libclang, first milestone finished
type: item
description: mirek wrote on Fri, 19 August 2022 07:05 Hopefully fixed Linux crashing in release mode. Thanks. TheIde works fine on Linux.
creator: Novo
date: 2022-08-20T18:00:24-00:00

Re: theide with libclang, first milestone finished
type: item
description: CPU usage. TheIde is heavily using CPU even if I change one character. This is a problem for my ten-years-old laptop. Is it possible to add an options, which will 1) disable libclang completely. 2) limit reindexing to a "Save" command. 3) keep libclang running all the time. TIA.
creator: Novo
date: 2022-08-20T18:33:27-00:00

Re: theide with libclang, first milestone finished
type: item
description: Hi, I had to move #include <X11/Xlib.h> in GtkApp.cpp in order to compile in Windows with clang: #include <CtrlCore/CtrlCore.h> #ifdef GUI_GTK #include <X11/Xlib.h> #define CATCH_ERRORS 1 Additionally, compilation fails to missing include in MSBT22x64: C:\upp-16323\upp.src\uppsrc\ide/clang/clang.h (5): fatal error C1083: Cannot open include file: 'clang-c/Index.h': No such file or directory Best regards, Tom
creator: Tom1
date: 2022-08-20T18:34:45-00:00

Re: theide with libclang, first milestone finished
type: item
description: Tom1 wrote on Sat, 20 August 2022 20:34 Hi, I had to move #include <X11/Xlib.h> in GtkApp.cpp in order to compile in Windows with clang: #include <CtrlCore/CtrlCore.h> #ifdef GUI_GTK #include <X11/Xlib.h> #define CATCH_ERRORS 1 Additionally, compilation fails to missing include in MSBT22x64: C:\upp-16323\upp.src\uppsrc\ide/clang/clang.h (5): fatal error C1083: Cannot open include file: 'clang-c/Index.h': No such file or directory Best regards, Tom Compilation of theide with MSC is not supported at the moment. Not sure it will be possible. Use CLANG64... Mirek
creator: mirek
date: 2022-08-20T22:42:58-00:00

Re: theide with libclang, first milestone finished
type: item
description: Novo wrote on Sat, 20 August 2022 20:33 CPU usage. TheIde is heavily using CPU even if I change one character. This is a problem for my ten-years-old laptop. Is it possible to add an options, which will 1) disable libclang completely. 2) limit reindexing to a "Save" command. 3) keep libclang running all the time. TIA. Disable is needed. However, reindexing is already issued only in specific moments (save being one of them). The problem is that initially it has to do it all. Also, changing headers can trigger reindexing of a lot of files. This is the price to pay.... (this is the reason I was reluctant to do abandon my parser....) Mirek
creator: mirek
date: 2022-08-20T22:46:22-00:00

Re: theide with libclang, first milestone finished
type: item
description: Good news: I have found a bug in my code (unitialised variable for all things) that was making autocomplete slow. It now seems to be superfast after the initial parse. Things appear to fall in place. The last major hurdle is to figure out Linux installation. The new idea to try is to use runtime .so loading... Mirek
creator: mirek
date: 2022-08-30T14:04:45-00:00

Re: theide with libclang, first milestone finished
type: item
description: Hi Mirek, Each time I have e.g.: void LeftDown(Point p, dword keyflags) override { I get a row: "WARNING: EMIT" in my Navigator. If I remove the override, it goes away. (I do not think this is anything new, but it shows up in the latest version too.) Second, there are some warnings (at least on MSBT): C:\upp-git\upp.src\uppsrc\Core\Other.h(123): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data C:\upp-git\upp.src\uppsrc\Core\Other.h(143): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data C:\upp-git\upp.src\uppsrc\CtrlLib/DropChoice.h(83): warning C4099: 'Upp::PopUpList::Popup': type name first seen using 'struct' now seen using 'class' C:\upp-git\upp.src\uppsrc\CtrlLib/DropChoice.h(54): note: see declaration of 'Upp::PopUpList::Popup' Best regards, Tom
creator: Tom1
date: 2022-08-31T08:00:43-00:00

Re: theide with libclang, first milestone finished
type: item
description: Tom1 wrote on Wed, 31 August 2022 10:00 Hi Mirek, Each time I have e.g.: void LeftDown(Point p, dword keyflags) override { I get a row: "WARNING: EMIT" in my Navigator. If I remove the override, it goes away. (I do not think this is anything new, but it shows up in the latest version too.) I have to say I am a bit confused here... Can you post a screenshot? (full window so that I can see the source etc.. too)
creator: mirek
date: 2022-08-31T08:13:21-00:00

Re: theide with libclang, first milestone finished
type: item
description: Hi, Sorry. This must be caused by something strange deeper in my own code, since creating a simple test case does not seem to trigger showing it. Anyway, I will stop wasting your time on this one for now. (I bet you have something better to do... Thanks and best regards, Tom
creator: Tom1
date: 2022-08-31T08:50:42-00:00

Re: theide with libclang, first milestone finished
type: item
description: Tom1 wrote on Wed, 31 August 2022 10:50 Hi, Sorry. This must be caused by something strange deeper in my own code, since creating a simple test case does not seem to trigger showing it. Anyway, I will stop wasting your time on this one for now. (I bet you have something better to do... Thanks and best regards, Tom Ah, ok, by the screenshot I immediatelly see this non-clang version anyway (new theide has arrow instead of ":") Mirek
creator: mirek
date: 2022-08-31T09:37:58-00:00

Re: theide with libclang, first milestone finished
type: item
description: mirek wrote on Wed, 31 August 2022 12:37 Tom1 wrote on Wed, 31 August 2022 10:50 Hi, Sorry. This must be caused by something strange deeper in my own code, since creating a simple test case does not seem to trigger showing it. Anyway, I will stop wasting your time on this one for now. (I bet you have something better to do... Thanks and best regards, Tom Ah, ok, by the screenshot I immediatelly see this non-clang version anyway (new theide has arrow instead of ":") Mirek Thanks Mirek! For some reason my upp.src had slipped back to master branch. Anyway, I switched back to clang and now all my problems are gone. Arrows are shown instead of colon and there are no more "WARNING: EMIT" for overrides. Thanks! Best regards, Tom
creator: Tom1
date: 2022-08-31T10:33:47-00:00

Re: theide with libclang, first milestone finished
type: item
description: Mirek, Would it be possible to add support for working with git branches in TheIDE menu? Support for 'switching branch' for any git repository currently in use would be most useful, but possibly also support for creating and merging of branches. Best regards, Tom
creator: Tom1
date: 2022-09-15T06:31:11-00:00

Re: theide with libclang, first milestone finished
type: item
description: Tom1 wrote on Thu, 15 September 2022 08:31 Mirek, Would it be possible to add support for working with git branches in TheIDE menu? Support for 'switching branch' for any git repository currently in use would be most useful, but possibly also support for creating and merging of branches. Best regards, Tom I would like to, I just do not quite know how to do it right. So far "Terminal at file directory" was my favorite tool in this regard... Plus it is not quite related to Assist/clang, correct? I plan a batch of another theide improvements after libclang is merged. Which now seems will be quite soon. Mirek
creator: mirek
date: 2022-09-15T07:48:18-00:00

Re: theide with libclang, first milestone finished
type: item
description: mirek wrote on Thu, 15 September 2022 10:48 Tom1 wrote on Thu, 15 September 2022 08:31 Mirek, Would it be possible to add support for working with git branches in TheIDE menu? Support for 'switching branch' for any git repository currently in use would be most useful, but possibly also support for creating and merging of branches. Best regards, Tom I would like to, I just do not quite know how to do it right. So far "Terminal at file directory" was my favorite tool in this regard... Plus it is not quite related to Assist/clang, correct? I plan a batch of another theide improvements after libclang is merged. Which now seems will be quite soon. Mirek Yes, sorry. This is not the place for this discussion. Anyway, I'm looking forward to progress on this too. Best regards, Tom
creator: Tom1
date: 2022-09-15T12:10:15-00:00

Re: theide with libclang, first milestone finished
type: item
description: Merged...
creator: mirek
date: 2022-09-16T09:28:14-00:00

Re: theide with libclang, first milestone finished
type: item
description: Merged version. Linux. libclang: crash detected during reparsing libclang: crash detected during parsing: { 'source_filename' : '(null)' 'command_line_args' : ['clang', '-I/home/ssg/dvlp/cpp/sergey/upp', '-I/home/ssg/dvlp/cpp/code/upp/git/uppsrc', '-I/home/ssg/dvlp/cpp/code/upp/git/bazaar', '-I/home/ssg/.local/tmp/dvlp/cpp/code/upp/.config/u++/ide/UppHub/SysExec', '-I/home/ssg/.local/tmp/dvlp/cpp/code/upp/.config/u++/ide/UppHub/Updater', '-I/usr/include/freetype2', '-I/usr/include/libpng16 ', '-I/usr/include/freetype2', '-I/usr/include/libpng16 ', '-I/usr/include/freetype2', '-I/usr/include/libpng16 ', '-I/usr/include/gtk-3.0', '-I/usr/include/at-spi2-atk/2.0', '-I/usr/include/at-spi-2.0', '-I/usr/include/dbus-1.0', '-I/usr/lib64/dbus-1.0/include', '-I/usr/include/gtk-3.0', '-I/usr/include/gio-unix-2.0', '-I/usr/include/cairo', '-I/usr/include/pango-1.0', '-I/usr/include/harfbuzz', '-I/usr/include/pango-1.0', '-I/usr/include/fribidi', '-I/usr/include/harfbuzz', '-I/usr/include/atk-1.0', '-I/usr/include/cairo', '-I/usr/include/pixman-1', '-I/usr/include/freetype2', '-I/usr/include/libpng16', '-I/usr/include/gdk-pixbuf-2.0', '-I/usr/include/libpng16', '-I/usr/include/libmount', '-I/usr/include/blkid', '-I/usr/include/glib-2.0', '-I/usr/lib64/glib-2.0/include ', '-I/usr/include/gdk-pixbuf-2.0', '-I/usr/include/libpng16', '-I/usr/include/libmount', '-I/usr/include/blkid', '-I/usr/include/glib-2.0', '-I/usr/lib64/glib-2.0/include ', '-I/usr/include/libpng16 ', '-I/usr/include/c++/10.2', '-I/usr/include/c++/10.2/x86_64-unknown-linux-gnu', '-I/usr/include/c++/10.2/backward', '-I/usr/local/include', '-I/usr/lib/clang/12.0.1/include', '-I/usr/include', '-I/usr/include/c++/10.2', '-I/usr/include/c++/10.2/x86_64-unknown-linux-gnu', '-I/usr/include/c++/10.2/backward', '-I/usr/local/include', '-I/usr/lib/clang/12.0.1/include', '-I/usr/include', '-DflagGUI', '-DflagPOSIX', '-DflagLINUX', '-DCLANG', '/home/ssg/dvlp/cpp/code/upp/git/uppsrc/Core/lheap.cpp', '-DflagDEBUG', '-DflagDEB…
creator: Novo
date: 2022-09-17T03:54:28-00:00

Re: theide with libclang, first milestone finished
type: item
description: Is it possible to disable libclang support during compilation? I'm having troubles trying to cross-compile Clang. Actually, I couldn't compile it at all. I'm trying to cross-compile TheIDE for Windows on Linux.
creator: Novo
date: 2022-09-17T04:44:49-00:00

Re: theide with libclang, first milestone finished
type: item
description: What is your HW configuration? I am setting some defaults for assist config, but the lowest HW I can test with is Ryzen 2400G 8GB Linux - they seems to work fine. Check Setup BTW, I would start with disabling automatic indexer... BTW, I might be wrong, but I think for theide, every "indexer thread" and "autocomplete file cache" is about 200MB of memory. Mirek
creator: mirek
date: 2022-09-17T09:04:26-00:00

Re: theide with libclang, first milestone finished
type: item
description: mirek wrote on Sat, 17 September 2022 05:04 What is your HW configuration? Intel(R) Core(TM) i7-3720QM CPU @ 2.60GHz (4 cores / 8 threads) 16GM RAM It is a Lenovo W530. (I also have a T530 ) I like the keyboard on these models
creator: Novo
date: 2022-09-17T10:23:38-00:00

Re: theide with libclang, first milestone finished
type: item
description: mirek wrote on Sat, 17 September 2022 05:04 Check Setup BTW, I would start with disabling automatic indexer... BTW, I might be wrong, but I think for theide, every "indexer thread" and "autocomplete file cache" is about 200MB of memory. Opening a project and closing it in a couple of seconds leads to "Invalid memory access" ... EDIT: Happens after full rebuild.
creator: Novo
date: 2022-09-17T10:37:03-00:00

Re: theide with libclang, first milestone finished
type: item
description: Novo wrote on Sat, 17 September 2022 12:23 mirek wrote on Sat, 17 September 2022 05:04 What is your HW configuration? Intel(R) Core(TM) i7-3720QM CPU @ 2.60GHz (4 cores / 8 threads) 16GM RAM It is a Lenovo W530. (I also have a T530 ) I like the keyboard on these models That is strange, that is about the same or better than my linux machine.... Maybe your libclang is somehow deffective? Mirek
creator: mirek
date: 2022-09-17T10:48:47-00:00

Re: theide with libclang, first milestone finished
type: item
description: mirek wrote on Sat, 17 September 2022 06:48 Maybe your libclang is somehow deffective? I'm using "Void Linux". Its default clang version is 12.0.1. Void Linux is like a stable version of Arch. I do not have any problems with ccls on these machines ...
creator: Novo
date: 2022-09-17T10:57:33-00:00

Re: theide with libclang, first milestone finished
type: item
description: I'm getting weird error messages in console on exit. /usr/include/glib-2.0/glib/gtypes.h:32:10: error: 'glibconfig.h' file not found /usr/include/glib-2.0/glib/gversionmacros.h:375:2: error: "GLIB_VERSION_MIN_REQUIRED must be >= GLIB_VERSION_2_26" /usr/include/glib-2.0/glib/gtypes.h:548:2: error: unknown ENDIAN type /usr/include/glib-2.0/glib/garray.h:49:3: error: unknown type name 'guint8' /usr/include/glib-2.0/glib/garray.h:75:36: error: unknown type name 'gsize' ... /home/ssg/dvlp/cpp/sergey/upp/amap/amap.cpp:81:2: error: cannot initialize object parameter of type 'Upp::TopWindow' with an expression of type 'amap' /home/ssg/dvlp/cpp/sergey/upp/amap/amap.cpp:83:2: error: no matching member function for call to 'Icon' /home/ssg/dvlp/cpp/code/upp/git/uppsrc/CtrlCore/TopWindow.h:174:13: note: candidate function not viable: no known conversion from 'amap' to 'Upp::TopWindow' for object argument /home/ssg/dvlp/cpp/code/upp/git/uppsrc/CtrlCore/TopWindow.h:176:13: note: candidate function not viable: requires 2 arguments, but 1 was provided ... fatal error: too many errors emitted, stopping now [-ferror-limit=]
creator: Novo
date: 2022-09-17T11:01:02-00:00

Re: theide with libclang
type: item
description: Seemed OK at first but: - Include file completion doesn't work (you're probably already aware of this) - Alt+C works in .cpp files (only one direction though, from inner class scope to outer) but not .h files. It also appears to sometimes get the outer class name wrong. Biggest issue thus far is 100% CPU and memory exhaustion. Before indexing, TheIDE uses around 50 megabytes, after first indexing it goes up to around 150, then eventually libclang breaks, autocomplete stops working for some symbols, then all - a second indexing increases memory usage until it reaches around 1.8 gig. This is with only one core enabled. With all cores, it uses all available system memory and freezes the OS... typically when compiling.
creator: jjacksonRIAB
date: 2022-09-18T10:22:57-00:00

Re: theide with libclang
type: item
description: jjacksonRIAB wrote on Sun, 18 September 2022 12:22 Seemed OK at first but: - Include file completion doesn't work (you're probably already aware of this) I went really far to make it work (is not really simple with libclang), so that is unpleasant surprise. Quote: Biggest issue thus far is 100% CPU and memory exhaustion. Before indexing, TheIDE uses around 50 megabytes, after first indexing it goes up to around 150, then eventually libclang breaks, autocomplete stops working for some symbols, then all - a second indexing increases memory usage until it reaches around 1.8 gig. This is with only one core enabled. With all cores, it uses all available system memory and freezes the OS... typically when compiling. Can you please tell me details about the machine? I was mostly testing / using in Windows; there with 32 indexing threads the memory goes to somewhere like 6GB.
creator: mirek
date: 2022-09-18T11:05:09-00:00

Re: theide with libclang
type: item
description: Linux mint, it's an older machine with 8 cores and 16 gig of memory, libclang should be version 12 IIRC. If 6 gig is common I'll probably have to turn it off because the compiler causes a lot of memory pressure as is, oh well
creator: jjacksonRIAB
date: 2022-09-18T11:27:32-00:00

Re: theide with libclang
type: item
description: In my case (Lenovo ThikPad T530 8GB RAM, 2 cores (4 threads), indexer is using 4 threads) when indexing TheIde itself: - 1.3GB when reindexing all src. Memory usage doesn't go down after reindexing. - 300MB when opening an already indexed TheIde project. I guess, memory allocation strategy is inefficient.
creator: Novo
date: 2022-09-18T14:57:16-00:00

Re: theide with libclang
type: item
description: valgrind explains everything.
creator: Novo
date: 2022-09-18T19:23:27-00:00

Re: theide with libclang
type: item
description: Novo wrote on Sun, 18 September 2022 21:23 valgrind explains everything. While it is definitely a good idea, only one of these errors seems libclang related and fixable and it happens on theide exit. Anyway, now fixed. That said, everything valigrind says is worth investigation. I was able to fix many of these issues (in master now), unfortunately it looks like majority of remaining problems is cairo related (these happen when we are reading images from gtk). Any help identifying the cause(s) is welcome. Mirek
creator: mirek
date: 2022-09-20T11:09:38-00:00

Re: theide with libclang
type: item
description: mirek wrote on Tue, 20 September 2022 07:09 Novo wrote on Sun, 18 September 2022 21:23 valgrind explains everything. While it is definitely a good idea, only one of these errors seems libclang related and fixable and it happens on theide exit. Anyway, now fixed. That said, everything valigrind says is worth investigation. I was able to fix many of these issues (in master now), unfortunately it looks like majority of remaining problems is cairo related (these happen when we are reading images from gtk). Any help identifying the cause(s) is welcome. Mirek Thanks a lot! TheIde doesn't crash anymore. I personally do not see any cairo problems. I guess Void Linux has a better valgrind exceptions list. The only problem I do see is an uninitialized value in Ide::Serialize. Interestingly, this problem depends on when you close TheIde. I guess this data is uninitialized during libclang indexing/reindexing. My command line: valgrind --track-origins=yes --log-file=memchk.txt ~/dvlp/cpp/code/upp/out/CLANG.Debug.Debug_Full.Gui.Shared/ide
creator: Novo
date: 2022-09-20T17:16:26-00:00

Re: theide with libclang
type: item
description: Quote: Thanks a lot! TheIde doesn't crash anymore. Really? I got the latest updates and it was completely broken for me. My assemblies/nests were all empty except some example code and then trying to open even one of those crashed the ide. I must be having a run of bad luck lately, I'll try to clone the whole thing again.
creator: jjacksonRIAB
date: 2022-09-20T17:21:00-00:00

Re: theide with libclang
type: item
description: Novo wrote on Tue, 20 September 2022 19:16 mirek wrote on Tue, 20 September 2022 07:09 Novo wrote on Sun, 18 September 2022 21:23 valgrind explains everything. While it is definitely a good idea, only one of these errors seems libclang related and fixable and it happens on theide exit. Anyway, now fixed. That said, everything valigrind says is worth investigation. I was able to fix many of these issues (in master now), unfortunately it looks like majority of remaining problems is cairo related (these happen when we are reading images from gtk). Any help identifying the cause(s) is welcome. Mirek Thanks a lot! TheIde doesn't crash anymore. I personally do not see any cairo problems. I guess Void Linux has a better valgrind exceptions list. The only problem I do see is an uninitialized value in Ide::Serialize. Interestingly, this problem depends on when you close TheIde. I guess this data is uninitialized during libclang indexing/reindexing. I am getting that one (in Ide::Serialize) too; my guts feeling is that it is not related to libclang, just some old serialisation bug like forgotten unitialised variable. Want to hunt that down, but it is time consuming.... What about those "_dl_catch_exception " things? Does not look like U++ fault, right?
creator: mirek
date: 2022-09-20T17:37:57-00:00

Re: theide with libclang
type: item
description: Quote: I am getting that one (in Ide::Serialize) too; my guts feeling is that it is not related to libclang Possible but I was getting crashes on close on the previous version and it was dumping a libclang exception to the terminal because it was indexing at the time, so it was reporting an indexing exception. I don't have the text for it anymore unfortunately.
creator: jjacksonRIAB
date: 2022-09-20T17:46:20-00:00

Re: theide with libclang
type: item
description: OK looks like I had a bad repo. I rebuilt and I'm back in. I'm still having excessive memory usage though and another curious thing I noticed is that it reindexes the file I have open in the editor regardless of whether it's changed. If I click away to another app and back to theide it will reindex. It never frees memory, even if I switch to another package entirely. I'm not getting the clang error in the console anymore though, so that appears to be fixed. I am, however, getting freezes if I cancel a build.
creator: jjacksonRIAB
date: 2022-09-20T18:13:12-00:00

Re: theide with libclang
type: item
description: OK so with address sanitizer and USEMALLOC enabled, I'm getting a bunch of reported heap overflows all on memcmp. lib/clang Signature.cpp line 171, for example: if(memcmp(s, " = {", 4) == 0) what happens if s is shorter than the sequence it's being compared against? Does it overrun?
creator: jjacksonRIAB
date: 2022-09-20T19:32:31-00:00

Re: theide with libclang
type: item
description: Once I replaced those memcmp instructions with strnlen it shut up address sanitizer. They're probably fine but at least I was able to get further and... there was nothing worthwhile except the usual font stuff. Found another errata to potentially test. If I go into any sufficiently complex package (like CtrlCore or CtrlLib) and hold down alt + down arrow to cycle through/open as many files as quickly as possible, libclang gets stuck in an infinite reparsing loop on one file until I click off to another file. I'm assuming this is because I'm leaving the last parsable .cpp/.h file and moving into other non-parseable file types rapidly. Edit also if you left ctrl click a symbol with indexing turned off it will still try to index but pin a core instead.
creator: jjacksonRIAB
date: 2022-09-20T22:33:41-00:00

Re: theide with libclang
type: item
description: mirek wrote on Tue, 20 September 2022 13:37 I am getting that one (in Ide::Serialize) too; my guts feeling is that it is not related to libclang, just some old serialisation bug like forgotten unitialised variable. Want to hunt that down, but it is time consuming.... One time I ran valgring I saw ZERO problems. I didn't believe my eyes. I reran it and got this Ide::Serialize problem. This is why I think it depends on something else. mirek wrote on Tue, 20 September 2022 13:37 What about those "_dl_catch_exception " things? Does not look like U++ fault, right? It is hard to tell. In my latest rerun I didn't get this "_dl_catch_exception " thing. I believe it happens when I close TheIde before it finishes loading of libclang data. I'm closing it via unorthodox window manager called herbstluft, which is sending a message to TheIde, I believe. So, something gets interrupted in the middle, I guess. Theoretically, a loader should automatically unload everything correctly ...
creator: Novo
date: 2022-09-20T23:33:00-00:00

Re: theide with libclang
type: item
description: I've got a different call stack with Ide::Serialize. Maybe it will help to locate a problem.
creator: Novo
date: 2022-09-20T23:56:10-00:00

Re: theide with libclang
type: item
description: Novo wrote on Wed, 21 September 2022 01:56 I've got a different call stack with Ide::Serialize. Maybe it will help to locate a problem. Thanks, fixed. (Unfortunately it was complete non-bug that could not have influenced anything real, just unitialized variable that is never used again).
creator: mirek
date: 2022-09-21T08:52:28-00:00

Re: theide with libclang
type: item
description: Thanks for doing this stuff Mirek. I was looking through the code trying to figure it out and learn about libclang ASTs but I simply don't know enough about TheIDE's internal operations (or libclang) to make heads or tails of it. I also swear I find you everywhere. I came across you on Stack Overflow asking about libclang, then a couple of weeks ago I was reading some posts from Andrei Alexandrescu and I saw you in there too. As for libclang one of the strange problems I had that shouldn't happen is that clang_parseTranslationUnit was returning a translation unit of 0 and as a consequence Dispose() was never getting called. I wasn't able to fix it because it mysteriously stopped happening and I don't know if it's anything I did. All I did was put in the hook for clang_parseTranslationUnit2 to see if it would give me a more meaningful error... but that turned out to be useless. It stopped happening even after I switched back. The only other change I made was I hooked clang_defaultReparseOptions and passed that to clang_reparseTranslationUnit. Supposedly it provides some default options that improve performance, don't really know if it did... so I flailed around uselessly for a few hours. The other behavior I had and am still having is that it's not simply parsing and then calling reparse after the initial parse, it's calling both every time. Once I got into CurrentFileThread it got too deep into TheIDE internals to understand what was going on.
creator: jjacksonRIAB
date: 2022-09-21T20:09:15-00:00

Re: theide with libclang
type: item
description: mirek wrote on Wed, 21 September 2022 04:52 Thanks, fixed. Thanks a lot! Could you also fix a problem below, please? ./umk reference SMTP CLANG -busr /home/buildbot/worker/l-upp/build/uppsrc/Core/SMTP/Smtp.cpp:457:2: error: expected expression DDUMP(GetMessage(true)); ^ /home/buildbot/worker/l-upp/build/uppsrc/Core/Diag.h:159:26: note: expanded from macro 'DDUMP' #define DDUMP(x) @ // To clean logs after debugging, this produces error in release mode ^ 1 error generated.
creator: Novo
date: 2022-09-23T03:41:41-00:00

Re: theide with libclang
type: item
description: mirek wrote on Wed, 21 September 2022 04:52 Novo wrote on Wed, 21 September 2022 01:56 I've got a different call stack with Ide::Serialize. Maybe it will help to locate a problem. Thanks, fixed. (Unfortunately it was complete non-bug that could not have influenced anything real, just unitialized variable that is never used again). I do see two problems with valgrind messages: 1) Tons of "uninitialized value" messages hide real problems. 2) Nobody will use Upp with commercial apps for a very simple reason: who knows what kind of problems can be caused by these "uninitialized values". IMHO, Upp should be checked with all available valgrind tools and all available sanitizers. At least before release.
creator: Novo
date: 2022-09-23T03:52:28-00:00

Re: theide with libclang
type: item
description: Mirek, Runaway memory allocations seem not to be an issue using it all day. The one thing I can report today is libclang getting stuck in a reparse loop on app exit and refusing to let the ide close. It happens rarely but I can reliably force it to happen by attempting to exit the ide during a build.
creator: jjacksonRIAB
date: 2022-09-23T05:09:47-00:00

Re: theide with libclang
type: item
description: Novo wrote on Fri, 23 September 2022 05:52 mirek wrote on Wed, 21 September 2022 04:52 Novo wrote on Wed, 21 September 2022 01:56 I've got a different call stack with Ide::Serialize. Maybe it will help to locate a problem. Thanks, fixed. (Unfortunately it was complete non-bug that could not have influenced anything real, just unitialized variable that is never used again). I do see two problems with valgrind messages: 1) Tons of "uninitialized value" messages hide real problems. 2) Nobody will use Upp with commercial apps for a very simple reason: who knows what kind of problems can be caused by these "uninitialized values". IMHO, Upp should be checked with all available valgrind tools and all available sanitizers. At least before release. I do not argue with that (as you have seen, I have fixed what I could). The comment was not meant as an apology for the bug, just a note that it cannot explain any weird behaviour with libclang.
creator: mirek
date: 2022-09-23T08:48:15-00:00

Re: theide with libclang
type: item
description: I don't know how much improvement we can expect in, say half an year's time. At the moment, I Ctrl+Left click on a member function in a moderately complex project, the parsing starts and goes on for ever. At least 30 minutes past and no idea how much longer it's going to take. In contrast, it used to work so smoothly. Theoretically moving to libclang is the right thing to do, but there is a huge chance it may render theide completely useless. Some system info Quote: Upp version: 16443 OS: Ubuntu 22.04.1 LTS x86_64 CPU: Intel i7-8650U (8) @ 4.200GHz GPU: Intel UHD Graphics 620 Memory: 3757MiB / 15875MiB PS: I tested with Examples/HomeBudget. In HomeBudget.cpp, Ctrl+left click some member function, eg LoadCategories(...) to start parsing. Now roughly 10 minutes past, still parsing. Something might not be right.
creator: Lance
date: 2022-09-24T02:41:10-00:00

Re: theide with libclang
type: item
description: Fortunately I can say, on my computer it works almost immediately. I am on Windows 10 64 bit. The computer is a Ryzen 7, 32 GByte Memory. I had a look into taskmanager. Total memory used was 12 GBytes. I had some other programs open, Browser, email, Fotoviewer Faststone, Fotosoftware. TheIde used 2 GBytes and had the biggest memory chunk in use. I can imagine, if the computer had less memory, if the drive is a harddrive and not a SSD, and if TheIDE temporarily wants more memory, then the computer would start swapping and it could terribly slow down. (If it had an SSD it could be faster, but would lead to unneccessary SSD wear) (I have a SSD but I hope it will not be used for swapping) Fortunately swapping does not happen here. It might be useful to monitor and report this machine data and properties. Possibly reboot and do not start other programs than TheIDE and compare the behavior and memory consumption. So far I know, libclangd can be disabled in the setup. This could be a solution as a last ressort if memory is low.
creator: peterh
date: 2022-09-24T05:26:23-00:00

Re: theide with libclang
type: item
description: Lance wrote on Sat, 24 September 2022 04:41 I don't know how much improvement we can expect in, say half an year's time. At the moment, I Ctrl+Left click on a member function in a moderately complex project, the parsing starts and goes on for ever. At least 30 minutes past and no idea how much longer it's going to take. In contrast, it used to work so smoothly. Theoretically moving to libclang is the right thing to do, but there is a huge chance it may render theide completely useless. It is just first week after the merge... Quote: Something might not be right. The is the most likely explanation. There is no reason for it to take so long, it is probably stuck somewhere. Is autocomplete working? Mirek
creator: mirek
date: 2022-09-24T06:08:35-00:00