Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » U++ Library support » U++ MT-multithreading and servers » httpRequests in secondary non-gui-main Thread (if memory leaks come from http.buffer?)
httpRequests in secondary non-gui-main Thread [message #55327] Fri, 30 October 2020 18:20 Go to next message
JeyCi is currently offline  JeyCi
Messages: 24
Registered: July 2020
Promising Member
if such http.buffer in debug (with multiple FreeFree_etc) means memory leaks?.. what does it mean?
http://savephoto.ru/Files/10ovf10znabl530_t.jpg
I'm using mingw32 compiler v.9.3 & getting memory leaks in the situation described in the topic name, but when using mingw_ delivered as a part of U++ v.13664 -- memory not leaking (though another my stuff gives some mistakes)... what explanations can be (for mingw 9.3)?.. and what to do?.. or should I do everything in main thread - concerning loading several urls in asynchronous mode - is it the way to avoid memory leaks?... ( Confused just don't want to refactor for single-thread if the problem can really be in something else)


Best regards.

[Updated on: Fri, 30 October 2020 18:25]

Report message to a moderator

Re: httpRequests in secondary non-gui-main Thread [message #55329 is a reply to message #55327] Sat, 31 October 2020 11:18 Go to previous messageGo to next message
Klugier is currently offline  Klugier
Messages: 795
Registered: September 2012
Location: Poland, Kraków
Contributor
Hello,

Could you post test case for this (optimally simplify version)? Without the code it will be difficult to diagnose what is causing the problem.

Klugier


Ultimate++ - one framework to rule them all.
Re: httpRequests in secondary non-gui-main Thread [message #55330 is a reply to message #55329] Sat, 31 October 2020 11:38 Go to previous messageGo to next message
JeyCi is currently offline  JeyCi
Messages: 24
Registered: July 2020
Promising Member
Klugier wrote on Sat, 31 October 2020 11:18
Without the code it will be difficult to diagnose what is causing the problem.

well... it can take time... I just saw such variable in debugger (with multiple FreeFreeFree...) when I had a memory leak... the first question is just about - does it really mean that the memory is being allocated, but not freed?..
p.s.
to extract a piece of code will be really not so easy - will took time... I will see...
just I saw some preventions in forum, that openSSL sometimes gives memory leaks when working in MT-environment (in my case in secondary thread, though when I do loop in main-gui-thread - no problems - already was tested)...
- if this problem can occur in v.13664-32x-windows or it depends on compiler (mingw or MSVC)?.. - I'm not sure, that the whole code is needed - especially if I'm restricted in using just last 32x-version of U++ in windows with mingw-compiler (optimized to 9.3 version in Build_Method_Settings & I've got away everything else that was there - perhaps I've lost something important?)


Best regards.

[Updated on: Sat, 31 October 2020 11:41]

Report message to a moderator

Re: httpRequests in secondary non-gui-main Thread [message #55333 is a reply to message #55330] Sat, 31 October 2020 15:02 Go to previous messageGo to next message
Klugier is currently offline  Klugier
Messages: 795
Registered: September 2012
Location: Poland, Kraków
Contributor
Hello,

Did you try to debug memory leaks? It will show you exactly in which place the leak is. The all information about detecting memory leaks and marking false positives you can find here.

Please noticed that today I updated this article significantly, so not all information will be there. Visit this site tomorrow and all new information should be there. Alternatively you could update to the latest SVN version and read new documentation page inside TheIDE or just read the raw data in commit content available on GitHub (Click on Load Diff for obtaining content).

Klugier


Ultimate++ - one framework to rule them all.

[Updated on: Sat, 31 October 2020 15:03]

Report message to a moderator

Re: httpRequests in secondary non-gui-main Thread [message #55334 is a reply to message #55333] Sat, 31 October 2020 16:01 Go to previous messageGo to next message
JeyCi is currently offline  JeyCi
Messages: 24
Registered: July 2020
Promising Member
Klugier wrote on Sat, 31 October 2020 15:02

Did you try to debug memory leaks?

I do have such main
GUI_APP_MAIN
{
#ifdef _DEBUG
	StdLogSetup(LOG_COUT|LOG_FILE);
	HttpRequest::Trace();
#endif

    MyAppWindow().Run();
}

and getting such LOG (under spoiler)
Toggle Spoiler

but didn't know what to do with it Embarassed - taking into consideration this part:
Quote:
Of course, the only problem with this is that the order of allocations must be exactly the same in the "test run". Which is often not, but usually it is possible to arrange it so after spending a couple of nights desperately looking for the source of leak in the code using other methods...

... while using ONLY U++ & neither VisualStudio tools nor Deleaker for Qt - it was problematic for me to deal with this LOG's info... therefore I have only debugger in my disposal & its FreeFreeFree_etc in one variable & my supposition that it concerns the leak?.. by the way original code, though not leaking is in examples in u++ (as in my screen was) & when using it not in main, but in secondary thread - I'm getting memory leaks (as I already mentioned)... therefore I supposed that httpRequest can give memory leak, but in ST-app these leaks are being hidden somehow, though in MT-app are being revealed by assertion... IMHO... if async loading takes place in secondary thread... BTW it's not a Server, just some requests to some html_pages (multiple)...
Klugier wrote on Sat, 31 October 2020 15:02

visit this site tomorrow and all new information should be there.

Smile ok - will wait till tomorrow - thank you!
Klugier wrote on Sat, 31 October 2020 15:02

Alternatively you could update to the latest SVN version and read new documentation page inside TheIDE

I do not see any updater in the ide itself... is it possible only from the git? - I am not connected to it anyway...
Klugier wrote on Sat, 31 October 2020 15:02

or just read the raw data in commit content available on GitHub (Click on Load Diff for obtaining content).

have got... thank you!
but my results you can see under spoiler in this message above here - until I'll find what to do with it from your paper... Rolling Eyes because I don't know yet... and you are not telling about openSSL behavior in my situation described


Best regards.

[Updated on: Sat, 31 October 2020 17:19]

Report message to a moderator

Re: httpRequests in secondary non-gui-main Thread [message #55336 is a reply to message #55334] Sat, 31 October 2020 17:34 Go to previous messageGo to next message
JeyCi is currently offline  JeyCi
Messages: 24
Registered: July 2020
Promising Member
Oh, I really have already got it into my U++ - thanks a lot! - for detailed description, code examples & even pictures == that is really very informative! - will learn... thank you for spending time to do such update into Help section
p.s.
now I even see that updating to the latest SVN is very simple Smile (just copy-paste needed files)


Best regards.

[Updated on: Sat, 31 October 2020 17:58]

Report message to a moderator

Re: httpRequests in secondary non-gui-main Thread [message #55359 is a reply to message #55330] Mon, 02 November 2020 11:57 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 12954
Registered: November 2005
Ultimate Member
JeyCi wrote on Sat, 31 October 2020 11:38
Klugier wrote on Sat, 31 October 2020 11:18
Without the code it will be difficult to diagnose what is causing the problem.

well... it can take time... I just saw such variable in debugger (with multiple FreeFreeFree...) when I had a memory leak... the first question is just about - does it really mean that the memory is being allocated, but not freed?..


If you see pointer pointing to "FreeFreeFree....", it most likely means that it points to the memory that was already deallocated.

In debug mode, U++ allocator fills the memory with this pattern on deallocation. Then when the same block is allocated again, it checks that the pattern is unchanged. If it is not, it would means something has written to that memory after it was deallocated, which is a sign of dangling pointer problem. In that case the app is aborted (in debug).

Mirek
Re: httpRequests in secondary non-gui-main Thread [message #55366 is a reply to message #55333] Tue, 03 November 2020 08:07 Go to previous messageGo to next message
JeyCi is currently offline  JeyCi
Messages: 24
Registered: July 2020
Promising Member
Klugier wrote on Sat, 31 October 2020 15:02
It will show you exactly in which place the leak is.

thank you all for your replies! - I've got the place where it leaks:
it leaks here when httpRequests are being done in secondary thread (not gui-thread) - when using --memory-breakpoint__ [num detected before] - myApp stops here in debug
index.php?t=getfile&id=6272&private=0
? can it be connected with OpenSSL, perhaps, as external library is considered to be?
or just the problem of my app_design?..
perhaps, declaring Vector <urls> as thread-local prior to start of thread can help?..
but frankly speaking I intend to consider the error arising just because of switching-context that takes place in MT-environment & not freeing http.buffer... though I added w.http.ClearContent(); in the code...
or perhaps really http.buffer for each request should have thread-local storage-duration?...
p.s.
(btw I'm gathering the Vector of links in the same secondary thread - not transferring it from main thread)
Anyway, thank you, - now I see exact place where it happens
  • Attachment: 03.11.jpg
    (Size: 61.89KB, Downloaded 166 times)


Best regards.

[Updated on: Tue, 03 November 2020 08:20]

Report message to a moderator

Re: httpRequests in secondary non-gui-main Thread [message #55367 is a reply to message #55366] Tue, 03 November 2020 09:08 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 12954
Registered: November 2005
Ultimate Member
JeyCi wrote on Tue, 03 November 2020 08:07
Klugier wrote on Sat, 31 October 2020 15:02
It will show you exactly in which place the leak is.

thank you all for your replies! - I've got the place where it leaks:
it leaks here when httpRequests are being done in secondary thread (not gui-thread) - when using --memory-breakpoint__ [num detected before] - myApp stops here in debug
index.php?t=getfile&id=6272&private=0
? can it be connected with OpenSSL, perhaps, as external library is considered to be?
or just the problem of my app_design?..
perhaps, declaring Vector <urls> as thread-local prior to start of thread can help?..
but frankly speaking I intend to consider the error arising just because of switching-context that takes place in MT-environment & not freeing http.buffer... though I added w.http.ClearContent(); in the code...
or perhaps really http.buffer for each request should have thread-local storage-duration?...
p.s.
(btw I'm gathering the Vector of links in the same secondary thread - not transferring it from main thread)
Anyway, thank you, - now I see exact place where it happens


The interesting info you should have sent is the backtrace. The best way is (after it stops on memory breakpoint) Debug / Copy backtrace of all threads.

HttpRequest should not leak (of course, it is always possible there is a bug). Any chance you are doing something like early exit from the thread?

Ah, and maybe the most important: Are you using Upp::Thread? Or are how do you start the thread?

Mirek

[Updated on: Tue, 03 November 2020 09:09]

Report message to a moderator

Re: httpRequests in secondary non-gui-main Thread [message #55374 is a reply to message #55367] Tue, 03 November 2020 13:40 Go to previous messageGo to next message
JeyCi is currently offline  JeyCi
Messages: 24
Registered: July 2020
Promising Member
mirek wrote on Tue, 03 November 2020 09:08
The best way is (after it stops on memory breakpoint) Debug / Copy backtrace of all threads.

? what do you mean "backtrace" - is it this dropdown
index.php?t=getfile&id=6273&private=0[
mirek wrote on Tue, 03 November 2020 09:08
Any chance you are doing something like early exit from the thread?

seems not - I'm simplifying testing case Smile
mirek wrote on Tue, 03 November 2020 09:08

Ah, and maybe the most important: Are you using Upp::Thread? Or are how do you start the thread?

member Thread work; - in class MyAppWindow : public WithMyAppWindowLayout<TopWindow> starting work from here:
void MyAppWindow:: btnStart_Click()
{
    work.Run ( [=]{ WorkerThread(); } );
}

I thought that native for U++ Thread is OK & I do not need creating my own wrapper_class for executing my thread - I like opportunities U++ gives with its Thread class... or do you think I'd better create my separate wrapper_class to work with Thread?.. something like RAII-style... though I don't need any special ways of synchronization - am just using THISFN for interaction with Gui, using GuiLock/Call in these functions...
===
So, here:
		    MemoryIgnoreLeaksBegin() ;
		    w.http.Do(); 
		    MemoryIgnoreLeaksEnd();

- with thus it is OK when being executed Workerthread() function...
or do you think it is better to create a separate class to wrap the Thread?..
===
BTW the situation still differs depending on chosen compiler - as I have written - with MINGW_built_in_U++ - OK working... but with my newer mingw_9_3 in Build Method I do have leak in the place I've shown in the code (switching-on MemoryIgnore in that line - function works good)...
because I deleted everything in BuildMethod from U++ but made settings (path, include, lib) only for foreign mingw (loaded with msys2) - perhaps I have lost something important in my settings (when removed native stuff)?
though in msys2 I have even done separate installation additionally to mingw itself:
Installation:pacman -S openssl

===
oh, mirek - I think I can suppose the reason - msys2 is also no more supporting x32 from May 2020... you was right - most modern CPUs support the possibility to increase RAM-size... really...
Anyway it seems I now do know how to find memory_leak in U++ - no other foreign tools needed -- Thank you...
and the testing code still works without memory-leaks in U++-v.13664 - the version in which mingw already build_in
  • Attachment: 03.11_2.jpg
    (Size: 181.64KB, Downloaded 156 times)


Best regards.

[Updated on: Tue, 03 November 2020 13:51]

Report message to a moderator

Re: httpRequests in secondary non-gui-main Thread [message #55375 is a reply to message #55374] Tue, 03 November 2020 13:57 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 12954
Registered: November 2005
Ultimate Member
You should have invoked that menu command I have suggested, that would put the whole backtrace snapshot on the clipboard that you could post here (e.g. as attachement). Much better than screenshots... Smile

Using Thread is fine! I was actually afraid that you are not using it, that would lead to leaks. You need to use Upp::Thread as it contains some allocator related stuff to release some allocation caches...

Anyway, from the very sparse info I am getting, I am starting to think that you migh not be locking things properly when communicating between threads. I would guess that 'showDone' is something that you call from the thread to show progress in GUI. Do you lock things propertly? E.g. with GuiLock __?

Mirek
Re: httpRequests in secondary non-gui-main Thread [message #55376 is a reply to message #55374] Tue, 03 November 2020 13:59 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 12954
Registered: November 2005
Ultimate Member
JeyCi wrote on Tue, 03 November 2020 13:40
mirek wrote on Tue, 03 November 2020 09:08
The best way is (after it stops on memory breakpoint) Debug / Copy backtrace of all threads.

? what do you mean "backtrace" - is it this dropdown


Once it stops, click "Debug" menu and choose "Copy backtrace of all threads". It will put the backtrace onto clipboard which you can then paste here.
Re: httpRequests in secondary non-gui-main Thread [message #55377 is a reply to message #55375] Tue, 03 November 2020 15:07 Go to previous messageGo to next message
JeyCi is currently offline  JeyCi
Messages: 24
Registered: July 2020
Promising Member
mirek wrote on Tue, 03 November 2020 13:57
I would guess that 'showDone' is something that you call from the thread to show progress in GUI. Do you lock things propertly? E.g. with GuiLock __?

no it isn't... I just didn't care about the name of the method... but as you can notice this method is being got from another class (asyncHttp)... so I'm starting the WorkerThread as I have shown above... & I separated the loading itself to the separate class that takes vector done in WorkerThread() as Constructor's parameter & several THISFNs as parameters of the method showDone (but its real purpose is exactly loading stuff & returning results to functions of gui_class, in which I of course use GuiLock & Call([=]{this->finished.SetArray(i, va)})...
well but it was prior design, then I of course refactored for not returning results to gui... just from the beginning I wanted to see what is returning - but in the refactored code everything goes to SaveFile(n,res)... I had finally decreased the quantity of synchronization points to zero now... just have now taken for the test the shorter code - to reduce it even more for testing purposes (but it just appered to be the code still with synchronization)...
but the problem in all my step_by_step dev.process is the same - the line I've taken in MemoryIgnore... perhaps you're right - perhaps I shouldn't do loading stuff in the separate class?.. not knowing how to make this class thread-safe?.. perhaps it is not thread-save in my situation described?
mirek wrote on Tue, 03 November 2020 13:57
You should have invoked that menu command I have suggested, that would put the whole backtrace snapshot on the clipboard that you could post here (e.g. as attachement).

Surprised ah, I see now... ok when I will shorten the whole code even more - perhaps... in order not to provide unneeded info... (if given description doesn't clarify the surrounding of the problem)


Best regards.

[Updated on: Tue, 03 November 2020 15:16]

Report message to a moderator

Re: httpRequests in secondary non-gui-main Thread [message #55379 is a reply to message #55377] Tue, 03 November 2020 15:46 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 12954
Registered: November 2005
Ultimate Member
Well, really, I would like to help, but I am completely in the dard about the code... I will stop guessing now and wait for simplified testcase.
Re: httpRequests in secondary non-gui-main Thread [message #55380 is a reply to message #55379] Tue, 03 November 2020 18:46 Go to previous messageGo to next message
JeyCi is currently offline  JeyCi
Messages: 24
Registered: July 2020
Promising Member
You already gave me all key points to check in my code - thank you - if it is working with mingw_that_is build in U++ - then will be ok anyway... just it started to work when I started to shorten it... I will check everything again taking into consideration all your advices - Smile it can appear to be quicker then to shorten the code & leaving the error...
p.s.
for any case - I've read this topic - Win32 TLS & this - release 2019.2 (Aug) - workaround for Mingw TLS performance issue - to remember whether I have this in v.13664 - if I will need it... or to change my design... just for info...
but I will check thoroughly how do I pass parameters to the Class doing loading stuff & how do I make destruction of its objects in order then to exit thread-function correctly


Best regards.

[Updated on: Tue, 03 November 2020 18:48]

Report message to a moderator

Re: httpRequests in secondary non-gui-main Thread [message #55392 is a reply to message #55380] Fri, 06 November 2020 03:57 Go to previous messageGo to next message
JeyCi is currently offline  JeyCi
Messages: 24
Registered: July 2020
Promising Member
JeyCi wrote on Tue, 03 November 2020 18:46
release 2019.2[/url] (Aug) - workaround for Mingw TLS performance issue - to remember whether I have this in v.13664

exactly the version I use - 13664


Best regards.
Re: httpRequests in secondary non-gui-main Thread [message #55395 is a reply to message #55375] Fri, 06 November 2020 13:03 Go to previous message
JeyCi is currently offline  JeyCi
Messages: 24
Registered: July 2020
Promising Member
mirek wrote on Tue, 03 November 2020 13:57
You put the whole backtrace snapshot on the clipboard that you could post here (e.g. as attachement). Much better than screenshots...

seeing under spoiler
Toggle Spoiler

index.php?t=getfile&id=6274&private=0
attempt to compile under mingw_9_3 (from msys2) seems to leak somewhere in System32??.. perhaps really the problem is in external lib... if I'm seeing correctly
p.s.
though it is even strange to see mingw-w64... as for my win32-system I probably need i686-w64-mingw32 from msys2 folder...
p.p.s
Smile though a little bit having refactored the whole code (not shortening) with new knowledge from this topic - I do not have leak with mingw-build-in-U++... seems OK
  • Attachment: 06.11.jpg
    (Size: 18.01KB, Downloaded 73 times)


Best regards.

[Updated on: Fri, 06 November 2020 13:18]

Report message to a moderator

Previous Topic: Encoding URL for HttpRequest
Goto Forum:
  


Current Time: Sat Nov 28 23:01:45 CET 2020

Total time taken to generate the page: 0.01381 seconds