|
|
Home » U++ Library support » U++ Core » Value question (memory consumption)
Value question (memory consumption) [message #29906] |
Mon, 29 November 2010 11:31  |
Factor
Messages: 5 Registered: June 2007 Location: Hungary
|
Promising Member |
|
|
I've made a simple test with Vector<Value>:
#include <Core/Core.h>
using namespace UPP;
#define ITEM_COUNT 1000000
CONSOLE_APP_MAIN {
Vector<Value> v;
getchar();
for(int i=0;i<ITEM_COUNT;i++) v.Add((int)i);
getchar();
v.Clear(); v.Shrink();
getchar();
}
1. Adding ITEM_COUNT items (ints for simplicity) to Vector v raises the memory consumption of the process to ~20MB (Win XP - Task manager).
2. After clearing and shrinking Vector v the memory usage drops to 17MB.
As I saw the reference counting works as it should and the "delete *this;" (Void::Release) part of the inner Void class is called, still the memory usage of the process remains high.
I've tried to directly call the ~Value() destructor for all the Vector items before clear, but the result was the same.
Is this the normal behavior?
|
|
|
|
Re: Value question (memory consumption) [message #29914 is a reply to message #29906] |
Mon, 29 November 2010 21:23   |
|
Hi Factor,
This is actually not related to Value at all. If you try the same with Vector<int>, you will notice the same behavior.
I think it is caused by the way Vector allocates memory when it grows. If you do v.SetCount(ITEM_COUNT,0), instead of series of Add() calls and then call Clear() and Shrink, you will get back at the memory usage you started with.
I am not sure if this is a bug. I have a strange feeling that this might be actually result of how OS manages the memory. If you ask it for a big chunk at once and then return it later, it might work different than when you are asking for chunks of size increasing as powers of two (which is what vector does).
Maybe someone knowing the memory handling better than me might shed some more light into this. But from what I've seen, I would recommend to use SetCount() every time you know how many items will be necessary.
Best regards,
Honza
|
|
|
Re: Value question (memory consumption) [message #29916 is a reply to message #29914] |
Mon, 29 November 2010 22:57   |
|
Memory consumption is higher with U++ allocator. I've tested with USEMALLOC flag and the memory consumption was half, but has same behavior, only part of the memory is released.
I've modified example to do new allocations and seems that memory get back to OS, at least here on Linux, if i don't use U++ allocator.
#include <Core/Core.h>
using namespace UPP;
#define ITEM_COUNT 1000000
CONSOLE_APP_MAIN {
Vector<Value> v;
getchar();
for(int i=0;i<ITEM_COUNT;i++) v.Add((int)i);
getchar();
v.Clear(); v.Shrink();
getchar();
for(int i=0;i<ITEM_COUNT;i++) v.Add((int)i);
getchar();
v.Clear(); v.Shrink();
getchar();
}
without U++ allocator:
292 KiB
38.4 MiB
30.8 MiB
38.8 MiB
420 KiB
with U++ allocator:
412 KiB
34.5 MiB (for a sec.) 70.2 MiB maybe Vector double amount of memory it need
62.5 MiB
70.2 MiB
62.5 MiB
Linux 2.6.35-23-generic #40-Ubuntu SMP Wed Nov 17 22:14:33 UTC 2010 x86_64 GNU/Linux
Andrei
[Updated on: Mon, 29 November 2010 22:58] by Moderator Report message to a moderator
|
|
|
|
|
|
Re: Value question (memory consumption) [message #29927 is a reply to message #29916] |
Tue, 30 November 2010 13:36   |
 |
mirek
Messages: 14255 Registered: November 2005
|
Ultimate Member |
|
|
andreincx wrote on Mon, 29 November 2010 16:57 | Memory consumption is higher with U++ allocator. I've tested with USEMALLOC flag and the memory consumption was half, but has same behavior, only part of the memory is released.
I've modified example to do new allocations and seems that memory get back to OS, at least here on Linux, if i don't use U++ allocator.
#include <Core/Core.h>
using namespace UPP;
#define ITEM_COUNT 1000000
CONSOLE_APP_MAIN {
Vector<Value> v;
getchar();
for(int i=0;i<ITEM_COUNT;i++) v.Add((int)i);
getchar();
v.Clear(); v.Shrink();
getchar();
for(int i=0;i<ITEM_COUNT;i++) v.Add((int)i);
getchar();
v.Clear(); v.Shrink();
getchar();
}
without U++ allocator:
292 KiB
38.4 MiB
30.8 MiB
38.8 MiB
420 KiB
with U++ allocator:
412 KiB
34.5 MiB (for a sec.) 70.2 MiB maybe Vector double amount of memory it need
62.5 MiB
70.2 MiB
62.5 MiB
Linux 2.6.35-23-generic #40-Ubuntu SMP Wed Nov 17 22:14:33 UTC 2010 x86_64 GNU/Linux
Andrei
|
In Win32, memory consumption estimated using task manager, it goes like:
USEMALLOC:
724KB
28.2MB
1.1MB
28.2MB
1.2MB
With U++ allocator:
624KB
20.4MB
16.5MB
20.4MB
16.5MB
So the maximum usage is significantly less than for M$ allocator..
I would like to know why Linux is so much higher.
If it is not because of Debug release, we should check the correctnes of core allocation routines in POSIX:
void *SysAllocRaw(size_t size)
{
sKB += int(((size + 4095) & ~4095) >> 10);
#ifdef PLATFORM_WIN32
void *ptr = VirtualAlloc(NULL, size, MEM_RESERVE|MEM_COMMIT, PAGE_READWRITE);
#else
#ifdef PLATFORM_LINUX
void *ptr = mmap(0, size, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
#else
void *ptr = mmap(0, size, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0);
#endif
if(ptr == MAP_FAILED)
ptr = NULL;
#endif
if(!ptr)
Panic("Out of memory!");
DoPeakProfile();
return ptr;
}
void SysFreeRaw(void *ptr, size_t size)
{
sKB -= int(((size + 4095) & ~4095) >> 10);
#ifdef PLATFORM_WIN32
VirtualFree(ptr, 0, MEM_RELEASE);
#else
munmap(ptr, size);
#endif
}
int s4kb__;
int s64kb__;
void *AllocRaw4KB()
{
static int left;
static byte *ptr;
static int n = 32;
if(left == 0) {
left = n >> 5;
ptr = (byte *)SysAllocRaw(left * 4096);
}
n = n + 1;
if(n > 4096) n = 4096;
void *p = ptr;
ptr += 4096;
left--;
s4kb__++;
DoPeakProfile();
return p;
}
void *AllocRaw64KB()
{
static int left;
static byte *ptr;
static int n = 32;
if(left == 0) {
left = n >> 5;
ptr = (byte *)SysAllocRaw(left * 65536);
}
n = n + 1;
if(n > 256) n = 256;
void *p = ptr;
ptr += 65536;
left--;
s64kb__++;
DoPeakProfile();
return p;
}
(the rest is the same for Win32, POSIX):
Mirek
|
|
|
Re: Value question (memory consumption) [message #29928 is a reply to message #29925] |
Tue, 30 November 2010 13:51   |
|
luzr wrote on Tue, 30 November 2010 13:21 |
andreincx wrote on Mon, 29 November 2010 16:57 | Memory consumption is higher with U++ allocator. I've tested with USEMALLOC flag and the memory consumption was half, but has same behavior, only part of the memory is released.
|
Have you tested in debug or release (Optimal)? Those numbers look quite high to me (for U++ allocator). In debug mode, U++ inserts a LOT of debugging info into the heap, so the consumption is higher.
(Will have to test myself... 
|
I tested with optimal and U++ allocator.
luzr wrote on Tue, 30 November 2010 13:25 | Yes, only "very large" (> app. 64KB) memory blocks are returned to OS. Rest is kept for later reuse. Anyway, this behaviour is pretty typical for most allocators, only maybe some of them have different threshold.
| This explains it. SetCount() allocates all in one big block so it is returned, but Add() causes allocations of 1+2+4+8+...2^16B (that is up to 64KB) which are not returned (any further are bigger than 64KB and therefore returned). That should make ~128KB of non-returned blocks and that agrees exactly with what I measured for Vector<int>. For Vector<Value> it leaves more non-returned blocks probably because there are some additional allocations when constructing the Value...
Thanks for explanation,
Honza
|
|
|
|
|
Re: Value question (memory consumption) [message #29931 is a reply to message #29930] |
Tue, 30 November 2010 16:48   |
|
Hello,
Sorry, I haven't had time to respond until now. Yes, i've tested in DEBUG mode only.
On release i've got same results as Mirek with U++ allocator, except for first allocation(364 KiB). Without U++ allocator, again same results, except first and last allocation (268 KiB, 396 KiB).
|
|
|
|
|
Goto Forum:
Current Time: Sat Apr 26 21:55:03 CEST 2025
Total time taken to generate the page: 0.00681 seconds
|
|
|