Only pointer is stored. Memory usage could be very slightly bigger (because the whole array is not used if there are no displays set), but 10-30 bytes/cell max.
I would say problem is somewhere else or more complicated. If you could post a testcase eating a lot of memory, I could investigate...