Also the change in BufferPainter::CharacterOp() seems to fix the issue. Similarly, the rendering speed drops.
The special thing about these maps I'm rendering is that they have an extremely small change in the text rotation thorough out the map page as they have been transformed from one coordinate projection to another (Transverse Mercator to Mercator in this case) and this involves a change in text rotation by the amount of meridian convergence value. This boils down to just parts of a degree through out the entire map width. I do not know if the caching gets affected by this or not.
I also generated a serialized painting, but it did not exhibit the issue. I think this is because the history for the cache is different and therefore the problem does not show up.