Well, I think that VectorMap is not that bad in this situation. Also, quite often it is possible to reorganize the code and remove 'id' from T.
As an option, you can consider using HashBase. It is not really documented (?yet), but it implements just hashing without values.
Also worth noting is that hashing itself costs about 20 bytes per element. In this light, wasting 16 bytes to store String key separately is not really critical.