A DisplayPort KVM Switch can be handy in your situation. A 4-port one off ebay can cost ~$100.
I used KVM Switches in the past before VMs got popular.
Yeah, well, got 4-PC / 3-USB switch.. Works great with normal PCs (Windows/Linux). Then got M2 mac mini. Of course the switch does not work with Mac... (But what did I expect, my apple experience was always crappy...)