How Dark Is Dark Matter?: Robust Limits On Dark Matter-Radiation Interactions From Cosmological Observations