On The Interplay Of 2D And 3D Representations For Robotic Perception