Quantization of random orthonormal matrices with application to adaptive transform coding