Growth Diagrams from Polygons in the Affine Grassmannian