Structured Sequence Modeling with Graphs, Flows, and Real-world Knowledge