Ultrafast manipulation of single photons using dispersion and sum-frequency generation