The big bugaboo in all of this is time itself. If the fact that it takes 100,000 years for light to travel across the galaxy or a million years to physically get there became irrelevant, then there is no problem.
And we are roughly 20,000 light-years from the edge of the galaxy. That means that a light ray from a distant object beyond our galaxy had already entered and had made 90% of the trip through our galaxy towards earth when Jesus and Julius Caesar were alive.
