No, but we can observe the movement of galaxy clusters and the microgave background radiation and then draw conclusions based on the available evidence. I find your position that if we don't observe an event happening in "real" time that you can't infer that it happened to be rather odd.
Maybe with the following analogy I can clarify why it is odd to me:
Imagine that you are driving down the road and you see two cars that appear to have crashed. There is an ambulance taking care of the wounded. You did not see the crash yourself but based on the available evidence you can draw the conclusion that both vehicules ran into each other.
The movement of galaxies and the microwave background radiation are the crashed cars and the ambulance in this analogy. The Big Bang is the event itself (the crash).
Currently there are alternative explanations to the Big Bang theory but they are not testable and we can't take them seriously...yet.
RE: The End Of Reason: A Glimpse Behind The Curve