The 2012 race was tight. There was doubt about its outcome even on election night, as the votes were being counted. After all, Obama had had to deal with a very bad economic situation for his first four years. And he was also inexperienced in national politics.
So if he hadn't won a second term, I don't think it would have been all that surprising. And Romney is no Trump.