As my local Congresscritter said, the simplest explanation is that Cantor lost the trust of the people in his district, for many reasons.
The main one though, they saw him as representing Washington, not them. When the people of your district think that, you're toast.
That's how we got rid of Chet Edwards who no one thought could ever be defeated. He didn't just lose, he got blown right out of the water.