A hot air balloon just caught fire in East Texas last week at the Great Texas Balloon Race. Luckily they came down and no one was hurt, this could have turned ugly quickly (note, this is not the one in the story, just an illustration how easily they catch fire).
Even though it does not require any sort of licensing like a pilots license, the expense of it still has kept it to some very serious pilots only, some dumb kid can't go pick one up and fly out of his back yard.
I personally love it. It is huge where I live, but I know when I go for a ride, even with an experienced pilot, there are a lot of factors that make it dangerous. (still probably safer than driving across town though).
