The scary part is having no way to guarantee that. Any civilization capable of traversing space will be so much more technologically advanced that we would just have to hope they wouldn't destroy/enslave humanity. Humans regularly destroy animal habitats because it helps our own survival by giving us space and resources. An alien race might see Earth as a great resource and our extinction as acceptable collateral damage.
I'm not sure the Earth would be seen as particularly resource-rich from the perspective of a species that can transport hundreds or even thousands of individuals within a time-frame that is no more than a small percentage of each individual's life span.
The comets, Neptune, and Uranus all contain substantially more water than does the Earth. The asteroid belt (and, most likely, the "junk" left over in the outer clouds orbiting the Solar System) contain more metals than does the Earth.
And those resources are all farther away from the Sun, and therefore not so deep into the Sun's gravity well. Going after those resources before you venture down the gravity well to Earth is much more likely.
