I think the Supreme Court, as well as James Madison, defined the way of leaving very clearly. A valid secession can only be done with the consent of a majority of the states.
It's an interesting issue. Let's play hypotheticals here. If were were to let everyone secede who wanted to, the US would probably end up a collection of city-states or fiefdoms. Almost literally anarchy.
And yet, if the UN were to annex the US, would we want to have the UN's permission to secede from the UN?
Somewhere in between must be the answer. An interesting question to ponder.