You could certainly draw a "bigger picture" conclusion based on states rights, which likely was the primary motivation for non-slaveholding southerners to support secession.
But the slaveholders themselves were disproportionately wealthy, powerful, and influential politically, and it was their refusal to accept Lincoln as President - including his position on the non-expansion of slavery into the territories - that led them to push successfully for secession.