The assumption, of course, is that every slaveholder was like something out of Uncle Tom's Cabin, and every slave was grievously mistreated.
While I have no love for slavery, slaves were seen as a valuable investment, and treated as such in order to get a good return on that investment.
But but...they didn't have television, cell phones, hot and cold running water, central air, central heat, flush toilets, or any of the necessities of life as we know it...
...but neither did their owners.
While I find the idea of one human owning another repulsive enough to justify the elimination of the institution, in practice I doubt that it was ordinarily as harsh as Ms. Stowe's novel portrayed it.