Most? That's a bold statement to make IMO. Anything to back that up? I do agree that not all were mistreated.
Uncle Tom's Cabin was a novel, and Abolitionist agitprop. While it likely portrayed the worst of the worst, slaves were a considerable investment.
Mistreating that investment made no sense, as their labor would bring success to their owners, their inability to be productive would harm them. Far better to treat them well and earn their loyalty. Keep in mind that while not all owner's wives were wonderful people, many were good Christians who would brook no mistreatment of their servants, except in the most extreme cases when severe discipline was seen as absolutely necessary. It was easier to sell recalcitrants down the river and keep morale up on the farm or among those in the business.
Do sample the accounts at this link:
http://dbs.ohiohistory.org/africanam/html/mss/gr7999.html as the accounts seem pretty typical of those I have read from other areas. Yes, there were incidents and some were known for cruelty, but it was far from universal, and by simple logic, in the minority.
That is, of course, putting aside for the moment the fundamental objection to owning another person, which I think we all share.