Can't judge them using today's rubric
It was a pretty complicated issue.
Even men who wanted to release their own slaves couldn't. Slaves were collateral on debts and if a man released his slaves without the debt being paid, the slaves belonged to the creditors.
James Armistead Lafayette is a good example of what a mess it was. He was a slave owned by William Armistead who went to join Benedict Arnold's camp (and later to Lord Charles Cornwallis's camp) as a spy who was actually a double agent spying for the Marquis de Lafayette.
After the war slaves who had fought as soldiers were free if they wished it but James Armistead was a spy and not considered to be a soldier so he returned to William Armistead's plantation.
In 1824 James Monroe invited the the Marquis de Lafayette to visit the USA. The Marquis was a big hero and crowds came out to see him everywhere he went. The Marquis spotted James and William in the crowd and found that James was still a slave. The Marquis with the help of William Armistead (who was himself a delegate to congress) petitioned congress for James to be freed. James Armistead took the name Lafayette to honor the Marquis.