The sad part is that Lee owned no slaves. He resigned his commission in the Union Army to go home to his home State of Virginia. At the time, yes, we were all Americans, but the States had more power than now, and were what the name implies, sovereign states, united to provide a better common defense and for purposes of trade. Each State had it's governor, legislature, Secretary of State, Treasurer, constitution and unique set of laws--in effect a State, a nation unto itself, bound only by an agreement that they would work together to secure the Liberties and Rights reserved for the States and the People in the US Constitution, the compact which formed the Republic.
When that compact was no longer working, many States chose to follow the advice of the Declaration of Independence
That whenever any Form of Government becomes destructive of these ends, it is the Right of the People to alter or to abolish it, and to institute new Government, laying its foundation on such principles and organizing its powers in such form, as to them shall seem most likely to effect their Safety and Happiness.
This was no spur of the moment decision, and tensions had been building between the Northern States and Southern States for decades. Virginia was slow to secede, so much so that Maryland lost the opportunity to do so and was invaded by the armies of Pennsylvania and Massachusetts before such a vote could be taken.
Lee faced a serious quandary, whether to remain in the Army of the Federal Government, or return to and be of service to his home State of Virginia.
Consider, in modern terms, if you were a British general of the Army of the European Union (should such exist), and had to decide whether to remain in the EU army, or go home with Brexit and be true to your roots.
Lee chose Virginia.
Not "slavery" but his home State, and trying to make the war solely about slavery (racism) only ensures that the other issues which were raised then will return, and quite possibly, bloody conflict with them.
Only the intense ignorance of honorable men will permit their memories to be thus sullied, and for the Capital city of the State, and indeed, the Confederacy, to tear down his statue is an egregious desecration of the memory of an honorable man.