You would be correct IF you knew for a fact that there were no other people besides Nazis/KKK in that park. I don't think you can make that assumption. There may have been people there who were merely opposed to statues being torn down. In the end, it just didn't matter.
The whole charade was designed to spin up the narrative that Trump was aligned with a massive White Supremacist movement. Nothing Trump said was going to be enough to satisfy the mob. The stories were written before he spoke.
Here's what happened: The Nazis showed up about 24 hours before the riot/killing of the woman, walking in a circle two abreast shouting racist slogans, fists in the air, holding racist signs denouncing Jews, non-whites etc. This went on for some time. Anyone who was not aligned with them mostly left the scene at that point, knowing that trouble was afoot.
The advertisements for the march were largely distributed by white supremacists, so it was not a surprise to see them there for anyone. Would YOU have attended such an event under those circumstances? I wouldn't.
Many of the Nazis in attendance had riot-gear and weapons obviously prepared for violence. Not exactly signs that they were focused on a peaceful march.
To be fair, the confederacy was an ill-advised, unnecessary movement. The leaders of the confederacy ignored Lincoln and his faction's attempts to dissemble them through their open offers to compromise on the slavery issue and also ignored Lincoln's promise not to be the side that drew first blood.
There are some arguments to be made that forces on the Union side promoted a situation where the confederates would attack Ft Sumter setting off the war, but if so, the confederates were duped into doing it, because it was not necessary (some options for negotiated settlement remained at the time of the start of the war).
Robert E. Lee was arguably a brilliant, honorable man. But he fought for a side that killed more Americans than any foreign enemy. If they want to remove the statue of Lee, I don't really care.
