Goddamn do republicans love to forget history. They hear "They used to be left" and then think that's the end of the story.
Republicans and Democrats switched sides during Lincoln's time. So anything that was "Left" around that time is now the right. Republicans love saying now they were the party that freed the slaves because of Lincoln, but really the democrats then have flipped.
So, to say "Weren't they left wing" is a bad notion, because those "wings" meant completely different things back then. The fact is that the KKK and racists have always been on the wrong side of history. Right now they label themselves Republicans. In Civil War era they called themselves Democrats, who at the time were anti-big government, pro-slavery, and dominated most of the south.