My GGGrandfather Samuel Stafford, 9th TN Cavalry, was captured near Bardstown Kentucky in 1863 during Morgan's Great Raid, and sent to Camp Douglas. He survived smallpox and the war and returned home, else there would be no HoustonSam today.
As bad as it was for him and his colleagues at Douglas, I'm sure the Union men imprisoned at Andersonville had it worse.