As much as I love genealogy and as much as I love the Civil War, it saddens me to know I don't have any direct ancestors who fought for either side. I think one of my g-g-grandfather's cousins/nephews fought for the Union, but have never been interested enough to confirm that, as my genealogy interests are more within the direct lineage. That's perhaps a bit of selfish attitude, but that's where my interest is.
On the other hand, every time I think about it, I realize that if one of my direct ancestors had been in the War (or any other war, for that matter), they may have been killed before having the children that eventually led to me, so I may never have been born. In that perspective, luck was on my side.
(A lot of it had to do with simple luck of when they were born - I know a couple of g-g-g grandfathers were born in 1814 and 1817 respectively, so they were a bit too old for the war (for the most part), but there children were born in the 1850s, and were too young.)
What service records have you been able to find of your ancestors - i.e. do you know their regiments, what campaigns they were a part of, which officers they served under, etc.? That's the kind of stuff I would find absolutely fascinating if I had any relatives who had fought.