Yup. And the temperatures were FAR higher.
Then the removal of that CO2 from the atmosphere, being trapped as coal or other fossil hydrocarbons, caused the temperatures to plummet down to where we are today.
If we want to go back to those conditions, all we have to do is burn the fossil fuels!
You oversimplify. From:
http://joannenova.com.au/global-warming-2/ice-core-graph/One of those graphs:

Please note that 12,000 years ago, where I sit was covered by an ice sheet over 1km thick (some estimates say 2km). This is roughly the time Clovis Man makes his appearance in North America.
Note, however, that the bulk of the warming occurred between 20,000 years ago and 12000 years ago, and did so without any coal-fired power plants, SUVs, or oil well flares. In fact, the whole industrial revolution thingy was well, between 19,850 and 11,850 years away.
But wait, there's more. According to the Vostok ice core data, the lag time between temperature going up and CO
2 increasing is roughly 800 years. That makes CO
2 a lagging indicator, not a forcing indicator.
So, why does the CO
2 content go up? Soluability. CO
2 dissolves more readily in colder water (70% of the earth's surface). As temperatures warm, the ability of the CO
2 to remain in solution declines, and it is precipitated to the atmosphere.
Just like the CO
2 leaving beer or pop as it warms, it left the oceans. Now, the concentration wasn't as high in the oceans as in the beverages, but there is a lot more marine habitat than pop or beer.