Neither Letterman or Colbert are particularly funny or entertaining, in my view.
At that time of the night, I'm on my Roku picking what I want to view, not settling for network slop between commercials.
Hence the problem facing most late-night TV. There won't be another Carson in our lifetimes... because 1) people use their DVR's now, and 2) the number of outlets for late-night TV are far greater. When Carson was host, he was it. Sure, you had Merv Griffin or Dick Cavett for a couple years, but by and large, Carson (benefiting from a network that was far quicker to put priority into late-night TV) was the only choice for a show of his format. Then Arsenio Hall came along and blew up the whole business model. Now you have ABC, CBS, NBC, Comedy Central, TBS, and Arsenio back in syndication... all fighting for that same piece of the pie.
The only real opening would be a weekly or biweekly show on Saturday/Sunday nights. NBC and CBS have NFL overruns on Sundays in the fall, making that slot more difficult. Fox and ABC have the same problems on Saturday (and NBC, of course, has SNL), and besides, no one watches TV on Saturday anymore. The other option is something a little more out of the mold, such as a more country-oriented show from Nashville or Texas, or maybe something out of Branson (where there are always celebrities swarming).