I can say with pretty close to 90% certainty that if somehow Trump is denied the nomination he'll declare as 3rd party. That is how narcissistic this SOB is.
I suspect not, simply because he likely would know what that would do to his legacy. He'd likely lose some of his own supporters, and every other conservative would despise him. He's got some smart people around him who would let him know what things would look like after the election, and it probably wouldn't be good for him. I think it's more likely that he just wouldn't endorse/support the actual winner, and at least implicitly suggest that his supporters should stay home.
I also think there is a decent chance that -- again, considering his narcissism, he
might demand and get a keynote speech to support the winner. Because after all, there's nothing a narcissist values more than the wild cheers of huge crowds.
Here's why I see that as a possibility. Imagine all the drama around "will Trump support the nominee, or not". Tons of media attention, which he would absolutely love. He then milks that for a speech at the Convention where he enthusiastically endorses the nominee - obviously while patting himself on the back for all the wonderful things he did. But
if he did that, he'd get
wild applause. The entire party would thank him, and he'd be a freaking GOP hero for doing that. I think that puts him in a much, much better place moving forward than would a doomed third party effort that would end in a whole lot of people really hating him. It'd be a perfect end for the ultimate showman.