If Trump isn't leading the delegate count, he has no claim. But this seems unlikely.
If he is leading the delegate count but fails to get a majority, he merely has a moral claim in a game with rules which allow a different choice to be made. On the other hand, you might argue that the moral claim lapses in the face of the fact that a majority of delegates oppose him.
If he is selected in those circumstances, I suspect many key people will avoid offering their support and will hope he loses the election. If he does, then the party turns the page on an embarrassing interlude.
If he is rejected having led the delegate count, my guess is that Trump would launch an independent bid. The Republicans would lose the election but retain their party. Trump would also lose. I doubt Trump will stick around to build a minority national party from the ground up, unless there's a branding opportunity available. The Trump Party. Pay the Donald on a subscription basis for use of the name. No returns policy applies. Read the small print.
If Trump wins a majority of delegates, I suspect the establishment conservatives will launch an independent bid. The Democrats would win the election but the establishment conservatives would retain a sense of dignity in putting the country above their party. The GOP would split, of course. Conservatives now know the danger of riding Tea Party nationalism. They'd need to start a new party, probably called the Conservative Party, like in the UK.
Maybe splitting the GOP would be a good outcome for the USA. A new, moderate party might emerge alongside the wingnuts. This would allow compromise in Congress and make the country governable for a change.
The worst outcome of all for the USA is that he wins the nomination and the election. Not sure how the country stays together under those circumstances. Would California wear being led by the new Mussolini?