It matters before he's elected. But it certainly isn't anything that justifies removal from office after the election, which is pretty much what some have been attempting to do since November 2016.
Frankly, that's why I support the guy. The concept of trying to force an elected President out of office because you don't like him, or don't think he's "fit" for office, is anathema to a republican form of government.
Au contraire............ it matters much after one is elected, and is the very reason that people should be smart enough, and care enough about a candidate's character to not allow him to get elected.
I'm not trying to get him out of office (though Pence is a thousand times better a man), but I don't think that it should be forgotten that he was elected by people who knew he was not trustworthy, and didn't give a rip.
He wasn't fit to be President before he was elected, but all we can do is endure the further disintigration of our republican form of government until someone who shares our principles can be elected.