I think you're trying to draw a definitive conclusion from incomplete data. I think there are lots and lots of reasons why folks vote (particularly in local elections). Yes, “midterms go to the party opposing the sitting President most of the timeâ€. The question is why? The answer COULD be dissatisfaction with that President’s performance and his policies (and a desire for more opposition in the legislative branch). I’m not saying that’s always the case, but as I said before, I just don’t think there’s any way to prove the motivations of voters one way or the other. Having said that, I definitely have my OPINION that the first mid-terms after an election are sometimes a referendum on the new President and his policies (but for any given election, there are a multitude of motivations for voters so it's tough to generalize one way or the other). That’s JMO though just as you have YOUR OPINION.
Concerned, I see the point you're making but I think you also have to account for the normal ebb and flow of support between the parties. One party wins the White House and holds congress...so motivating partisans of the other party to activate. Since off year elections CONSISTENTLY are won by the party that does NOT hold the White House, clearly they are about much more than being a referendum. This is not to say that the popularity of the incumbent President is a non-factor, just that you must evaluate how MUCH he factors by looking at the "norm"...and then seeing how strongly the election deviates from the norm.
So, I don't think you are wrong per se when you assert that mid-terms can be a referendum on the presidency, but I would argue that there IS a built in set of factors that advantage the out of power party...to the extent that the loss of 15-20 House seats, for example, should be looked at not as a negative reflection on presidential support...but rather as essentially being a wash in terms of the public's review of that president's performance.
In other words, set the bar for evaluating the referendum in a fair context. Its not realistic to think a party that's out of power in both Houses and the The White House will not naturally make gains in an ensuing mid-term...regardless of the performance of the President.
The most analogous electoral cycle with 2018 would be that of 1994. Clinton and the Dems took a beating, in part because the public perceived the pendulum had swung too far in one direction...and in part because the GOP was activated and motivated to turn out given its recent defeats. By 1996, Clinton had regained his balance and comfortably won the 1996 election. So you could SAY 1994 was a referendum...and in part it was...but it was so much more, and its impact was somewhat fleeting. I would expect something of a repeat of that pattern in 2018 and 2020.