Yes, better to punish hard working taxpayers.
If you contend that higher education is a net cost to "taxpayers," then every student is a cost too.
If you argue that taxation of graduates over their lifetimes, repays the costs then yoou get another answer.
That is another issue, however. I went to a state university, and received the GI Bill, too. I have paid income, property, sales, excize, utility taxes for over 50 years, too.