The government "gave" him the pass, a physical item.
To take it away, the court would argue that their should be a process to remove that pass, once granted.
I don't agree with this, but I think that is the position they are taking.
Think of it like being fired, with getting previous warnings, or write-ups, etc.
More like a labor board issue than a 1st amendment issue.
Again, I don't agree, because he hasn't lost his job, he was just "demoted" which happens in the real world, too.