If the hypothesis could be weakened to the point of irrelevance it would have been by now. The problem is none of us really knows, so all we can do is speculate. We need to further our understanding of the universe before we can answer some of these questions definitively.
The problem with that is you're contradicting yourself. If the hypothesis cannot be weakened, then it must be the case that we're alone. Given that, no amount of studying the Universe is going to find other intelligent life, so why bother on that account?
The simplest refutation, but also the most depressing, is that there is other intelligent life out there but it's simply impossible for that life to travel from star to star.
The other argument - persuasive, but not disproof - is that we're simply on the wrong side of probability at this point: i.e., since the Universe has only been around for a finite amount of time, the probability of having found, or been visited by, other intelligent life is less than one. Even if the odds are small, they still imply that there is a possible state of affairs in which they exist but we haven't yet made contact.
Fermi's argument is essentially a variation on the law of large numbers
In probability theory, the law of large numbers (LLN) is a theorem that describes the result of performing the same experiment a large number of times. According to the law, the average of the results obtained from a large number of trials should be close to the expected value, and will tend to become closer as more trials are performed.
The LLN is important because it "guarantees" stable long-term results for the averages of some random events. For example, while a casino may lose money in a single spin of the roulette wheel, its earnings will tend towards a predictable percentage over a large number of spins. Any winning streak by a player will eventually be overcome by the parameters of the game. It is important to remember that the LLN only applies (as the name indicates) when a large number of observations are considered. There is no principle that a small number of observations will coincide with the expected value or that a streak of one value will immediately be "balanced" by the others. See the Gambler's fallacy.
Fermi's argument essentially posits that there have been enough samples already that the hypothesis has been sufficiently tested. I'm not sure that that's correct.