Could be an anthropic explanation for that. In a solar system with a hypothetically-"normal" Oort cloud, comets and debris from the cloud might wipe out life on the habitable inner planets every few hundred million years, never allowing it to advance to human-like levels.

So we might be here only because our solar system is surrounded by an unusual amount of nothing.

But we also look at a lot of other stars in the sky. If every single one (or almost every single one) had a massive 5x mass Oort cloud around it, it would affect the light we see from that star.

Consider that we can currently detect differences in luminosity small enough to tell whether an Earth-size planet is passing between us and the star. A 5x mass Oort cloud would be thousands of times more mass than that. It would have noticeable effect on luminosity.

And, while our sun has an Oort cloud, there are a lot of stars out there that probably don't--too small, too big, too hot, too young, too old, etc.

Amazingly small! It extends halfway to the next star (a lightyear or so) which seems like it would add up to a lot.