Exactly.
And when I talked about having a set number of rolls... the point isn't that I "need" them. I've been running these silly games for forty years. But it can be nice, especially for newer GMs, to have some guidelines to prevent falling into one of the common probability traps that people do fall into - the most common being "roll until you succeed" and "roll until you fail".
It's also about pacing - how long do we want to focus on this problem? Combat provides hit points as a guideline, but non-combat doens't have the equivalent. So if I say "yeah, this should be about 5 rolls" then that's just the amount of focus I want to give it. It means that any action either should progress the situation about 1/5th of the way, or shouldn't be rolled. But of course if somebody does something that shortcuts it (either way!) then that takes precedence. It's a guideline, not a straitjacket. It's really something I use to make sure that I'm playing fair and giving players a reasonable chance, and keeping things moving in a reasonable way.
Note that it's perfectly valid to dislike aspects of the 4e implementation, but this concept is used in tons of games and can be helpful when approached properly. I think a lot has to do with how you approach it, as RedMage points out. I treat them (and similar) less as discrete "now we're doing this thing!" and more as a rough scaffolding over the players just doing.... stuff... to ensure pacing is reasonable.