I think about this all the time.
Consider Astral Projection: As a 9th level spell it's one of the most powerful spells in the game, and the only thing it does is let you go somewhere. You can even count on your DM making all the protections of the spell useless, as you'll definitely run into silver swords or creatures who can sever your cord - why wouldn't they, amirite?
So, long story short, you have to cast a 9th level spell to fulfill, "you're allowed to go here now." And how long will that last? Well, since 5e refuses to make high level games balanced/playable, it won't last long.


But, yes, I agree with your premise, OP.
The story your DM wants to tell will accommodate the party as it needs to. If "do you know this spell?" is going to be a wall, the game just grinds to an immediate halt and no one has fun when that happens.