Interesting pet peeve here. I'll put it this way.... If you are fast enough for that to happen then you should be allowed to pass through.
On the other hand, if you are just paying attention to the group ahead of you, and that is the thing they do that ticks you off, then that is a different matter. So long as the group in front is at pace of play with the group preceeding them, this isn't shouldn't be an issue.
My cue while I wait is to do all my pre-shot during this time so I'm focused on my play and not my group or the group in front or behind me. If "pace of play" is correct, then I want to worry about me and no one else.