Would it annoy you if someone stood by a tank of human waste and waited for you to walk by before they stirred it?
St. John Chrysostom writes,
For God will determine your sentence, not only from the nature of your transgressions, but from the judgment which you have passed upon others. Therefore He gave the admonition, “Judge not, that ye be not judged.” For the sin, of whatever kind, will not appear the way it did when committed, but it will receive a great and unpardonable addition because the judgment you passed on your fellow servants...
...If any one were to stir up a cesspool, when you were passing, say, would you not reproach and hate the man who did it? This then also do with respect to the slanderer. For the stirred cesspool does not so grossly offend the sense of those who smell that ill savour, as the stirring up other men’s sins, and the exposure of an impure life, offends and disturbs the soul of those who hear of it. Therefore let us abstain from evil speaking, from foul language, from blasphemy; and let us not speak ill of our neighbour, nor of God!*
In other words, listening to gossip affects us in two ways: First, it leads us into the silliness of passing judgment on another person, thereby increasing the weight of our own judgment. Second, it exposes us to the ugly "odor" of the sin our poor brother/sister fell into(stirring the cesspool), making us more susceptible to that very sin.
So, if we knew what was good for us, gossip and judging would make us angry, not eager to hear more.
*Homilies on the Statues to the People of Antioch, Homily III, ¶ 16