When we hear that phrase, it is normally accompanied with an assumption that one person is not equipped to tell another person that something they are doing is wrong. Pick your favorite issue here.
The underlying reasoning is quite often that individual’s desire to justify an action or idea they are engaged in that is being criticized or condemned.
The accuser is told emphatically that they cannot tell another person what is wrong for them. The implication following that assertions is that the accuser must therefore agree that what the person is doing is permitted (since he has no right to condemn or question it).
If one person is not allowed to tell another that what they are doing is wrong, how is it that they are fully expected to tell them it is right? If it is judgmental to say it is wrong, then the same moral certitude is required to say it is right.
Let’s at least be honest enough to answer those who accuse us of judgmentalism that in order to avoid that unsavory attitude, we cannot agree that what they are doing is OK, because we don’t want to appear judgmental.