Was looking into the visibility checks because I need them for something different and noticed the checks are more complicated than they have to be.
The rule is just: user/org is visible if
- The doer is a member of the org, regardless of the org visibility
- The doer is not restricted and the user/org is public or limited