There is a chance that you don't have read permission to tokenGroups for a specific set of users because the ACL on those objects does not have "pre Win2K" group granted read access and no other ACEs grant generic read either. It may be that these users are either all admin users and have inheritance blocked on their objects or that it is the normal users that have this problem whereas the admin usrs in the directory DO have the pre-Win2K with the blocked security descriptor.
Either way, you'll need to ensure that your ID has read rights on tokenGroups for this to work. The bottom line is that in typical AD installs, normal users have read rights on tokenGroups but it CAN be taken away with certain default ACL setups and that can cause your app to break because you take a dependency on a permission that you don't actually have.
Hopefully that provides some info. If you need more detail on why some objects have ACL inheritance blocked by default, do some searches on adminSDHolder.
Best of luck!