CB moderators were also groupjive admins by default and may change everything in groups and categories.
Yup, it's designed this way. You'll notice many parameters in configuration directly state moderators are exempt. Category owners also have power over all the groups in their category.
Is there a way to change this so that we may give to CB mods ability to act on all groups or not ?
No.
After CB 2.0 where we hope to have a better permissions system in place this will of course change, but for now it's working as intended.