Conditions will likely reverse the hidden state due to the condition trying to show the field and the empty behavior trying to hide it. Have added a bug ticket for CB Conditional to see if this can be fixed, but it's a complicated since depending on the situation one hide/show behavior needs priority over the other.
forge.joomlapolis.com/issues/8874
For now it's probably best to not hide it with CB Conditional and let its empty state hiding behavior apply.