Megara Wench

My company and I play on xbox. We are trying to figure out the grouping to switch crew members from left to right and front to back. It is not working because we are unable to deselect the right and back groups. We click the button to deactivate those two groups, but the slot stays highlighted purple and nothing happens. Is there something we are missing? We use to be able to deselect them.