Yes! Batteries must be removed from the derailleurs in order to connect them to the charger.
Take a look at our Overview video of the SRAM RED eTap system for a tutorial on the meaning of the LEDs and the removal, installation, and charging of the batteries.