Learning to Shut Your Mouth and Open Your Ears