Change "Remove selected items" to "Exclude selected items" and grey them out instead of removing them. If the user wants to include them again they check the box and click a new button "Include selected items". This way the user dont have to start over if the remove the wrong item accidentally.
Great idea! Promoted to a feature.