Device list in profile

Issue: Device list is not updating for devices removed from profile OR new devices added to profile are not showing up.

Question: How do I remove devices from the drop down list without deleting a profile and reconfiguring everything?