You need to review the expiration dates and re-set those which should remain active or set to expired those students who have left the district.

Go to Alerts System > Add/Modify Alerts and use filtering options to select a listing by grade or single Alert type. Click on the Student ID to open up the Alert. Re-set the Expiration Date to a future for those that should remain active date (enter the end of new school year or the year the student graduates) or to terminate, set the Expiration Date to a day before the first date of school.