As mentioned, Havre De Grace, St. Michaels and Easton are nice. You might also hit downtown Annapolis and include a tour of the Navel Academy ... I found it quite interesting and my 11 year old son thought it was "very cool."
A little further down the road on the eastern shore is the Ward Museum in Salisbury, MD. You'll have a chance to see decoys and wildfowl art from the top names in carving, past and present. Then head over to the Ocean City area ...
A couple stops in the DC area I would definitely take in, if they aren't already on you list is, George Washington's Mt. Vernon and the new portion of the air and space museum at Dulles airport.
Headed the other direction, you could head down to the Charlottesville area and check out Thomas Jefferson's Monticello and then take in part of skyline drive.
Enjoy your trip