I just came back from there.
Been to both. I would recomend the VA side. A little more to get to it, but a lot more wild and secluded. There are a couple of nice visitor's centers there. After 3 pm, they open the "Wildlife Loop" on the island to cars. It's a great little 3 1/2 mile ride. You can walk it, but I wouldn't. Mosquitos are more than bad this time of year, more like "unbearable". Ponys are usually around somewhere. Sometimes they'll be up by the road, sometimes only visible with binoculors. You just never know.
Also along the main drag is a neat little Merritime Museum. It's free as are the visitor's centers. Doesn't take much time to go thru, but interesting nonetheless.
Couple pretty good places to eat on the Island too. We like the Crabshack for lunch. If you want something a little nicer, AJ's on the Creek or Bill's Seafood are good. The Village is our favorite. There is a McD's and Subway too if you like.
Ponys you should be able to find:
Unfortunately, the Refuge Waterfowl Museum has been closed for a couple years. I did notice the outside of the bulding had a fresh coat of paint on it this year though, so maybe there is hope for the future.
Just to let you know what you're missing in the museum:
If you happen to get there, let us know how everything went.
We LIKE pictures...
Jon