WW2 museum is a must - absolutely fantastic.
Joey K's on Magazine is a great place for lunch for red beans and rice - and fabulous cornbread if you hit them on the right day (can't remember which day). Jacques-Imos is one of our favorite restaurants and a stop everytime we go to NO. Camelia Grill for a great burger and local experience.
If you want a fancy dinner, Galatoires or Commanders Palace. You'll pay, but exceptional food and great service.
Audubon Park makes for a lovely walk, as does the Tulane campus. Both easy to get to by trolley car if you're staying somewhere on the line.
I generally agree about avoiding the French Quarter... except, if you haven't been you need to at least once. It's worth a walk through and maybe a drink. Or tie a walk in with dinner at Galatoires.
My wife and I love New Orleans. It just has its own rhythm, which is awesome if you go to immerse yourself and can let go of its many flaws. Great people, too. But you do have to pay attention to where you are and to your surroundings. If you're into music, I'd highly recommend timing your trip with JazzFest and spending a day there. We've been a couple times and can't wait to go again. An exceptional event.
Damn, I'm jealous.