I mainly have experience with Quebec City, from living in Maine and going there several times with either my wife, or my brother & Dad, etc....
First, it's a really hilly city and not really built for cars (not a lot of parking) so be prepared to walk a lot, especially up and down stairs.
Second, the area around the Chateau Frontenac is definitely the most touristy part, especially the part down the hill to the east. It's still fun, but prepare for swarming tourists. In this area, the restaurants just down the hill (between 136 and the hill) are so touristy as to not be worth it. However, the restaurants in the area just to the west of the Chateau (on the same level, generally) while still touristy, are pretty good. In general, it's worth doing some research before you go, because the good food in QC is
really good.
Third, I do recommend hiking around Mt. Saint-Anne, about a 45-minute drive away up the St. Lawrence, if you like hiking. Beautiful area, well-maintained, etc....
Fourth, the Musee des beaux-arts (right on the central plain over looking the river) is totally worth it. Check out every nook and cranny of the several wings. The surrounding parkland area, and old walls, are also a fantastic visit.
But fifth, my favorite museum, and quite possibly one my favorite museums in the world, is the Museum of Civilization, just down the hill. While they have a lot of fun special exhibits (and a good cafe), just the multiple exhibit areas that walk you through the history (including pre-colonial) of Canada and Quebec are utterly fantastic and captivating.
Hope that helps!
