Many people skip Boulogne on route to elsewhere, which is a pity. The old town is full of character and the ancient city walls are quite impressive, you can walk around the parapit with wonderful view across this part of Northern France. Stay one night, or at least have lunch in the Old Town, Le Pierre Chaud is a favorite, but there are quite a few good resturants to choose from.
A little shopping, Auchan is good for everyday drinking wines and other French goods, this is based out of town at St. Martin. It's worth filling up with fuel here as it's very good value in France and positively cheap compared to the UK.
So, if you are in Boulogne, stay awhile, you might be presently surprised with this pretty little coastal town.