...If you are even mildly interested in art, architecture, or design, or if you would like to see a small island village, I cannot recommend highly enough to go to Naoshima (ferry from Takamatsu).
Benesse House,
http://www.naoshima-is.co.jp/english/index.html , the Chichu Art Museum, and the Art Houses project are so good that they would be the first place I would want to revisit in Japan (after 4 trips, going to all the main islands, etc. My wife shares my view on Naoshima.)
It's expensive to stay at Benesse, but I'd say it's definitely worth it, and would happily pass up indulging in lots of my favourite treats at home in order to save to stay there again. If you do, the most wonderfull thing is to have dinner in the museum. All the visitors go home. Most of the other guests at Benesse are eating in the restaurant in the hotel, but you (and about 10 other tables, iirc, have the whole museum to yourself for a night and it is magical.
Anyway, you can probably guess that I could go on about the place forever.
Ritsurin Koen is pretty good. If I were to rank the ones I've been to, I'd put the Adachi or Kenrokuen ahead of it, but they're all three very good in my book.
There's not much else in Takamatsu town (that we found, anyway), but it's easy to navigate and there are some good small bars and restaurants (the local noodles are worth a try), and I suppose it will be your base for the whirlpools.
Dogo is pretty, and if you enjoy onsens then it would be a shame to miss at least a flying visit to kame no yu (I think that's what the famous old bath house is called). We just did that, and then spent the rest of our afternoon and evening there eating and bathing in a lovely ryokan.
The train ride from Matsumoto to Takamatsu gives some good views of the coast at points.
Hope you have a lovely time