I've just returned from a Golden Week vacation to Matsumoto in which we tackled the castle on a very crowded day (likely to be similar to the crowds at cherry blossom time).The stairs at Matsumoto Castle aren't anything to balk at for the most part, but the very last two flights of stairs to the very top of the structure are rather daunting. I found myself having to be extra careful on the extremely steep and narrow steps in this section.
On extremely crowded days, large groups of visitors are herded through the castle in lines. There are few opportunities to escape and quite a bit of pressure to descend and ascend stairs quickly so as not to hold up the line.
With this in mind, I would recommend NOT going during the most crowded seasons to ensure you'll be able to explore the castle in your own time and opt out of the last two flights of stairs if you feel uncomfortable with them.
That said, castle's interior is far less impressive than its exterior. You might find yourself more than satisfied just exploring the grounds and looking at it from outside (especially if you go in a time when there are crowds).