National Holiday

Hi, we'll be in Tokyo for 5 days during Christmas 2012. As 23 December 2012 is Emperor's Birthday & falls on Sunday, will 24 December be considered a national holiday as well? Will most places like onsen or museums be closed?

23 December 2012 is Emperor's Birthday & falls on Sunday

Yes; therefore the 24th will be a substitute holiday.

Will most places like onsen or museums be closed?

Not in general.
I suppose that for most facilities in Japan, Emperor's Birthday or Christmas is not a reason to be closed.
Some facilities might be closed for another reason such as greeting New Year or being busy (e.g. sake breweries).

Facilities which are usually closed on Mondays may be open when a public holiday falls on a Monday; some of them are closed alternatively on another day, for example on the following day.
The 24th December 2012 will be a public holiday but not Emperor's Birthday. So, even the East Gardens of the Imperial Palace will be open that day and be closed on the following day (= 25 December).