Water in Japan

I am wondering whether the water that comes from the water supply (eg. from sinks) is drinkable or do you have to buy bottled water in stores?

Tap water in Japan is quite clean and safe, it is no problem to drink it.
Whether it is a great water or not varies depending on the water source. Generally speaking, in countryside closer to mountains water is much more "tasty".

I had no problem, drinking it every day in Tokyo, Kyoto and Hiroshima. All tasted fine to me too (as someone who always drinks tap water).

drinkable all over Japan.

Tokyo tap water doesn't taste great, but it's fine if you filter it. I've read that over 50% of people in Tokyo drink either filtered or bottled water.

Unless there is a sticker that says it is not drinkable, then you can assume it is drinkable. I drink tap water in hotels and ryokan across Japan. The tap water in cities sometimes has slight strange tastes, but the water in the countryside tends to be clear and delicious.

Generally tap water is drinkable.

The only places I see "don't drink" signs are on the trains.

I drink water straight from my sink and it's fine. Plus, buying the big bottles of water from the store adds up in cost and it's a hassle buying it from the store and recycling it.