I tried counting up, then down as Dave in Saitama suggested, and wow, yes, when counting up I say "shi," and when counting down I say "yon," without thinking. Wow!
This is so far what I found as explanation, at least for 4:
- Normally, when numbers are read out individually, for example, simply counting, or reading out phone numbers, postal code, street address or the year, etc., it is read as "yon." It may become "yo-," when followed by a counter. For example, "4 (people)" would be "yo-nin," and "4-year-old" would be "yon sai.
- Only when in idioms or fixed phrases, "shi" comes up.
When we read the numbers "counting up" from one through five, we do say: ichi, ni, san, shi, go..." This is explained saying that the speakers (native Japanese speakers) consider this 1, 2, 3, 4, 5 as one single phrase rather than individual numbers, and that is why we say "shi." However, when you count down, we do say "go, yon, san, ni, ichi." This can be attributed to us no longer thinking this a natural string of words that constitute a fixed phrase, thus we use "yon," as it is considered an individual number read out.
One of the numerous links that I referred to, for other Japanese readers who may be curious:
http://www5b.biglobe.ne.jp/~aiida/gimon15.html
