I stick out like a sore thumb. I'm 6'0" but very heavy set, caucasian and always wearing shorts and/or short sleeve polo shirts. In Japan, short sleeves aren't an all-season thing, so I do look like a tourist. I don't care. I dress comfortably for me and as long as I don't offend anyone, I'm fine.
Yes, location is a big factor. I was at Okayama Station just last October and at the very end of the platform was a group of elementary school second-graders out there for their school initiation trip on the Shinkansen. I walked towards them as I was very curious and when I got close, some of the children yelled, "gaijin!" to which I responded, "hai, gaikokujin desu." When they heard me say that, they started jumping for joy and laughing. I was invited to board the Kodama service train with them although I wasn't allowed to photograph them. The teachers let me show them photos of my visit to another elementary school in nearby Onomichi and the kids were glued to my iPad. They allowed me to sit with them and one girl wrapped her arms around my massive arms and rested her head on my shoulder and wouldn't let go. The initiation trip is only a one-stop trip to the next station where they alight. They all lined up again at the platform and waved goodbye to me as the train rolled away. It was definitely worth the extra 45 minutes to Hiroshima that I had to spend on the train versus the Nozomi service.
Even in Onomichi, people look but are always friendly and say, "hello". Onomichi is famous for a cycling path nearby and many people use it, but don't stay in town much. When I go to shops and they ask me why I'm there, I always mention the school I go to annually and after showing them photos of my visits, they suddenly point to some of them and say,"oh, my son is in the same grade as this boy" or "my daughter is friends with this kid!" and they're excited to tell me. I'm sure they're a little surprised I visited the school.
I've been to Japan on 10 separate occasions and have traveled from Kagoshima in Kyushu to Otaru in Hokkaido and many places in between. I don't know much Japanese but I can get by on most occasions with a smile and an online translator to get my point across. I will always stick out because I don't look like a Japanese person. And that doesn't matter. Kids will look at me and say, "herrow". Some will play "touch" (offer their hand as in a 'high five') and they'll dare each other to say something. I had a group of Jr. High School students waiting for the tram across from the Atomic Bomb Dome and one of them said, "ice cream". I repeated it to them back and they said it again, so I decided to playfully mess with them a little and say, "Ice cream, you scream, we all scream for ice cream" and that must have blown their minds, because they said, "you are so coooool!" and started taking photos with me with their phones.
As long as you as comfortable with yourself and don't mind a little attention, you'll be fine. In most instances, people won't even acknowledge you unless you say something to them. That is especially true in the major cities and tourist spots. In the small towns, like Minamisanriku, where I was stopped by the police, or Kesennuma where they wondered if I had gotten lost, people will perhaps pay more attention to a foreigner. At the school I visit, the second graders stared at me closely when I knelt next to them for a photo. When asked why, the teacher told me that most of the kids hadn't seen blue eyes before and were curious.
Things have definitely changed from the first time I went to Japan in 1987. Back then, even the high school kids were very shy and giggled at the sound of me speaking in Japanese. And that was a town next to the airbase I was working at. Don't worry about sticking out and embrace the culture, food, and especially the people there....they're great!