Is the term gaijin for any non-japanese

Or is that mostly used for white people and us non white's get called a different term.

As I am Indian, I just would like to know, just for curiosty reasons

Gaijin simply means foreigner (lit: 'outside person'). Any non-japanese is considered a gaijin in Japan.

I heard it had the implication of a white person. Though, for the Japanese at least, it's not a good word.
Personally, I don't care and I haven't known any foreigners who do care. But it's not a polite word to use. (I'm half-Japanese if you're wondering)
But in any case, correct me if I'm wrong, but i've heard that gaijin holds an implication of a Caucasian but it literally means "outside person". Gaikokujin is more polite and seems to represent every foreigner.

As far as I know, a Chinese is a gaijin just as much as an American is. I was under the impression it means anyone who isn't Japanese. When I have heard Japanese people refer specifically to Caucasians they usually say "hakujin".

白人 the kanji literally are "white" and "person" respectively. Is that world considered to be rude as well? I haven't heard it often, so I'm guessing so.

the kanji for gaijin is 外人 not 白人.
To be grammatically correct "foreigner" should be "gaikokujin".
After the second war, the word gaijin ("people from outside") appeared and was to designate american people and by extension white people. Today gaijin is just slang for foreigner and has kept its insulting notation (somehow).
So i would say its traditionnally for white people but nowadays used for any foreigner.

the kanji for gaijin is 外人 not 白人.
did you see the post above mine?
I was referring to "hakujin" which is 白人. And i later explained the kanji meanings. I know the kanji for gaijin well enough.