It totally depends on who you are talking to.
"Otaku" ORIGINALLY was a word that was respectful, "your (honorable) home," thus, "you" keeping a polite & respectful distance. But eventually as the word was used by people who could not relate well with real-life people but only with characters in manga/anime (two-dimentional beings), who would typically use the word "otaku" to refer to "you" not as a polite term but to keep a distance from people, the word came to be associated with people who were too much into anime/manga, and detached from real life, so the English equivalent word would be "geeky" in a negative sense.
Of course, there are people who are into manga/anime to a pleasant or normal degree. Those would be normal manga/anime fans, who love to talk about that topic among other topics in real life as well. But if the first thing that you bring up on a date is anime/manga and you demonstrate an extraordinarily deep knowledge about it, you could be considered "geeky" right from the start.
In any case "otaku" culture, while by now it is well established, is not that prevalent in Japan - some people like it, others definitely not into it.
Now, the above is "generally speaking," including Japanese and non-Japanese.
Assuming you are non-Japanese:
If a non-Japanese person starts talking about anime/manga on first meeting, and that continues to be the main topic, or the only topic, then the Japanese person you are talking about might get the impression that oh, that's how you know of Japan, and that's the ONLY aspect of Japan you know, and that's how you see Japan... through the tinted glass called "anime."
Yes, it originated from our country, and yes, it is nice you like it, but if that's the only thing that is ever mentioned from a non-Japanese, that might be a bit off the balance.
I guess that's the reason why the poster you bumped into must have warned you against talking about anime on a first date. And unless you are in manga publishing, animation movies, anime-inspired fashion, or gaming industry, bringing up the topic of manga/anime right up front upon meeting (a business partner from Japan, since you say your company?) might be considered a bit childish (the business partner might not be into manga) and the Japanese visitor might be smiling later thinking "oh, after all Japan is known only for anime!" :)
Nothing is really "wrong" with the word otaku itself, and no offense, but please not that that "might" be the impression you give.
Now, this comes from Japanese woman in her 40s, likes manga in general but definitely not "into" it, and definitely not an "otaku" :)