Spouse visa if spouse lives abroad?

I'm at American woman married to a Japanese man for almost two years, and we live in New York while his family is in Japan. Unfortunately his job needs to keep him in New York but I would really like to start living with his family in Japan. Is there a possibility of getting a Spouse of a Japanese National visa in this case if my husband won't be living in Japan?

Purpose of that visa is to allow you to live together in Japan. Given that you don't fall into that category, while you can certainly apply, I think it would be difficult.

As Dainichi Heater says, that is the intention (to allow the couple to live "together" in Japan). But how long is your husband's job going to keep him occupied in the States?

If he too is eventually moving back to Japan, you could possibly apply for this visa through the path of applying for CoE (Certificate of Eligibility) from outside Japan simply based on your marriage, and if you get it, you can apply for the visa in the States. The husband, due to his professional commitments, decides to stay a "short term" alone in the States, but will be joining you in Japan for sure. If this is true, you might be able to do that.

If you come alone just with your passport (as US citizen you'd get 90-day stay as Temporary Visitor), and apply for a change of resident status from "temporary visitor" to "spouse of Japanese," there will be little chance because you need to show a resident's record showing him and you at the same address.

applying for CoE (Certificate of Eligibility) from outside Japan

That is not possible.

As mentioned above, I do not think you would be eligible for spouse visa. Spouse visa obviously granted to the foreigners to be their Japanese spouses not to be with spouses families.
Also, considering that you are not living together in Japan, I doubt that that you will get a spouse visa.
For example, even when people apply for renewal of spouse visa, the immigration officers usually check the location of the work and home address, if they find the husband lives in a different city like Osaka and his home is in Tokyo, the spouse visa renewal either will be 6 months, max 1 year or it can be denied if the immigration officers had strong reasons to think that it is a sham marriage, marriage to get visa to Japan only. Again, this may not be 100% relevant to your case but it guess it gives you an idea.