US International Drivers Permit (IDP)

I'm from the US, but have lived in Japan since 2015.
From 2015 to 2016 I had an international drivers permit from AAA in the US.
I never used it and I never got my Japanese license.
I live in a big city and my work is within walking distance to home.
But now I'd like to travel to places only accessible by car.
AAA said I can get another IDP through the mail.
Will this new international drivers permit be acceptable/legal in Japan?
Will I be able to rent cars and drive?
I was under the impression I had to return home for 3 months in order to get a new one.

Your impression is completely correct. You will need to return to the US for at least 3 months before the international drivers permit will be recognized by Japan for a year. Otherwise, you would be driving without license.

From 2015 to 2016 I had an international drivers permit from AAA in the US. I never used it
Actually, that is not true. The fact you had it is the "use", the fact you may not have done any driving is irrelevant. BTW, automobile associations will happily issue IDPs even if they are not usable due to local rules.
A useful link:
https://english.jaf.or.jp/driving-in-japan/drive-in-japan
Check the section "Period during which a Person Can Drive in Japan" towards the bottom of the page.

You have to be outside of Japan for 3 months, then you could use the IDP.

However, if you are under a long-term visa or permanent residency, better for you to get a Japanese license. If you already had your foreign license for 2 years, you can get the Japanese license cheaply.

You will still have to take the written exam (easy) and driving test. The driving instructors are usually a bit mean to foreigners and usually will speak only in Japanese. This is said to be because foreigners don't pay full price for Japanese licenses (if you have valid foreign license). So even if you are an experienced driver, do take the Japanese driving test seriously, as they enjoy failing foreigners. They are said to do this so they can get them to pay more money (you pay for each test).