Can I ignore my COE after it is issued?

Hi..
Three months ago, I applied in a Japanese Language School so they're processing for my COE. But, last month I got accepted into a university (not in Japan) so I couldn't go to Japan like what I planned. I already inform the school in Japan but they said I can't cancel my application when the COE is under process, so they told me to wait and hope the immigration will deny my COE application. Then, last week I got my COE issued.
Can I just ignore my COE? or should I contact the immigration myself? But I don't know how. The school in Japan told me if I cancel my application, I have to pay some fine and I will be BANNED to enter Japan for another 5 years. Is that even true? I'm so afraid to be banned, it's not like I did some crime, but I don't really want to blindly believe them too. Does anyone know anything about it, I will appreciate it a alot, thanks.

The school in Japan told me if I cancel my application, I have to pay some fine and I will be BANNED to enter Japan for another 5 years. Is that even true?

No, that's ridiculous. Just do nothing, your CoE will expire, and that's that.

I second that, absolutely ridiculous!
I believe that the school you have applied to is trying to use scare tatics on you in order to gain some profit/compensation. You will not be banned for not using the COE, as stated above, it will simply expire.

Thanks a lot for the answer. Actually, that school even sound like a fraud now. They want to get money from me by forcing me to sign a contract that says I'm willing to pay the fine meanwhile I never know how much the fine is. I tried to ask but they never said any number.
Tomorrow I'm going to call the embassy in my country and ask for it. I wonder if the school really have any right to make me pay some fine. It's not like the school giving me any advantages, it's not even scholarship or working contract, just why..
Anyway thanks a lot