When you are leaving Japan for good, you are required to return your Certificate of Alien Registration to the immigration officer at the airport (not to the City Hall that issued it).

If you have a valid re-entry permit you will not be asked to return the Certificate of Alien Registration when you leave the country, but nonetheless you should mention to the customs officer that you have no intention of returning to Japan, and hand the card back. If you leave the country without returning your Certificate of Alien Registration, you will continue to be registered as a citizen and will be liable for Residents' Tax.