Yes, it is quite safe to drink tap water in Japan. Tokyo Waterworks writes on their homepage that the tap water is completely safe to drink and has a soft taste. Technically the water is soft and is that gives a strange taste for me who is used to hard water. If you don’t like the taste or need water when outside, then every convenience store sells a big water bottle for ~100 yen (7-10 NOK).