Obviously, you need to visit since this is one of the most important sites in Buddhism. It doesn't matter that you don't get to see the tooth (and that it's probably not the real deal). It's the emotional and philosophical experience at being at such an important place.
More