It matters because where your hosts' server is can affect it's speed of connectivity. The more telecomunications backbones in the area (that it connects with) the better the speed should be.
That's a matter of connection though, not actual location. Physical location itself shouldn't matter at all, as long as the server itself is well cared for.