Physical location, in this case, does not refer to political boundaries or geographic location, but rather to network topology.
Just as the address prefixes of telephone numbers (the country and area codes) are tied to the topology of the telephone network, just as the address prefixes of postal codes are tied to the topology of the postal distribution network, IP address prefixes are tied to the topology of the Internet.
The converse, assuming that address prefixes are not related to physical addresses, would require address tables containing entries for all of the millions of IP addresses. Clearly this is infeasible, just as it is infeasible for a mailman to know the location of every potential address that might appear on his envelopes. The mailman only knows about the handful of addresses he is directly responsible for, and uses the hierarchical postal codes to forward mail addressed to other regions. The Internet router only maintains detailed information about the handful of devices it can directly deliver packets to. Packets addressed to other devices are delivered using the location information encoded in their addresses.