It also doesn't specifically restrict the federal government's ability to regulate borders. And common law precedent at the time of the adoption of the Constitution generally provides governments the ability to regulate borders. Such a right by a government is as old as the practice of exiling people, as well as declaring someone an "outlaw", which traditionally gave free citizens the right to kill those declared as "outlaws" if discovered within the government's borders. It's also why the Constitution specifically prevents the *states* from limiting interstate travel by citizens of any state, preserving that traditional power for the federal government.