Since the 16th century when Jesuit priest Jose de Acosta created the theology of extraction giving a Christian rationale for exploiting indigenous people on land with valuable minerals we have been dealing with the affects of racialized geography. In a speech for Louisville Presbyterian Theological Seminary Dr. Willie Jennings discusses the way the rich and powerful manipulate the system to ensure that things stay the way they are. This applies to race too, the rich and powerful (mostly the dominant white culture in the United States) manipulate the system to ensure that property is divided up so the rich and powerful have their part of cities/towns/etc. and the not rich and powerful are kept out.
The reason why many white people become defensive when talking about white privilege is because white people have never had to reckon with whiteness as an identity. Author, professor, and theologian Dr. Willie Jennings says that we have to recognize whiteness and it's deep roots in American society and the American church before reconciliation can happen. If you are white, this is a difficult, but needed, critique to listen to.