Racial residential segregation is still very high in many American cities. Some portion of segregation is attributable to socioeconomic differences across racial lines; some portion is caused by purely racial factors, such as preferences about the racial composition of one’s neighborhood or discrimination in the housing market. Social scientists have had great difficulty disaggregating segregation into a portion that can be explained by interracial differences in socioeconomic characteristics (what we call structural factors) versus a portion attributable to racial and ethnic factors. What would such a measure look like? In this paper, we draw on a new source of data to develop an innovative structural segregation measure that shows the amount of segregation that would remain if we could assign households to housing units based only on non-racial socioeconomic characteristics. This inquiry provides vital building blocks for the broader enterprise of understanding and remedying housing segregation.