Yep. That's what I mean. The legitimate issue of the scale of immigration and its effect on US culture is easily turned into ugly and potentially violent thoughts about the racial composition of the US.
We had something similar when Angela Merkel invited overnight a million Syrians into Europe without political consultation. This was one factor in why the UK chose to reestablish its sovereignty. But the reasoning - we can't adopt vast numbers of immigrants over such a short timescale - became something that was treated as a racial question in the minds of some. Probably on both sides.
For myself, aside from the numbers, I think there was a concern about absorption of a whole load of people with a very different culture. This wasn't a skin colour thing. It was a residue of 9/11 thing about cultural trust. How do you know if there's an AQ needle in the Moslem haystack? [Moslem being not a race but a belief system]
Societies do need to be able to have those conversations without either kneejerk assumptions of racism, or, of course, actual racial ideas overwhelming the discussion. A very hard thing to do.