Racist? Me? aw cmon....

So yesterday, I was called a racist by this liberal, let-everyone into the country no matter their criminal background, type.
Now, for those of you who know me, that's damn near as ridiculous as calling me a liberal. But I thought "How in the hell do these people get this sort of idea?".

I mean, who does illegal immigration hurt? White people? Not usually. I mean, it does, but not to the degree that it hurts legal immigrants. Heck, it's the liberal whites who are fighting to keep their low-cost nannies (illegal), gardeners (illegal), and other folks. It's not the mexicans or black crowd.
This is not a racist statement, but a statement of fact...white collar folks tend to be, well....white. Illegal immigrants don't tend to diplace them, they displace blue collar folks. While the majority of folks in the blue collar are still white (we're the majority in the country, get over it, I'm not being racist and saying minorities don't work), lots of blue collar folks are minority. Probably a larger percentage of their population than white are in blue collar rather than white collar. I don't have specific numbers that I can quote, but I'm sure the commerce department could provide such.
So, is it racist for me to want people to come to our country legally? Hell no. In fact, it seems to me that it's the opposite. It's racist for those who want the cheap labor here to encourage the law breaking to continue. It's racist to say "I don't want to have to pay a fair wage for my child care, so bring in some illegals".
The bastion of conservativism and racism, Cesar Chavez (/sarcasm off) advocated severe limitations on illegal immigration, because it lowered the wages of his union members. He even offered his union staffers to be the first incarnation of the Minutemen, because when a strike would be organized, the farmers would send trucks to the Mexican border, load them up with illegals, and bring them back to the farms, to try to break the unions.

Now, I'm sure there are other reasons for advocating illegal immigration (human rights, blah blah etc), but calling someone a racist for advocating much more restrictive immigration regulations just doesn't seem quite logical to me.