No, that's not really the case. It doesn't matter if something is done on the basis of race or sex. If the end result hurts those groups disproportionately it's racist/sexist.
I guess this just may be a difference in definitions, but I think when most people are talking about racism/sexism on a macro level, they are using this definition.
You're right, the broader issue is worker's rights, but it's definitely worth pointing out that among workers with the least rights, minorities are overrepresented at Amazon.
Getting pedantic here. Folks can 'see' race and not be racist. The Census taker for instance. Or the affirmative-action recruiter.
To be meaningful, conventional racism has to include damage. Its an old debate technique for racists to complain that affirmative action is 'reverse racism'. Because, you see, it sees race and takes action.
I think it does matter if you want to address the root cause of the problem.
If the problem in this case is a lack of warehouse worker bargaining power vs profit motive, then solutions to a prejudice problem will not be helpful to the workers.
If not, this is just race/gender baiting and it detracts from the broader issue of worker treatment.