It is hard to overstate how many supremely qualified applicants these universities get each year. Far more than they have space for. Most of the people I interviewed despite amazing applications would invariably not be admitted. These are extremely selective schools- Brown University for example has a 5% acceptance rate.
I don't know much about the legal landscape of affirmative action, but a lot of the university admissions process is a crapshoot among fantastic applicants. So I've certainly more appreciated the randomness of it all in terms of this debate. I think it is a disservice to ever insinuate affirmative action led to the admission of a student that was not qualified- and more to the point I think it is just a laughable notion if you consider the distribution of SAT scores and GPA of applicants to schools like Harvard.
I find legacy admissions a much more pernicious state of affairs.