Therein lies the problem with the Rooney Rule, and even DEI as a whole.
It completely undermines the ability, skill, or intelligence of someone of color (or of a certain gender). Even if you're perfectly competent at your job, people will think you only got the job because of your skin color, race, or what's in between your legs.
You'd only think that if you were already prone to think that those people were undeserving of the job. Believe it or not, there are those of us who don't assume that the default for qualification is 'white male'.