I don't know that it matters if he is or isn't racist. The problem is that some players will think he is and that can cause locker room issues and make it difficult to sign some free agents and re-sign some players.
While the players didn't say anything about Gruden in public, it was made clear by Underhill that they talked to the players before doing it and they were okay with it, but only because it was as an advisor and not as the OC or an assistant coach. And, given the turnover that is needed on this team, it's not just guys on this team that matter and younger players may only know him by reputation and have no idea that he was ever considered a good coach.
But, even if we ignore all that, he was 22-31 in the last job he had.
I just don't think he's a good enough coach at this point to risk the serious downside.