Like it or not, right or wrong, if their is a perceived association to the point of it being brought up in a debate then its a question the voters want an answer to.
Farrakhan is obviously not an accepted figure to the vast majority of americans views or else he would run for president himself or at the least have a positive influence. If the association is there and it were me, I would go to lengths to make sure people knew I did not have the same views.
What I notice so far of Obamma is that he appears to be fighting a battle of his internal beliefs and refuses to make a decisive stance. JMHO.