20 years worth of baptisms invalidated

It depends on what they believe

Is the baptism a symbolic ceremony or absolutely necessary for their immortal soul?

And if it's the latter and the church says it's the latter is there any legal recourse for the families of those who had an invalid baptism but passed away before it was discovered?

Good question - I can't think of any kind of applicable traditional tort or wrongful act, apart from general negligence. I suppose you could craft a persuasive claim of general negligence: the priest agreeing to fulfill this sacrament by performing the baptism has a duty to use reasonable care and he breached that duty with his mistake that the church has now said was critical, and invalidated the baptism. I think any real damages would be highly difficult to prove (you can't prove it prejudiced that person's soul). But I think a case for some kind of presumed/nominal damages might be possible.

Another question is who could bring that claim, I would think it would have to be the estate (or equivalent form of the deceased's personal representative). Not all claims are viably brought after death, but I don't know how this would fit into that.

The family members could argue some kind of negligent infliction of emotional distress - but those claims are difficult and typically only apply to a situation where the claimant was in the vicinity of some kind of serious incident ("zone of danger" rule). I don't think that would apply to a case like this.