I think the question is a little unclear. I assume we mean "if someone is claiming to be certified, but is not" (but it could be interpreted as "if a project applies for certification and we put effort into this but then they fail"!)
The challenge here is that it is unlikely there would be much legal basis for imposing a financial or formal penalty (unless we set out to create one). Without a legal basis, only 'well behaving' projects would pay any penalty (and of course such projects would likely not be failing to meet the criteria, or would fix any issues if they were told about them). "malicious" projects simply would not pay (and even if they stopped using the certification, would probably already have benefitted). To achieve the goals of the certification project, greater understanding and engagement of open hardware, such malicious projects need to be firmly discouraged.
A legal basis for imposing a penalty could be something like having the certification logo and name trademarked, at which point OSHWA could use trademark law as the basis of a legal complaint against a project using the logo/name inappropriately.
Such a legal basis would also be necessary were OSHWA to wish to be able to demand that a product cease using the logo.
A "list of shame" could be effective if OSHWA was sufficiently well known that consumers would avoid companies that OSHWA disrecommended, and of course that consumers, or review sites or media drew attention to a company or project being on the list of shame. Such a list would need some curation as projects were added and removed etc. Of course, some projects would find being on such a list, and discussed in the usual open/FOSS/etc forums would be a bad thing and they would avoid this; but larger projects seeking to profit off the 'open hardware' concept in more mainstream markets would likely not (and the sales they would lose would probably be very small).