Garbage In, Garbage Out

As an organization whose mission is to conduct fact-based analyses with the highest level of intellectual honesty, accuracy, and objectivity, we find it troubling when a deeply flawed analysis is used to inform policy debates. A recent report, Cost of State Regulations on California Small Business Study, is the latest example of pseudo-research we’ve heard cited in policy circles, and it prompted us to write a memo warning that the report’s findings don’t hold up under scrutiny. While this report should not be used to inform policy decisions, it does serve one useful purpose: to illustrate important lessons about how to spot shoddy research.

Lesson 1: Be wary of research that can’t be independently verified. Sound studies meet the basic academic research standard of validity: A valid study is one that measures what it claims to measure. The regulations report fails to meet this standard because its analysis relies on poorly documented information on state regulatory climates that cannot be independently assessed. That means it’s impossible to determine whether the key measure upon which the report’s findings are based is legitimate, and that makes all of the report’s conclusions dubious.

Lesson 2: Don’t be duped by the use of complex econometric models – basic methodology and the integrity of the underlying data matter more. Just because a report appears on the surface to have the elements of economic research doesn’t mean that it actually follows sound methodology. The regulations report plugs numbers into complicated models and tosses around sophisticated-sounding econometric terms, but in reality, its analysis is replete with serious statistical errors. In fact, correcting for a single methodological error in the report reverses its key finding, rendering all of the report’s conclusions invalid.

Lesson 3: You don’t have to be an expert to evaluate economic research – if the results don’t make sense, something is wrong. Sometimes researchers put so much faith in their models, they fail to think about whether their results make sense. The regulations report is filled with illogical and implausible findings. For example, the report’s analysis suggests that state economies produce substantially more when the cost of doing business increases or when workforce quality deteriorates. Such findings should have been blazing red flags that something was wrong with the report’s analysis.

— Alissa Anderson

Bookmark and Share