Risks of Quantitative Studies (Jakob Nielsen's Alertbox)

Risks of Quantitative Studies (Jakob Nielsen's Alertbox) - So Jakob Nielsen gets on his soap-box and tells everyone to go do ethnography. Thank you very much, this is wonderful... so you are saying that studies that report correlations may be faulty? no really? and significance testing might actually get you to wrong conclusions? really? wow... I guess Jakob Nielsen assumes (unfortunately rightly) that most readers haven't had a basic class in statistics. Unfortunately, many qualitative researchers have skipped that same class as well and harbor many of the misconceptions that Nielsen elaborated in his soap-box statement. Quantitative studies are not just numbers!

Quantiative data collection is steeped in theory, based on theoretical predictions, where outcomes must "make sense" in order to be believed. Just like qualitative observations (which may be woefully wrong and completely misleading if observers use bad qualitative techniques and lack theoretical background, which is often the case in usability studies unfortunately) must be interpreted with care and an eye towards unobserved phenomena that may have driven observed phenomena, quantitative analyses must be approached with caution and deep considerations of causal direction and possible mediators and moderators.

Nielsen insists that its easier, cheaper and better to do qualitative studies but I am not convinced. Nowhere does he point out that badly done qualitative studies can be just as false, biased, misleading and overly narrow as quantitative studies and for similar reasons - lack of knowledge in theory, background, technique application. The argument between quantitative and qualitative researchers has been an ongoing disagreement. Taking one side over the other is a myopic acceptance of one side of an old argument. I believe that a good, proficient amalgam of the two approaches is the most efficient method of research. Unfortunately, we are far from this stance being accepted in research and academia. Statements like the latest one from Jakob Nielsen don't help, they spread more disagreement and misconception than provide understanding and caution. Its unfortunate and it keeps reducing my respect for Nielsen with such sophomoric statements.