Is Misguided Research Failed Research?

I don’t like Self-Determination Theory (SDT). The constructs of autonomy, competence, and relatedness, of which it is made, seem like ill-defined semi-arbitrary categories that are not really MECE (mutually exclusive and comprehensively exhaustive).

No doubt my criticism, in part, stems from the fact that SDT claims to explain intrinsic motivation, which is another theory I dislike. The distinction between intrinsic and extrinsic motivation seems conceptually confused, and empirically short-sighted. (See the criticisms of Steven Reiss)

And part of my criticism and confusion around THAT distinction might stem from my constant confusion over what the word “motivation” even means, as the word seems to imply agency, which goes against my science-is-deterministic training.

All of this is to say, this whole area of research seems empirically and theoretically misguided to me. I see conceptual confusion three layers deep, and believe the field needs an overhaul.

Why is this relevant

Gordon Pennycook polled Twitter, asking the following:

You’re at the end of your career and you managed to publish some influential and highly cited research. However, with new and more advanced methods, this work was ultimately discovered to be largely misguided empirically.

Do you consider this

A. A success

B. A failure

I’m not surprised at how many answered “a failure”, but I think many who did are confused about what science is and how it works.

So let me rephrase Pennycook’s question to make it more concrete.

Isaac Newton managed to publish some influential and highly cited research. However, with new and more advanced methods, Newtonian Mechanics was ultimately discovered to be largely misguided empirically.

Do you consider Newtonian Mechanics

A. A success

B. A failure

Newtonian Mechanics is a model which does not describe the laws of physics, and turned out to be wrong because of flaws in the methodologies used. But that doesn’t mean it isn’t (or wasn’t) useful. There is more to science than pure truth value.

I wonder if you sat down those who voted “a failure” (a large portion of whom are likely academics in psychology) and asked them about various theories whether they would be willing to claim that any of their favorite theories will be unchanged and mainstream in 100 years. Especially those in CogPsy, where the subject matter is complicated and obfuscated, and the problem of underdetermination is particularly relevant. I hope most would admit that the majority of theories will adapt or no longer be mainstream in 100 years. To do otherwise I think portrays a profound lack of humility about our current knowledge, and stems from a “the science is settled” mindset which is harmful to the field.

Personally, I see a successful and healthy science as one which is constantly discovering how the past generation of scientists were “misguided”. In this view, failure is not when a theory gets overturned, as was Newton’s, but rather, when it doesn’t. The failure is when we consider something settled and no longer question it. When we explain away the inconsistencies, and never find a different way of talking about the phenomenon that better explains the data.

Scientists after learning that the science is settled

But if you approach every discovery and every theory with a “the science is now settled” mindset, then you contribute to that failure. And it is that mindset that I think led many to answer “a failure” on the poll. Because if you think every scientific theory is attempting to be the final word, then of course you’ll see it as a failure if a theory turns out to be “misguided”.

The science isn’t settled. The science is never settled. There is no theory today that you shouldn’t approach with the mindset that “this is probably misguided”. Or what kind of scientist are you? A theory getting replaced by something better shouldn’t be surprising, that’s the expectation. No theory is the final word, and you shouldn’t act like it is.

This is (partly) why I prefer a pluralistic approach to science. The concept of isomorphisms, along with the problem of underdetermination, free us from settling with one theory. There is always more than one way to understand and conceptualize anything in this world. In part because there are always an infinite number of isomorphisms for the same information, and also because there can never be enough data to ensure that any particular theory is the final word. And so regardless of whether we even like a theory, we should still seek for new ways of conceptualizing what is happening. We should constantly look for alternative explanations, as different as the Copenhagen Interpretation is to the Many Worlds interpretation. Not in antagonism, but as a joint venture, because there’s always the possibility of yet another way to represent and explain the same data in a more useful, beautiful, and parsimonious way.

That’s where I am at with SDT, Dual Process Theory, motivation, the Heuristics and Biases approach, and many other things. I think each is conceptually confused, and would be extremely surprised if in 100 years psychologists hadn’t changed their understanding of them, or abandoned them completely. In fact, I will consider that a failure if they understand them in the same way current psychologists understand them. But I don’t necessarily “disbelieve” these theories and approaches, and I still talk about them and use them in conversation. I don’t have a better framework yet (with the possible exception of SDT), and am looking for a better way of framing the data. But until then, I am fine using SDT as a rough theory for what is going on.

Am I side stepping the issue at the center of Pennycook’s poll? My examples are more conceptual, and about theories and models rather than specific empirical results which may be what most voters had in mind. And isn’t the largely still-predictive-and-useful Newtonian Mechanics example quite different than the failures in psychology?

Ignoring the fact that Newtonian mechanics fails to replicate in the same way Loss Aversion and many other phenomena fail to replicate (e.g., they are models which are only predictive under the right circumstances), I agree I am side stepping a little. There’s a difference between representing and explaining something, and discovering a phenomenon which was never there to begin with.

There may be true instances of failure. Where a line of research is truly useless and pointless, and a distraction of time and resources. Maybe an effect that turns out to not exist, and we learn nothing from having even considered the effect. That’s possible. But I do think it’s the exception, and more often caused by outright fraud and misinterpretation than anything else. (And would be extremely rare in a scientific regime where attempts at falsification are the norm, btw. Something which would be the norm if less people took a “the science is settled” approach.)

Basically, as with all things, the answer to the question posed in the title is, “it depends”. But as for SDT, a theory I actively dislike? I still won’t call it a failure. That’s such a narrow view of science. The only failure is to consider it settled science. The only failure is to never be able to say how it was misguided because we just accepted it as the final word. And I think that is as true of theories I dislike as well as the theories I do like.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s