Reflecting on world views and mixed methods designs of research

Comparison of Creswell’s continuum of research and Edmondson’s methodological fit chart.

1. Creswell describes a single continuum that connects qualitative and quantitative research at the endpoints with mixed methods in the middle. This model implies that there is a single dimension along which methodology can be arrayed. I don’t think this holds up to scrutiny for very long.

2. He repeats the simple assertion that qualitative measures are in captured in words while quantitative measures are captured in numbers but glosses over the distinction that these two different research methods may be assessing entirely different dimensions altogether. For example quantitative measures are generally of physical objects that exist in the world or the qualities that exist independent of human sense making, whereas qualitative measures are describing aspects of meanings that people create an attribute to situations as well as physical objects. I submit that the objects of inquiry are quite different and don’t exist along a single continuum as his model would array.

3. Furthermore I don’t necessarily agree that mixed methods simply split the difference between qualitative and quantitative methods although it’s true that there are elements of both inside mixed methods. The reasons to choose one or the other have everything to do with a combination of the purpose of your inquiry and the nature of the research subject and the state of its theory as outlined in Edmondson. These are multidimensional considerations not easily arrayed along a linear continuum. For example qualitative and quantitative methods might be chosen because their subject could be contained sufficiently inside a single methodology where is a mixed methods approach might be chosen because of the complexity of the topic area which could not be properly described and analyzed inside a single discipline. In that case the very nature of the research subject cannot be expressed along a single continuum that is qualitatively different.

4. Moving on to the four point world view model I sense that the top two quadrants consisting of post-positivism and constructivism align very closely with quantitative and qualitative methods respectively. These two approaches seem to be value neutral and outcome neutral and relate to an ontological orientation about the nature of the world and the topics that are of interest.

5. The bottom two members of the quadrant: advocacy and pragmatism, are different in nature in that they focus on outcomes and problem solving first and only secondarily on methods. Their focus on methods matters only in so far as they facilitate the achievement of their outcome and not necessarily related to the nature of the research question itself. For example, both a pragmatist and an advocate might use a sub optimal research strategy simply because it might increase the odds of approval by the chief decision-maker because of their cognitive biases. In this case the search for truth and expansion of knowledge and contribution to theory is subordinate to the achievement of specific outcomes. The clear distinction between the advocate and a pragmatist is that a pragmatist takes his cue from the environment and participants and derives measures of success that are related to the situation whereas the advocate brings his values and predispositions to bear before diving deep into the situation.

6. It’s clear that the worldview quad chart is not simply a two-dimensional array but really represents four distinct perspectives that have different assumptions, different relationship to the truth as an end or a mean.

7. Compared to Edmondson’s methodological fit model, the differences could not be more dramatic. Edmondson’s approach is driven strictly by the nature of knowledge at a given moment compared to the method chosen. This is a strictly functional or utilitarian model that establishes metrics for common interpretation by a uniformed group of members of a profession who want to make functional judgments. This is a much more narrow application of a model. It explicitly is limited to feel research, where is Creswell treats the entire range of human experience in his model. Creswell goes much deeper and includes an assessment of the researcher and his motivations and purposes as well as the knowledge domain itself.

Reading notes for the Edmondson article: (I need these for later anyway 😀 )

1. a powerful article that is the single best argument for assigning this class early in the program. Would’ve saved me a year of spinning around in circles.

2. article purpose: provide a contingency framework to classify and recommend type of research based on proposed research question and status of previous work in the field.

3. Real-world complexity comes from trade-offs, constraints, timing and funding issues among others and can steer the research logic and the research connection to theory.

4. Authors intend to promote methodological fit as a criterion and have operationalized the end for elements consisting of research question, prior work, research design and contribution to literature.

5. The elements are mutually reinforcing and getting a good fit is seen as an iterative process consisting of feedback loops read thinking and revising. This reminds me that good writing is good rewriting.

6. this method is to be applied to feel research which is defined as systematic collection of original data in real organizations.

7. Methodological fit has deep roots back in the 60s when field study was recognized as being useful for explorations while mine field studies were used to advance..

8. Borchaud observed that good research consisted of asking the right question and then picking the most powerful method to answer.

9. Authors cautioned against the hammer and analogy in which proficiency with the hammer makes everything look like a nail.

10. Lee observed that qualitative research is good for theory generation, elaboration and testing.

11. The central debate in field research includes dialogue on whether or not mixed methods are really appropriate to combine and if qualitative and quantitative can be reasonably integrated. The issue concerns whether or not the same object can be compared using to radically different methods. There is consensus emerging that mixed methods can increase the living through triangulation or permit a deeper dive into understanding from an alternative point of view. Care must be exercised to not overstate the conclusions from two different methodologies.

12. Insight: revealing an unfinished manuscript resembles Karl Weick’s theorizing article which treats the rising as cooking and progressive theory making.

13. Reviewing makes it easier to see the lack of methodological fit in other people’s works, probably because we are outside of the fishbowl.

14. Insight: mature theory fills the room taking away space for things that have not yet been done. For new researcher it’s like going to a **tail party or you don’t know anybody or what they’re talking about and are trying to fit in.

15. Insight: have to read Baker’s article from 1993. The story of his inquiry resembles mine in many respects although he was looking at self-directed work teams and private companies were as I’m looking at military education and a period of transformation. Everything else is pretty close.

16. Authors assert that mixed methods offer a way to gain both deeper insight from qualitative and deeper he or from quantitative. The insight comes from answering the question what’s occurring behind the numbers. Depending on how you organize the research and what’s appropriate one of the methods can support the other in the following way:
a. qualitative can support quantitative by helping to make meaning out of numbers
b. quantitative can support qualitative by testing associations and insights that seem to emerge from a situation

17. the iterative nature of field research with mixed methods is like sailing back and forth between theory creation and theory testing.

18. For mature theory, triangulation of known measures is not necessary and will be seen as wasting time.

19. Premature quantitative research runs the risk of locking you into a theoretical construct too soon before the concrete is set. The final diagram is a nice job of demonstrating the iterative and focusing function of multiple research cycles.

A friend observed:
While your comments [on… ]Edmondson and McManus in general [are] accurate, I am nonplussed about the assertion that you make in connection with that of Creswell. I read and read the chapters on the research methodology by Creswell. And yet, I could draw the inference that mixed method is a simply split the difference between qualitative and quantitatively. In fact, he suggests that we should use both to bolster our claim.

I replied:
What I am saying is that to represent the choice of a research as a single straightline continuum, in a single dimension between Pure Quant and Pure Qual, with mixed method simply at a trade off point, rests on the assumption that there actually is a single dimension to consider when choosing a method.I believe this is too simplistic a notion for mixed methods.

There are times when a single approach may be appropriate.

But I think we know enough about complexity to say that there are problems and situations that are too large to be contained in either pure methodology by itself, and that mixed methods are a co-equal approach to the research design question.

For example Complex Adaptive Systems (CAS) of sufficient size cannot be “computed” (too many variables for pure quantitative research to fully appreciate), and at the same time cannot simply be understood or appreciated by one particular style of qualitative inquiry (too many points of view to select just one). The situation cannot be reduced beyond a certain degree of complexity, and may only be known and appreciated indirectly or approximately. The circumstances require a multiplicity of approached (ie mixed methods) and not simply as an “average” of 2 styles.

I suggest that Complex Adaptive Systems require “sense making” that is informed by multiple points of view, but which will not make us “certain” of the results. The insights from the mixed methods are not averaged, but allow us to define a space between the methods where truth may lay. A mixed methods “results” in this case would not be the same as the degree of objective certainty we get from quant, and the metaphorically “true” story we get from qual, but resides in a 3d place off the qual-quant single dimension continuum.

So I suggest mixed methods are not just what you do when you don’t know what to do, but are indicated as a preferred method for specific environmental reasons. I argue for a co-equal “theory of when to use mixed methods” that stands on its own. There surely can be some times when it is simply a blend, but other times when its more than that.

another friend observed:

I like your example of CAS, but I’m lost on your conclusion. I thought that Creswell (2009) indicates that you use the mixed method process not to get some 3rd answer but to narrow the banding on your question (pg. 18), and thus the answer to the question. In addition, Creswell seems to agrue that mix methods should be the perferred methodlogy for reasearch for this very reason, which I think is your thougth as well. So I guess I don’t understand your conjecture that mixed methods would not give us the same degree of accuracy using Quant alone, or truth as using qual alone. As an auditor, I’ll use either quant or qual analysis to determine a problem. But sometimes I have to use qual analysis to better define the data points for quant type research. Or, vise versa.

I replied:

i am coming at this from the perspective of chaos and complexity theory, which concerns situations which cannot be reduced by analysis/research to increase our certainty/knowledge.

Normally we think of research as a way to increase our knowledge so that we can predict & control; this comes from the positivist school and quant methods.

Constructivists and the qual methods may still be motivated by understanding and knowledge and increasing certainty and may want to achieve control & predictability (with less confidence in causality because of free will and the complexity of the human condition).

I sense there are some  challenging problems of a certain degree of complexity which the mixed methods may serve to increase our knowledge and move us towards certainty, where we can actually expect to know things with more certainty afterwards as a result of the inquiry.  Example:  trying to increase manufacturing production in a new multinational venture in a way that values and incorporates the culture, welfare and autonomy of the workers. As a result of “normal” mixed methods in a small, semi-controlled problem space  we might expect to come out really knowing something and have confidence in our ability to shaoe and control our situation. These are the situations I see Creswell as describing.

What I am talking about are problems on a much larger scale, where the complexity is essentially intractable, non-computable (for the quant methods) and where qual methods alone are unsatisfying in giving us deep enough insights to even hint at cause and effect or probable relationships.  i am thinking of things like nation building, and nuclear non-proliferation, and normalizing global financial  systems, where the situation is reinvented so frequently that its always a new start.

It strikes me that simply mixing a little qual and quant in an effort to traingulate an answer is materially different than saying these “wicked problems”  require a mixed methods approach that goes beyond simply taking an “average of the qual and quant”.

Complexity theory suggests that there are categories of situations we want to try to manage, improve, endure, survive that dont fit the simple formula of “pick a research method to learn something so we can act with some confidence to get more predictable results as a consequence of our study”

wicked problems are always morphing so that each cycle of research and insight is unique, and there is no historical theory or case study or “research that has gone before” that can guide.  These are the kinds of situations where I believe a theory of mixed methods is needed that goes beyond a simple blend of qual/quant, because the very premise of the purpose of the research and findings is so different

he said, further:

Next, I guess the logical leap of your discussions about a research topic with enough complexity, without any hope of really getting a real answer, means that no matter the outcome, if you can write with enough zeal and engagement you can give the appearance providing real data and support. At which point even the most complex issue, i.e. world hunger, can be answered by your research. Then if you get political support, the perception of your report is truth – right?

Not that any of us would do that, but just food for thought.

i replied:

i think what’s called for is a new understanding oif what real answers are for complex situations. I dont accept that all we can do is dance around a fire, and elect shamans and offload responsibility for results on to them, ritualistically, and feel like we have understood our world

I believe that a theory of mixed methods that accounts for the nature of complexity and wicked problems, that moves beyond the basic Creswell model is needed in order to explore what it is we can say and do and understand about complexity.

I appreciate the power of rhetoric, but I am wary of rhetoricians and cults of personality.

I wont trade my liberty for that false comfort. And so that’s why I am engaged in mixed methods research at our college where we propose to educate leaders in managing and appreciating complexity

Here is an essay I also posted in our college’s blog to challenge our leadership model which emphasizes leadership traits and our notion of the responisbility of the heroic leader, leader as savior.

http://kansasreflections.wordpress.com/2009/03/30/leadership-in-complex-adaptive-systems/

I believe a very different sensibility is needed for complex times. Which has implications for how we think, how we define success, how we work in groups, and yes, how we conduct research to determine what we can/might learn about CAS, which is quite different than the epistemological models we use for smaller scoped studies.

I know all about unscrupulous honey tongued serpents, and I believe the answer includes scholarship and integrity and peer review 😀

Related articles by Zemanta

• Qualitative Research4 bahman taherkhani (slideshare.net)
• Qualitative Analysis Presentation (slideshare.net)
• Using the Case Study Method in PhD Research (brighthub.com)
• Go Beyond Metrics For a Better User Experience (web2expo.com)
• IBM SPSS for Introductory Statistics (psypress.com)
• Introduction to Qualitative Market Research (corporate-marketing-branding.suite101.com)
• Research methodology insaniah (slideshare.net)
• Qualitative Research3 bahman taherkhani (slideshare.net)
• Qualitative Research Vs Quantitative Research (slideshare.net)
• Scott Meis: Adding Qualitative to Your Social Media Measurement Mix | Chris Bailey :: Thinking Big Thoughts on Business, Work, and Life (baileyworkplay.com)