James, Bret and I published our definition of context-driven testing at http://www.context-driven-testing.com/.
Some people have found the definition too complex and have tried to simplify it, attempting to equate the approach with Agile development or Agile testing, or with the exploratory style of software testing. Here’s another crack at a definition:
Context-driven testers choose their testing objectives, techniques, and deliverables (including test documentation) by looking first to the details of the specific situation, including the desires of the stakeholders who commissioned the testing. The essence of context-driven testing is project-appropriate application of skill and judgment. The Context-Driven School of testing places this approach to testing within a humanistic social and ethical framework.
Ultimately, context-driven testing is about doing the best we can with what we get. Rather than trying to apply “best practices,” we accept that very different practices (even different definitions of common testing terms) will work best under different circumstances.
Contrasting context-driven with context-aware testing.
Many testers think of their approach as context-driven because they take contextual factors into account as they do their work. Here are a few examples that might illustrate the differences between context-driven and context-aware:
- Context-driven testers reject the notion of best practices, because they present certain practices as appropriate independent of context. Of course it is widely accepted that any “best practice” might be inapplicable under some circumstances. However, when someone looks to best practices first and to project-specific factors second, that may be context-aware, but not context-driven.
- Similarly, some people create standards, like IEEE Standard 829 for test documentation, because they think that it is useful to have a standard to lay out what is generally the right thing to do. This is not unusual, nor disreputable, but it is not context-driven. Standard 829 starts with a vision of good documentation and encourages the tester to modify what is created based on the needs of the stakeholders. Context-driven testing starts with the requirements of the stakeholders and the practical constraints and opportunities of the project. To the context-driven tester, the standard provides implementation-level suggestions rather than prescriptions.
Contrasting context-driven with context-oblivious, context-specific, and context-imperial testing.
To say “context-driven” is to distinguish our approach to testing from context-oblivious, context-specific, or context-imperial approaches:
- Context-oblivious testing is done without a thought for the match between testing practices and testing problems. This is common among testers who are just learning the craft, or are merely copying what they’ve seen other testers do.
- Context-specific testing applies an approach that is optimized for a specific setting or problem, without room for adjustment in the event that the context changes. This is common in organizations with longstanding projects and teams, wherein the testers may not have worked in more than one organization. For example, one test group might develop expertise with military software, another group with games. In the specific situation, a context-specific tester and a context-driven tester might test their software in exactly the same way. However, the context-specific tester knows only how to work within her or his one development context (MilSpec) (or games), and s/he is not aware of the degree to which skilled testing will be different across contexts.
- Context-imperial testing insists on changing the project or the business in order to fit the testers’ own standardized concept of “best” or “professional” practice, instead of designing or adapting practices to fit the project. The context-imperial approach is common among consultants who know testing primarily from reading books, or whose practical experience was context-specific, or who are trying to appeal to a market that believes its approach to development is the one true way.
Contrasting context-driven with agile testing.
Agile development models advocate for a customer-responsive, waste-minimizing, humanistic approach to software development and so does context-driven testing. However, context-driven testing is not inherently part of the Agile development movement.
- For example, Agile development generally advocates for extensive use of unit tests. Context-driven testers will modify how they test if they know that unit testing was done well. Many (probably most) context-driven testers will recommend unit testing as a way to make later system testing much more efficient. However, if the development team doesn’t create reusable test suites, the context-driven tester will suggest testing approaches that don’t expect or rely on successful unit tests.
- Similarly, Agile developers often recommend an evolutionary or spiral life cycle model with minimal documentation that is developed as needed. Many (perhaps most) context-driven testers would be particularly comfortable working within this life cycle, but it is no less context-driven to create extensively-documented tests within a waterfall project that creates big documentation up front.
Ultimately, context-driven testing is about doing the best we can with what we get. There might not be such a thing as Agile Testing (in the sense used by the agile development community) in the absence of effective unit testing, but there can certainly be context-driven testing.
Contrasting context-driven with standards-driven testing.
Some testers advocate favored life-cycle models, favored organizational models, or favored artifacts. Consider for example, the V-model, the mutually suspicious separation between programming and testing groups, and the demand that all code delivered to testers come with detailed specifications.
Context-driven testing has no room for this advocacy. Testers get what they get, and skilled context-driven testers must know how to cope with what comes their way. Of course, we can and should explain tradeoffs to people, make it clear what makes us more efficient and more effective, but ultimately, we see testing as a service to stakeholders who make the broader project management decisions.
- Yes, of course, some demands are unreasonable and we should refuse them, such as demands that the tester falsify records, make false claims about the product or the testing, or work unreasonable hours. But this doesn’t mean that every stakeholder request is unreasonable, even some that we don’t like.
- And yes, of course, some demands are absurd because they call for the impossible, such as assessing conformance of a product with contractually-specified characteristics without access to the contract or its specifications. But this doesn’t mean that every stakeholder request that we don’t like is absurd, or impossible.
- And yes, of course, if our task is to assess conformance of the product with its specification, we need a specification. But that doesn’t mean we always need specifications or that it is always appropriate (or even usually appropriate) for us to insist on receiving them.
There are always constraints. Some of them are practical, others ethical. But within those constraints, we start from the project’s needs, not from our process preferences.
Context-driven techniques?
Context-driven testing is an approach, not a technique. Our task is to do the best testing we can under the circumstances–the more techniques we know, the more options we have available when considering how to cope with a new situation.
The set of techniques–or better put, the body of knowledge–that we need is not just a testing set. In this, we follow in Gerry Weinberg’s footsteps: Start to finish, we see a software development project as a creative, complex human activity. To know how to serve the project well, we have to understand the project, its stakeholders, and their interests. Many of our core skills come from psychology, economics, ethnography, and the other socials sciences.
Closing notes
Reasonable people can advocate for standards-driven testing. Or for the idea that testing activities should be routinized to the extent that they can be delegated to less expensive and less skilled people who apply the routine directions. Or for the idea that the biggest return on investment today lies in improving those testing practices intimately tied to writing the code. These are all widely espoused views. However, even if their proponents emphasize the need to tailor these views to the specific situation, these views reflect fundamentally different starting points from context-driven testing.
Cem Kaner, J.D., Ph.D.
James Bach
Cem,
This post helped me much more to know what I was thinking, practicing and questioning each time when I said myself “I am a Tester!”.
I want to know and understand, whether I am heading in right direction. Please correct me and guide me to walk in the right path, for giving the best, credible, motivating service and information from my-side to the people who trusts me, when they give me to test.
In the below illustration,
Mission: To test a pen and its worthiness of having it.
The context driven approach what I brought here, for testing a pen are like this:
1) Who is/are the user(s) of this pen?
2) What claims does the pen manufacturer makes to the market?
3) What features the pen, have in to fulfill the need of the purpose of using it. Does all those features are met in various scenarios and
condition, while the user have that pen with her/him? If doesn’t meet so, what are those contexts, scenarios and tests that uncovers
them?
4) How does the pen, satisfies its users besides the other competitor pens? If not so, what are those which makes non-worthiness of having
this pen?
5) Does the pen breaks any regulations? If so, when does it breaks? If not so, what are those things, which is making the pen to be
consistent with the regulations? And, what are those regulations that the pen should abide by?
6) How does this pen suits to the user, who have used the other manufacturer’s pen? If doesn’t suit, what are the reasons that makes
the pen doesn’t suit the user?
Like this I go on identify the contexts and scenarios to test a pen, where I test for each of those contexts and scenarios with complex set of tests, which gives the test result — which is easy to evaluate with accountable information.
Am I, in the right direction of thinking, questioning and testing w.r.t the Context Driven Approach?
Ravisuriya
Well said, Cem.
Anything that uses the suffix “-driven” must have a starting point. As you have so clearly pointed out, context-driven testing has a well-defined starting point, that often distinguishes it from other testing approaches.
Thanks for the excellent clarification.
-joe
I think this is a useful expansion of the original definition. Thanks for publishing this.
This is a great post, it help me understanding context-driven better.
Thanks
Thank you for the great post, Dr. Kaner. One of things I do when talking about Agile testing is to use a phrase like this:
“I found the Agile movement to be a breathe of Fresh Air because I found that the approach they were advocating mapped well to business software development in North America in my experience.”
A few key terms I use:
(1) Business Software Development – That is the environment that Agile Methods (Scrum, DSDM, XP) came out of
(2) In North America – Development cultures in other cultures are beyond my experience. I understand Germany is similar to US-Cubeville, but South Africa, for example, is not
(3) In my Experience – There are lots of companies that I have not worked out. They could do things differently.
I /hope/ this terminology is helpful so that the listener doesn’t go off and say “Matt Heusser said I should do agile-testing for (defense work) (avionics) (embedded) (heavily-regulated work), etc”)
I define Context-driven as a breathe of fresh air for different reasons, first of all, it’s intellectually honest and stimulating. It’s intellectually challenging and can make the work satisfying in the Malcolm Gladwell, not-brave-new-world sense. It’s humble and doesn’t claim to have all the answers – and yet it /can/ scale to different domains whereas agile-testing is more limited. Most importantly, context-driven testing asks “how can we (testers) do our job better?” instead of asking “how caIn I get everyone else to change to make my job easier?”
On a judgemental day, I would admin that admonitions to get involved in the process early, have the devs do unit tests, to “get management buy-in”, get better requirements, and so on, are really requests to have someone /else/ change to make testing easier. That’s not test process improvement, that’s test process simplification.
Again, Good post, thank you.
–matt heusser
I agree that this is a very usefuil clarifying post.
However, I can see how the conflating of the context-driven approach and exploratory-testing technique comes about. James, in particular, along with Bolton and others are strong advocates of both and often in the same fora. I think a person new to the conversation might surmise that one is a part and parcel of the other.
The dog ate my original reply (Did I spell the password incorrectly? Why oh why does the text box not get preserved when one backspaces after the error message?!), which appeared shortly after the original post.
To Anthony:
However, I can see how the conflating of the context-driven approach and exploratory-testing technique comes about. James, in particular, along with Bolton and others are strong advocates of both and often in the same fora. I think a person new to the conversation might surmise that one is a part and parcel of the other.
For me, it’s not that the exploratory approach is fundamental to context-driven testing; the exploratory approach is fundamental to skilled testing of any kind, even when the influence from the scripted end of the continuum dominates, or even when “best practices” drive the project focus. The exploratory mindset forms the basis of the individual tester’s ability to continue to think new thoughts and ask new questions. When you’re not looking for new information, you need not take an exploratory approach, but if you’re not looking for new information, I’d argue that you might as well not test at all. I’m reasonably sure that Jerry Weinberg, James, and Cem would agree. In any case, there’s more on my point of view here.
Note also (should I update that post?) that exploratory approach are not incompatible with planning or documentation; it’s simply that the more the planning and documentation controls the thinking and actions of the tester, the more we’re in a scripted mode and the less we’re in an exploratory one. But, as you say, this is subtly distinct from the context-driven-or-not dimension, and so maybe there’s some work to do on that. Thanks for pointing that out.
—Michael B.
Hi Cem,
It’s a wonderful post and I wish it were there when I was compiling the ideas on schools of testing about an year back.
I have got a question, which I don’t feel, is still answered. Do you define a school based on the Way One Thinks or the Way One Tests (as a part of current project)? Now, there’s a difference between the two which is quite understandable.
To elaborate on the above point, should we say that a project is being executed in Context-Driven way or that the project has team members who subscribe to the Context-Driven school? Let’s consider some mix-up situations for this. A project being executed in context-driven way by testers who actually advocate best practices but are bound by the wishes of the manager who subscribes to context-driven school. Another situation can be the opposite one.
Cem replies — I haven’t really thought much about this. My experience has been that ideologically prescriptive managers are USUALLY not context-driven. On the other hand, when I was managing testers, I pushed them pretty hard to think for themselves and to design their own tests. If they could not/would not do that, or if they had contempt for my style, they left. But that’s about group management, not about context-driven versus not context-driven.
How does it help a tester to practice day-2-day testing in “School A” way and come back to the web/forums/conferences and talk in context-driven way? Is it practically possible? It’s like dreaming…it could have been better if I had executed this test in this way……anyways…let me complete this as per the prescribed guideline…
Cem replies — I don’t think I understand what you are asking. I think it is desirable for a person to speak congruently with how she acts and thinks.
What are your thoughts and suggestions on the above?
I like the ideology behind Context-Driven testing and related literature. But subscribing to the “school”, supporting only those who belong to the school…I’m really not sure how helpful it would be for the testing community.
Cem replies — Who said you have to support only those who belong to the school? Or who support the principles of the school? I can espouse a point of view without insisting that everyone else adopt it.
I have had discussion with Pradeep and Shrini. On the web, there were many arguments with them. I met them and did not find many differences in the way we think or approach testing. Still, they “belong” to the school, I don’t (atleast that’s the current state of my mind).
On the other side, I met a couple of others, who do not like the idea of schools of testing, and they speak totally against the Context-Driven school.
Cem replies — some people dislike the “context-driven school” because they think the idea of schools is (a) divisive or (b) a gimmick (a trick for marketing to make money). I think those people are mistaken. I think (a) the field is severely divided, and that the question is not whether there are fundamentally different views in the field but how we can characterize them. The schools we’ve described may or may not be the best characterizations, but they’re what we’ve got now. As to (b) the idea that this is a marketing gimmick, I think that a few people will use anything to peddle their services and some of those people have discovered agile development, context-driven testing, exploratory testing, etc. But I think there are easier things to sell than context-driven testing, and we’re not selling a certification or certification courses so one of the big money-makers that a gimmick would support is just not available.
Cem replies — On the other hand, some people dislike our ideas, and of those, some people are so attached to their view that they think what we teach is ethically wrong as well as pragmatically wrong. I think that opens interesting opportunities for discussion and debate.
Are we missing out on something? I have always considered you as one of my gurus in the field of software testing. How can I be left out from “your testing community”, just because I do not subscribe to the context-driven school?
Cem replies — I don’t know what you mean by being left out of my testing community. There are a few individuals in the field who I hope to never interact with again. But that’s personal. There are some viewpoints that I’m tired of debating and some ways of thinking about other people on projects that I think are ethically wrong or counterproductive. On the other hand, I make my course materials available (for free) to everyone and I’ve spoken at a pretty diverse set of conferences, including conferences organized by people whose views were very different from mine.
Regards,
Rahul Verma
Hi Cem,
I know this response is late. Apologies. I checked for a response from your end quite regularly for some days. I lost any hopes for the same after a couple of weeks. I thought I have offended you in some way.
Your reply has cleared many a doubts in my mind. I will soon revisit my consolidation of thoughts on schools of testing, documenting my current viewpoint. To bring into your information, I have had a good amount of discussion with Pradeep and Shrini on the subject.
There are many lessons that I have learnt after reading literature on testing, the context-driven way. As mentioned by you, the free BBST course prepared by you and James is an excellent resource for learning testing. Added to this, the RST workshops taken by Pradeep in India have also made me think about testing from a different perspective.
Regards,
Rahul Verma
Cem,
Thanks for the excellent post!
I came late, but found your definition (and ‘Testing as a Social Science’) a true breath of fresh air!
Lately, my studies have focused on Behavioral Economy and Neuroeconomics, which have generated a tremendous amount of NEW knowledge regarding how we, as humans, make decisions which involve risk/unknown factors. Unlike our 80 yrs of neoclassical economic assumptions, this knowledge points to a boundedly rational behavior, influenced by emotions and cognitive biases.
I’m working on bringing this knowledge into our world (IT, quality, ALM etc).
Let me know if you’d like to exchange more info on how to incorporate behavioral economy and neuroeconomics into testing and quality assurance os systems.
Best regards,
Pedro Correa
Cem:
Very well stated. You have articulated a lot of things I have thought while approaching different test projects in my work, because we are in a process of transitioning into a more Agile approach (although of course they don’t call it that).
I especially like the following part. It’s brilliant! 🙂