Introduction
First, this article was already posted on my personal blog, but as it is highly relevant to what we do at Northscaler, I've moved the blog to here.
Waterfall, agile, iterative, whatever your process, your chance of success can be measured by the number solid feedback loops between the developers and the domain experts.
In this article I’ll try to argue that no matter your process, you have to find ways to insert frequent checkpoints. I’ll argue that agile development makes this easier, but I’ll also give you an insight into how we do this at Northscaler. At Northscaler we’ve developed a platform that allows to go through iterations in minutes by use of gamification and it’s our belief that such approaches will become commonplace in the future. This platform was originally developed at SciSpike, but as we merged with Northscaler, it is now one of the assets of Northscaler.
What’s the Problem?
The picture above illustrates our inherent problem. The systems we’re building are improving a domain. However, the builders are not necessarily experts on this domain.
Say for instance we’re building a remote patient monitoring application. The physicians understand the domain and know what they want to monitor. However, they don’t have the ability to convert their knowledge into running systems and must rely on programmers to do so.
The programmers don’t understand the domain and have to rely on the physicians to explain it to them. Their language and abstraction level are dramatically different. A typical sign of such semantic gap is a mismatch between what has been built and what was expected by the business. As late as 2012, McKinsey & Company and Oxford University did a survey that showed that on average IT projects are 45% over budget and deliver 56% less value than predicted. Also, 17% of IT projects fails in way that threatens the existence of the commissioning company.
We typically get conversations like the ones below:
Sounds familiar??? I’m sure it does.
Business Analysts...
The fallacy of the modeling approach for software is that it is very difficult for domain experts to translate the notation into requirements they understand. Most people can easily take an architectural plan and visualize what the building would look like when it’s done, however, I’ve yet to find business people that can do the same with a UML model.
When Are Errors Detected? Errors are rarely detected before the end users or domain experts see the software in action. It’s when they use the software or see it demoed for the first time that they understand how it works. For complex systems, hours of use may be required before they discover missing requirements or incorrectly implemented requirements.
No Waterfall Then?
Waterfall is expensive because it tends to increase the time between when requirements are discussed and when the resulting system is verified. There are still business domains where waterfall may be appropriate, however, waterfall almost always results in increased risk and project cost.
Government organizations seem to still prefer waterfall approaches and there are some practical reasons why they do. However, in almost all other business domains, waterfall approaches are to be avoided.
Is Agile the Answer?
Agile methodologies focus on increased communication between the domain experts and the developers. In particular, most agile methodologies promote frequent demos of the application. I typically see demos scheduled with regular intervals (once per week to once per month is common).
The demos improves communication and allows the domain experts to make course corrections early. In well functioning agile teams, we see that they rarely implement more than “one iteration worth” of code before they get verification of correctness through demos.
So, agile development is clearly an improvement over waterfall, but is it good enough? I’ll argue that even agile development can be improved. I see two major issues with agile:
An iteration-long-delay before verification is still too long (in particular in typical Scrum projects where the iteration length is 4 weeks).
In many areas, it is impractical to get the domain experts together at regular intervals. They may have to be flown in from various places or they may simply not have the time to come. Most demos I see, the domain experts are glaringly absent…
A Better Approach
At Northscaler, we’ve developed a special platform for requirements gathering. We’ve created a language that allows us to specify the application Intent. The Intent language allows us to hold requirement-gathering sessions where the specification is tested in real-time using executable software that we generate from the Intent language. This yields some very substantial benefits:
Shorten the iteration cycle (we’re aiming at minutes)
Provide formality to the requirements that minimizes the potential for misunderstanding
Generate documentation that the parties can review and verify after the requirements sessions
The platform is also used to generate partial implementation (including business logic and a simple UI) that further ensures that the software development progresses in the desired direction (more of that in another blog post, I’ll just focus on the requirements gathering in this post). We call this an Executable Requirements Specification.
The approach I’ll discuss below yields a very large set of meaningful iterations even in short requirements sessions. The iterations may just be a few minutes long. The course corrections discovered in our minute-long-iterations are usually discovered in week-long iterations in traditional agile projects.
The Intent Languages
We have two main languages that we use to describe Intent:
Conversation Language. This language is a language to describe collaborative behavior. It may at first encounter look similar to workflow languages, but there are some important semantic differences. We are using a computing model based on Carl Hewitt’s actor model (we’ve changed the semantic of actors quite a bit, so we call it an agent-model instead).
Domain Modeling Language. This language allows us to specify conceptual information models. This language is very similar to the kind of models you can create with a UML class model.
We decided to keep our languages textual, although we have mapped (and generate) graphical views of the textual languages. We decided not to build a graphical editor because:
The speed of editing (much faster when textual).
The advantage of using textual merge/diff tools. It is now easy to support concurrent updates of the models.
We want users of the tools to be able to create models without having to install elaborate tools/IDE’s. Ideally, we want developers of the Intent language to only need a simple text editor.
Gathering Requirements
To have an effective requirements session, we need the following roles present:
Intent language recorder. This is a person fluent in our Intent languages that record decision and makes modification to the Intent Language.
Moderator. Someone driving the requirements session,
Domain expert(s). Someone that fully understand the domain.
Product owner. Someone able to make decisions as to what the product we’re building is supposed to support.
We often find people that can play multiple of the roles above. In particular, SciSpike usually provide resources that can play the roles of both Moderator and Intent Language Recorder.
We run the requirements sessions in the following way:
Each iteration may literally only take a few minutes. In a few hours, we would typically have made more iteration that most projects do in years!
Typical Requirement Capture
Most of you will probably used to capture requirements in one of the following three forms:
Use-cases (perhaps supported by UML diagrams)
User stories
Functional specification
Although each of the forms has a long history of being used to capture requirements, I would suggest that all three forms are suboptimal.
Use-cases (often supported by UML models) are a semi-formal form that tries to strike a compromise between being precise and at the same time being readable by the various stakeholders. I have definitely seen some well defined use-cases, but most of the ones I see suffer from two fatal flaws:
They are ambiguous (I can almost always create a simple questionnaire (usually with multiple choice questions), ask the creators/reviewers of the use-case to read through them and finally administer the questionnaire. Usually, I get an almost random distribution of answers across the questionnaire.
They are cumbersome to construct. I’ve witnessed business analysts spending weeks to define even the simplest use-case.
User stories may work out great when the domain experts are readily available. They are somewhat of a reaction to the formality of the use-cases. However, in most cases, all they do is to defer the discussion to later. They act a simple hook for us to remember a feature, but the details have to be worked out later. In many cases, the domain experts are not available when the user stories are being expanded and we often waste multiple iterations where the developers interpret what they think the stakeholders want, demo it to learn the “real requirement” then finally reimplement it to fulfill the actual requirement.
Example
Let me show an example of how such a requirement session may work. I think it will be very difficult to explain this in text, so I have made a small video that may help illustrate how it all works.
The below videos are the same. I've just made it available both on YouTube and on Vimeo. I've had some bad luck with YouTube where they sometimes shorten the videos, so I decided to make it available on Vimeo also.
Alternatives
Unless you also built a tool like us, you may now ask, how can I do what you showed? Being selfish, I would say, contact SciSpike and something can be arranged…
However, if you don’t want to go down that route, there are other alternatives that you can apply to decrease the time required for a single iteration.
The main thing to focus on is:
How do you capture requirements and are those requirements meaningful and precise enough for all the stakeholders?
I’ve seen very meaningful sessions using tools like storyboarding, manual role-plays, screen mockup tools, etc. Try to gamify the requirements gathering. Make the specifications as executable as possible.
Conclusion
In this article I argued that the cost and risk of software projects are inverse proportional to the length of each meaningful iteration. By a meaningful iteration, I mean an interval where the stakeholders explained some features, the development organization captured the requirements of the features and the stakeholders are able to verify that the development organization has understood the feature.
In a typical software organization, the length of an iteration varies from 1 week to multiple months. We at SciSpike have developed a platform that allows us to shorten the iteration length to minutes. It is our experience that this dramatically reduces risk and shortens the duration of a software project.
References
Survey’s of project failures?
Communication through intermediaries
Actor Model
Hewitt: Universal Modular Actor Formalism for Artificial Intelligence (1973)
Agile and iterative development