Documentation is an important part of every system, agile or otherwise
By Scott Ambler,
Feb 05, 2007
Scott is a DDJ Senior Contributing Editor and author of numerous IT books. He can be contacted at www.ambysoft.com/ scottAmbler.html.
Documentation is an important part of every system, regardless of the paradigm followed to develop that system. Although agilists are sometimes maligned for what is perceived to be their cavalier attitude towards documentation, the fact is that we take documentation very seriously. Agilists adhere to the philosophy that your team's primary goal is to develop software, a message that sometime drowns out its sister philosophy that a project team's secondary goal is to enable the next effort. Because the "next effort" typically involves running, supporting, operating, and maintaining the system, chances are incredibly good that you're going to need to write some documentation along the way. It's possible to do so in an agile manner, and this month, I explore strategies for doing exactly this.
For some reason, the agile community seems to struggle to describe how we approach documentation. There are several explanations for this. First, the majority of agile developers prefer to focus on technical practices (such as regression testing, refactoring, and continuous integration) instead of "softer" practices (such as modeling, documentation, and governance). Hence, we talk about those things a lot more. Second, because the agile community is growing quickly, we have large numbers of novices who are in the early stages of learning—in their exuberance, they are all too eager to share their misperceptions with others, and a common misperception is that agilists don't document. Third, modeling and documentation practices weren't explicit in most first-generation agile methodologies, unlike second generation agile methods such as Open Unified Process (OpenUP) and Microsoft Solutions Framework (MSF) for Agile.
A Few Myths and Misconceptions
To understand the agile approach to documentation, you must first understand the potential misunderstandings about documentation that have seeped into traditional thinking:
- • Documentation isn't the primary issue—communication is. For example, your primary goal isn't to document requirements, it's to understand them so that you can implement them fully. Your primary goal isn't to document the architecture, it's to formulate a viable one that meets the long-term needs of your stakeholders. This isn't to say that you won't decide to invest in such documentation, but that isn't the raison d'tre for your project.
- • Comprehensive documentation does not ensure project success. Just because you have a detailed requirements specification that has been reviewed and signed off, that doesn't mean that the development team will read it, or if they do, that they will understand it, or if they do, that they will choose to work to the specification. Even when the "right thing" happens, comprehensive documentation still puts the project at risk because of the perceived need to keep the documentation up-to-date and consistent. Your real goal should be to create just enough documentation for the situation at hand; more on this later.
- • Documentation doesn't need to be perfect. People are flexible, they don't need perfect documentation. Furthermore, no matter how hard you work, it's very unlikely that you could create perfect documentation—even if you could, it isn't guaranteed that the readers will understand it anyway.
- • Few people actually want comprehensive documentation. When was the last time you met a maintenance programmer who trusted the system documentation that they were provided? Or an end-user who wanted to sit down and pore over a massive user manual? What people typically want is well-written, concise documentation describing a high-quality system, so why not invest your time to deliver that instead?
- • Documentation doesn't need to be standardized. Many organizations have adopted the unfortunate philosophy that repeatable processes lead to success, yet in reality what you really want is repeatable results (that is, the delivery of a high-quality system). The quest for repeatability often leads to templates, and because each system has its own unique set of issues, these templates have a tendency to grow over time into all inclusive monstrosities that do little more than justify bureaucracy. Try to achieve a reasonable level of consistency, but don't go overboard doing so.
Rethinking Documentation
Documentation is as much a part of the system as the working software itself. Your goal should be to develop documentation that is sufficient for the needs of its audience, and to do that you must work closely with your stakeholders to understand their actual requirements. I believe that this is where agilists truly differ from their traditionalist counterparts: Agilists treat documentation as a stakeholder requirement, not as something dictated by the process. Because documentation is a requirement, it is something that should be estimated by the development team and prioritized by the stakeholders. Because the resources that you have to invest in an IT project are finite, and because it's your stakeholder's money being spent, they should decide which documentation will be created and how comprehensive it needs to be.
Because your stakeholders are a varied bunch, they consist of end users, business management, IT management, operations staff, support staff, enterprise architects, and many more, chances are pretty good that most someone will identify the need for a document. But let's assume that they forgot to ask for system overview documentation. Does that mean that you don't create it? No, of course not. What you do is suggest to your stakeholders the need for such documentation, justify why it's important, estimate the total cost of ownership (TCO) of creating and maintaining it, and then let them decide whether the wish to invest in the documentation.
This is a very different approach than what traditional teams typically take. Traditional software processes define which documents need to be created, will often provide detailed guidelines and templates to help ensure completeness, will indicate when the documentation should be created, and who should receive it. The underlying assumption is that the stakeholders, the people who were smart enough to earn the money, aren't smart enough to determine how to spend the money and therefore those decisions are taken away from them in traditional processes. Not only is this incredibly arrogant, it is also ripe for abuse—hence the overly bureaucratic and documentation-heavy processes that we see in many organizations.
Many organizations justify their burdensome documentation practices on industry regulations such as the Food and Drug Administration (FDA) regulations, the Sarbanes-Oxley (SOX) act, or the Basel-II regulations. I've worked in organizations that have had these regulatory requirements inflicted upon them and I can safely assure you that it is possible to remain agile. The first, and more important, step is to actually read the regulations. Nowhere in them does it say that you need to be ineffective and wasteful, yet if you allow the paper-pushers within your organization to interpret the regulations (and doesn't it seem that they're always lining up to do so?) then inevitably you'll find yourself in a bureaucratic morass. The second step is to be actively involved with interpreting the regulations to ensure that your resulting process is as streamlined as possible yet still in compliance. Third, be prepared to evolve your process to reflect new versions of the regulations as they evolve over time. For example it is likely that SOX will be simplified soon, and I think that it will be interesting to see how many organizations choose to tear down the political empires that were justified based solely on SOX.
When to Document
Documentation should be created on a just-in-time (JIT) manner when you need it, and only if you need it. You should document something only when it has stabilized, otherwise you will be updating it endlessly as the situation changes. This is one of several reasons why it proves foolish to write comprehensive requirements documentation early in the system lifecycle—because your stakeholders will change their minds, usually for completely valid reasons, you discover that the investment in detailed documentation was counter-productive. The implication is that detailed documentation should be written after, or at least towards the end of, the development work. Yes, you likely need to do some very high-level modeling to think things through, but investing in detailed documentation too early is questionable at best. I present a critical examination of the big-requirements up front (BRUF) approach at www.agilemodeling.com/essays/examiningBRUF.htm.
Agile Modeling (AM) advises that you should update documentation only "when it hurts". People are flexible, if the documentation isn't perfect that's okay, they can figure it out. Millions of software systems are successfully running around the world as you read this paragraph, and how many do you honestly think have perfect documentation that is up to date and fully consistent? Perhaps a handful? How many billions of dollars, if not trillions, do you think has been wasted over the decades striving to produce perfect documentation? Your goal should be to produce documentation that is good enough for the situation at hand—once it's good enough any more investment in it is clearly a waste.
Documentation Alternatives
Agilists have rediscovered the concept of literate programming, of writing high-quality, easy-to-understand source code that contains embedded documentation. Furthermore, teams that take a behavior-driven design (BDD) approach consider their test suites to be executable, detailed documentation. Agile acceptance test suites forms a significant portion of the requirements documentation and developer test suites the design documentation. We follow AM's Single Source Information practice by having our tests do double-duty as specifications and therefore we require significantly less external documentation.
The "documentation is in the source code" philosophy covers the vast majority of technical documentation, but system overviews will still be required as will documentation for non-development staff. Andreas Rueping's book Agile Documentation: A Pattern Guide to Producing Lightweight Documents for Software Projects (John Wiley and Sons, 2003) defines a wonderful pattern language for improving the efforts of technical writers. Good documentation is concise—it overviews critical information such as design decisions and typically provides visual "roadmaps" showing the high-level relationships between things. Agilists focus on this style of documentation, and only write detailed documentation when it is needed and no better alternative exists.
Agile software developers travel as light as we possibly can, creating just enough documentation for the situation at hand in a just-in-time (JIT) manner. We believe that the benefit of having documentation must be greater than its total cost of ownership (TCO), that our stakeholders should decide whether to invest their money in it, and that we should strive to maximize the return on investment (ROI). Agilists are really smart about the way we write system, user, operations, and support documentation and as a result we tend to write significantly less documentation than our traditionalist colleagues. If you're interested in learning more about agile strategies for writing documentation, I highly suggest www.agilemodeling.com/essays/agileDocumentation.htm.
No comments:
Post a Comment