Closing the Gap
Introduction
Ramsay Millar spends his working days as close as anyone can get to applying emerging trends in software development to real-world challenges. Millar travels the U.S. and Canada facilitating workshops for developers and IT professionals looking for better ways to deliver and evolve business-critical software.
"Currently we are all focusing on closing the gap between business requirements and software delivery by moving away from the dysfunction of code-and-fix legacy thinking," said Millar.
Each week he mentors a different group of workers. Over the course of a year Millar works with hundreds of software-development professionals -- software designers, code engineers, project managers, business analysts, testers, CIO's and enterprise architects. This wide-ranging experience with his clients provides him with insights that vary from traditional dysfunction on one hand to outstanding examples of process and tools efficiency on the other.
All this front-line experience leaves him with an interesting perspective towards the ongoing struggle within corporations and government agencies looking for new solutions for building software. This brings him face to face with a wide range of experiments for applying better practices to the complicated process of delivering software to business stakeholders.
"Software development is a high risk business when we look at the industry record of success," said Millar. "It's really astonishing how commonly we face IT project failure -- CIO Magazine recently reported 'as many as 71 percent of software projects that fail, do so because of poor requirements management, making it the single biggest reason for project failure.' "
After three decades as a software engineer -- and ten years as a software road-warrior and mentor -- Millar currently sees a shift in industry practice as significant as the introduction of object-oriented programming a generation ago -- a shift that is so new it does not yet have a name.
"We're seeing three trends maturing and starting to converge," said Millar. "When you layer all three trends together the cumulative effect is a solution to that high historic failure rate."
The result is a profound change in IT culture and practice. This change improves business agility and creates repeatable software quality along with dramatic reductions in the need to re-work code. This kind of improvement appeals not only to the CIO team but also to the comptroller, who is always looking for ways to increase value-for-money.
Millar said a focus on quality costs far less than wasteful code-and-fix legacy thinking. "The manufacturing industry discovered this years ago," he said. "The excitement today is now more clients realize we can do this with software also!
"Every one of my clients is actively engaged in improving the way they deliver and maintain software," he said. "My clients are tired of traditional IT dysfunction and wasteful re-work practices. "When we can move beyond these issues, the reward for improving the process is a happier customer and an improved business bottom line."
Millar said the three trends of this "quiet transformation" are:
- A wish to close the gap between business requirements and software delivery by applying an agile process and best-of-breed automation to the entire software development life cycle. This gap is closing across the lifecycle -- from process workflows, through rigorous use-case-driven requirements to well formed re-useable services and architecture.
- An increasing acceptance and sophistication of the Unified Modeling Language (UML) standard -- evidence of which is found in the rapid growth of reference models in Model Driven Architecture (MDA), the recent merging of UML and Business Process Modeling Notation (BPMN) by the Object Management Group (OMG), and increasing popularity of next-generation UML tools.
and most significantly:
- A shift towards more collaborative, role-based teamwork and away from the "lone wolf" IT culture of code-and-fix.
Millar said while there is widespread awareness of each of these changes on its own, industry leaders are just beginning to realize what happens when all three trends are combined within a team to produce changes in culture, process, and toolsets.
First Trend: Closing the Gap
The oldest of the three trends is the increasing automation of the software development process.
"As software engineers we've spent the past two decades automating every other aspect of the organization -- manufacturing, engineering, accounting, human resources and customer services for example," said Millar. "But IT has been slow to bring the same degree of sophistication and practice to its own work --to the process of creating and evolving software to meet continual business demands." Millar said the situation reminds him of the old adage about "the shoemaker's children having no shoes."
"The IT worker was so busy reengineering and automating everyone else to prepare for the digital revolution -- now that revolution has taken place, IT practices have some catching up to do," he said. "That's where I see this profound change -- I'm really starting to see a strong appetite for software process improvement."
This appetite for improvement is growing, says Millar, he sees this quiet transformation is moving from Moore's early-adopter phase into the early-majority phase. In 2008 the notion of successfully automating the development of large software systems is not leading-edge in itself. "What is recent," says Millar, "is that practice, process and tools for achieving this kind of automation are now coming together."
Millar points to Ivar Jacobson, principal of Ivar Jacobson Consulting (IJC) -- a founding father of the Unified Process, UML and the use-case-driven approach. Jacobson has customized his Essential Unified Process to make it work with the Microsoft Solutions Framework and to integrate with Microsoft's VSTS and the related Team Foundation Server (TFS) and .NET.
Jacobson's IJC has joined Microsoft's Visual Studio Industry Partner (VSIP) program. Essential Unified Process targets small teams interested in agile and iterative development and an agile Unified Process for software process improvement. Millar also cites the work of process gurus David Anderson, author of Agile Management, and Scott Ambler, author of Agile Modeling. "It was just a few years ago when both Anderson and Ambler were breaking radical new ground challenging organizations to approach software development in entirely new ways," he said.
"Anderson now works for Bill Gates' Corbis, and Ambler is now an Agile practice leader with IBM," said Millar. "Their thinking today is every bit as brilliant as five years ago, but it's now becoming accepted by progressive CIO's, their IT workers and managers. Millar said that every week he works with corporate clients who have adopted approaches from Jacobson, Anderson and Ambler and have increasingly turned their once-radical concepts into standard industry best practice. "I sense this quiet transformation is on the way to becoming mainstream."
"The leading sector for this 'early majority' phase is what Gartner refers to as 'small and midsize enterprises' (SMEs), usually because many of the new tools are affordable and SMEs are less invested in the old ways and tools of the more traditional vendors," said Millar. "But we're also seeing small pockets within the Fortune 500 take a nimbler approach. The leaders are starting to change their software development culture, adopt better practices and ito standardize on UML tools to close the gap between business process, requirements, coding and testing."
Second Trend: Increased Growth and Sophistication of the UML Notation
Millar, who has used UML case tools since 1997, said the second trend in his "quiet transformation" -- in addition to closing the gap -- is the increasing use and sophistication of Model Driven Development, most commonly using the UML notation.
"When OMG introduced UML ten years ago, industry skeptics dismissed it as nothing more than a marginally-useful 'Visio on steroids,' " said Millar. "But in 2007 history has proven them wrong. IBM sources currently estimate UML adoption at 42 percent."
Millar pointed out that while UML was initially considered "somewhat exotic," in the past two or three years it is increasingly accepted as the industry standard by CIO's, enterprise architects and other software leaders. Millar said the UML notation is about to become even more powerful. The Object Management Group (OMG), which sets industry standards, is in the process merging the notation used in UML with the traditional notion of workflow called Business Process Modeling Notation (BPMN).
This merging of UML and Business Process Modeling Notation (BPMN) provides a new standard for orchestrating business process workflows across a re-usable services architecture. "Enterprise architects will be able to orchestrate business-process changes across a Service Oriented Architecture (SOA) using web services delivery language to execute well designed service components," said Millar.
As further evidence that UML modeling now exceeds the expectations of early critics, Millar points to Microsoft's decision two years ago to integrate SPARX Systems' and support SPARX's integration of Enterprise Architect software (a full-lifecycle UML modelling suite) into its Visual Studio Team System (VSTS) and the underlying Team Foundation Server (TFS) environment. (The SPARX tools have similar capacity for integration with Oracle, Eclipse/Java/J2EE, IBM WebSphere, and testing tools like HP Quality Center.)
Microsoft has since joined the Object Management Group and have announced that the upcoming VSTS 2010 Architecture will enable both technical and nontechnical users to create and use UML models for collaborating and defining business and system functionality in graphic formats.
"This kind of two-way integration actually changes the software development process," said Millar. "Microsoft and SPARX have knocked down the firewall that used to separate the software engineer from the world of the business systems analyst.
"We're finding when business analysts and requirements people avoid silos and work collaboratively with the design architects, software engineers and testers -- we see software code re-work decline by 30 to 50 per cent," he said.
Jacobson cites a case study that makes a similar point in his February 2007 Americas Newsletter. "A highly mature US Insurance provider uses the Jacobson Essential Unified Process to dramatically increase software development agility and capacity with a whooping 30 per cent productivity improvement."
Millar agrees -- and he characterizes this degree of improvement as "astonishing." "What we're experiencing here is much more profound than simply 'tweaking' the process," said Millar. "This changes our mindset -- when you add all of these process changes together we have fundamentally changed how IT workers collaborate.
"Improving the process and adopting current best-of-breed tools actually changes the software business case," he said. "This brings about a vast improvement in both software quality and return-on-investment (ROI)."
Millar -- who mentors and consults with clients on the use of case tools and process -- said the next generation tools he currently uses have evolved from single-purpose UML tools into what he calls "cleverly-designed Swiss Army knives."
"The newer tools allow team-wide model sharing with complete traceability across all work items including business process models, software requirements, analysis, design, code and testing artifacts. "These modeling tools are capable of anchoring an iterative and agile process," he said. "But all this capability is relatively new. Neither the process -- as it is possible today -- nor the tools even existed just four years ago."
Millar said the ROI on current-generation UML tools has also improved because of a dramatically lower per-seat cost -- in some cases less than one-tenth the cost of first generation UML tools
The lower price-point and increased functionality of these newer products not only changes the ROI calculus -- but it also deliberately turns the "guru-based" deployment model on its head. Instead of equipping a small corps of modelling experts within a larger team, the new pricing model is designed to put UML tools (and working access to the resulting models) on the desktop of every single person involved in the software development lifecycle. One of the many benefits of this change is that it facilitates greater collaboration.
Third Trend: Teamwork and Collaboration
The increased growth of UML and the arrival of cheaper, more powerful UML tools bring Millar to the third trend in his "quiet transformation:" a shift towards teamwork, collaboration and improved quality. When Millar works with clients to create their in-house wish list for improving the process and selecting modeling software, he finds that a "strong, robust capacity for collaboration" is usually at the top of the list.
The shift towards teamwork in IT is much broader than simply UML case tools or modelling -- but in the context of UML tools -- the current best-of-breed can improve team performance by providing:
- Integration with industry-standard development environments like Java/Eclipse/J2EE, IBM WebSphere, Microsoft Solutions Framework and Microsoft's Visual Studio Team System (VSTS) and the related Team Foundation Server (TFS) and .NET, and Oracle/TOAD and UML models.
- Complete work item traceability created, maintained, and updated by all team members.
- Built-in online, cross-team discussions that involve everyone -- local or international -- in an open chat that references (and is linked to) all work items using an extended UML model.
- Shared Model Driven artifacts -- documentation (conveniently in RTF or HTML); software code; business rules; UML diagrams -- readily available to all team members.
- Tools for accurate team based project management -- supporting resource allocation, work in progress, backlog items, velocity, quality indicators and highly accurate use case cost estimates.
Every week, as Millar's consulting practice takes him to a new work group in a new city, he sees more and more evidence of the shift towards more IT collaboration. While this kind of culture change is never easy, he said his clients share a strong recognition of the need for change.
Millar also sees evidence of this culture change in the evolving makeup of audiences for his mentorship and consulting work. Six years ago the typical client work group consisted almost entirely of object-oriented software engineers and their software managers.
Today that profile is changing. In 2008 the typical client working group includes not only software engineers but also business analysts, requirements analysts, project managers, design architects and testers. "This just never happened before the availability of affordable, collaborative software development tools started bringing everyone in the team together, as opposed to earlier tool solutions that split us apart with too many expensive, hard to use, and badly integrated tools." said Millar.
Under the Radar
Millar says the reason he characterizes this transformation as "quiet" is because the adoption of next generation UML tools and best practice techniques -- and the changes they enable -- is largely happening incrementally and under the radar. This is because many of the best tools are less expensive and so acquisitions are happening without high level approval. On top of this many companies are secretive about their work in this area because they see it as a significant competitive advantage. He illustrates this by citing one of his favorites amongst the new-generation modeling tools: SPARX Systems' Enterprise Architect .
Millar often demonstrates SPARX Enterprise Architect in his webinars to provide an initial overview to clients who are just beginning to explore UML tools to adopt standards. "The high-level presentation almost always leaves the IT workers astonished," he said.
"People new to UML tools, or people who have only used older tools, have no idea how the newer tools have evolved," he said, adding the characteristics that consistently cause a buzz with newcomers typically include:
- An open and shared SQL database (and other collaboration tools) that unites the entire software development team;
- Comprehensive integration of a single, versatile model that creates Model Driven Architecture used to transform and generate code, create documentation, publish websites, deliver test plans, specify use-case contracts and deliver project-management reports;
- Ease of use -- intuitive productivity features that enables team members to learn UML rapidly.
- The combination of agility, flexibility and power at an affordable per-seat cost.
Millar agrees that final proof of his view that a "quiet, major transformation" has actually arrived lies in front-line practice, and not in classroom theory. "If it's all talk, then it's a little early to call it a transformation,'" he said. "However every week I see people and organizations 'walking the walk.'"
International Institute of Business Analysts
The recent certification for Business Analysts is another indicator of the software industry's recognition of the importance of closing the gap between business requirements and code. Business Analysts may now choose to be certified by their own professional association: the International Institute of Business Analysts (IIBA).
People have really struggled with IT going in one direction and the business going in the other.
Adam Honor, Senior analyst Aite Group
The Voice of Millar's Clients
The real-life clients Millar sees have three things in common:
- The organization is shifting (or has shifted) to a software-development process that is increasingly nimble, automated, and more customer-driven.
- The organization is adopting (or has adopted) leading edge, standards based UML tools to improve communication, software quality, reduce project risk and improve ROI.
- The software development process is increasingly more efficient and collaborative -- not just amongst software engineers, but amongst everyone involved in the process.
"This is all very real -- but virtually everyone I work with is keeping the details of their own process improvement very low profile," said Millar. "They see their innovations as a competitive advantage -- slightly ahead of the curve -- and they don't want to share outside their home base."
While preparing for this article Millar asked many of his clients to openly discuss details about their process improvement, and the only ones prepared to do so insisted their companies go unnamed. "Until these new approaches become ubiquitous, many current adopters are treating the details as 'trade secrets,' " he said.
First Client
Profile: What Gartner would call a "small and midsize enterprises (SME's)." A competitive company whose primary business is developing innovative and practical software products used primarily in the education sector: universities, colleges and public-school districts. Their solutions are very much shaped by customer needs, and they have emerged as a leader in web-based solutions tailor-made for education professionals.
Process improvement within the company is led largely by the Director of Software Quality Assurance who oversees a shop with 32 IT staff . He commented on how some of the newer tools have reshaped their development process in the past couple of years:
We are cutting back on the "scope creep" that comes from misunderstandings by using a combination of Microsoft's Visual Studio Team System (VSTS) and SPARX Systems' Enterprise Architect and integrating them through Microsoft's Team Foundation Server. In a VSTS/.NET environment this combination closes the gap between the business analysts on one hand and software engineers and QA testers on the other. We recently did a pilot project, and the clarity provided by the very first UML modeling project, produced with the new process and the new tools, repaid the entire cost of retooling.
Second Client
A very large national U.S. retail company that just celebrated its 20th anniversary, and is "very excited about the future." The retail operation includes outlets across the country from coast to coast. Their business strategy includes continually introducing improvements and innovations to make shopping easier and more convenient for the customer, while continually searching the globe for merchandise that gives customers "exactly what they're looking for." The retail industry has very small margins and improving software agility and services is viewed as a company-confidential competitive advantage.
The Chief Software Architect was prepared to talk for this article on condition that his organization went unnamed. As with the first example (above), his development team is using SPARX Systems' Enterprise Architect.
I've been able to impose a new level of discipline on the development process -- we've eliminated the 'weak link' by creating a common language (UML) between the business people coming in with requirements and the architects who are designing the new systems. The newer software-development tools are a breath of fresh air when I compare them to their predecessors -- that performed like a herd of elephants.
No Longer "Pioneering"
Millar said he personally no longer feels like a "in the wilderness" when he works with clients who are part of this change -- because he sees so much of it now in his professional travels
"As recently as five years ago the people bringing me in to an organization were the early adopters and ahead of their colleagues in the industry," he said. "I would facilitate workshops with a group of keeners, and it would take a day or two before the others in the team got excited at all the new and emerging possibilities."
"Today it's different," said Millar. "Now when I go in and work with a client -- typically everyone in the room is already enthusiastic about trying out new tools and learning how to improve the processes of evolving and delivering software."
When Millar talks about a "quiet, major transformation" in process improvement, standards and tools, he really sees a fundamental change in the way software development teams work together. He sees software development professionals embracing change as never before, and happily leaving behind dysfunctional software development practices.
"The light at the end of the tunnel is getting clearer."
Resources
Scott Ambler
http://www.ambysoft.com/
David Anderson blog
http://www.agilemanagement.net/Articles/Weblog/blog.html
Ivar Jacobson
http://www.ivarjacobson.com/
Ramsay Millar
http://www.in2grateIT.com/
Microsoft Solutions Framework
http://www.microsoft.com/technet/solutionaccelerators/msf/default.mspx
Microsoft Foundation and Visual Studio Team System
http://msdn.microsoft.com/vstudio/teamsystem/msf/
SPARX Systems
- Enterprise Architect modeling Tool)
http://www.sparxsystems.com - Integration with Visual Studio
http://www.sparxsystems.com.au/products/mdg_integrate.html
International Institute of Business Analysts (IIBA)
http://www.theiiba.org
OMG (Object Management Group)
Useful Books
Agile Modeling
by Scott W. Ambler
Wiley; First Edition (2002)
ISBN-13: 978-0-471-20282-0
Amazon: http://tinyurl.com/2otnm5
Requirements in Context
by Daryl Kulak and Eamonn Guiney
Addison Wesley; Second Edition (2004)
ISBN-13: 978-0-321-15498-3
Amazon: http://tinyurl.com/3ddk9v
No comments:
Post a Comment