Winston W. Royce, the man who was the first to describe the Waterfall model for software development, although he did not use the term “waterfall” nor advocated it as a working methodology.

As I explained in a previous post, the last year I have been involved in a renewal process of all of our application development tools. One of the first things we did when we started the program was to apply the most fundamental lean principle: eliminate waste. To lean thinking, waste is anything that does not create value for a customer. For those of you who are not familiarized with the lean principles, I recommend the “Lean Software Development: An Agile Toolkit” book. In this book, Mary and Tom Poppendieck translated the seven wastes of manufacturing identified in the Toyota Production System into the seven wastes of software development: Partially Done Work, Extra Processes, Extra Features, Task Switching, Waiting, Motion and Defects. In this entry I would like to explain some of the problems we have encountered while trying to eliminate waste and some lessons learned.

The first waste we tried to eliminate was the extra processes. In other word, we tried to eliminate paperwork that does not means adding value for our users or for our organization. In our case, this task was not related to the development of our tools, it was about eliminating extra processes that were embedded in the tools we developed which forced our users (developers) to execute some unnecessary processes. This action produced some surprises, since near the end of the development, in one of the latest functional demos, there was a crisis moment. Some developers reminded us an essential process they were following in the old tool: they defined the batch programs in a product repository. This process was removed deliberately, because it does not provide any value, so it was a surprise for us that our users asked for this. When we asked them why this process is necessary, they answered that they did not know, but it was something they used to do because someone told them that they must do this task. “Is this useful?” we replied. “No, but we must continue doing that because … we must do that”. Wow, it remembers me the monkey experiment. Obviously, and despite the laments of our users, we did not add again that process. So lesson learned: in order to eliminate waste you need to break the status quo, you need to break the corporate culture.

The second waste we tried to eliminate was the extra features, because this was one of the biggest mistakes we did in the past in other tools. Some years ago we started developing a new tool and, using the usability argument, we added a lot of features that lately nobody used. Some of the hitches you may suffer adding those features are that your code-base grows uselessly, increases maintenance costs and makes future developments more complex. This must not be a big problem if users really need these features, but why must we maintain them if they are not being used? Users, moreover, feel that the tool is more complex, so your usability argument disappears, and they reproached us that we are not focusing on what it is really important for them. Now, I am proud to say that our tools have less features, but, at least, the ones we implemented are really used. So two more lessons learned: 1) more features does not mean better tools; 2) usability does not mean more features.

I am going to stop here. I am sure most of you who have tried to eliminate waste have found these or similar problems. But I do not want to conclude this entry without explaining one of my latest lessons learned. It is not strictly related to lean thinking, it applies to software development in general, albeit it could only apply to some organizations. Sometimes, I believe it is better not to explain that you are using an “x” methodology, or to intensify your position saying that you learned those practices from whatever methodology. Sound strange, isn’t it? But I have discovered that lots of developers hate the words “methodology” and “process”, and they have adverse reactions when they hear them. I find easier to explain practices without any reference to the original methodology. Lots of times, using the common sense is better to prove the goodness of a practice. And if it does not sound good, perhaps it does not match your organization. OK, maybe I am generalizing. In every change process you will find resistance, so perhaps if it does not sound good it is because fear. So, my last lesson learned: use common sense, do not arbitrarily adopt new practices.

Recently, I have been involved in a major IT program, framed on a high demanding business strategic plan, which aims to renovate all of our core banking system. One of the program’s first steps was changing our IT organizational and governance structure, the enterprise architecture, the application development tools and the software development methodology.

Although my responsibility in this program only relies on the application development tools, as one of the PMO leaders, I was able to follow closely the rest of the items. One of the topics that I was specially interested on was the software development methodology, because among other things, before the program, it was one of my responsibilities, and, after the program, it will be, again, my responsibility. The truth is that I had high hopes for change (perhaps influenced by the “Yes, we can!” slogan), but the fact that they came to conclusion that we need another waterfall methodology, perhaps a bit stricter than the one we use today, disappointed and frustrated me.

But please, that nobody misunderstood me. I deeply respect the work and decisions of my colleagues. I have had the opportunity to explain my thoughts. I gave them some books on the topic. I tried to influence the people who had the task to define the new methodology in order to introduce more innovative methods/process/practices of software development, but, maybe, being too innovative in a very classic environment doesn’t helped me. I think that changing the software development process in a big organization requires lots of effort, education, very slow and gradual steps, …, and I do not want to miss this opportunity, I believe that now is the moment to do that, taking advantage of the whole changing program. Well, at least, now some people knows that there are more life after the waterfall model, there’s a gray scale between white and black.

As I’m a bit nonconformist, I tried to find out which were the reasons behind that decision. So I decided to ask some developers and project managers to get an exact idea of what they think/know about software development methodologies. Although most of the answers were what I expected to find, I also found some interesting frustrating observations. I want to share with you some of them:

• Most developers and project managers are unaware of the existence of another methodologies, and they do not have any interest in learning them. Methodologies and process are something bored and do not provide any value to what they are doing actually. Although they recognize there are lot of inefficiencies in the way they work, they don’t want to change it. When I ask some of them if they follow our actual methodology, their answer is NO. “Well, at least, are you following any predefined process?” again, the answer is NO.
• Most project managers feel that a plan-driven methodology provides them more control over the whole project, that their projects are more predictable. But when I ask them if their projects are on time, most of them recognizes that NO. They also doesn’t have/use information from previous projects in order to estimate or improve the next ones, every project is different.
• They see iterative / agility methodologies as a chaos, the wild west, where there isn’t any discipline. W00t? discipline and agile are not conflicting. XP, for example, requires high levels of discipline.
• Some people told me that waterfall methodologies encourages a comprehensive documentation. “Well, could you show me your last project’s documentation? no, we didn’t have time to write it. OK, doesn’t mind. Could you show me any documentation of any project you’re involved? no, we’re still working on it, you know, we’ve tigh schedules”.
• Some answers reflected the “We’re Special” syndrome: “Agile is suitable when you want to develop software for an Iphone, but not for financial applications” (paraphrased).
• Some project managers get annoyed when they must talk with their clients or stakeholders, they hate them (I believe this is a mutual feeling). Yeah, those evil people that everyday changes the requirements and doesn’t have any idea how hard is the software development process.

I also tried to analyze some projects, and, surprisingly, I discovered that most of them are short term projects (2-3 months). I expected longer projects, as corresponds to a waterfall process. So I ask myself if we are really using a waterfall methodology, or we’re using a masked iterative process?

How about you? Did you find these kind of answers in your company? Any ideas on how to address this situation?

Don’t miss the No Silver Bullet Reloaded Retrospective OOPSLA Panel Summary post by InfoQ. It’s a worth read. Great panelists and great content.

Some statements that I’ve found interesting:

Dave Parnas:

Desire for people to seek better tools rather than “actually learning the trade.” Dave used the metaphor: “there is an old saying: ‘the poor workman blames his tools’ – people are poor workmen, I think most programmers are poor workmen.” Dave also said that people are looking for silver bullets to avoid learning the trade.

Linda Northrop:

We still need great designers, and I think we still have far too few, and we still need to cultivate an atmosphere of hard work but also an inner-disciplinary perspective that takes us uncomfortably out of our coding world.

Dave Thomas:

He called today’s state of the art in software development (especially middleware) a gratuitous disaster due to overly quick fast pace of change in API’s and frameworks which create immature software and also create too much difficulty for average software teams to track and stay up to date with. Dave claimed that the majority of today’s enterprise systems are basically just CRUD applications: “In the end these people are trying to do things that are fairly straight forward, and if they were working on a mainframe using a 4GL, they would actually have the thing done.“

He has seen “real successes” in niches, citing highly specialized domain-specific and specialized language applications in airlines reservations and hedge funds;

Martin “Werewolf” Fowler:

You can’t measure your output, you can’t measure your productivity, no way you can run sensible experiments to figure out whether one technique is better than another. …

The biggest difficulty in software development is the communication between the people writing the software and the people for who the software is being written. The conceptual attacks, the essence problems, focus a great deal on trying to improve that communication. The trouble of doing that, of course, is that it’s hard to get that communication flowing. Some of the business people don’t want to talk with software people, and fortunately the software people are hopeless when it comes to social interaction, they can’t talk to business people.

You seem to think that with hardly any information at the beginning of the project you can map out exactly what’s going to happen, pin down costs, commit yourselves to all sorts of unrealistic plans. And then you wonder why it is that I always manage to turn up. That illusion of control, the idea that on basis of hardly any data you can let make these grand predictions, that very folly is one of my greatest strengths.

Fred Brooks:

I know of no field of engineering where people do less study of each other’s work, where they do less study of precedents, where they do less study of classical models. I think that that is a very dangerous thing.

Some days ago, I wrote about a QCon 2008 session where Erich Gamma spoke about transparency and how it is related to the IBM’s Open Commercial Development model (OCD). As I have been involved since last year as a beta customer in one of the projects where OCD is been applied, the Rational Jazz Project, I want to discuss with all of you some thoughts and opinions about this software development style.

First of all, and just to be clear, I’m not going to talk about which is the license model behind OCD. The words “Open” and “Commercial”, when seen together, produces some unexpected chills and thrills, and they have generated some controversial discussions out there, mainly in the open source community (maybe, as Stephen O’Grady points out, it will be more accurate to characterize this as transparent development). In the case of the Jazz Platform, there is also some confusion, because we don’t know if IBM is going to release it as an open source software and then develop commercial products based on this platform (like Eclipse), or if they, both platform and products, will remain as commercial products. I vote for the first option, but the word “commercial” in OCD suggest me the second one.

Instead, what I’m going to talk about is how transparency in a great feature in the software development process. So go ahead.

During my career, I have dealt with lots of products, from both open and non-open source software providers. One pattern that I always find in traditional proprietary software is that you never interact with the development team; there is a barrier between you, the customer, and the vendor’s developers. If you need to report a bug or to ask for an enhancement, you can only interact with a service desk. Usually, they have a support web site, where you can see your own tickets, but you can’t see any other bugs or enhancements reported by others companies. You never know when they are going to deliver a solution for your problem (except if it is a blocker), which will be the way they are going to implement your enhancement, if there are more people interested in some enhancement, … You can only check if the bug is fixed or the enhancement implemented when the vendor delivers a new version of the product, and, sometimes, results are not what you wanted. Furthermore, sometimes, you will have to deal with lots of useless questions, mainly due to misunderstandings between you, the service desk and the development team. This is what Erich Gamma calls “Swiss bank approach to software development”.

This firewall between customers and developers is really very frustrating, not only for the customers, but also for the developers. Some companies try to supply this lack of communication organizing user’s conferences (where you can meet some developers), meetings with whatever worldwide VPs, or through a customer advocate. In some cases, frustrated users set up unofficial forums to share their problems or to try to join forces so that the vendor accepts an enhancement. They try to establish some kind of user’s community, but without the vendor involvement. In my experience, and without intention to offend anyone, you will get lots of nice words, but you rarely archive a real solution.

With open-source products, there is a really different way of relationship between customers and providers. There is an open participation and customers can influence easily in the development process in several ways. And I’m not talking about having access to the source code, which is important, but also having access to the bug tracking system, the development mailing-lists, user’s forums, and, in some cases, the development plans, all of them maintained by the development team (fewer misunderstandings). This well-known transparent and collaborative model usually produces enhanced feedback, which leads to deliver better products (in terms of user’s expectations). It’s about archiving customer value, instead of vendor value.

This is the same interaction I have found while working with the Jazz Project. I have had access not only to the source code but also to the latest integration builds (so I can check how my enhancements are implemented), a wiki with technical information about the platform (if you care about the extensibility), a community forum, the development plans and a dashboard to monitor the health of the overall project. I have had also full access to the issue tracking system, where I can submit my own bugs and enhancements, but I can also see which other enhancements are requested by other customers (and to enroll them if I found someone interesting), in which version are they planned … Summarizing, full transparency in the development process. One of the consequences is that I’ve felt, and this is personal and subjective opinion, more involved in the development process, more biased towards submitting enhancements and more confident about future problems that could appear. This feeling is nothing new if you have previously worked with open source projects, but it is something strange coming from an proprietary product.

However, there is a huge difference between open source and OCD models. While in open source software you can contribute to the code base by fixing a bug or improving some features, in OCD it is not clear which will be the contributor’s role. It will depend on the license model selected, which, in turn, it will establish if there will be a vibrant community and ecosystem outside IBM or if there will be a vibrant IBM’s customer community. Anyway, if the final decision is to keep the software as a commercial product, the transparency applied in this model it will represent a great improvement in the proprietary software development process, for both customers and vendors.

But after using this software development style for some time, I believe that this model it is not useful only to software vendors, but it also could be applied to any IT department, especially in big enterprises, in order to improve their software development process. I also believe this is one of the objectives pursued by Rational Team Concert, the first product based on the Jazz Platform (I will talk about this product in future posts). But this is something that I need to try by myself in the company where I work. As most of my readers already knows, I am in charge of the application developments tools in one major Spanish savings bank. Looking through the development process we use, I am sure that if I ask the users of the tools we develop (our internal customers) how much transparent are we, they are going to complain me. Although we meet periodically, it is hard to achieve transparency only with meetings, we need to adopt different and innovative approaches, the ones I have told you in this post.

And finally, although this is what I honestly think about transparency and OCD, I want to hear other opinions. Do you think this is a good software development model for proprietary software? Do you think this model could be applied inside enterprise firewalls?

Via Anant Jhingran blog, today I discovered MIKE2.0 (Method for an Integrated Knowledge Environment), a nice initiative from BearingPoint which aims to be a collaborative, open source methodology for Enterprise Information Management and to provide an organizing framework for Information Development:

MIKE2.0 is a comprehensive methodology that can be applied across a number of different projects within the Information Management space, including those that apply to: Information Governance, Data Quality Improvement, Data Integration, Data Migration (once-off batch migration and parallel runs), Data Warehousing, Master Data Management and IT Transformation.

I didn’t had time to review the wiki contents (I’ll do in the next few days ), but what I really appreciate is the fact that they are releasing most of their content using a Creative Commons license and that they are using a collaborative model. Quoting David Weinberger:

If You Love Your Information, Set It Free.

They encourage every information management professional to join this open source community, to fully participate in this effort and to let them know what you think about the intellectual property they have shared. So if your job is related to the IM field, take a look and share your opinions with them.

Per Kroll (EPF Project Lead) has announced that yesterday it has been released EPF 1.2, which you can download on the EPF Composer Download page.

EPF Composer 1.2 – New Features and Key Improvements

• Improved Diagram Editor
• Diagram editor has been completely reimplemented
• Provides swimlanes, control flow labels, free-form drawing, and font styles
• Files can be saved in GIF, JPG, or BMP format
• Improved Rich Text Editor
• Font family and size can be changed
• Content cut from Microsoft Word can be pasted as plain text
• Links and images can be added in the HTML view
• Improved HTML error markers and correction features
• Improved Configuration Editor
• Processes can be fine-tuned by adding or removing categorized elements
• Improved configuration error reporting for Method Configurations and Method Libraries
• Current view is highlighted in configuration editor
• Configuration views can be ordered
• Improved Process Editor
• Activities are displayed with indented and outdented elements
• All tasks at any level can be suppressed
• Activities can have multiple descriptors
• Additional platform support
• Vista
• Internet Explorer 7
• Mozilla FireFox 2
• Subversion
• Eclipse 3.3
• Other Improvements
• New plug-in naming convention allows plug-ins to be displayed in hierarchical lists
• Published sites on application servers can implement server-based search
• GUIDs are no longer present in published filenames
• A new variability type is provided: Extends and Replaces
• Elements in categories can be ordered alphabetically, manually, or by type

The May/June 2007 issue (subscription required) of the IEEE Software magazine has a special section devoted to Test Driven Development:

This special issue of IEEE Software includes seven feature articles on various aspects of TDD and a Point/Counterpoint debate on the use of mock objects in applying it. Notably, these articles demonstrate the ways TDD is being used in nontrivial situations (database development, embedded software development, GUI development, performance tuning). This signifies an adoption level for the practice beyond the visionary phase and into the early mainstream.

LogicaCMG has created a new site, called EPF Wiki, aimed to contain free software process descriptions, created using the Eclipse Process Framework (EPF) Composer, that everyone can edit using Wiki technology.

EPF Wiki is Wiki technology designed to be used together with Eclipse Process Framework (EPF). This offers the best of two distinct worlds: the worlds of powerful process frameworks and Wikis. It offers an process engineering infrastructure that combines a modular method construction approach and the flexibility and ease of use that is the defining characteristic of a Wiki.

EPF Wiki is an innovation that adds Wiki features to the hypertext process descriptions created with EPF Composer.

There is a discussion in epf-dev maillist about how Wikis can be accepted as a means of contributing as expected by the Eclipse organization, similar to Bugzilla and the official discussions lists.

This new site is considered an unofficial EPF community site, as this technology is still under Eclipse Legal Review, and it is not available as a download. It is planned to become available in the third quarter of 2007.

Mark Dickson has announced that the Eclipse PMC have approved the donation of the DSDM Plugin for OpenUP/Basic through the Eclipse Process Framework.

The plug-in has been donated by the DSDM Consortium. It is designed to extend OpenUP/Basic, combining elements of DSDM with the Open Unified Process. Central to this is the idea of enhancing the existing Stakeholder role through the addition of 4 business roles from DSDM, namely:

• Ambassador User
• Executive Sponsor
• Visionary
• Advisor User

The objective is to increase active stakeholder participation within a project by making these roles responsible for performing tasks and delivering products.

An Ambassador User comes from the business and ensures that the system being built is the right one to meet requirements. An Executive Sponsor is a high-level user who will champion the project and is ultimately the owner of the system being developed. A Visionary is a user committed to the project and its business goals, who will ensure that objectives are being met. An Advisor User brings day-to-day knowledge of the job being carried out into the project.

The plugin will shortly be added to the EPF CVS repository.