The Rise And Fall Of Waterfall
Winston W. Royce, the man who was the first to describe the Waterfall model for software development, although he did not use the term “waterfall” nor advocated it as a working methodology.
Winston W. Royce, the man who was the first to describe the Waterfall model for software development, although he did not use the term “waterfall” nor advocated it as a working methodology.
Interesting initiative to revitalize the software engineering discipline. SEMAT (Software Engineering Method and Theory), launched in October’09 by Ivar Jacobson, Bertrand Meyer and Richard Soley, is trying to recognize the fundamental problems and to develop a sound and general theory for software engineering.
They’ve started with a “Call for Action”:
Software engineering is gravely hampered today by immature practices. Specific problems include:
- The prevalence of fads more typical of fashion industry than of an engineering discipline.
- The lack of a sound, widely accepted theoretical basis.
- The huge number of methods and method variants, with differences little understood and artificially magnified.
- The lack of credible experimental evaluation and validation.
- The split between industry practice and academic research.
We support a process to refound software engineering based on a solid theory, proven principles and best practices that:
- Include a kernel of widely-agreed elements, extensible for specific uses
- Addresses both technology and people issues
- Are supported by industry, academia, researchers and users
- Support extension in the face of changing requirements and technology
and the vision statement (PDF), specifying what the community want to achieve in the next twelve months.
Despite agreeing with the above statements and the impressive expert lists who had signed the call for action, I’m quite sceptical about the results. Starting with the discussion whether software development is an engineering discipline or not, to the danger that this initiative will become just another fad. Anyway, interesting enough to keep an eye on it.
Several months ago, I promised to write about the IBM Rational Jazz platform and IBM Rational Team Concert. As you may have noticed, I have not yet write about them, but in my defense I can say that I have not had much time to devote to this nor other posts in this blog. As I mentioned in some others posts, lately I have been leading a major renovation of our entire suite of custom development tools, and these last 3 months I have been fairly busy managing all this change. Taking advantage of Easter holidays, I finally found the right time to get to write about the Jazz platform.
But before proceeding, a disclaimer. What I am going to write about Jazz is just a personal opinion, may or may not be wise, may or may not have something to do with reality, but I want to make clear that this is an entirely personal opinion, and do not mean any endorsement from my current employer.
To describe the Jazz platform, I think we should go back to the past, because in my opinion, Jazz is trying to evolve the vision/mission/wildest dream that Lee Nackman had in 1998: to create a single technology platform on which to build the various IBM’s application development tools. The objectives Lee had at that time were:
With the help from the autonomous OTI subsidiary (acquired by IBM in 1996), and overcoming an enormous amount of skepticism within IBM, Lee and his team delivered a technology platform that became what today is known as the Eclipse platform. Looking at the success of this platform, especially in terms of IBM adoption across the different brands and tools, it seems that the main objectives were reached. Not to mention also that open sourcing the platform and several projects (as the JDT), they killed lots of competitors, shrinking the Java tools market, and created a great ecosystem around it.
But the knowledge and tool set that IBM acquired when they bought Rational Software in 2002, mixed with a retrospective analysis they did based on the experience they gained in the Eclipse development, helped them to figure out which were the new challenges for the software delivery process. I’ll try to summarize, IMHO, some of the improvements they realize:
So in my opinion, and as I told previously, the Jazz platform is the evolution of the original Eclipse vision, keeping in mind the above and some more other concerns, with an special focus on teams and collaboration. But its aim is not to replace Eclipse, they are distinct platforms with different goals, although Jazz seems to be the perfect complement to Eclipse. This new vision is well summarized at the About Jazz page:
Our goal is to provide a frictionless work environment that helps teams collaborate, innovate, and create great software. To that end, we are focusing on driving fundamental improvements in team collaboration, automation, and reporting across the software lifecycle.
When trying to describe what compose the Jazz platform, albeit IBM have split the original Jazz project in several projects at the jazz.net site, I still have some problems trying to draw the line between the platform and the applications, to see which components are part of the Jazz platform and which are part of the different products based on it. So I will try to use the below picture, that I have borrowed from the IBM Rational guys, in order to clarify my ideas:

To enable a seamless and higher level of integration between tools, IBM has defined a reference architecture, API specifications, and a set of common services and tool building blocks, that together are called the Jazz Integration Architecture (JIA). At the center of this integration architecture we found the Jazz Team Server (that may consist of one or more physical servers that act together as a single logical server), which provides foundational services (RESTful web services) to enable groups of tools to work together. Let’s summarize each of these foundational services:

Anyway, the Jazz platform is still in its early stages, and it is constantly evolving to meet additional challenges. What I have summarized previously is what it is know as the Jazz Platform 0.6, but a new version is expected to be delivered on June with a new name, the Jazz Foundation. So if you are interested in more deep details about the above or new services that are going to be delivered, I recommend that you go through the development team wiki (registration required).
There is also an interesting parallelism between how Eclipse and Jazz has been developed. In order to convince other IBM’s development tools product managers to adopt the Eclipse platform, Lee and his team decided to build a Java IDE. There were two reasons behind that decision: 1) to provide a real example (a killer-application) of a tool developed on the platform, proving in that way its benefits; 2) to help the Eclipse development team to better understand the needs of future consumers of the platform and to discover areas that required further development. This strategy was a success and the Eclipse platform and the Java IDE were quickly adopted inside IBM.
The Jazz project seems to use the same kind of strategy. They are developing a real tool using the Jazz platform. This killer-application is called Rational Team Concert and as far as I know it is also rapidly adopted inside IBM. I hope to write about this product in the near future.
Looking at the jazz.net site the increasing number of IBM tools that are adopting the Jazz platform, I have no doubt that it will be another success in terms of IBM adoption. But …
So in terms of external adoption/extension, it seems that IBM is focusing only in the interface as a way to integrate non-IBM tools, encouraging developers, customers and ISVs to participate in the development of the Open Services for Lifecycle Collaboration initiative, something like an open standards consortium.
Starting looking at the original Eclipse vision and how IBM revisited it after the Rational Software acquisition and the Eclipse success, I have attempted in this post to describe what it is the Jazz platform. I am sure some of you have realized that some of the improvements that I have described previously can be easily or are already implemented through Eclipse plugins, but I think these are the minor ones. There are three main conceptual differences between Eclipse and Jazz:
And in my opinion, these conceptual differences can only be implemented through the creation of a new platform. Instead of solving some particular problems in an isolated way, Jazz is trying to attack the essence of the software development process. Does it means that we must convert to a new religion, drop the Eclipse platform and adopt the Jazz platform? No, Jazz is not going to replace Eclipse. There will be a strong relationship between the Jazz and Eclipse environments, yet the
two are distinct and can run independently. Jazz is going to complement Eclipse in some particular scenarios:
Eclipse for Individuals, Jazz for Teams
We have also seen that IBM’s decision is not to open source this new platform, but to create a community around the jazz.net and OSLC websites. However, it will need to attract a broad and active participation from a wide external community in order to be a great success outside IBM, as it was Eclipse, something that I believe it is not a current IBM goal.
What will happen in the future? Sincerely, I don’t know, time will decide. From my particular point of view, this new vision match up what we have been doing for a long time and that has lead us to extend the Eclipse platform in order to create a custom collaborative tool set. So, as we are already an IBM Rational customer and we have licensed some of the Jazz products, I will be very happy if we can integrate, in an easy way, our custom tools with the IBM Rational tool set.
Before concluding this long post, I would like to ask you:
PS: One of the latest tasks Lee Nackman did before his retirement at IBM, was to help spur the development of the Jazz platform.
PS: Most of the people actually involved in the development of the Jazz project were part of the Eclipse platform development team, so it is not strange to see that they are applying the same strategy, but also adopting the best practices they learned during the Eclipse development process.
PS: There is an excellent case study on IBM’s strategy and process for creating Eclipse at the Harvard Business School. It’s a worth read.
PS: Another interesting read is a paper on Collaborative Development Environments (PDF) by Grady Booch and Alan W. Brown, which seems to be the “spark” that started the new vision.
As I mentioned in a previous post, last week I attended to the Eclipse Banking Day in London. It was an enriching experience for me, not only in terms of speaking in a foreign language in front of large crowd but also seeing what other financial institutions are doing with Eclipse. I also met great people, during and after the event, so, who could ask for more? It was a really great event!
Most of the presentations are now available on the event wiki page. I encourage you to take a look to some of them. Anyway, here there are some few personal thoughts about the presentations:
And that’s all folks. Any other attendees would like to share their thoughts?
As I explained in a previous post, the last year I have been involved in a renewal process of all of our application development tools. One of the first things we did when we started the program was to apply the most fundamental lean principle: eliminate waste. To lean thinking, waste is anything that does not create value for a customer. For those of you who are not familiarized with the lean principles, I recommend the “Lean Software Development: An Agile Toolkit” book. In this book, Mary and Tom Poppendieck translated the seven wastes of manufacturing identified in the Toyota Production System into the seven wastes of software development: Partially Done Work, Extra Processes, Extra Features, Task Switching, Waiting, Motion and Defects. In this entry I would like to explain some of the problems we have encountered while trying to eliminate waste and some lessons learned.
The first waste we tried to eliminate was the extra processes. In other word, we tried to eliminate paperwork that does not means adding value for our users or for our organization. In our case, this task was not related to the development of our tools, it was about eliminating extra processes that were embedded in the tools we developed which forced our users (developers) to execute some unnecessary processes. This action produced some surprises, since near the end of the development, in one of the latest functional demos, there was a crisis moment. Some developers reminded us an essential process they were following in the old tool: they defined the batch programs in a product repository. This process was removed deliberately, because it does not provide any value, so it was a surprise for us that our users asked for this. When we asked them why this process is necessary, they answered that they did not know, but it was something they used to do because someone told them that they must do this task. “Is this useful?” we replied. “No, but we must continue doing that because … we must do that”. Wow, it remembers me the monkey experiment. Obviously, and despite the laments of our users, we did not add again that process. So lesson learned: in order to eliminate waste you need to break the status quo, you need to break the corporate culture.
The second waste we tried to eliminate was the extra features, because this was one of the biggest mistakes we did in the past in other tools. Some years ago we started developing a new tool and, using the usability argument, we added a lot of features that lately nobody used. Some of the hitches you may suffer adding those features are that your code-base grows uselessly, increases maintenance costs and makes future developments more complex. This must not be a big problem if users really need these features, but why must we maintain them if they are not being used? Users, moreover, feel that the tool is more complex, so your usability argument disappears, and they reproached us that we are not focusing on what it is really important for them. Now, I am proud to say that our tools have less features, but, at least, the ones we implemented are really used. So two more lessons learned: 1) more features does not mean better tools; 2) usability does not mean more features.
I am going to stop here. I am sure most of you who have tried to eliminate waste have found these or similar problems. But I do not want to conclude this entry without explaining one of my latest lessons learned. It is not strictly related to lean thinking, it applies to software development in general, albeit it could only apply to some organizations. Sometimes, I believe it is better not to explain that you are using an “x” methodology, or to intensify your position saying that you learned those practices from whatever methodology. Sound strange, isn’t it? But I have discovered that lots of developers hate the words “methodology” and “process”, and they have adverse reactions when they hear them. I find easier to explain practices without any reference to the original methodology. Lots of times, using the common sense is better to prove the goodness of a practice. And if it does not sound good, perhaps it does not match your organization. OK, maybe I am generalizing. In every change process you will find resistance, so perhaps if it does not sound good it is because fear. So, my last lesson learned: use common sense, do not arbitrarily adopt new practices.
After the success of the Eclipse Banking Days in Frankfurt and New York, the Eclipse Foundation hast just announced the Eclipse Banking Day in London on February 12, 2009.
Eclipse Banking Day is a day-long event for senior technical developers, architects and managers in the finance industry to learn how to better leverage Eclipse technology and the Eclipse community as part of their development strategy. The event will focus on three themes:
- Eclipse as a platform for application development;
- Leveraging Eclipse modeling technology for data exchange; and
- Collaborating with the open source community.
Attendees will have the chance to hear speakers from leading financial institutions and experts from the Eclipse community.
It is not a secret that Eclipse is being used by some of the major banks and financial institutions around the world. Well, as it could not be less, we are also using Eclipse, both as a tools integration platform and as a branch teller workplace. Some days ago, Ian Skerrett (Eclipse Marketing Director) asked me if we would be interested in sharing our experience with other banks. So … we decided to accept the invitation and I’m going to present at the Eclipse Banking Day in London how we are using Eclipse at “la Caixa“. Despite the below pompous abstract, I think it will be an interesting presentation.
Repository Based Application Development Environment for Banking Systems
“la Caixa” is currently the leading savings bank in Spain and the third largest financial entity in the country. With a large network of more than 5.500 offices, more than 8.100 automatic cashpoint machines, a staff of more than 26.000 employees and more than 10,5 million clients, ”la Caixa” has positioned itself as a leading entity and referent within the Spanish financial sector.
In this talk, we will explain how “la Caixa” is using Eclipse to create a repository-based application development environment that successfully empowers its +1000 developers to create first-class custom enterprise banking applications in a fast-changing market. We will take a brief tour of “la Caixa”‘s enterprise architecture and we will take an inside look at some custom Eclipse plugins built at “la Caixa”. We will describe how using a collaborative environment, visual designers and code generators “la Caixa” allows its developers to create rapidly all the software components, from web UI to IMS-PLI-DB2 transactions, but also archiving software reuse across the whole organization and enforcing governance in an unobtrusive way.
This presentation will also explain briefly how “la Caixa”‘s 24.000 tellers are using Eclipse as a branch teller workplace. We will describe “la Caixa” bank teller evolution, and how using Eclipse it is possible to integrate in a common workplace from a custom legacy UI render to a modern web UI.
If you are interested in attending, you need to pre-register (early, space is limited!). There is no cost to register but you must work for the financial services industry.
Hope to see and meet great Eclipse enthusiasts there! I will try to share my thoughts on the conference after the event.
Recently, I have been involved in a major IT program, framed on a high demanding business strategic plan, which aims to renovate all of our core banking system. One of the program’s first steps was changing our IT organizational and governance structure, the enterprise architecture, the application development tools and the software development methodology.
Although my responsibility in this program only relies on the application development tools, as one of the PMO leaders, I was able to follow closely the rest of the items. One of the topics that I was specially interested on was the software development methodology, because among other things, before the program, it was one of my responsibilities, and, after the program, it will be, again, my responsibility. The truth is that I had high hopes for change (perhaps influenced by the “Yes, we can!” slogan), but the fact that they came to conclusion that we need another waterfall methodology, perhaps a bit stricter than the one we use today, disappointed and frustrated me.
But please, that nobody misunderstood me. I deeply respect the work and decisions of my colleagues. I have had the opportunity to explain my thoughts. I gave them some books on the topic. I tried to influence the people who had the task to define the new methodology in order to introduce more innovative methods/process/practices of software development, but, maybe, being too innovative in a very classic environment doesn’t helped me. I think that changing the software development process in a big organization requires lots of effort, education, very slow and gradual steps, …, and I do not want to miss this opportunity, I believe that now is the moment to do that, taking advantage of the whole changing program. Well, at least, now some people knows that there are more life after the waterfall model, there’s a gray scale between white and black.
As I’m a bit nonconformist, I tried to find out which were the reasons behind that decision. So I decided to ask some developers and project managers to get an exact idea of what they think/know about software development methodologies. Although most of the answers were what I expected to find, I also found some interesting frustrating observations. I want to share with you some of them:
I also tried to analyze some projects, and, surprisingly, I discovered that most of them are short term projects (2-3 months). I expected longer projects, as corresponds to a waterfall process. So I ask myself if we are really using a waterfall methodology, or we’re using a masked iterative process?
How about you? Did you find these kind of answers in your company? Any ideas on how to address this situation?
Cameron has been blogging about new features in our product.. In a recent post he used the term Explicit Design. I’ve been reflecting on this, and I like it. In software development we really do need to capture design data that is not just the code, but should be saved and versioned just like the code. What do we call it? We could call it “models” but “model” and “model driven development” are subject to so much historical baggage and methodology and terminology arguments. “Model” just seems to imply baked-in code generation and round tripping, when there is so much more that you can do with it: planning, verifying, testing, refactoring. We need new vocabulary that represents our ability to capture versioned design data at a more abstract level than the code without simultaneously implying the history of CASE.
I have to agree: changing the name doesn’t solve the root of the problem, but perhaps we start thinking in a different way.