“Collaborating with others is at least as important as having great technical skills”. Great Goggle I/O 2011 video:

Winston W. Royce, the man who was the first to describe the Waterfall model for software development, although he did not use the term “waterfall” nor advocated it as a working methodology.

Interesting initiative to revitalize the software engineering discipline. SEMAT (Software Engineering Method and Theory), launched in October’09 by Ivar Jacobson, Bertrand Meyer and Richard Soley, is trying to recognize the fundamental problems and to develop a sound and general theory for software engineering.

They’ve started with a “Call for Action”:

Software engineering is gravely hampered today by immature practices. Specific problems include:

• The prevalence of fads more typical of fashion industry than of an engineering discipline.
• The lack of a sound, widely accepted theoretical basis.
• The huge number of methods and method variants, with differences little understood and artificially magnified.
• The lack of credible experimental evaluation and validation.
• The split between industry practice and academic research.

We support a process to refound software engineering based on a solid theory, proven principles and best practices that:

• Include a kernel of widely-agreed elements, extensible for specific uses
• Addresses both technology and people issues
• Are supported by industry, academia, researchers and users
• Support extension in the face of changing requirements and technology

and the vision statement (PDF), specifying what the community want to achieve in the next twelve months.

Despite agreeing with the above statements and the impressive expert lists who had signed the call for action, I’m quite sceptical about the results. Starting with the discussion whether software development is an engineering discipline or not, to the danger that this initiative will become just another fad. Anyway, interesting enough to keep an eye on it.

Several months ago, I promised to write about the IBM Rational Jazz platform and IBM Rational Team Concert. As you may have noticed, I have not yet write about them, but in my defense I can say that I have not had much time to devote to this nor other posts in this blog. As I mentioned in some others posts, lately I have been leading a major renovation of our entire suite of custom development tools, and these last 3 months I have been fairly busy managing all this change. Taking advantage of Easter holidays, I finally found the right time to get to write about the Jazz platform.

But before proceeding, a disclaimer. What I am going to write about Jazz is just a personal opinion, may or may not be wise, may or may not have something to do with reality, but I want to make clear that this is an entirely personal opinion, and do not mean any endorsement from my current employer.

The Eclipse vision

To describe the Jazz platform, I think we should go back to the past, because in my opinion, Jazz is trying to evolve the vision/mission/wildest dream that Lee Nackman had in 1998: to create a single technology platform on which to build the various IBM’s application development tools. The objectives Lee had at that time were:

1. to solve one of the most customer complaints: instead of having tools with their own “personality”, customers demanded a common look and feel;
2. to be able to integrate different tools, especially from IBM, but also from external ISV in order to complement IBM’s product line;
3. all reducing the development costs, as at that time each IBM tooling group used its own specific platform.

With the help from the autonomous OTI subsidiary (acquired by IBM in 1996), and overcoming an enormous amount of skepticism within IBM, Lee and his team delivered a technology platform that became what today is known as the Eclipse platform. Looking at the success of this platform, especially in terms of IBM adoption across the different brands and tools, it seems that the main objectives were reached. Not to mention also that open sourcing the platform and several projects (as the JDT), they killed lots of competitors, shrinking the Java tools market, and created a great ecosystem around it.

The Eclipse vision revisited

But the knowledge and tool set that IBM acquired when they bought Rational Software in 2002, mixed with a retrospective analysis they did based on the experience they gained in the Eclipse development, helped them to figure out which were the new challenges for the software delivery process. I’ll try to summarize, IMHO, some of the improvements they realize:

• When IBM built Eclipse, their focus was on the developer productivity. But the software development process usually involves some more skills, specialists, roles, levels, … and they need to work together, they need to share information, and when all members of the team work in a geographically dispersed manner, the conflicts inevitably appears. So there is a need to involve all the team in all the phases of the software lifecycle regardless of their location and role, and instead of improving the productivity of the developer, we need tools to improve the productivity of the entire team, and directly or indirectly, the productivity of the whole organization.
• When talking about covering the overall development cycle, we usually find that we need several tools, and sometimes these tools are outside of the scope of Eclipse. And we also find that there are lots of barriers to share resources between these heterogeneous tools, as they use private vocabularies, formats and stores. So the integration between these external tools are usually built on bridges, and lots of times, highly cobbled (so they require updates with every interface change). There is a need to raise the level of integration. We need to be able to integrate and share cross-repository information using open interfaces and a loosely coupled approach.
• When thinking about non coding activities, we realize that not each role or tool needs a heavy desktop client. There are some situations where a web UI (or another type of client) is more suitable. Despite some incubators (e4 Bespin or Eclifox), Eclipse nowadays only supports its desktop client. It’s true that using Equinox and its underlying OSGi services, you can deploy Eclipse plugins into the server-side, but there isn’t any “standard” way to share user interfaces or a framework for the web UI. Which will be the problem? the same Lee discovered in 1998: tools with their own “personality”, tools without a common behavior.
• Process, process and process. How much we love them and how much we hate them also? Why is so hard to try to follow a process? Why the only Eclipse tools available (RMC or EPF) only try to author and then publish a static document? Why tools doesn’t live the process?
• Creating a new Eclipse environment with all of the required plugins, configuring the project, setting-up the build process and all the other little pieces that come into play to give code life could be a mess for a new team member. This kind of manual tasks are tedious and error-prone, and they are perceived by developers as a waste time. So it is not strange that tools like Maven had a great adoption.

So in my opinion, and as I told previously, the Jazz platform is the evolution of the original Eclipse vision, keeping in mind the above and some more other concerns, with an special focus on teams and collaboration. But its aim is not to replace Eclipse, they are distinct platforms with different goals, although Jazz seems to be the perfect complement to Eclipse. This new vision is well summarized at the About Jazz page:

Our goal is to provide a frictionless work environment that helps teams collaborate, innovate, and create great software. To that end, we are focusing on driving fundamental improvements in team collaboration, automation, and reporting across the software lifecycle.

The Jazz Platform

When trying to describe what compose the Jazz platform, albeit IBM have split the original Jazz project in several projects at the jazz.net site, I still have some problems trying to draw the line between the platform and the applications, to see which components are part of the Jazz platform and which are part of the different products based on it. So I will try to use the below picture, that I have borrowed from the IBM Rational guys, in order to clarify my ideas:

To enable a seamless and higher level of integration between tools, IBM has defined a reference architecture, API specifications, and a set of common services and tool building blocks, that together are called the Jazz Integration Architecture (JIA). At the center of this integration architecture we found the Jazz Team Server (that may consist of one or more physical servers that act together as a single logical server), which provides foundational services (RESTful web services) to enable groups of tools to work together. Let’s summarize each of these foundational services:

• Presentation: in a multi-tool integration scenario we usually found lots of linked resources that may not be familiar to a particular client tool or this tool may not be able to provide a user interface for theses resources. The services provided by the presentation foundational services enables a client tool to find and invoke a suitable user interface for any resource URL in order to present the relevant data. There are also two main components (I believe, but I am not sure, they belong to the presentation services) that allow tools developers to implement specific user interfaces: the web dashboards component, that provides the infrastructure and UI for creating and presenting dashboards in a web browser, and the web UI component, that provides a framework for rich web user interfaces (based on the Dojo toolkit).
• Process enactment: these are the services that allows to define and implement a wide range of processes. It is focused on agile processes, but it can also be used in highly-structured processes, as it provides the essential components of a development work flow, such as operations, roles, permissions, preconditions or follow-up actions. By default, it is packaged with several process, as Scrum, OpenUp or the Eclipse Way (PDF), and it has an editor to be able to modify the process configuration. Each time you create a project you must assign a process, but you can have several projects and each project can follow a different process. It governs all activities, artifacts, artifact relationships, and operations that are pursued within the context of the process area, and it works in a seamless and unobtrusive way, as it manifests itself through artifacts types, operations manipulating the artifacts, and artifact change events.
• Administration, users, projects, teams: For dealing with users, projects, security, and licenses, each server hosts a set of core administration services. For example, these services can provide a common user identity in order to support authentication (establishes user identity) and authorization (a particular operation can be performed) based on the team membership or role in a project.
• Collaboration: Collaboration between the team members of a project can be performed in real-time, but also asynchronously (especially important for teams working across time zones). It also occurs at different contexts: around tools, process, tasks or data elements. The collaboration services in the Jazz platform supports and enables some of these core functions, for example, instant messaging, sending email and SMS, maintaining subscriptions, etc. It is something like a mix of the Eclipse ECF and Corona projects (and I wonder why they did not use these projects).
• Storage, data warehouse and search: You may have noticed that I have deliberately grouped 3 core foundation services in only one. The reason is because the Jazz repository model is composed by three logical DB in one, working together in order to provide the above 3 services. I am going to use another “stolen” picture to describe it:
  
  
  • Instead of having a fixed schema (that make integration hard) or a very generic schema (that makes writing tools tough), the Jazz repository allows tools to store their data their way. So content resources are created in a particular representation by the client, and can only be retrieved in that representation. The server doesn’t know enough about the content to transform it into an alternate representation. The storage services provides a completely RESTful framework for CRUD operations on resources stored in the Jazz private DB.
  • For every resource stored in the private DB, there are a set of “indexed properties” that are stored automatically in another DB using RDF. The indexing process is able to extract RDF triples from some resource representations and able to extract text streams from some resource representations as well. This process extracts asynchronously each tool’s data into searchable indexes, consolidates them, and provides centralized query services for searching across the consolidated index. In this way, the search foundation services are able to provide common queries, both structured queries (based on SPARQL) and full text search (based on Apache Lucene).
  • And finally, we find the data warehouse DB, a periodically snapshot of all the information, used for public reporting. The data warehouse services relies on the Eclipse BIRT project for its reporting system.

Anyway, the Jazz platform is still in its early stages, and it is constantly evolving to meet additional challenges. What I have summarized previously is what it is know as the Jazz Platform 0.6, but a new version is expected to be delivered on June with a new name, the Jazz Foundation. So if you are interested in more deep details about the above or new services that are going to be delivered, I recommend that you go through the development team wiki (registration required).

The killer-application

There is also an interesting parallelism between how Eclipse and Jazz has been developed. In order to convince other IBM’s development tools product managers to adopt the Eclipse platform, Lee and his team decided to build a Java IDE. There were two reasons behind that decision: 1) to provide a real example (a killer-application) of a tool developed on the platform, proving in that way its benefits; 2) to help the Eclipse development team to better understand the needs of future consumers of the platform and to discover areas that required further development. This strategy was a success and the Eclipse platform and the Java IDE were quickly adopted inside IBM.

The Jazz project seems to use the same kind of strategy. They are developing a real tool using the Jazz platform. This killer-application is called Rational Team Concert and as far as I know it is also rapidly adopted inside IBM. I hope to write about this product in the near future.

Adoption

Looking at the jazz.net site the increasing number of IBM tools that are adopting the Jazz platform, I have no doubt that it will be another success in terms of IBM adoption. But …

• Will it be a success outside IBM as was the Eclipse platform? IBM has not contributed the Jazz platform to the open source world (in terms of a free software license), and nor it is licensing it in any way (as far as I know). The only way to get this platform is licensing some of the Jazz based products. I am sure they are going to attract some more new customers looking for a complete lifecycle solution, but I believe it will not be a great success as Eclipse was. Anyway, I think the current goal for the Jazz platform is different from the Eclipse platform goal.
• Are ISVs going to adopt this platform for their own products? There are the usual business partners that are complementing/extending the IBM’s Jazz based products with some new features, but it does not seem that they are going to adopt the Jazz platform for their own products. And it does not seem that IBM is trying to convince them to adopt the platform, as they did with Eclipse, or licensing it on an OEMs basis.
• Will this platform attract external developers? IBM is not encouraging them to contribute in terms of code (the rights of what you contribute are transfered to IBM). They are only encouraging people to influence the direction of products through direct, early, and continuous conversations at the jazz.net website. So it will be very strange to see any non-customer developer.

So in terms of external adoption/extension, it seems that IBM is focusing only in the interface as a way to integrate non-IBM tools, encouraging developers, customers and ISVs to participate in the development of the Open Services for Lifecycle Collaboration initiative, something like an open standards consortium.

Conclusion

Starting looking at the original Eclipse vision and how IBM revisited it after the Rational Software acquisition and the Eclipse success, I have attempted in this post to describe what it is the Jazz platform. I am sure some of you have realized that some of the improvements that I have described previously can be easily or are already implemented through Eclipse plugins, but I think these are the minor ones. There are three main conceptual differences between Eclipse and Jazz:

• a server centric approach instead of a local workbench in order to leverage the team concept;
• a persistent storage using a federated cross-linked repository to store resources;
• a seamless integration between tools using standard loosely coupled open interfaces and web protocols.

And in my opinion, these conceptual differences can only be implemented through the creation of a new platform. Instead of solving some particular problems in an isolated way, Jazz is trying to attack the essence of the software development process. Does it means that we must convert to a new religion, drop the Eclipse platform and adopt the Jazz platform? No, Jazz is not going to replace Eclipse. There will be a strong relationship between the Jazz and Eclipse environments, yet the
two are distinct and can run independently. Jazz is going to complement Eclipse in some particular scenarios:

Eclipse for Individuals, Jazz for Teams

We have also seen that IBM’s decision is not to open source this new platform, but to create a community around the jazz.net and OSLC websites. However, it will need to attract a broad and active participation from a wide external community in order to be a great success outside IBM, as it was Eclipse, something that I believe it is not a current IBM goal.

What will happen in the future? Sincerely, I don’t know, time will decide. From my particular point of view, this new vision match up what we have been doing for a long time and that has lead us to extend the Eclipse platform in order to create a custom collaborative tool set. So, as we are already an IBM Rational customer and we have licensed some of the Jazz products, I will be very happy if we can integrate, in an easy way, our custom tools with the IBM Rational tool set.

If you reached this point, please, participate in the conversation

Before concluding this long post, I would like to ask you:

• To the IBM/Rational guys: As I assume that I could make mistakes (history, goals, …) , if you want to add or point out something wrong, please write me, better as a comment in this post although I will also accept private emails, and I will correct it.
• To the non IBM/Rational guys:
  • How much of you have heard about Jazz? How much of you have experimented with it? Which is your (technological) opinion?
  • Must IBM open source the Jazz platform? Do you think it will be interesting and wide adopted? Why?

Additional information

PS: One of the latest tasks Lee Nackman did before his retirement at IBM, was to help spur the development of the Jazz platform.

PS: Most of the people actually involved in the development of the Jazz project were part of the Eclipse platform development team, so it is not strange to see that they are applying the same strategy, but also adopting the best practices they learned during the Eclipse development process.

PS: There is an excellent case study on IBM’s strategy and process for creating Eclipse at the Harvard Business School. It’s a worth read.

PS: Another interesting read is a paper on Collaborative Development Environments (PDF) by Grady Booch and Alan W. Brown, which seems to be the “spark” that started the new vision.