“Collaborating with others is at least as important as having great technical skills”. Great Goggle I/O 2011 video:

Interesting initiative to revitalize the software engineering discipline. SEMAT (Software Engineering Method and Theory), launched in October’09 by Ivar Jacobson, Bertrand Meyer and Richard Soley, is trying to recognize the fundamental problems and to develop a sound and general theory for software engineering.

They’ve started with a “Call for Action”:

Software engineering is gravely hampered today by immature practices. Specific problems include:

• The prevalence of fads more typical of fashion industry than of an engineering discipline.
• The lack of a sound, widely accepted theoretical basis.
• The huge number of methods and method variants, with differences little understood and artificially magnified.
• The lack of credible experimental evaluation and validation.
• The split between industry practice and academic research.

We support a process to refound software engineering based on a solid theory, proven principles and best practices that:

• Include a kernel of widely-agreed elements, extensible for specific uses
• Addresses both technology and people issues
• Are supported by industry, academia, researchers and users
• Support extension in the face of changing requirements and technology

and the vision statement (PDF), specifying what the community want to achieve in the next twelve months.

Despite agreeing with the above statements and the impressive expert lists who had signed the call for action, I’m quite sceptical about the results. Starting with the discussion whether software development is an engineering discipline or not, to the danger that this initiative will become just another fad. Anyway, interesting enough to keep an eye on it.

As I explained in a previous post, the last year I have been involved in a renewal process of all of our application development tools. One of the first things we did when we started the program was to apply the most fundamental lean principle: eliminate waste. To lean thinking, waste is anything that does not create value for a customer. For those of you who are not familiarized with the lean principles, I recommend the “Lean Software Development: An Agile Toolkit” book. In this book, Mary and Tom Poppendieck translated the seven wastes of manufacturing identified in the Toyota Production System into the seven wastes of software development: Partially Done Work, Extra Processes, Extra Features, Task Switching, Waiting, Motion and Defects. In this entry I would like to explain some of the problems we have encountered while trying to eliminate waste and some lessons learned.

The first waste we tried to eliminate was the extra processes. In other word, we tried to eliminate paperwork that does not means adding value for our users or for our organization. In our case, this task was not related to the development of our tools, it was about eliminating extra processes that were embedded in the tools we developed which forced our users (developers) to execute some unnecessary processes. This action produced some surprises, since near the end of the development, in one of the latest functional demos, there was a crisis moment. Some developers reminded us an essential process they were following in the old tool: they defined the batch programs in a product repository. This process was removed deliberately, because it does not provide any value, so it was a surprise for us that our users asked for this. When we asked them why this process is necessary, they answered that they did not know, but it was something they used to do because someone told them that they must do this task. “Is this useful?” we replied. “No, but we must continue doing that because … we must do that”. Wow, it remembers me the monkey experiment. Obviously, and despite the laments of our users, we did not add again that process. So lesson learned: in order to eliminate waste you need to break the status quo, you need to break the corporate culture.

The second waste we tried to eliminate was the extra features, because this was one of the biggest mistakes we did in the past in other tools. Some years ago we started developing a new tool and, using the usability argument, we added a lot of features that lately nobody used. Some of the hitches you may suffer adding those features are that your code-base grows uselessly, increases maintenance costs and makes future developments more complex. This must not be a big problem if users really need these features, but why must we maintain them if they are not being used? Users, moreover, feel that the tool is more complex, so your usability argument disappears, and they reproached us that we are not focusing on what it is really important for them. Now, I am proud to say that our tools have less features, but, at least, the ones we implemented are really used. So two more lessons learned: 1) more features does not mean better tools; 2) usability does not mean more features.

I am going to stop here. I am sure most of you who have tried to eliminate waste have found these or similar problems. But I do not want to conclude this entry without explaining one of my latest lessons learned. It is not strictly related to lean thinking, it applies to software development in general, albeit it could only apply to some organizations. Sometimes, I believe it is better not to explain that you are using an “x” methodology, or to intensify your position saying that you learned those practices from whatever methodology. Sound strange, isn’t it? But I have discovered that lots of developers hate the words “methodology” and “process”, and they have adverse reactions when they hear them. I find easier to explain practices without any reference to the original methodology. Lots of times, using the common sense is better to prove the goodness of a practice. And if it does not sound good, perhaps it does not match your organization. OK, maybe I am generalizing. In every change process you will find resistance, so perhaps if it does not sound good it is because fear. So, my last lesson learned: use common sense, do not arbitrarily adopt new practices.

Recently, I have been involved in a major IT program, framed on a high demanding business strategic plan, which aims to renovate all of our core banking system. One of the program’s first steps was changing our IT organizational and governance structure, the enterprise architecture, the application development tools and the software development methodology.

Although my responsibility in this program only relies on the application development tools, as one of the PMO leaders, I was able to follow closely the rest of the items. One of the topics that I was specially interested on was the software development methodology, because among other things, before the program, it was one of my responsibilities, and, after the program, it will be, again, my responsibility. The truth is that I had high hopes for change (perhaps influenced by the “Yes, we can!” slogan), but the fact that they came to conclusion that we need another waterfall methodology, perhaps a bit stricter than the one we use today, disappointed and frustrated me.

But please, that nobody misunderstood me. I deeply respect the work and decisions of my colleagues. I have had the opportunity to explain my thoughts. I gave them some books on the topic. I tried to influence the people who had the task to define the new methodology in order to introduce more innovative methods/process/practices of software development, but, maybe, being too innovative in a very classic environment doesn’t helped me. I think that changing the software development process in a big organization requires lots of effort, education, very slow and gradual steps, …, and I do not want to miss this opportunity, I believe that now is the moment to do that, taking advantage of the whole changing program. Well, at least, now some people knows that there are more life after the waterfall model, there’s a gray scale between white and black.

As I’m a bit nonconformist, I tried to find out which were the reasons behind that decision. So I decided to ask some developers and project managers to get an exact idea of what they think/know about software development methodologies. Although most of the answers were what I expected to find, I also found some interesting frustrating observations. I want to share with you some of them:

• Most developers and project managers are unaware of the existence of another methodologies, and they do not have any interest in learning them. Methodologies and process are something bored and do not provide any value to what they are doing actually. Although they recognize there are lot of inefficiencies in the way they work, they don’t want to change it. When I ask some of them if they follow our actual methodology, their answer is NO. “Well, at least, are you following any predefined process?” again, the answer is NO.
• Most project managers feel that a plan-driven methodology provides them more control over the whole project, that their projects are more predictable. But when I ask them if their projects are on time, most of them recognizes that NO. They also doesn’t have/use information from previous projects in order to estimate or improve the next ones, every project is different.
• They see iterative / agility methodologies as a chaos, the wild west, where there isn’t any discipline. W00t? discipline and agile are not conflicting. XP, for example, requires high levels of discipline.
• Some people told me that waterfall methodologies encourages a comprehensive documentation. “Well, could you show me your last project’s documentation? no, we didn’t have time to write it. OK, doesn’t mind. Could you show me any documentation of any project you’re involved? no, we’re still working on it, you know, we’ve tigh schedules”.
• Some answers reflected the “We’re Special” syndrome: “Agile is suitable when you want to develop software for an Iphone, but not for financial applications” (paraphrased).
• Some project managers get annoyed when they must talk with their clients or stakeholders, they hate them (I believe this is a mutual feeling). Yeah, those evil people that everyday changes the requirements and doesn’t have any idea how hard is the software development process.

I also tried to analyze some projects, and, surprisingly, I discovered that most of them are short term projects (2-3 months). I expected longer projects, as corresponds to a waterfall process. So I ask myself if we are really using a waterfall methodology, or we’re using a masked iterative process?

How about you? Did you find these kind of answers in your company? Any ideas on how to address this situation?

Don’t miss the No Silver Bullet Reloaded Retrospective OOPSLA Panel Summary post by InfoQ. It’s a worth read. Great panelists and great content.

Some statements that I’ve found interesting:

Dave Parnas:

Desire for people to seek better tools rather than “actually learning the trade.” Dave used the metaphor: “there is an old saying: ‘the poor workman blames his tools’ – people are poor workmen, I think most programmers are poor workmen.” Dave also said that people are looking for silver bullets to avoid learning the trade.

Linda Northrop:

We still need great designers, and I think we still have far too few, and we still need to cultivate an atmosphere of hard work but also an inner-disciplinary perspective that takes us uncomfortably out of our coding world.

Dave Thomas:

He called today’s state of the art in software development (especially middleware) a gratuitous disaster due to overly quick fast pace of change in API’s and frameworks which create immature software and also create too much difficulty for average software teams to track and stay up to date with. Dave claimed that the majority of today’s enterprise systems are basically just CRUD applications: “In the end these people are trying to do things that are fairly straight forward, and if they were working on a mainframe using a 4GL, they would actually have the thing done.“

He has seen “real successes” in niches, citing highly specialized domain-specific and specialized language applications in airlines reservations and hedge funds;

Martin “Werewolf” Fowler:

You can’t measure your output, you can’t measure your productivity, no way you can run sensible experiments to figure out whether one technique is better than another. …

The biggest difficulty in software development is the communication between the people writing the software and the people for who the software is being written. The conceptual attacks, the essence problems, focus a great deal on trying to improve that communication. The trouble of doing that, of course, is that it’s hard to get that communication flowing. Some of the business people don’t want to talk with software people, and fortunately the software people are hopeless when it comes to social interaction, they can’t talk to business people.

You seem to think that with hardly any information at the beginning of the project you can map out exactly what’s going to happen, pin down costs, commit yourselves to all sorts of unrealistic plans. And then you wonder why it is that I always manage to turn up. That illusion of control, the idea that on basis of hardly any data you can let make these grand predictions, that very folly is one of my greatest strengths.

Fred Brooks:

I know of no field of engineering where people do less study of each other’s work, where they do less study of precedents, where they do less study of classical models. I think that that is a very dangerous thing.

Jurgen Appelo has collected a list of the top 100 blogs for development managers. The list of blogs is quite impressive, rated using five different statistics (Google Page Rank, Technorati Authority, Alexa Rank, Hits, Comments). You can find the usual A-List bloggers, but also some other interesting blogs you might not know about (oh no! no more feeds please!).

This is the top 10. Check his post to discover the rest.

1. Joel on Software – Joel Spolsky
2. Coding Horror – Jeff Atwood
3. Seth’s Blog – Seth Godin
4. Paul Graham: Essays – Paul Graham
5. blog.pmarca.com – Marc Andreesen
6. Rough Type – Nicholas Carr
7. Scott Hanselman’s Computer Zen – Scott Hanselman
8. Martin Fowler’s Bliki – Martin Fowler
9. Rands in Repose – Michael Lopp
10. Stevey’s Blog Rants – Steve Yegge

Bonus track: It seems that Jurgen’s hobby of collecting lists comes from afar. Check these other lists he has collected:

• Top 100 Best Software Engineering Books, Ever
• Top 20 Best Agile Development Books, Ever
• Top 25 Most Popular Software Engineers

I’m back from vacation … just to discover again the cruel reality:

Some days ago, I wrote about a QCon 2008 session where Erich Gamma spoke about transparency and how it is related to the IBM’s Open Commercial Development model (OCD). As I have been involved since last year as a beta customer in one of the projects where OCD is been applied, the Rational Jazz Project, I want to discuss with all of you some thoughts and opinions about this software development style.

First of all, and just to be clear, I’m not going to talk about which is the license model behind OCD. The words “Open” and “Commercial”, when seen together, produces some unexpected chills and thrills, and they have generated some controversial discussions out there, mainly in the open source community (maybe, as Stephen O’Grady points out, it will be more accurate to characterize this as transparent development). In the case of the Jazz Platform, there is also some confusion, because we don’t know if IBM is going to release it as an open source software and then develop commercial products based on this platform (like Eclipse), or if they, both platform and products, will remain as commercial products. I vote for the first option, but the word “commercial” in OCD suggest me the second one.

Instead, what I’m going to talk about is how transparency in a great feature in the software development process. So go ahead.

During my career, I have dealt with lots of products, from both open and non-open source software providers. One pattern that I always find in traditional proprietary software is that you never interact with the development team; there is a barrier between you, the customer, and the vendor’s developers. If you need to report a bug or to ask for an enhancement, you can only interact with a service desk. Usually, they have a support web site, where you can see your own tickets, but you can’t see any other bugs or enhancements reported by others companies. You never know when they are going to deliver a solution for your problem (except if it is a blocker), which will be the way they are going to implement your enhancement, if there are more people interested in some enhancement, … You can only check if the bug is fixed or the enhancement implemented when the vendor delivers a new version of the product, and, sometimes, results are not what you wanted. Furthermore, sometimes, you will have to deal with lots of useless questions, mainly due to misunderstandings between you, the service desk and the development team. This is what Erich Gamma calls “Swiss bank approach to software development”.

This firewall between customers and developers is really very frustrating, not only for the customers, but also for the developers. Some companies try to supply this lack of communication organizing user’s conferences (where you can meet some developers), meetings with whatever worldwide VPs, or through a customer advocate. In some cases, frustrated users set up unofficial forums to share their problems or to try to join forces so that the vendor accepts an enhancement. They try to establish some kind of user’s community, but without the vendor involvement. In my experience, and without intention to offend anyone, you will get lots of nice words, but you rarely archive a real solution.

With open-source products, there is a really different way of relationship between customers and providers. There is an open participation and customers can influence easily in the development process in several ways. And I’m not talking about having access to the source code, which is important, but also having access to the bug tracking system, the development mailing-lists, user’s forums, and, in some cases, the development plans, all of them maintained by the development team (fewer misunderstandings). This well-known transparent and collaborative model usually produces enhanced feedback, which leads to deliver better products (in terms of user’s expectations). It’s about archiving customer value, instead of vendor value.

This is the same interaction I have found while working with the Jazz Project. I have had access not only to the source code but also to the latest integration builds (so I can check how my enhancements are implemented), a wiki with technical information about the platform (if you care about the extensibility), a community forum, the development plans and a dashboard to monitor the health of the overall project. I have had also full access to the issue tracking system, where I can submit my own bugs and enhancements, but I can also see which other enhancements are requested by other customers (and to enroll them if I found someone interesting), in which version are they planned … Summarizing, full transparency in the development process. One of the consequences is that I’ve felt, and this is personal and subjective opinion, more involved in the development process, more biased towards submitting enhancements and more confident about future problems that could appear. This feeling is nothing new if you have previously worked with open source projects, but it is something strange coming from an proprietary product.

However, there is a huge difference between open source and OCD models. While in open source software you can contribute to the code base by fixing a bug or improving some features, in OCD it is not clear which will be the contributor’s role. It will depend on the license model selected, which, in turn, it will establish if there will be a vibrant community and ecosystem outside IBM or if there will be a vibrant IBM’s customer community. Anyway, if the final decision is to keep the software as a commercial product, the transparency applied in this model it will represent a great improvement in the proprietary software development process, for both customers and vendors.

But after using this software development style for some time, I believe that this model it is not useful only to software vendors, but it also could be applied to any IT department, especially in big enterprises, in order to improve their software development process. I also believe this is one of the objectives pursued by Rational Team Concert, the first product based on the Jazz Platform (I will talk about this product in future posts). But this is something that I need to try by myself in the company where I work. As most of my readers already knows, I am in charge of the application developments tools in one major Spanish savings bank. Looking through the development process we use, I am sure that if I ask the users of the tools we develop (our internal customers) how much transparent are we, they are going to complain me. Although we meet periodically, it is hard to achieve transparency only with meetings, we need to adopt different and innovative approaches, the ones I have told you in this post.

And finally, although this is what I honestly think about transparency and OCD, I want to hear other opinions. Do you think this is a good software development model for proprietary software? Do you think this model could be applied inside enterprise firewalls?

Over the course of my career working as a software engineer, I have had the opportunity to play with different data structures (arrays, vectors, trees, …) and different persistence data systems, from hierarchical databases (IMS DB) to indexed and relative files (VSAM) to RDBMS. Even so, I consider myself as an apprentice, and I still have the insane curiosity to learn new persistence data models. That’s the reason why, recently, I have been playing with CouchDb.

For those of you who do not know CouchDb, it is a distributed, fast, reliable, fault-tolerant and schema-free document-oriented database accessible via a RESTful JSON API. It has been getting a lot of attention lately, specially after Amazon launched its cloud database system, SimpleDB, which has some similarities with CouchDb, and after IBM hired its creator, Damien Katz, that caused the donation and acceptance of CouchDb as an incubation project by the Apache Software Foundation. Despite this hype, it is still an alpha level software, it’s stable enough for testing but not ready for a production environment.

The first interesting feature about CouchDb is that, unlike typical relational systems, it will not enforce you to use an schema. Instead of designing tables, CouchDb stores in a flat file, called database, documents, which are lists of key-value pairs, called fields, with no predefined structure. If you want to think in a relational way, databases are tables, documents are rows, and fields are columns, but in the same database (table), you could have different documents (rows) and each row could have any number of different fields (columns) without the need to structure their data types (field = dynamic key + value). So you don’t need to structure your data in advance, you can do it later if you want using views, which will organize your data adding structure back to semi-structured data. It’s a “Data First vs. Structure First” strategy.

Here it is an example of a document (using the built in web front end):

This lack of schema, however, is not a suite-for-all solution, but it could be advantageous in some situations. I’ll give you a simple example. Imagine you need to create a repository of information related to all the software components of your information system (something like a CMDB), that is programs (or classes), databases, configuration files, web pages, … Components has common attributes and specific attributes depending on the type of product. For example, common attributes are type of product, name, description, … and specific attributes for a program are language, technology used (DB2, MQSeries, …) … You will have several approaches to store these information:

1. Design a relational table for each type of product with all the attributes (common and specific);
2. Use only one table with all the common attributes plus some generic attributes (for example, attrib1 varchar(255), …), and an external data dictionary that identifies and describes the format of each generic atribute;
3. No predefined structure, each row stores different attributes depending on the type of product.

From a pure relational perspective, it seems natural to use the first option. But what happens if you’re a tools vendor and you let the users add they own products? You still have the choice to use the first option, but the problem is a bit more complicated, as you’ll have to deal with some unknown tables (and, perhaps, not normalized). OK, no problema here, we’ll just use the second alternative. It’s a perfect choice (in fact, I’ve used this approach occasionally), but doesn’t seem very natural to the relational discipline. And here is when it comes the third approach, each row will have different columns (as a key+value pair) representing the product attributes. In summary, all three options are correct (and yes, I’m polite because I don’t want a flame war now), but in this case, it’s more advantageous to use the schema-free approach.

Another interesting feature is that CouchDb uses the Erlang OTP platform (internally, you’ll never see it), which is a perfect choice for highly concurrent access and fault-tolerant systems. But being highly concurrent means that you must try to avoid locks, something hard to design if you want to enforce all ACID properties. To deal with this situation, CouchDb uses a MVCC and a lockless optimistic writer mechanism (in comparison to Mnesia, Erlang’s DBMS, which uses a pessimistic locking). Using these approaches, readers could access documents without being locked by a writer, seeing only the version of the document that existed when he began reading (despite a writer updated the document at the meantime), because data is always versioned by the writers. But this strategy involves big database files, because they are growing all the time (remember, writers never overwrites data, they only append it, even if you’re deleting data). And even though CouchDb has an automatically compaction process that clones all the data without any service interruption, it would be nice to see some performance metrics to check which will be the performance parsing large files in Erlang.

The last interesting feature is its simple REST API to manage the data, using JSON for data transport. It is very easy to create, read, update or delete documents and views, that have a unique URI, using HTTP resources. Yes, and that means no painful database drivers, just regular HTTP connections, so you can use your favourite HTTP library. Let’s see some examples using cURL:

Getting a list of all databases on my local CouchDb server:

curl -i http://localhost:5984/_all_dbs

And the response:

HTTP/1.1 200 OK
Server: inets/develop
Date: Sat, 16 Feb 2008 23:57:32 GMT
Cache-Control: no-cache
Pragma: no-cache
Expires: Sat, 16 Feb 2008 23:57:32 GMT
Transfer-Encoding: chunked
Content-Type: text/plain;charset=utf-8

["ferdydb","test_suite_db","test_suite_db_a","test_suite_db_b"]

Retrieving a document:

curl -i http://localhost:5984/ferdydb/post-272/

And the response. Remember that documents are simple JSON objects:

HTTP/1.1 200 OK
Server: inets/develop
Date: Sat, 16 Feb 2008 23:58:25 GMT
Cache-Control: no-cache
Pragma: no-cache
Expires: Sat, 16 Feb 2008 23:58:25 GMT
Transfer-Encoding: chunked
Content-Type: text/plain;charset=utf-8
Etag: 1882588955

{
 "_id":"post-272",
 "_rev":"1882588955",
 "Type":"post",
 "Status":"publish",
 "Title":"Playing with CouchDb",
 "Author":"Ferdy",
 "PostDate":"2008-02-17T00:30:52+01:00",
 "Tags":["couchdb","erlang","json","rest","schema-free","simpledb"],
 "Contents":"<p>Over the course of my career working as a softw..."
}

Did you see the Etag header? Let’s see if it works:

curl -i http://localhost:5984/ferdydb/post-272/  \
     -H "If-None-Match: "1882588955""

It works! Here is the response:

HTTP/1.1 304 Not Modified
Server: inets/develop
Content-Type: text/html
Date: Sun, 17 Feb 2008 00:26:26 GMT
Cache-Control: no-cache
Pragma: no-cache
Expires: Sun, 17 Feb 2008 00:26:26 GMT
Etag: 1882588955

Enough experiments for today, right?

After reading this brief summary, and if you come from a relational model discipline, your first reaction may be “WTF“. But I would encourage you to go deeper with this new persistence data model approach checking the technical overview page at the CouchDb Documentation Wiki or reading these slides by Jan Lehnardt (don’t forget the comments also). You’ll find a lot more features that I didn’t explain in this post. And of course, share with me your opinion!

The IT Metrics & Productivity Institute is hosting a series of free, weekly webinars about software development and maintenance.

The Software Best Practices Webinars Series is dedicated to improving the practice and management of software development and maintenance world wide. All live webinars are FREE and have been accredited with PDU credits by PMI’s ISSIG group.

Each webinar will feature an expert speaker who has extensively researched and successfully applied best practice principles to the development and maintenance of software.

The Fall 2007 program includes:

• November 20th, 2007: The Journey towards Effective Portfolio Management
• November 27th, 2007: Ed Yourdon On Managing Death March Projects
• November 28th, 2007: Implementing Agile Development in New Teams
• December 3rd, 2007: Dr. Howard Rubin on Technology Economics and the Importance of Software Process
• December 3rd, 2007: Effective Vendor Management
• December 4th, 2007: 10 Things You Need to Know about Earned Value Management
• December 11th, 2007: Project Disasters: Predict Them, Prevent Them, or Pull the Plug on Them
• December 12th, 2007: Building a High Performance Risk Management Team
• December 13th, 2007: Requirements Errors – Sources and Avoidance Strategies

If you are interested in knowing the full 2007-2008 program, you will find more information on The 2007-2008 Software Best Practices Webinar Series website.

(Via Ed Yourdon)