Steve Cook:

Cameron has been blogging about new features in our product.. In a recent post he used the term Explicit Design. I’ve been reflecting on this, and I like it. In software development we really do need to capture design data that is not just the code, but should be saved and versioned just like the code. What do we call it? We could call it “models” but “model” and “model driven development” are subject to so much historical baggage and methodology and terminology arguments. “Model” just seems to imply baked-in code generation and round tripping, when there is so much more that you can do with it: planning, verifying, testing, refactoring. We need new vocabulary that represents our ability to capture versioned design data at a more abstract level than the code without simultaneously implying the history of CASE.

I have to agree: changing the name doesn’t solve the root of the problem, but perhaps we start thinking in a different way.

A recent post from Markus Voelter replying some articles from Martin Fowler has resulted in an excellent discussion about repositories for MDSD and textual vs graphical DSLs. I was thinking about joining the discussion in the original post, but as my answer is a bit long to be added as a comment, I’ll reply him here.

In his post, Markus raised a question that recurrently appears in my thoughts: where is it better to store models? as files in a VCS (like CVS/SVN) or as structured data in databases?

Unlike Markus, my first answer is usually: without any doubt, in databases. I suppose that this answer has much to do with my professional background, since in the past I have worked extensively with CASE tools. Yes, I believe I’m one of these people that M. Fowler describes in his post about MDSD:

The MDSD vision evolved from the development of graphical design notations and CASE tools. Proponents of these techniques saw graphical design notations as a way to raise the abstraction level above programming languages – thus improving development productivity. While these techniques and tools never caught on too far, the basic core ideas still live on and there is an ongoing community of people still developing them.

But also, it is because I’m used to working with very large repositories, where there are lots of relationships between models. If you work with lots of components but few relationships, you usually do not have problems working with local files (such as Eclipse does). VCSs are able to deal quite well large amount of files. But what happens when you have, for example, your entire transactional system modeled (> 30,000 components), with a very high index of reusability between models (lots of links), and each model can belong to a different owner (only the owner can modify it)? As Markus points out at the comments:

Also, text files tend to be a bit hard to scale. Often the minimum you need is some kind of “cross-indexer” via a database so you can efficiently cross-ref, search, etc. In a “real repository” that’s easier.

Consider the Xtext case. What do you do once you have hundreds of Xtext resources? Each linking into each other. How do you efficiently load, unload, search, find-refs, etc? You need some kind of (in memory or persistent) index.

Exactly. When you work with huge amount of data and links, you need to provide some kind of impact analysis or cross-reference functionalities, you must be able to do complex queries, you must version not only the components but also the relationships, and you must be able to link to other models without the need to download them locally. Yes, I know that there are some solutions out there that provide some of those facilities also for files, as text search engine libraries (Apache Lucene) and query languages (for example, XQuery for XML or SPARQL for RDF). But IMHO all of these solutions, although they work very well with few components, are not as powerful as what you can get *for free* using relational databases.

But I must also say that my opinion has changed somewhat over the years, due to my experience working with repositories. The approach of working with databases also has some problems. One of them is that in order to work with the tool you always must be connected to the database, and, although this situation sounds silly, this may limit the productivity of some developers. With VCS, you only need to be connected while you perform the checkout of the component, but after that, you can work locally with it. Another pain point is that in relational databases, you must to create a fixed schema (no matter if you use a metamodel, you always must create it), and that could be a mess when you need to modify the data structure, since RDBMS doesn’t provide schema versioning facilities. Fortunately, some new approaches has appeared in the market in the last years, as schema-free databases, that will help in this task. Another side effect is that if you want to preserve the integrity of the models and the relationships, you have to deal with locking mechanisms, so the scenario become worse, and usually, the system tends to be over-engineered. And finally, there are also some functionalities not provided by databases, as versioning, accountability (who, when and what) and in some cases traceability (why), so you must develop it by yourself (yay! we love to reinvent the wheel!). Almost all VCSs provide these facilities.

Let’s go back again to the original post. Markus talks about some conditions where repositories could fit well in this scenario:

My point is that a repository is not per se a bad thing, provided the following criteria: (1) you store all your relevant stuff in it (2) it provides versioning facilities (3) supports diff/merge on a meaningful abstraction level.

Ummm, I agree with almost everything, but I’ve some concerns:

1. Not sure, if he talks about storing all of the data that belongs to a model in the repository, then I agree. But if he talks about storing the model and the code together, then I disagree. There are some scenarios where this is not convenient. For example, when you want to be platform independent (and I’m not talking about all the MDA stuff). The various parser/generator/interpret could run and store the code/binary on several platforms, and not always the same platform where you store the model.
2. I agree, versioning is an essential facility.
3. Diff/Merge works well with textual DSL’s, with a concrete syntax. But, although this is a great feature, is it mandatory? I have worked a long time without this feature and I assure you that you can survive without it. And what happens with graphical DSLs?

Before concluding, I would also like to comment one of the latest projects where we applied MDSD. At this project, we decided to use a Oracle XML DB to store our models in XML (something like to what Eurocontrol-CFMU have done for their UML models), but we added also some metadata. By storing the XML directly in the database, we avoid the need to decompose the XML into a relational schema, and allow developers to download the XML and work locally without the need to be connected to the database. We could use also all the SQL query facilities, and for those situations where the performance could be a problem, then we use the metadata to store some relevant data and relationships. Oh, and this RDBMS provides us also with versioning facilities. At this moment, we don’t have enough data in the repository to tell you if this approach will be a success or not. Let’s see!

To sum up (or not!). I believe you should never reject the database approach (nor the VCS option). I can not give you a “Golden Rule”, but my advice is that if you are not going to have lots of relationships between models, then use the VCS approach. If not, then analyze first what a VCS approach could offer you, and if it doesn’t fit well with your requirements, then use the database approach. But please, be careful and don’t tend to design the metamodel too much complicated, or you could have lots of performance problems.

As the post is quite long, I will leave for another post my thoughts about textual vs graphical DSLs. In the meantime, what is your opinion? I would love to hear stories from other folks on what people are doing in their companies.

Well, I just finished the work I have been involved during the last month. The task consisted mainly to assemble a RFP document to renew all of our modeling and automatic code generation toolset. Next Monday, we will introduce to some selected providers which are the project objectives and which are the requirements.

Advancing some details, I will tell you that in a first stage we will start designing and developing an infrastructure where all the development tools will rely on. We will also ask to design and develop a tool to model the business logic flow and to automatically generate the code that will execute on a IMS/PL/I/DB2 mainframe environment under a property framework (we call this framework “Application Services Integration”).

The architecture of this framework could be compared to a service oriented architecture (SOA), thought in this case services are coupled, mainly because performance (we have constant peaks of 1.500 tx/s and one requirement is that the response time must be lower than 1 second).

This RFP is our second attempt to renew our development toolset. First attempt, developed under something similar to the MDA paradigm (or should I say DSL as we didn’t use UML?) and based on Eclipse PDE framework plus a metamodel that relies on a central repository database, was a complete failure. I will explain the reasons (a forensic analysis) in another post, although it’s about performance and scalability in a geographically disperse environment.

Why I’m telling you this? Well, because I want to share with you similar experiences. Unfortunately, this kind of experiences are usually hidden “behind the firewall”, and, in my opinion, it’s positive to share these details with others as we can learn about success and failures (as my manager usually says to me, “you always must assume that the rest of the people are cleverer than you”).

I’ve found an IBM Systems Journal article about the AD/Cycle strategy and architecture that makes me cry. AD/Cycle was one of the first projects that I had been involved as a software engineer (1991).

Years ago, repository technology promised to change the way corporate IT units build mission-critical software. Development teams could store meta data in a single warehouse and access that meta data using a variety of tools from multiple vendors. IBM’s AD/Cycle CASE initiative, based on Systems Application Architecture, was an attempt to centralize the management of mainframe application development, supporting the integration of tools through a consistent user interface (CUA), workstation services, an AD information model, tool services, Repository Services, and Library Services that provide control for defining and sharing application development data. The centerpiece was the Repository Manager MVS (RM/MVS) running on a mainframe (MVS) and DB2, that was built somewhat on the lines of the IRDS though not quite compliant with the specifications. In addition, IBM also developed SCLM to manage the various files of application development code. The architectural vision was to provide seamless operation between the IBM Repository and the SCLM.

Unfortunately, repository-based development failed to deliver the goods to IT development shops. Since then, IBM has switched to the client/server model, and its repository technology has evolved through Configuration Management Version Control, from 1991, to newer products in the Rational product line.

In our shop, we finally dropped AD/Cycle and developed an in-house meta data repository that it’s still running in production, providing a useful integration through our mainframe toolset.

I have enjoyed reading the “Object Oriented CASE Tools: Lost Opportunities and Future Directions” article at developer.*

I have spent more than 15 years designing and developing in-house case tools for our legacy systems (more than 1.000 developers). I have implemented almost all of the “Traditional CASE tools” features described in the article, and they have proved that they are necessary to archive a very productive development environment.

Now, we are focusing our efforts on distributed systems, and specifically, in OO environments. We have a current discussion about which is the best strategy towards our toolset in these environments. On one side, there is the opinion that we must use OO COTS tools; and on the other side, that we must developed our own tools following the guidelines of the traditional case tools. The absence of some of the features stated in the article is inclining the balance towards the second option (the traditional one). Some features, as the central repository relying on databases, are essentials in environments focused in the reusability of the artifacts, and with OO COTS tools you can’t implement them, with or without “frameworks”. They seem designed for individual developers or small teams and not for a big enterprise.