Updating the Common Lisp standard

Short answer -

  1. It involves more expense than anyone is willing to finance.
  2. The feature you want probably already exists.
  3. If not, it can be made via implementation-independent libraries.
  4. There is probably a good reason that the specification does not have that feature, and it will probably not be added in a revision.

Long answer -

Frequently, we receive questions why the Common Lisp standard has not been updated, and whether there is any project to update it. The answer is that there are no serious plans to update it in its current form. It has been estimated that the creation of the current standard cost around 1 million USD at the time (1994). It would be much more expensive now. But why is it so expensive?

These questions typically come from people who are used to languages that have no independent standard, and often from people who are used to languages with a single implementation. To clear things up, let us first explain the difference between different existing programming languages.

Languages and specifications

Most programming languages used today are not really programming languages, but something we can call programming systems, i.e., some software that is designed to make the task of writing programs tolerable. Such a system would typically have some specification or documentation that users can refer to in order to understand how to use it, without having to study the source code.

Specifications are incomplete almost by necessity, in that there are situations where the behavior of the system is not described in detail. The implicit specification, then, is how the system really behaves. When the creator of a programming system wants to change its behavior, perhaps in order to add new features, this can be done without consulting anybody, though it would of course be polite to ask if current users agree to the changes, especially if the changes might alter the way existing user code behaves.

Using a programming system as described above for some serious project comes with a great deal of risk. If the system is changed in some ways, the project members must decide what consequences the changes might have on the project code base, and make a decision to either stay with the current version of the system, or update the code base to incorporate the updates required as a consequence of the change. The first option may not be possible for a longer period of time because of lack of maintenance of older versions of the programming system.

Worse, support for the programming system may disappear altogether. Some systems are created as research projects, and when grants dry up, the initial creator may no longer have the means to continue its maintenance. If the code is free or open source, some other group can take over the maintenance, but there is no guarantee this will be the case. Some other systems are created by a single person or a small group of volunteers. These individuals may lose interest or financial means to continue the work on the system, again leaving the system unmaintained.

For something to be called a programming language, we think the minimum requirement is that it has a specification that is independent of its implementations, i.e., the programming systems that claim to be software that behaves according to the independent specification. A programming system can still evolve in that it can provide additional features and tools that are not covered by the independent specification. Programs written according to the independent specification are then guaranteed to work with any of the implementations, and users use additional features at their own risk.

Often, an independent specification of a programming language takes the form of a standard. Standards are written and published by standards organizations. Contrary to common belief, these organizations do not have to be internationally recognized, nor associated with a particular country. In fact, they do not even have a particular status that can identify them as standards organizations. Some such organizations are associations, some might be commercial companies, etc. Also, contrary to common belief, the documents published by standards organizations are not free to copy or use in any old way. They are documents covered by copyright, and if the standards organization is a commercial company, their business model is to make money out of selling standards documents. These organizations are not (or at least should not be) suppliers of software that claims to implement the standard that they publish.

A software company or some other group of people may decide to create an implementation of some independent standard. The standard typically contains information about the required features of the implementation, and perhaps about restrictions on the implementation that must be respected in order for the implementation to be considered an implementation of that particular standard. Many standardized programming languages have several implementations, and a project that decides to use a particular language can often choose between several suppliers of programming systems. As long as the project uses only those features that the standard specifies, changing to a different supplier is fairly straightforward, should one supplier disappear for some reason.

Updating a standard

Now let us consider why it might be desirable to create an updated standard, and if that might be the case, what such a process would involve. Clearly, users of programming systems that implement some standard might want new features that the standard does not mention, and that not all (or perhaps none) of the available implementations provide.

However, changing the standard will in no way force the creators of implementations to implement additional features of a revised standard. The creators of implementations still have a choice between announcing that they conform to the initial standards document and the revised standards document. So in order for a revision of the standard to have the desired effect on the features of implementations, the proposed revisions must be such that the creators of such implementations agree to change their implementations accordingly. And if only a subset of implementations agree to the changes, the users will find that they have less of a choice between different suppliers of implementations.

Furthermore, if the desired changes consist mainly of new features, such features can often be added to an existing implementation as an independent library. And if the library is written using only features of the standard, then it can be added to any conforming implementation of the standard. This way, there is no particular reason to update the standard at all, and the suppliers of implementations do not have to agree on anything. Users can simply choose to use such independent libraries if they feel the need for the additional features.

But the situation is actually much more complicated than that. A programming language is not just a collection of features. A serious programming language requires careful consideration of each feature and how it is best specified. A small mistake can have dire consequences in terms of the work required by a compiler writer, in order to achieve acceptable performance of user code. Some features may in fact make it next to impossible to create a compiler that generates high-quality code. And, conversely, omitting a feature that might be trivial to include can make the language considerably less attractive to users. In other words, to determine what features to include or exclude, and how the included features can be presented to the application programmer, requires detailed knowledge of programming-language design and compiler design, not to mention intimate knowledge of the programming language being worked on.

So the situation we often find ourselves in, is to answer questions about why the Common Lisp standard has not been revised, and whether there are any plans to revise it, from people who have neither of the required qualifications. They most often do not master Common Lisp, have no training or experience in programming-language design, and no knowledge of compiler design. Worse, some of these people go so far as to suggest that a particular feature be added to the Common Lisp language, without any idea of the consequences that adding the feature would have to the implementations that incorporate the features. As an example of such a suggestion that often comes up is to give the function eval access to lexical environments (we will not explain the consequences, and those who do not already know should go study compiler design).

But let us say for a moment that we have found a feature that would be desirable to have in Common Lisp, perhaps threads (though all implementation have it and it behaves in the same way across implementations) or a module system, and we know that it could be implemented without any negative consequences on performance of compiled code. Now let us see what it takes to accomplish this revision of the standard in a way that the vast majority of the creators of Common Lisp implementations would agree to include it.

The first step would be to find an existing standards organization that would be willing to publish a revised standard. For an organization to be interested, they would at least be convinced that they would not lose any money doing so (recall that standards organizations are not charities or government agencies). If none can be found, one would have to be created, which has its own cost associated with it.

Next, it would be necessary to create some kind of group of people that represent the vast majority of creators of Common Lisp implementations. There are currently two commercial implementations (Allegro and LispWorks) and several free or open-source implementations (SBCL, CCL, ECL, ABCL, CLISP, and perhaps more). Most of the free or open-source implementations have very limited resources and are usually maintained by volunteers. The people in the group would have to be given resources to meet, discuss, implement prototypes, and perhaps travel, and the work would likely take several months or even years. These resources would have to be found somewhere. For the existing Common Lisp standard, many of the resources were supplied by the US government. It is doubtful if any organization would be willing to supply these resources today.

Conclusion

In summary, it is not likely that the Common Lisp standard will be revised. And it is even less likely that it will be revised to include a random feature suggested by a relative newbie without knowledge of programming-language design or compiler design. The good news, though, is that there is very likely no need to revise the standard. Many people suggesting revisions to the standard gladly use programming "languages" with absolutely no independent standard or specification whatsoever on a daily basis. And the suggested features likely already exist in the form of libraries, often possible to incorporate into most Common Lisp implementations. And if the desired feature does not exist, it is highly likely that there is a good reason for it to be excluded. As mentioned before, it may have disastrous consequences to the ability of the compiler to generate good native code.


robert.strandh@gmail.com