Real programming languages are living things, changing and evolving. As with any production code, most of their designer’s time is spent on bug fixing and small improvements, rather than on the radical new features. One of the unique things about Kotlin is that it has been evolving in the use-case and community-driven fashion for years, starting way before it went to the stable 1.0 release in 2016, even for some time before it went public in 2011.
Story of Kotlin Null Safety
Take Kotlin null safety for example. The first question that any language design must answer is why would this or that feature be even included into the language, given so many potential interesting features to include from the research literature and other languages.
Kotlin was designed as a better Java for people who were already programming in Java, so its design goals were centered around fixing all the known shortcomings that Java programmers suffer from most. In this regard, adding null safety was a natural choice, since NullPointerException is the most common problem that real-life Java code suffers from.
Null safety has been present in a number of research languages together with its question-mark syntax for many years prior to Kotlin. The conceptual idea was straightforward and time-proven, so taking it and integrating it to the practical language looked like a no-brainer. Yet, as this straight-forward design with non-null and nullable types started to be used on the real code, it quickly began to conflict with another goal for Kotlin language design — the seamless interoperability with Java.
Most of the Java code Kotlin has to interoperate with was not marked for nullness. A null safe language must assume that Java methods can return null, but giving every Java method a nullable result type in a null safe language results in a very verbose code — not a pragmatic thing to do. It took a number of experiments on real-life projects, dead-end approaches, and a Kotlin-specific research collaboration with Ross Tate of Cornell University to come up with a solution for Kotlin’s problem in the form of flexible types, which are colloquially called platform types in Kotlin.
The basic idea behind flexible types is that for interoperability with less-strictly typed languages like Java, we don’t use a wider nullable type, such as String?, which includes all strings and a null, or a narrower type, such as String, but we use a flexible type — a range of types from String to String? denoting an unknown type coming from Java that lies somewhere in this range. The type system is relaxed to permit all operations that are allowed on any type in the range of the flexible type, resorting to runtime checks for correctness. This solution strikes a pragmatic compromise in terms of developer experience, so that Kotlin developers are not worse off working with Java APIs than in Java itself, but can still enjoy a safer type system when working with Kotlin APIs. See JVMLS 2015 – Flexible Types in Kotlin for details.
Why has no one done this before Kotlin? No one tried to integrate null safety into a language’s type system while maintaining safety and interoperability at such a scale before. The same collaboration yielded a solution for mixed-site variance that Kotlin needed for a similar Java-interoperability reason (see FOOL 2013: Mixed-Site Variance). In fact, Java interoperability consumes a considerable fraction of the time spent on Kotlin language design even today.
Evolution and Coroutines
In the initial design of the language, the most important consideration is what features to drop, rather than which ones to include. Many research languages center around a few core ideas. A pragmatic language ends up being more inclusive, especially when you take into the consideration that it has to be understandable and easy to learn by professional developers who are used to writing in other industrial languages. Yet, initially it is easy to keep the language small. It happens naturally due to the limited resources in the language development team anyway.
As the language gets more real-life use, it faces real-life industrial code with its idiosyncrasies, quirks, and patterns. Real-life languages face pressure to support all of them better. As the language grows, the language design focus inevitably shifts from whatever the original goals of its design were to the feature interactions and support. The challenge becomes maintaining conceptual integrity of the language, making sure that new features not only can be implemented, but can be also easily understood and adopted by the existing users of the language and fit into its ecosystem.
Kotlin Coroutines were added to the language after it already went to a stable 1.0 version, with the first experimental support coming out in 2017. The Kotlin coroutines were heavily inspired by C# async/await, yet the final Kotlin design has largely diverged from it as explained in Onward! 2021: Kotlin coroutines: design and implementation.
One of the reasons for that divergence was hindsight. By that time, we were already aware that C# has almost the same internal implementation mechanics both for its yield keyword that supports synchronous enumerator coroutines and for its async/await mechanism for asynchronous coroutines. A natural desire was to unify the two. This way, the language team can spend less effort, having to implement only one, simpler language feature and compiler support for it. The variety will be then provided via libraries, implementing separate support for synchronous and asynchronous coroutines.
The other reason was the aforementioned conceptual integrity. The Kotlin language already had its traditions and a lot of code written in it, so a new feature for the support of coroutines had to fit into the existing codebase, and had to help its existing users. So, a lot of emphasis was made on interoperability with all the asynchronous and reactive Java programming frameworks that were used by Kotlin developers and its performance and ease-of-use for desktop UI and mobile applications, which were getting much traction in the Kotlin ecosystem at that time.
The difference in emphasis and use-cases on the table inevitably lead to differences in the design. Instead of future/promise-based design, which would introduce yet another type of future to the already diverse ecosystem, the design was directly based on underlying continuations and a LISP-inspired call-with-current-continuation primitive (called suspendCoroutine in Kotlin) was introduced, making it straightforward to integrate Kotlin coroutines with all the existing libraries.
Tradeoffs
The design of many new features is riddled with tradeoffs. For example, we’ve been recently improving type inference for recursive generics in Kotlin 1.6 (see KT-40804 Inferring types based on self upper bounds). The original request for enhancement came from the users of APIs that use recursive generic types for a builder pattern, where the result of the function is materialized, without explicitly specifying the type parameter for the function, nor having any context to infer it from. Users expect that a wildcard type, representing a family of types, gets inferred in this case.
However, Kotlin was designed to suppress type inferences in such cases. In Kotlin a call to the function listOf(1) infers the resulting type of List<Int>, because the type of the parameter gives the hint on the type. However, the call to listOf() without parameters and without type from the context fails to compile, even though, technically, it could have been inferred to List<Any?>, which represents the widest type this function could return. Instead, Kotlin forces developers to explicitly specify types in the call, like listOf<Int>(). This avoids the compiler having to guess the developer’s intent, since this guess would most often be wrong in real code, and thus prevents further errors in the code.
The conundrum for recursive generics is that Kotlin does not have an explicit syntax to specify such a recursive type to make the code compile. So, we had a variety of choices. One of the runner up choices was to use a special syntax that tells the compiler to infer the type parameter to its upper bound. In practice, though, that would mean that in all the use-cases we had on the table one will have to write some additional boilerplate code to make the compiler happy. So, we ended up with a set of ad hoc rules that detect such patterns of recursive generic usage in the called function and turn on type inference to upper bounds on all such calls automatically.
In the end, the language became less regular and more complicated with one more ad hoc rule, yet more straightforward and simpler to use for real-life code that developers write with it.
Minor Tweaks and Improvements
Most of the language design work is not about big features, but about small things and inconveniences that need to be fixed here and there. Those small things are usually inconsistencies in the language design. Let us first discuss how they might appear in the first place.
When a new feature is added it starts to interact with all the other language features. These interactions tend to create lots of corner cases. Designing for all those corner cases is very time consuming and often becomes impossible in the absence of real life use-cases for those corner cases. The Kotlin approach here is pragmatic. If we cannot find or imagine a use-case for a specific corner case, then we forbid it, giving a compilation error when the corresponding combination of features is used. Sometimes there are known use-cases, but they do not outweigh the design and implementation effort.
For example, when Kotlin Coroutines became stable in Kotlin 1.3 they introduced a new class of functions — suspending functions and the corresponding suspending functional types. Yet, the usage of suspending functional types as supertypes was not allowed. It needed non-trivial design on how to represent them at runtime, while still supporting runtime type checking with the is operator in Kotlin. That was added later, in Kotlin 1.6, as the usage of coroutines grew and more demand to implement this feature interaction piled on (see KT-18707 Support suspend function as super type).
Sometimes inconsistencies are historic, predating even the initial release of the language. Right now, the Kotlin team is in the midst of a large-scale engineering project of rewriting the whole Kotlin compiler. The architecture of the compiler is being reworked for performance and future extensibility. During this work we have encountered dozens of corner cases, where the compiler that is written from scratch based on a consistent set of rules starts to behave differently on some real-life code. Some of those findings come back to the language design to rethink whether the old compiler’s behavior makes sense or needs replacement. We’ve discovered things starting from the quirks in type inference to the behaviors that depend on the order in which supertypes are mentioned in the source code.
Deprecations
When the language is stable and changes need to be made, then it is not often possible or practical to do it in a fully backwards compatible manner, especially if you intentionally want to fix some older design bugs. It is fortunate when the bug was such that the previous version of the compiler crashed or produced code that would crash right away. But sometimes, it did work and might have produced code that did some sensible things.
A lot of the design effort goes into evaluation of impact of such changes and into designing a migration plan for introducing such changes to the language. In some cases, when the potential impact of the change is non-negligible, the migration plan may span multiple versions and many years in total. We have cases where we needed to implement warnings and automated code fixes in the old version of the compiler and in the IDE, so that the developers affected by the change will have plenty of time to replace the code in advance, before the new version of the compiler, that treats this code differently, will be released.
This work is all about tradeoffs, too. The easiest decision is often not to change anything, but to carry old behaviors, even with bugs, forever. However, it accumulates design debt in the language and technical debt in the compiler. It is not a sustainable approach, as it will make further progress on the language harder and harder. So, the balance has to be found between maintaining backwards compatibility and evolution of the language.
For example, historically, the way the original compiler treated combinations of safe calls and various Kotlin operator conventions like a?.x += 1 was quite inconsistent. So, it had to be redesigned in such a way as to minimize the disruption to the existing code that might have been relying on some of those behaviors. We’ve run multiple experiments on the existing Kotlin codebase with various prototypes of solutions to pick such a design. See KT-41034 for details on the original problem and the design we’ve ended up with.
Conclusion
Language design in the real world is a maintenance of a complex system. We believe that with care we can keep Kotlin modern and relevant for dozens of years to come. And that is an amazingly interesting design and engineering challenge to have.
On this road we’ll continue to run into novel research questions with respect to type systems, feature interactions, usability, real-life code patterns in big code, etc. Research collaborations in those areas are paramount to place all of the improvements on a sound footing.
Bio: Roman Elizarov is a Project Lead for Kotlin at JetBrains and currently focuses on the Kotlin language design in the role of Lead Language Designer. He has been working on Kotlin in JetBrains since 2016 and has contributed to the design of Kotlin coroutines and the development of the Kotlin coroutines library.
Disclaimer: These posts are written by individual contributors to share their thoughts on the SIGPLAN blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGPLAN or its parent organization, ACM.
