What is an Open Transpiler and why do we need it?

A transpiler is an Open Transpiler if programmers can open the hood of the transpiler and tune it for their own needs. In this post, I will explain what it means exactly, why we need it, and how it works. I’ll end up showing some specific JSweet v2 examples.

A little bit of history

Talking about Open Systems brings me back to the 80s. Back to the years when reflection and meta-level architectures where an active field of research already deeply bound to AI. At these glorious times, I was really impressed by Pattie Maes research. In 1987, she published a notorious paper called Concepts and experiments in computational reflection, which talks about how programs can self-modify their own behaviors.

At this time, many researchers (including myself as a newbie) worked on how to find the most elegant way to open languages (or more generally systems). One of the most elegant way was to make the language/system to be self-described. Some API would allow the programmer to access the system within the system itself and eventually modify it (which could in turn impact the description of the system itself, which is scary). The most famous try of such a self-describing language is probably Smalltalk, an object-oriented language that brought up the concept of a meta-class in order to describe and modify how classes work within the very same language. Extending that idea, I was always fascinated by OpenC++, a project created by Shigeru Chiba (also creator of Javassist), who had the smart idea to use meta-classes at compile time to tune the C++ compiler.

Because reflection was complicated for programmers and quite heavy for most use cases, finding the right way to open a system eventually led to Aspect Oriented Programming in 1996. AOP grew big and made a lot of buzz, including in the industry. However, in a way, the outcome of this field of research was disappointing, at least for me. Firstly because the leading team on the topic pushed Aspect Orientation through a language called AspectJ, while I still believe that Aspect Orientation is more about a programming/design style rather than about a language. Secondly because of the focus and objectives of Aspect Orientation itself. AOP was centered on means to handle the modularization of so-called crosscutting concerns. However, crosscutting concerns are intimately bound to how these concerns are implemented, which means that there are many manners to deal with crosscutting concerns, including OOP, Generative Programming and Reflection. Pretty early, I felt like the community was looking in a lot of directions, but lost the purpose of the core issue. AOP became some sort of big lab or container to talk about virtually any kind of language or system design and implementation issue. It is funny to think that AOP, in a way, failed to apply separation of crosscutting concerns to itself.

In 2004, after being working actively on AOP during 6 years or so (my Ph.D. thesis was mainly about it), I decided that it led to too many issues (i.e. nowhere). So I went back to fundamentals and better scoped issues. That’s when I got interested in open transpilers such as Spoon, which I developed when I was working at INRIA. With Spoon, I worked on more pragmatic approaches such as annotations and templates to tune source code generation, which falls in the field of so-called Generative Programming. I ended up selecting various simple and pragmatic ways to open a transpiler. My main interests being about:

• keep the tuning simple and understandable for any programmers (from simple tuning to advanced ones);
• include type safety in that tuning: in short, if the tuning leads to generating incorrect code, it should be caught as early as possible by the open transpiler;
• keep the tuning light and performant: the tuned transpiler should not imply any visible loss of performance for most transpilations, which in a way excludes complex code analysis and transformations.

The JSweet Open Transpiler (v2) was designed and implemented following these principles.

Why do we need Open Transpilers

The motivation behind Open Transpilers comes from the distinction between a Transpiler and a Compiler. To me (and most people I guess), a compiler transforms a source code written with a high-level programming language to a low-level machine-understandable bytecode. Because it is machine-understandable, the target code is normalized against a set of canonical concepts, which are similar to the ones found in Turing machines. Although Turing machines can calculate anything, the way things are calculated are not understandable for humans in an efficient way. Since compilers take care about mapping source code to the machine, it is important that the compilation semantics is rigorously defined and that to a given input always corresponds the same predictable output.

On contrary to a compiler, a transpiler transforms a source code written with a high-level programming language, to a source code written with another high-level programming language. IMO, automatization of high-level source code generation is not as simple as compiling to machine bytecode. The reason is that within a high-level language, there are several strategies to implement the same program (and I am not talking about algorithms only). That’s related to the language syntax and structure, which is not as canonic or linear as bytecode, but also because of the vast amount of available libraries and APIs available to achieve complex goals. To illustrate what I am trying to explain, please imagine a simple GUI, and try to imagine how different the code would be if it had to be written using React.js or if it had to be written using Angular.js.

Because it works at a source-code level, a transpiler is a tool that should help the programmers automatizing various programming tasks, in order to simplify source-code maintenance and breakdown languages boundaries. In 2017, when AI is such a buzzword, it feels that there is something wrong being stuck in a language. It feels like it should be a given that the machine should understand the intents of the programmers better, without having to rely on a given syntax or a specific framework.

Open Transpilers ultimate goal is that programmers do not need to rewrite or duplicate code anymore. When a new language or libraries comes up, programmers should be able to use and tune transpilers to take them into account, so that legacy code does not turn out to be obsolete. Longer-lasting code is what we aim at with Open Transpilers.

JSweet: an Open Transpiler

Starting at version 2, JSweet is an Open Transpiler from Java to TypeScript/JavaScript. It means that it provides ways for programmers to tune/extend how JSweet generates the intermediate TypeScript code. Since generating JavaScript code can happen in many different contexts, tuning the transpiler is a solution to avoid repetitive tasks and automatize them. It can be useful to reuse and adapt legacy Java code, but also to share code between Java and JavaScript (such as DTOs and services).

In JSweet, you can tune the generated code programmatically by implementing so-called adapters. An adapter is an object that implements and uses the JSweet extension API. Let us take some simple examples to illustrate what you can do.

• Say you have a legacy Java code that uses the Java API for serializing objects (writeObject/readObject). With JSweet, you can write an adapter to erase these methods from your program, so that the generated JavaScript code is free from any Java-specific serialization idioms.
• Say you have a Java legacy code base that uses a Java API (for example the java.math.BigDecimal API), which is close to (but not exactly) a JavaScript API you want to use in your final JavaScript code (for instance Big.js). With JSweet, you can write an adapter that will automatically map all the Java calls to the corresponding JavaScript calls. It will works at transpile-time, which means that the generated code will be free from the Java API.
• Last but not least, you can tune JSweet to take advantage of some specific APIs depending on the context. For instance, you may use ES6 maps to replace Java maps if you know that your targeted browser supports them, or just use an emulation or a simpler implementation in other cases. You may adapt the code to avoid using some canvas or WebGL primitives when knowing they are not well supported for a given mobile browser. An so on…

Using an Open Transpiler such as JSweet has many practical applications, which you may or may not have to use, but in any case it is good to be aware of what is possible.

How does it work?

As shown in the figure below, a transpiler consists of a parser, which takes the source code to be transpiled as an input and creates an AST (Abstract Syntax Tree). From that AST and potential transpilation options, the transpiler transforms the source code for the source language to the target language (output) by using a pretty printer to print out the result. A transpiler works very similarly to a compiler and when it is implemented in an Object-Oriented programming language, it relies on a Visitor design pattern for efficiency and separation of concerns.

In some open transpilers such as Spoon, the AST can be modified. However, in JSweet, the AST is the javac AST, which is immutable. So the transpiler creates intermediate data, which is stored in a transpilation context. This context will store any data that will be useful to the transpilation, including metadata for tuning. There are two main sorts of metadata.

• Soft annotations: exactly like Java annotations, but they are not physically present in the source code. They are installed by the configuration options or programatically by adapters.
• Type mappings: these are in charge of mapping Java types to TypeScript ones.

For JSweet programmers, tuning the transpiler consists of programming adapters that may modify the metadata in the context and may override the default printer behavior. Adapters follow a decorator design pattern, which means that the are chainable. The adapter chain is declared in the jsweetconfig.json file and created when the transpiler is initialized. Then, the printer will delegate to the adapter chain various key printing functions such as printing method invocations, variable accesses, object creations, and so on. Each adapter can take care of adapting a specific aspect of the transpilation, and will delegate to the parent adapter when the current expression or statement to be printed is out of its scope. As a consequence, a programmer may write an adapter to use a given JavaScript API, and compose it with other adapters specialized for other APIs.

Good to know: by default, JSweet uses the RemoveJavaDependenciesAdapter to erase most JDK uses and replace them with inlined standard JavaScript.

Finally, note that one key aspect of JSweet design is that it generates intermediate TypeScript code, which is compiled and typechecked, so that if an adapter generates wrong code, it will be immediately reported to the programmer.

Getting started: “Hello World” adapter

Okay, extending a transpiler can rapidly become quite complicated. So for the sake of getting started I will step through a very simple example to generate strings in place of dates when finding java.util.Date types in the Java program.

Say we want to share a DTO with a JavaScript client. Here is the code:

package source.extension;
import java.util.Date;
/**
 * A Hello World DTO.
 * 
 * @author Renaud Pawlak
 */
public class HelloWorldDto {
    private Date date;
    /**
     * Gets the date.
     */
    public Date getDate() {
        return date;
    }
    /**
     * Sets the date.
     */
    public void setDate(Date date) {
        this.date = date;
    }
}

As you can see, this DTO contains a java.util.Date field. If you use JSweet to transpile that code, it will be usable from any JavaScript program. The java.util.Date objects will be transpiled to standard JavaScript Date objects. However, in some contexts, for any reason we don’t really care about (that’s personal), we may want the JavaScript clients to work with strings or numbers instead of dates. Let us say we want to work with strings.

With JSweet, you can create an adapter, which is a subclass of the org.jsweet.transpiler.extension.PrinterAdapter class, which will take care of mapping the date types to strings. First, create the HelloWorldAdapter.java file in the jsweet_extension directory at the root of your project. Copy and paste the following code in that file:

import org.jsweet.transpiler.extension.PrinterAdapter;
public class HelloWorldAdapter extends PrinterAdapter {
    public HelloWorldAdapter(PrinterAdapter parent) {
        super(parent);
        addTypeMapping(java.util.Date.class.getName(), "string");
    }
}

Second, in the project’s root directory, create the “jsweetconfig.json“ file with the following configuration:

{
  adapters: [ "HelloWorldAdapter" ]
}

Now, if you run JSweet on the HelloWorld DTO, the generated code should look like:

/* Generated from Java with JSweet 2.XXX - https://www.jsweet.org */
namespace source.extension {
    /**
     * A Hello World DTO.
     * 
     * @author Renaud Pawlak
     * @class
     */
    export class HelloWorldDto {
        /*private*/ date : string;
        public constructor() {
            this.date = null;
        }
        /**
         * Gets the date.
         * @return {string}
         */
        public getDate() : string {
            return this.date;
        }
        /**
         * Sets the date.
         * @param {string} date
         */
        public setDate(date : string) {
            this.date = date;
        }
    }
    HelloWorldDto["__class"] = "source.extension.HelloWorldDto";
}

Note that all the date types have been translated to strings as expected. By the way, note also the JSDoc support, which makes JSweet a powerful tool to create well-documented JavaScript APIs from Java (doc comments are also tunable in adapters!).

Other examples

For more details and more examples, please refer to the JSweet specifications. You will find various simple examples such as:

• an adapter that renames non-public members by adding two underscores as a prefix,
• an adapter to use ES6 Maps,
• an adapter to support Java BigDecimal by mapping it to the Big.js library,
• an adapter to map enums to strings (all accesses to @StringType enum MyEnum { A, B, C } will be modified to use the "A", "B", and "C" strings),
• an adapter to generate JavaScript JAX-RS proxies/stubs,
• an adapter to disallow global variables…

For a real-world example, you can look at Sweet Home 3D version 5.5, which uses JSweet to generate JavaScript out of the Java model and show 3D models in the browser. Thanks to JSweet, Sweet Home 3D was able re-use a lot of Java code and will save a lot of time in maintenance, because code evolution and debugging will not be duplicated between Java and JavaScript.

Conclusions

Open Transpilers are transpilers that are tunable so that programmers can change the way the code is generated. It is useful because when working with high-level languages, the wide variety of contexts and APIs makes it quite difficult for a generic transpiler to anticipate all cases.

With Open Transpilers, when new APIs or new languages appear, you can adapt your transpiler to take them into account instead of rewriting or duplicating code. Your legacy applications can last longer and programmers can focus on the features rather than having to keep up with constant environment changes. It is particularly useful in innovation-driven fluctuating contexts such as Web and Mobile development.

JSweet v2 is the outcome of many years of R&D around open languages and transpilers. It is also answering real-world cases and applications that we have met while working with JSweet. For example, JSweet is now used in Sweet Home 3D version 5.5 to generate JavaScript out of the Java model. Thanks to JSweet, Sweet Home 3D was able to read sh3d files in a browser with minimum code rewriting.

Eventually, in a not-too-far future, JSweet (and maybe Open Transpilers in general) may get smarter and adapt automatically to the context. Machine learning algorithms could be used to automatically generate better code, that corresponds better to the programmers needs. In the meantime, JSweet 2 proposes a pragmatic, efficient and flexible solution that should be applicable to most commonly encountered use cases.