JSweet transpiles Sweet Home 3D open source project to help creating a web version
A real-life example of migrating a Java application to JavaScript.
In this post, I will explain how JSweet helps building the online JavaScript Web version of an Open-Source interior design application called Sweet Home 3D developed and maintained by Emmanuel Puybaret.
Sweet Home 3D represents 130K Java Lines of Code (LoC). Amongst these, Emmanuel and I are able to transpile automatically around 50K LoC, leading to 68K JavaScript-generated LoC (by automatic transpilation, I mean that the generated JavaScript code can be run in the browser as is, without any manual adaptation of the code).
To make it work, JSweet also transpiles automatically general-purpose Java packages that are part of the AWT/Swing library, such as the java.awt.geom and javax.swing.undo packages (resp. 20K and 1.3K LoC).
The UI is mostly rewritten manually, with the help of JSweet that performs one-shot transpilations to give a head start to the programmer. The I/O part is mostly changed since the client/server Web environnement is fundamentally different from the desktop environment. However, we are able to automatically transpile some important parts such as the XML SAX Handler, which reads XML home models (the .sh3d file format).
For more details on the resulting application, you can follow the quick links below:
- Watch the demonstration video here.
- Check out the project on SourceForge.net here.
- Try it live here.
- Sweet Home 3D 6.4 release announcement here.
In total, we estimate that the automated translation saved about 100 days of development, which represent about 25% to 30% of the total development effort. The main remaining efforts mainly consist in re-implementing the UI with JavaScript WebGL and canvas APIs, and to deal with resources and files (I/O).
Besides saving development time, the other great benefit is that there is an automated bridge between the Java core code base and the JavaScript one. It means that all changes in the model or in the controller layers in the Java code base are immediately and automatically applied to the JavaScript counterparts, thus making maintenance and evolution of both applications much simpler.
A quick note on Software Design
Software Design is the key for software construction, performance, and evolution. Not surprisingly, it is also a key point for transpiling from Java to JavaScript.
Every software has a design, which is more or less stable and well defined. The design will tell us what are the modules of the application and what are their internal and external dependencies (dependencies to other modules or external libraries).
A good design aims at minimizing dependencies and remove circular dependencies. It also aims at extracting reusable modules through abstraction, so that the software can be used in various contexts without losing too much performance (it is sometimes a tradeoff). For instance, since Sweet Home 3D can run under Windows or MacOS, it is important to design the software to make it oblivious to the OS.
In general, the overall design is known by the software creator and unfortunately, it lives in the creator’s brain, with a lot of implicit constraints that are not visible when reading the code. Because the design is not explicit and requires a lot of effort to be fully understood and inferred, it is in general very easy to break. Fortunately, in the case of Sweet Home 3D, Emmanuel is the only contributor and makes sure that the design remains sound, which made our task much easier.
Sweet Home 3D design analysis
Sweet Home 3D’s design is based on a rigorously-implemented MVC pattern. Emmanuel had this design in mind since day one. He knew exactly where to go and what to transpile. However, it was not my case. So, in order to get a better view of how it looks like and what are the implications on JSweet transpilation, I have developed a small visualization of the dependencies of Sweet Home 3D Java packages.
For my visualization, I use JSweet to analyse all the dependencies (method invocations) between each file. It generates a graph in the CSV format, which I import and visualize in the Cytoscape Open Source software (which is, by the way, yet another great piece of software written in Java).
The following figure shows the result of the analysis. In Cytoscape, I use the AutoAnnotate plugin to cluster and layout all the files w.r.t. their packages. Each node of the graph (colored rectangle) is a Java file and the graph shows the invocations as edges. File clusters correspond to Java packages.
In order to better see the design, we can collapse the graph to have a more synthetic view of the dependencies (including directions) between each package. In the collapsed graph below, the colored rectangles now correspond to packages and contain many files.
We find the typical MVC pattern, where all the views are in the c.e.s.swing package, all the controllers are in the c.e.s.viewcontroller package, and all the models are in c.e.s.model. Links are directed as expected and we find no cyclic dependency issues.
Additionally to dependencies, I generate an attribute on the nodes, which I call RELATIVE_SUPPORT (RS). RS is a indicator that measures how easy it is to run the transpilation result in a JavaScript environment. For instance, if a node uses the Java function Math.cos(x), it does not impact its RS. However, if it uses the Thread.interrupt() function, it impacts its RS negatively, since it would be extremely difficult to support threads in a JavaScript environment.
In the graph visualization, green means good RS, and red means poor RS. By looking at the graph again, you can immediately see that packages c.e.s.viewcontroler and c.e.s.model are easy to translate, while the c.e.s.swing package would be difficult.
Note: if you are interested in how to create such visualizations for your own projects, please don’t hesitate to contact me and I’ll be glad to explain more.
Transpiling the Controller and Model Layers
As our design analysis demonstrates, we can transpile the model and controller packages. Thanks to JSweet open compiler, we were able to do it entirely automatically (no human intervention). To do so, we use the JSweet’s adapter API available since version 2.
Because it is fully automatized, the maintenance and evolution of the software’s core can now be done directly on the Java code base and is instantaneously synced as is on the JavaScript side. We are talking here of about 50K LoC in Java, automatically generating 65K LoC in JavaScript.
Maybe some programmers will not realize the advantage and may consider that it would be easy to rewrite the code manually. However, on a legacy software that took 15 years to build, the sum of knowledge that has been accumulated in the source code is huge. We are talking here about the functional part of the application that was iteratively created with the help of a vast community of users.
JSweet adapter
Many specific adjustments are required when transpiling a Java program to JavaScript. This is done in the SweetHome3DJSweetAdapter (source code available here). In this adapter, we wrote specific code to perform the following adjustments:
- Erase all the classes that are not part of the controllers and of the model and other features that we do not want to see in the JS version (for instance version update and damaged file management that are really desktop-specific issues).
- Erase the Java packages so that all the SweetHome3D classes are top level (each Java class has a unique name in the SweetHome3D API).
- Map core JDK types (IllegalArgumentException, IllegalStateException, PropertyChangeEvent, EventObject, PropertyChangeListener, PropertyChangeSupport, etc) to a simple default JavaScript implementation that can be found in core.js. These correspond to JDK types that are not supported by default by JSweet.
- Map BigDecimal Java class to Big.js library. Note that Big.js does not behave exactly like the Java implementation, but it is largely enough for the SweetHome3D use case.
- Map java.text.Format and java.net.URL Java classes to become strings in JavaScript.
- Map com.eteks.sweethome3d.tools.ResourceURLContent to a generic JavaScript client/server implementation (see URLContent.js).
- Map all SweetHome3D enums that end with “Property” to string values in JavaScript (by default enums are transpiled to integers). Note that the SweetHome3D model objects are similar to Java Beans and each property in a model object can be listen by a PropertyChangeListener. Enum properties are used in the SweetHome3D model to deal with property change events. Note that this mapping from enums to string poses overloading problems in the generated TypeScript code, which we solve by erasing overloads and substituting addPropertyChangeListener and removePropertyChangeListener implementations in the adapter.
- Erase some methods that depend too much on Java specific API. Amongst those erased elements, we have:
- readObject and writeObject methods (Java-serialization specific);
- all the methods that use the Java’s GregorianCalendar class, which has no direct equivalent in JavaScript (we have to rewrite manually these methods);
- the com.eteks.sweethome3d.model.LengthUnit class, that we had to write manually here because its implementation uses a lot of Java enum tricks that are not easy to map to JavaScript (and also a lot of text-related Java-specific APIs);
- the com.eteks.sweethome3d.model.UserPreferences, that we implemented manually here, since it is using a lot of language-aware resource management (I will explain later how we deal with resources).
- Adapt the XML Parser uses so that it uses lib/jsXmlSaxParser.min.js, a slightly modified implementation of the jsXmlSaxParser library.
- Map java.awt.BasicStroke to use our own implementation of stroke (lib/stroke.min.js). Note that this implementation was created by transpiling the Java source code with JSweet.
- Map java.util.UUID.randomUUID() to our own implementation of randomUUID().
- Automatically add the list of transient fields in the JS constructors so that we can ignore them in the serialization of our incremental recorder protocol (that I will explain later).
- … (and some other adaptations that you can check in the source code)
As you can see, there are many specific adaptations required to be able to transpile fully automatically. It requires a lot of work and a good knowledge and experience in both Java and TypeScript/JavaScript (and of course JSweet). But, above all, it requires a clear vision of what we want to achieve and why. As the final outcome of our work, here is the Ant task that transpiles automatically the functional core of SweetHome3D to JavaScript (see the build.xml file). Note the –factoryClassName parameter, which takes the class that instantiate the adapter (SweetHome3DJSweetFactory).
<!-- Java to JavaScript and TypeScript transpilation with JSweet -->
<target name="transpiledLibraries"
description="Transpiles Sweet Home 3D classes to tools/JSweet/build">
<!-- Compile classes which adapts JSweet transpiler to Sweet Home 3D JS needs -->
<mkdir dir="tools/JSweet/build/SweetHome3DJSweetExtension"/>
<javac srcdir="tools/JSweet/src"
destdir="tools/JSweet/build/SweetHome3DJSweetExtension"
classpath="tools/JSweet/lib/jsweet-transpiler-2.4.0-RC1-jar-with-dependencies.jar"
source="1.8"
encoding="ISO-8859-1"
includes="com/eteks/sweethome3d/jsweet/*" />
<!-- Transpile Sweet Home 3D classes -->
<mkdir dir="tools/JSweet/build/ts" />
<mkdir dir="tools/JSweet/build/js" />
<java classname="org.jsweet.JSweetCommandLineLauncher" fork="true" failonerror="true">
<arg value="-v" />
<arg value="-b" />
<arg value="--workingDir" />
<arg value="tools/JSweet/build/jsweet.tmp" />
<arg value="--factoryClassName" />
<arg value="com.eteks.sweethome3d.jsweet.SweetHome3DJSweetFactory" />
<arg value="--header"/>
<arg value="tools/JSweet/header.txt" />
<!-- ts output dir -->
<arg value="--tsout" />
<arg value="tools/JSweet/build/ts" />
<!-- js output dir -->
<arg value="-o" />
<arg value="tools/JSweet/build/js" />
<arg value="--declaration" />
<!-- input dir -->
<arg value="--jdkHome" />
<arg value="${java.home}" />
<arg value="--encoding" />
<arg value="ISO-8859-1" />
<arg value="--candiesJsOut"/>
<arg value="tools/JSweet/build/js" />
<arg value="-i" />
<arg value="../SweetHome3D/src:tools/JSweet/src" />
<arg value="--includes" />
<arg value="def:com/eteks/sweethome3d/model:com/eteks/sweethome3d/tools:com/eteks/sweethome3d/viewcontroller:com/eteks/sweethome3d/io" />
<sysproperty key="java.ext.dirs" value="../SweetHome3D/lib"/>
<!-- Transpilation target may contain almost all com.eteks.sweethome3d.viewcontroller classes or only the subset needed to run
Sweet Home 3D JS Viewer (in which case swingundo.js extracted from tools/JSweet/build/js/j4ts-swingundo-1.8.132-20170726/bundle.js isn't required) -->
<sysproperty key="transpilationTarget" value="${buildType}"/>
<classpath>
<fileset dir="tools/JSweet/lib" excludes="generated/" />
<pathelement location="tools/JSweet/build/SweetHome3DJSweetExtension"/>
<pathelement location="../SweetHome3D/libtest/jnlp.jar"/>
<pathelement location="../SweetHome3D/libtest/AppleJavaExtensions.jar"/>
</classpath>
</java>
</target>
Dealing with files, resources and localization
In Java localized resources are implemented with localized property files that can be accessed through the resource bundle API. This has two caveats. First, the property file format is Java-specific. Second, it relies on local files and we intend our JavaScript version to run in a browser and access its resources from the server (not from local files). So, we implement our own resource management layer in src/CoreTools.js.
The principle is the following. First we transform all Java property files to JSON files. This is done server-side so it can be implemented in Java, and the implementation is quite straightforward using the JSON.org API. Find the source code in the PropertiesToJson class. Second, we implement a JS function to load a resource bundle from an URL (for a given language).
/**
* Loads resource bundles for a given base URL and a given language.
*
* @param baseURL the base URL of the localized resource to be loaded
* @param language the language to be loaded (Java conventions)
* @returns an array of bundle objects, starting with the most specific localization to the default
*/
CoreTools.loadResourceBundles = function(baseURL, language) {
var resourceBundles = [];
if (language) {
resourceBundles.push(CoreTools.loadJSON(baseURL + "_" + language + ".json"));
if (language.indexOf("_") > 0) {
resourceBundles.push(CoreTools.loadJSON(baseURL + "_" + language.split("_")[0] + ".json"));
}
}
resourceBundles.push(CoreTools.loadJSON(baseURL + ".json"));
return resourceBundles;
}
And a function to get a localized string from the given key.
/**
* Gets a string from an array of resource bundles, starting with the first bundle.
* It returns the value associated with the given key, in the first bundle where it is found.
*
* @param resourceBundles {Object[]} an array of bundle objects to look up the key into.
* @param key {string} the key to lookup
* @param parameters {...*} parameters for the formatting of the key
* @returns the value associated with the key (in the first bundle object that contains it)
*/
CoreTools.getStringFromKey = function(resourceBundles, key, parameters) {
for (var i = 0; i < resourceBundles.length; i++) {
if (resourceBundles[i] != null && resourceBundles[i][key]) {
return CoreTools.format.apply(null, [resourceBundles[i][key]].concat(Array.prototype.slice.call(arguments, 2)));
}
}
throw new IllegalArgumentException("Can't find resource bundle for " + key);
}
Note that in SweetHome3D, all the localization and resource access is abstracted and centralized in src/UserPreference.js, which we re-wrote manually. So, in the end, thanks to this design, it was fairly simple to switch to JS all the localization and resource management.
Reading and saving
The SweetHome3D file format is mainly a zipped XML file describing all the home objects (and additional resource files such as 3D models and textures). The core of the read function is a SAX XML handler, which is automatically translated by JSweet, and can run as is in JavaScript thanks the jsXmlSaxparser library.
Last but not least, we save the user changes thanks to a protocol that sends all the edits that the user makes on the home model (see the IncrementalHomeRecorder class in JavaScript). The protocol stores the edits in a queue at client side, and when a connection to the server is found, it send the list of pending edits to the server. In turn, the server deserializes the edit list, apply them to the current home, and saves the file. The advantages of this protocole are the following:
- it minimizes network traffic because it only sends the edits (differences) and not the entire home model;
- it allows working offline with no storage necessary (when the connection is back, the pending edits are sent);
- in future versions, it will make it possible to deal with multiple users working on the same home.
Note that it was fairly simple to implement this protocol, since we just connected to the undo/redo design pattern that SweetHome3D uses. In SweetHome3D, the controller layer implements each action by creating an instance of an javax.swing.undo.UndoableEdit specialization and store them in the undo manager (javax.swing.undo.UndoManager). Such edits can be undone (by calling the undo() method) or redone (by calling the redo() method).
Since we have also transpiled the javax.swing.undo package with JSweet, on the client side (JS), we simply serialize all the undoable edits and send them to the server. On the server side (Java), we deserialize and we apply all the received edits by calling the redo method. That’s another good example to demonstrate that design is the key. Don’t tell me that design patterns are useless please 😉
Translating the 3D viewer and the 2D plan editor component
As expected, these components are a big part of the migration work. Emmanuel had to rewrite his own implementation of Java3D-like library in JavaScript. For the 2D plan component, I had to implement the drawing using the canvas API, while the Java version uses the AWT/Swing graphics/painting API.
Both these tasks took benefit of JSweet. First, both 3D and 2D components make intensive use of a great piece of library embedded in AWT in the java.awt.geom package. This package is actually an abstract geometric manipulation package (it contains primitives to deal with points, lines, shapes, matrices, affine transforms, etc.). It does not depend on AWT at all and only uses very standard Java APIs, so we transpiled it as is and made it available to JavaScript thanks to JSweet (you can find the J4TS candy that contains awt.geom here).
For the 2D plan editor, we took the PlanComponent Java class, which is part of the c.e.s.swing package (holding the views), and I performed a one-shot basic translation with JSweet. Then, I finished the adaptation manually. A noticeable trick is to re-implement the java.awt.Graphic2D API for a JS canvas (see src/graphics2d.js).
Conclusions and remaining work
To conclude, I would like to say that I am really proud to collaborate with Emmanuel Puybaret on this project. Transpiling SweetHome3D would not have been possible without his deep knowledge of the code, and without his rigorous design. I believe that it is rare to see such well-designed code on a 15-year old piece of software that is still evolving fast. Emmanuel and I share a common vision of Software Engineering: we are bored with the hype and the new languages/frameworks/API popping up everywhere all the time for no good reasons. We like to focus on the value of functionalities: the services that the software implements and the knowledge it holds.
Our next step is to extend the UI to support more features of the Java version. Typically, editing preferences and object properties. It would probably be possible to transpile the Swing UI to do that (at least partially). However, since we have to deal with responsive UX, we both agree that the ROI would not be that interesting.
We may use an existing JavaScript framework for the UI, or not, since we do not want to rely on something that will be out of fashion in 2 years or so. In any case, if we use an external framework, we will make sure that our software design will make our implementation independent from it.
Remember: software design is the key 🙂