For an open source project as old and well-known as TinyMCE the primary repository is more than just a collection of code; it’s home base, the first place to look for help and the foundation for everyone who forks or contributes to the project.
So you can imagine that making perhaps the biggest change this repository has ever seen was quite daunting. What started as a simple proposal became an evolving background priority task that has taken 5 months for me to become confident it is stable.
As TinyMCE transitioned to a modern codebase through 2017 and 2018 many external library dependencies were added from previously closed source projects. This multi-repository development style worked well enough in small teams, but as we grew the team on TinyMCE 5 we hit the pain points harder and harder slowing the team down significantly. It’s a common story in “I switched to a monorepo” blog posts.
As TinyMCE 5 wound through beta and release candidate at the end of 2018 I decided enough was enough. In consultation with Spocke the decision was made to bring our 22 library projects together alongside TinyMCE in a monorepo. I volunteered to take this on as a pet project, expecting the scope of changes in TinyMCE 5 to take priority and it would be a long slow burn.
I don’t want to create another tedious “how to monorepo” article, but I do want to give a high-level overview of why and how we did it for posterity and conversation.
Based on my expectation of delays I started with the best decision of the entire project; I did not build the monorepo by hand. I used a script to do the heavy lifting and import both the existing code and templates of new files. This script could rebuild my monorepo fork based on the master branch of TinyMCE and 22 source repositories in just 2 minutes. This gave me freedom to progress, experiment and iterate in my own little sandbox while also keeping it up to date.
Early on we decided to convert the TinyMCE repository rather than start a new one. The popularity of the project meant we could not break the master branch; the whole monorepo update for our developers had to come down with a single
git pull. Next, we settled on Lerna as the basis for our monorepo. It is fairly well known and seemed to be strongly recommended.
Side note: the decision to use
modules instead of the Lerna default
packages is a whole other tale, one I tried to cover in the TinyMCE readme. It’s quite possible we will eventually drop Lerna, a yarn workspace gives us most of the benefits I was looking for, and there are definitely stories of people outgrowing Lerna. But for now it’s working well.
I quickly abandoned the low-level management functions and settled on Yarn workspaces instead, but Lerna’s help with publishing independently versioned modules is essential as we wished to continue publishing the libraries even after their source code was merged into the monorepo.
The base setup of my monorepo script was fairly simple:
yarn config set workspaces-experimental true
mkdir -p modules/tinymce && mv .gitignore * modules/tinymce
- create new
- Update lerna.json
- Update the default package.json created by lerna:
From there I needed to start adding the package repositories into the monorepo. There are many and varied resources to help with creating a monorepo but the conversion I had planned was more complicated than most articles I found on the subject.
My initial attempt followed the simplest possible approach, git subtree:
git subtree add -P modules/name ../name master
This retained the commit log history, but it didn’t show file diffs in each commit (I guess due to the file location changes?). I’m not sure what the use case for this command is.
My second attempt was with Lerna’s
import command. This filtered every commit, making it very very slow (8867 commits across 22 repositories) and the resulting git history structure did not impress me.
After digging through more articles, and finding the right words to search for, I came across a detailed approach to merging git repositories without losing file history by Eric Lee (via his Stack Overflow post).
This technique checks out the source repository
master, moves all files into a subfolder, then merges that into the monorepo
master. SeemsGood. I only needed to make small adjustments to this process, I can post the script if there are requests but most of the details are specific to TinyMCE and our internal git server.
Once the repositories are imported my script uses
sed and a few other tricks to adjust common tooling so it works inside the monorepo:
- Move all
devDependenciesto the monorepo root to avoid diverging versions
- Switch all packages to TypeScript project references and build mode
- Switch all packages to LGPL, matching TinyMCE (most of the source projects were Apache 2.0 licensed)
npx cmdno longer work
- This was a simple fix, we now use
- This was a simple fix, we now use
load-grunt-tasksfound no tasks to load because it required the task modules be in a local
node_modulesbut the monorepo moved all of those to the root. So I had to get creative:
pattern: ['grunt-*', 'rollup']
grunt.loadNpmTasks()also found no tasks to load
- This was deleted, replaced by pattern additions to the
- This was deleted, replaced by pattern additions to the
From here I made constant tweaks to the import script and began finding issues I could fix in the source repositories instead of patching with my script:
- Repositories with CRLF in their files (TinyMCE standardises on LF). The way
git mergeis configured by the script it was performing renames + line ending conversions but only committed the rename. This did not happen often so was easy to fix when it came up.
- In late 2018 we built a more modern API for our AJAX library, jax, but only deployed it to premium plugins. We decided that rather than standardise the monorepo on the old API, we would completely replace the open source jax with this new code.
- I took the opportunity to use BFG Repo-Cleaner to strip out binary history from two projects before they were imported. This brought the monorepo
git pullsize down from 23MB to 11MB.
- We use webpack a lot in development, and for a long time have relied on
awesome-typescript-loader. We are big fans of
atl, and we still use it for the demo environment, but testing in the monorepo was just too slow without support for TypeScript project references. So we switched testing to
ts-loader, which does support them, via a seemingly innocuous commit message 😉
TS2563: The containing function or module body is too large for control flow analysis.
This one turned out to be easy to fix and sped up all projects, so in it went to the open source project months ago.
- I sneakily deployed a monorepo-specific change to resource loading in tests. The test framework added resource aliasing for yarn workspaces, with an otherwise pointless local alias, except I switched TinyMCE tests to make use of the local alias allowing all tests to run inside the monorepo without patching from the script.
I poked and prodded at this on and off for a couple of months until I had my monorepo dev environment working, the tests all passed, and I could start thinking about versioning / publishing. I’m the sort of person who doesn’t totally trust documentation; I want to mess around and explore the commands to see what they do.
This required extreme caution. I had imported live NPM module source code so a rogue
npm publish could have dire consequences. After spending some time logged out of NPM to be safe I hit upon Lerna’s
publishConfig setting which let me constrain all publishing to a specific NPM registry.
We publish to NPM from CI very regularly, so at first glance
from-package seemed like the best path forward to match how we had been developing. After some discussion we decided to switch to automated
lerna publish patch. Independent versioning on 22 packages will potentially create hundreds of version tags a year but we love automation and cleaning up a tag mess can be scripted. We do still use
from-package to account for manually updating minor and major versions, but we are hoping to explore conventional commits to later automate the entire release process.
As a final step I leveraged
lerna changed to create a new CI script that only runs tests on changed packages. This reduces CI build times and further improves the iteration speed of developing in the monorepo.
Five months in the planning, and after a week of internal testing, nearly 9000 new commits and the TinyMCE monorepo are now live. I’ve had a lot of fun building it and I hope it works as well for our contributors as it has already done for our own development.
Today marked the release of TinyMCE 5 and the launch of our shakeup of the project with the hope to see it modernise and move forward into the future. This shakeup, from my perspective, comes in four parts.
The first big change is that version 5 was developed under the stewardship of the Editor Platform team in Brisbane, Australia, where I am the Technical Lead. Our developers in Sweden are still involved – TinyMCE wouldn’t be TinyMCE without them – but now the team assigned to TinyMCE 5 is triple the size we had previously (and there was an expansion to 4x at one point during the project). Everyone is excited about the things this makes possible.
The second big change is the switch to a more rigorous Functional Programming style of development, not only in the code but also the project. We have many small self contained libraries that we now compose together particularly in the new
The team in Brisbane has been working in this way for a number of years while developing our Textbox.io editor technology; moving onto TinyMCE we have begun to apply those techniques there as well. I could (and should) do an entire post on the details, but if you browse the codebase and see imports from weird-sounding libraries like
boulder that’s us.
The libraries we built had been making their way into TinyMCE in bits and pieces over the last year or two, but their use went into full effect with this release.
That brings me to the third big change and headline feature of TinyMCE 5, the new
silver theme and
oxide skin. The entire
src/ui folder has been deleted, along with the
modern theme, and we started fresh. With the blessing and help of TinyMCE lead developer Johan “spocke” Sörlin we built a new API designed to be modern, flexible, and most importantly abstracted away from the DOM. Taking inspiration from virtual DOM style libraries this structure serves two high level goals:
- Custom dialogs fit into the new TinyMCE style without any effort, including the multitude of UI variants the new skin makes possible.
- Implementation details of components can be changed without breaking the API. This was a very clear goal to not replicate the “everything and the kitchen sink” approach of previous TinyMCE interfaces.
This breaking API change is the one that will be most obvious for developers migrating to TinyMCE 5, and also the most contentious. We have already received feedback during the pre-release cycle that the new API does not replicate v4 functionality in areas that are important to our developers. We hope to address these needs in the near future but we will be doing our best to not compromise either goal.
The final change I’d like to talk about is one that we are working on but isn’t yet ready. In the last few years it has became clear to the Editor Platform team that while our split library approach allowed for excellent separation of concerns, it was frustrating to develop with when changes were required across multiple libraries. This manifested in a significant way during TinyMCE 5 development, is a common story as the number of libraries in a project expands, and the common answer to these problems is a monorepo. It isn’t a simple change, although it appears to be easier than it used to be, I’m keeping notes and plan to do a detailed write-up of the process.
So the shake up still in progress is that at some point soon the TinyMCE repository will most likely be converted to a monorepo – everything there today will move into a subfolder, and the entire 7000 commit history of the libraries that are used in TinyMCE 5 will be merged into the master branch under the same subfolder. How that looks and when it happens is still yet to be determined, but initial signs are that it will ease these pain points for us.
I hope this change in direction for TinyMCE is well received by the community. We’re trying something new, initial feedback on version 5 has been positive and I’m confident the changes are for the best. We are listening, however, so if you think we’ve done something wrong or not well enough feedback is greatly appreciated.
I’ve had a John Carmack quote at the top of my CV pretty much since I started taking it seriously (which was actually after I got my first job with Ephox who I still work for).
This morning, I mentioned it to him in response to a speech he gave recently:
And well… the stats on my tweet tell most of the story, but a picture tells a thousand words:
Today is a good day.
There has been a lot of change in the OCaml -> JS landscape over the last couple of years, and I regularly see questions about what all the names mean. This is my attempt to sort out the world as I know it.
As you navigate the world of compiling OCaml to JS, most if not all of the following will come up at some point:
That’s a lot to remember, but each works in their own similar yet completely distinct way. They can also be mixed and matched as projects see fit. I’m going to use diagrams to attempt to explain all of this and hopefully provide some clarity.
With apologies to those who understand how over-simplified this is, here is a rough overview of how compilers work, specifically in the case of OCaml.
Code flows from top to bottom in three distinct phases:
- Syntax is read into an Abstract Syntax Tree
- Checks are done to confirm the AST has valid semantics in the OCaml language
- For example, type checking
- Optimisation is performed at this level too
- The AST is written out to a machine executable
- OCaml supports both native output, like a C compiler, and platform-independent bytecode output like a Java compiler (although not using JVM bytecode, OCaml has its own unique VM).
Are you with me so far? Good. Next we’re looking at the development that happened in 2011 with the creation of js_of_ocaml.
- It can be a bit slow, JSOO is effectively a second compiler
- The resulting JS is mostly unreadable machine code (since that’s more or less what JSOO had to start with). Source maps help here, but they aren’t a silver bullet.
In early 2016, Bloomberg open sourced their answer to this process. Instead of treating the compiler as a black box and working with the result, they dug in and replaced the output phase with Bucklescript.
This no doubt took a lot of effort to achieve, but the result has some unquestionable benefits:
- By working with the compiler internals, their output retains the structure of the original code
- Most if not all of the compiler speed is retained
Really the only downside I can see is that by working so deep in the compiler they have to chase new compiler releases (as I write this it is still based on the OCaml compiler released in July 2015). Not that this is a particularly bad thing as the OCaml compiler is fairly stable – and it has since been further mitigated as we’ll see in a moment.
But here is where the real fun begins. If you compare the reason diagram to earlier diagrams, you’ll begin to wonder if it could be combined with with either js_of_ocaml or bucklescript. And that’s exactly what happened. When I first looked at this, efforts were made to support both as ways to compile reason source code to JS. That’s still possible, but like all good communities a single recommended approach is appearing and the tools are moving in that direction.
In the last few months the community has settled on the solution that is very fast and produces output JS that can be very similar to the input reason code. Native compilation and tooling still uses the reason compiler, but bucklescript has added first-class support for reason syntax directly into their compiler.
Wow. Check out where we are now.
Come join the fun in the reasonml discord!
My post describing why I was interested in OCaml was originally intended as an internal document at my office arguing that we should invest in an AltJS project. The failure of that effort was unfortunate, but the post opened up a gold mine of support. Not just once, but twice (we’ll get to that in a moment).
I saw confusion from people who hadn’t heard of OCaml, and some attackers, but plenty of people defending it. It was a fairly normal programming language discussion, to be honest, which struck me as impressive for a language that wasn’t mainstream and vindicated my choice.
Writing up my thoughts on this would’ve been a good idea three years ago, and certainly two years ago after the second traffic bump, but despite the probable lack of value in these links now I still want to get these thoughts out on record.
So let’s start with the morning after the post went up (due to time zones most of the discussion happened while I was asleep). I saw wordpress notifications that my stats were booming, most referrals pointing to this reddit thread. A coworker captured a screen shot of the hacker news front page for me, with discussion happening in this Hacker News thread. More than 15000 views on that single day.
But almost a year later, in February 2015, it happened again:
Thanks to someone else posting it to Hacker News:
The total stats are what I really want to highlight. 18309 views in March 2014, plus another 15101 in February 2015, have raised this post to great exposure. It has legs, too, continuing to drive 100-200 views a month.
And yet I’ve been silent since then. Looking back at the two hacker news threads now, I don’t think I even read the 2015 comments in great detail. The failure of my efforts in 2014 pretty much lead to burnout on all coding outside of work hours. I didn’t even celebrate the 10th anniversary of creating this blog (November 2015). It was all just… nothing.
I was very excited about all of this in 2014. Thanks to some prompting from Jordan Walke on twitter, who gave me a really interesting gist link to the ocaml-based API that he was hoping to use for react, I embarked on creating a good quality ocaml-js wrapper. One that maintained pure ocaml as much as possible, unlike later efforts. I have kept it private all this time due to the embarrassing lack of progress since burning out, but it’s relevant now so here we go.
Note the distinct lack of commits after about a month. I am very proud however that the MyComponent file makes no reference to js_of_ocaml, and the example code only uses it to pass a DOM element to the render function. This is how web coding should be, and I’m very happy to see that it is the direction we’re headed.
So why am I surfacing now? In a similar parallel to last time, it’s because discussions at work have finally returned to AltJS. I have a new post brewing in my head about the current state of OCaml and JS, but I wanted to get this one out first. I’m doing it for my own record and for posterity, but feel free to get the discussions rolling again 😀
yes I do. That draft was created just after I joined twitter, and in the details I had written that I had just passed 400 tweets. I’m now closing in on 13,000. I am, however, in the process of scaling back my twitter and Facebook usage – I identified last week that as I reduce my tweeting I might return to blogging for my creative output.
So. 15,500 hits and counting for my first real post in 3 years. That’s a pretty high bar to live up to 😉