Why YS Chose Clojure
So YS compiles to Clojure.
It also has a Clojure runtime and access to the Clojure ecosystem.
It's also written in Clojure. (It didn't have to be. The compiler could have been written in any language.)
Why Clojure?
I'll explore that topic today.
Do I have a Lisp?🔗
I thought of YS as a concept in mid 2022, but didn't go far with it. One thing that I decided (possibly years earlier) was it should be a Lisp. A Lisp in the sense that Lisp is code as data. Since YAML is a data language, it felt like code in YAML was code as data, which pretty much is the definition of Lisp.
But...
I didn't really know what a Lisp was or how to make one. I knew that a Lisp expression was parens around a function and arguments. That's about it.
(print (join "2 + 2 = " (+ 2 2)))
Something like that.
How to Make A Lisp🔗
I said "I didn't know how to make a Lisp" on purpose.
In early 2023 I googled "how to make a Lisp" and found this amazing project
called mal 
(Make A Lisp).
It's a complete 11 step tutorial on how to implement a rudimentary Lisp in any
language you want to.
Each of the 11 steps has a test suite that you need to get to pass before moving
on to the next step.
Last I counted, there were over 100 implementations in over 75 languages!
It took me 2 weeks to implement a Lisp in Perl. The "mal" Lisp is Clojure inspired. The final step is to use your mal to run (self-host) a mal implementation written in mal and have it pass all the tests! It still blows my mind a bit to think about it.
When I was done, I decided to see what the mal Lisp would look like in YAML. It looked awful! But I kept playing with it and it kept getting better. Soon I was pretty sure that YS could be a language that I would actually like programming in.
That said, it would still be a lot of work.
The Holy Graal(VM)🔗
I decided not to stop working on my Perl mal/Clojure Lisp, and to see how far I could get with making a Clojure hosted on Perl. I called the project Lingy.
I was attending a Perl conference in Toronto in July 2023 and decided to give a talk called "Lingy and YAMLScript".
Somehow this talk got noticed by the Clojure community and I was invited to join
their Slack.
I met the right people and soon learned about GraalVM
and SCI
.
SCI is a Clojure runtime and GraalVM's native-image tool can be used to
compile jar files (what Clojure compiles to) to native executables.
With these 2 things I realized that I could make YS be a real language in a
matter of months, rather than years.
To a large degree, I was right.
I had a usable language by the end of 2023.
When I realized that I could make YS a real language, I was fully committed from that point on. I've been working on YS ever since.
Of course, programming languages are never finished, and there's more to do than ever to have all the YS tooling and ecosystem in place for the best possible YS user experience.
Shared Libraries🔗
Maybe more important than GraalVM compiling YS to a native executable is its
ability to also compile YS to a shared library: libys.so.
With that, I knew I could make a YS capable YAML loader for almost every common programming language. In fact, without the shared library realization, I probably wouldn't have started working on YS in earnest.
I think we had 5 or 6 YAML loaders by the end of 2023. Today we have 15. The goal has always been (at least) 42!
Note
I literally have a list of 45 FFI capable programming languages that also have library registries. So I know we can do this.
The amount of code for a typical libys binding is about 100-200 lines of code. It involves FFI stuff, but with the help of LLMs these days, it only takes me a day or two to release a new language binding.
More often than not, learning how to use a new publishing registry is the hard part.
What's Good About Clojure?🔗
Probably the best things about Clojure is that it has a great core library and a great ecosystem.
If you want to make a new programming language these days, you really need to be building on the ecosystems of giants.
Clojure itself was built on top of Java; one of the biggest giants of the programming language world. From all I've read, that was very much on purpose.
Building YS on top of Clojure started with the promise that it would have a giant, Rich and robust ecosystem from the start.
One other awesome thing about Clojure is its commitment to not breaking code. It's ridiculously backwards compatible. I've yet to encounter a single bug introduced by a new version of Clojure (or Java for that matter).
What's the Problem with Clojure?🔗
From a YS perspective, the biggest problem with Clojure is that people assume it needs a JVM or Java knowledge to use it well. That's just not true.
The ys command line interpreter is just a single standard binary executable.
You don't even need to know Clojure to be able to get value from YS. On the other hand, to be a great YS programmer, you do need to know quite a bit about it.
I'll say personally that I think Clojure is a great language to learn. Especially if you don't already know a Lisp. Learning Lisp should be a requirement for anyone who wants to be a well rounded programmer, and Clojure is a great modern Lisp.
What's the Problem with GraalVM?🔗
Maybe my biggest current misgivings around YS and Clojure is that the GraalVM compiler:
- Is very slow to compile things (over a minute)
- Only supports Windows, Linux and macOS
- Creates large binaries
These aren't deal breakers, but they keep me thinking about other options.
One option that I haven't done much with yet is simply using the JVM. Once a JVM is started, it's very fast. Faster than the native compilations. The native compilations just start very fast. Alas, the JVM solution doesn't really fit the shared library binding model.
Another option is to explore Clojure hosted by other languages. ClojureScript is a great example of this. It runs on the web and NodeJS.
I'm even more interested in exploring Clojure hosted on Go or Rust. There are a few projects out there already playing with this.
There's also the promise of being able to compile Clojure to WebAssembly. It's gonna be ready any day now. Or so I've been told. (More than once. ;)
I hope so, because I think it would be a very big deal for YS.
Conclusion🔗
I'm quite happy with the choice of Clojure for YS. I think it's a sound foundation. It's like a Lego set that you can use out of the box and also build new domain specific Lego sets with.
It also has the potential to reach beyond the JVM and NodeJS to new exciting platforms.
YS effectively lets you use Clojure in YAML with a more YAML appropriate syntax, taking it places it may never have gotten to otherwise.