Skip to content

Run a YAML File with Bash!

Wait, what?!!?

Run a YAML file?

And with Bash?

How is that possible?

Step 0🔗

Let's run this simple YAML file.yaml:

name: Fido
age: 3
dog years: age * 7

When we run this file, we'd like to see this JSON output:

{
  "name": "Fido",
  "age": 3,
  "dog years": 21
}

$ bash file.yaml
file.yaml: line 1: name:: command not found
file.yaml: line 2: age:: command not found
file.yaml: line 3: dog: command not found

Well that didn't work...

Let's not give up yet.

Step 1🔗

Let's forget about Bash for a moment and just load the YAML file with YS External link :

$ ys -J file.yaml
{"name":"Fido", "age":3, "dog years":"age * 7"}

Close, but look at the dog years value. It's just a string.

We need to tell YS to evaluate the expression age * 7 as a math expression.

YS won't do any evaluation unless we tell it to with the !YS-v0: tag. Then we need to tell YS that just the dog years value should be evaluated. We do that with the :: syntax:

!YS-v0:
name: Fido
age: 3
dog years:: age * 7

Let's run it:

$ ys -J file.yaml
Error: Could not resolve symbol: age

Oh, right. The age symbol is not defined here. Luckily we can define it inline:

!YS-v0:
name: Fido
age =: 3
age:: age
dog years:: age * 7

Again:

$ ys -J file.yaml
{"name":"Fido", "age":3, "dog years":21}

Sweet! We have the JSON output we wanted.

Note: We could do the same thing with an anchor and alias:

!YS-v0:
name: Fido
age =: &age 3
dog years:: 7 * *age

Now Back to Bash🔗

OK. But this YAML file is definitely not a Bash script.

But maybe we can make it one?

Let's try this:

#!/usr/bin/env ys-0
source <(echo 'export YS_FORMAT=json' && curl '-s' 'https://getys.org/run') "$@":
---
!YS-v0:
name: Fido
age: &age 3
dog years:: 7 * *age

Now:

$ bash file.yaml
{"name":"Fido", "age":3, "dog years":21}

Woah! It worked! We just ran a YAML file with Bash!

But how?

Well I cheated a bit and showed you the second time I ran it. The first time I ran it, I got this:

$ bash file.yaml
Installing YS CLI '/tmp/yamlscript-run-ys/bin/ys-0.1.95' now...
Ctl-C to abort
See https://yamlscript.org/doc/run-ys for more information.

Installed /tmp/yamlscript-run-ys/bin/ys - version 0.1.95
--------------------------------------------------------------------------------
{"name":"Fido", "age":3, "dog years":21}

The first time you run the file, it installs YS under /tmp and then runs the file with YS. The second time you run it, it just runs the file with YS.

The Polyglot Magic🔗

It turns out that the first two lines of the file are both valid Bash and valid YS!

The first line is a Bash shebang that invokes YS, but we ran the file with Bash.

Let's change the shebang to call Bash instead:

#!/usr/bin/env bash
source <(echo 'export YS_FORMAT=json' && curl '-s' 'https://getys.org/run') "$@":
---
!YS-v0:
name: Fido
age: &age 3
dog years:: 7 * *age

Then let's make the file executable, and run it:

$ chmod +x file.yaml
$ ./file.yaml
{"name":"Fido", "age":3, "dog years":21}

Works like a charm!

Ok so what about the second line?

For YS, the source command is a no-op. A function that ignores its arguments and returns nothing.

For Bash, we are sourcing a "process substitution" that curls a tiny script that will install a YS binary under /tmp and use it to run that file.

We also set the YS_FORMAT environment variable to json, which tells YS to --load the file starting in data mode.

Update

It seems that you can't source a process substitution in Bash 3.2 (the default /bin/bash on MacOS). Fortunately, this variation works on all the versions of Bash used on Linux and MacOS:

source /dev/stdin <<<"$(echo 'export YS_FORMAT=json' && curl '-s' 'https://getys.org/run')" "$@":
How can that be valid YS?

Again, this is both valid Bash and valid YS! Here's how YS compiles the source line to Clojure code:

$ ys -c -
!YS-v0
source /dev/stdin <<<"$(echo 'export YS_FORMAT=json' && curl '-s' 'https://getys.org/run')" "$@":

(source
#"dev"
stdin
<<<
(echo "export YS_FORMAT=json" && curl "-s" "https://getys.org/run")
"$@")

The YS source function ignores its arguments, but those arguments need to be valid YS code. Ironically here /dev/stdin compiles as the regex form /dev/ followed by the symbol stdin! Also <<< is valid syntax for a YS operator, but currently there is no operator called <<< in YS.

But Why?🔗

Ok, neat trick, but how is this useful in the real world?

First off it's a great way to write a YS program for someone who might not have YS installed. You don't even need to tell them about YS at all.

But for data files there's a more interesting use case.

Say you have a config.yaml config file for some service that you'd like to be able to use YS for instead of plain YAML. If that service already used YS to load its YAML configs then you'd be all set. But most services don't. Yet!

What if you could put a config.ys file in the same directory as the config.yaml file and made it executable? And what if the config.ys file wrote its output to the config.yaml file when it was run? As long as you could set things up to make sure the config.ys file was run whenever it (or one of its dependencies) changed, you'd have a generated config.yaml file that would do everything you wanted it to do.

Let's try it out. We'll make a config.ys file that loads an other.yaml file and uses environment variables to set some values:

!YS-v0:
name: My Thing
stuff:: load('other.yaml').stuff
:when ENV.DEBUG.?::
  environment:: ENV

And an other.yaml file:

stuff:
  some: data

If we load it with YS we get:

$ ys -Y config.ys
name: My thing
stuff:
  some: data

This pulled in a part of the other.yaml file. It also would add a map of all your environment variables if DEBUG was set.

Let's add 2 lines to make it write to config.yaml: 

!YS-v0:
name: My thing
stuff:: load('other.yaml').stuff
:when ENV.DEBUG.?::
  environment:: ENV

--- !YS-v0
write FILE.replace(/\.ys$/ '.yaml'): _:yaml/dump

Now we can run it:

$ ys config.ys && cat config.yaml
name: My thing
stuff:
  some: data

Perfect! Since YS evaluates to the last document in a multi-document file, we get what we want by running it.

The only remaining problem is that we need to have ys installed to run this...

Or do we?

Let's try it with Bash:

#!/usr/bin/env bash
source <(curl '-s' 'https://getys.org/run') "$@":
--- !YS-v0:

name: My thing
stuff:: load('other.yaml').stuff
:when ENV.DEBUG.?::
  environment:: ENV

--- !YS-v0
write FILE.replace(/\.ys$/ '.yaml'): _:yaml/dump

Now:

$ chmod +x config.ys
$ ./config.ys && cat config.yaml
name: My thing
stuff:
  some: data

Voila! We just replaced a config.yaml file with a config.ys file for any YAML configured service.

A More Solid Approach

The strategy for using YS files to generate YAML files is solid. But you probably don't want to actually run them (with Bash or YS).

A better way is to simply configure your pipeline to install YS and then run ys -Y config.ys > config.yaml at the appropriate time.

Conclusion🔗

I hope you've enjoyed this little journey into the world YS, YAML, and Bash.

Hopefully it gives you some interesting ideas for how to use YS in your own projects.

Please let us know if you come up with any super cool uses for this technique.

Happy YAMLing!

PS Ingy (that's me!) will be at KubeCon in London External link next week.

Let's talk about YS, HelmYS, or whatever you'd like!