Lua is a much cooler and more useful language than I think it, publicly, gets credit for. I know it’s widely used in many industries but so many times I’ve read an article like this and thought that it could be even more widely useful.
I’m currently writing a just for me static site generator in Fennel and it’s sad to see just how many powerful packages are abandoned. A lot of it is because they are mostly complete and part of why I like it is lack of churn but a last update on 18 months ago is very different from 10 years.
I want to host knitting patterns with my site generator and techniques like this make it super easy to build a little pattern dsl. It’s stuff like that which makes me wish it was more widespread.
so many times I’ve read an article like this and thought that it could be even more widely useful
Lua is not widely used because it doesn’t exist. Lua {5..1-4,JIT} do exist, but these are different languages between which you and others can easily port code sometimes. It’s sort of cool in a moment, but if (when) you live through a couple of versions, that just gets old. It’s not immediately visible, so you’ll see reviews from 5 stars to zero at all times.
Actually, Lua exists all over the place, in all kinds of amazing applications - but because Lua is an embeddable VM, you mostly don't know about it that much. Its not exposed because - when you use Lua professionally, you tend to bury it deep in the problem it solves for you.
> Lua is a much cooler and more useful language than I think it, publicly, gets credit for.
I think it’s gets about the right amount of credit. It isn’t obscure by any means, openresty is used quite a bit in odd places, so many games have used it. PyTorch (well what became it) was originally written in it (it was just torch but basically LuaTorch)
It doesn’t get more adoption because its warts and limitations really start to show once pushed into more into a “real” language. It doesn’t have the browser monopoly. It’s also quite slow, and LuaJIT doesn’t always magically fix that.
Lua without JIT is one of the fastest interpreted languages. In fact it might just be the fastest. LuaJIT is also one of the most advanced JIT compilers out there, frequently beating V8 and others.
There are very valid complaints about Lua. It lacks useful quality of life features. Some of its design decisions grate on users coming from other languages. Its standard library is barebones verging on anemic, and its ecosystem of libraries does not make up for it.
But after using Lua in many places for well over a decade, I’ve gotta say, this is the first time I’ve heard someone claim it’s slow. Even without JIT, just interpreter to interpreter, it’s consistently 5-10x faster than similar languages like Python or Ruby. Maybe you’re comparing it to AOT systems languages like C and C++? That’s obviously not fair. But if you put LuaJIT head to head with AOT systems languages you’ll find it’s within an order of magnitude, just like all the other high quality JITs.
Vanilla Lua traditionally doesn’t show any substantial difference to python or ruby. You’re probably talking about LuaJIT in interpreter mode (jit off).
> Lua without JIT is one of the fastest interpreted languages.
Maybe I should say it is "still slow". Of course you can find examples of 5-10x over python performance (not that this is saying much), this is not the norm. There's just no way, PUC-Lua is still fundamentally dynamic hash tables for lookup with a big switch statement, there's a certain limit to how fast this can go.
Let's just say there are plenty of game devs that have found that Lua quickly becomes too slow for the task it has been pressed into which then becomes an exercise of exporting more and more stuff to C++ or whatever. There is Luau of course, which sort of proves this point.
These interlanguage comparisons are mostly meaningless anyway. Python may be slow, but you do the heavy lifting in other ways and that ecosystem is huge. Since the Lua ecosystem is small though you are usually on your own. LuaJIT FFI is sorta cool, but wrapping can still be a PITA.
> Maybe you’re comparing it to AOT systems languages like C and C++? That’s obviously not fair.
Or C#/.NET/Java/Kotlin/JVM. Which you can often use as part of a plugin system.
> this is the first time I’ve heard someone claim it’s slow.
Quite ironic to say this, because I'm pretty sure this is why Mike Pall created LuaJIT in the first place. For example http://lua-users.org/lists/lua-l/2011-02/msg00742.html (but there are many other great in depth essays in the archive)
> LuaJIT is also one of the most advanced JIT compilers out there, frequently beating V8 and others.
No it's not these days. LuaJIT is awesome and may have been alien technology in 2005 but has a lot of missing functionality compared to the most advanced JIT implementations.
There were of course many examples of LuaJIT beating V8 on particular microbenchmarks through the years, and many of these are rehashed from over a decade ago and are outdated. I wouldn't say this means it "frequently" beats V8, especially in a practical sense.
Anyone that has much experience with LuaJIT and has stared at a few traces is aware of "NYI" - there is a ton of stuff that is not implemented in the JIT and means fallback to the interpreter, including even pedestrian uses of closures: https://github.com/tarantool/tarantool/wiki/LuaJIT-Not-Yet-I.... There is a ton of stuff that current V8 does to optimize real world larger applications that LuaJIT JIT will bail on.
LuaJIT does well when you write code in a particular style that exploits its strengths. It's not quite as amazing for general purpose use.
Elsewhere you suggest that a GC rewrite is actively being developed, but LuaJIT development has slowed tremendously, the mailing list is not very active and I wouldn't be holding my breath for anything major (this new GC has been talked about for well over a decade).
Not to mention there are some significant performance issues on aarch64. Love2D, a popular game framework, opted to not use JIT on macOS for this reason.
This is strictly about how much memory LuaJIT can address on that platform. There’s no “significant performance issues”. If your workload fits inside 2 GiB (IIRC) it will be fast. LuaJIT is a world class JIT.
For a long time a new garbage collector has been an open issue for LuaJIT, which would fix this particular issue, and make the language even faster. Last I checked this was actively being worked on.
> If your workload fits inside 2 GiB (IIRC) it will be fast.
This has nothing to do with the issue being mentioned here.
You are understating the severity of this issue and seem to be confusing it with a different issue that is no longer relevant in 2.1.
This is a nice example of one way to implement DSLs in a language that doesn't support them well.
IIUC, they're using Lua's language's function application syntax, combined with the ability to introduce a different namespace environment for function lookup, and then they're evaluating the tree of function calls at run time.
In languages that support doing this at compile time, through compile-time evaluation or compile-time syntax transformation, you can make run time faster. For example of syntax transformation, https://www.neilvandyke.org/racket/html-template/ shows an example of expanded syntax, where it's rendered some of the angle brackets and coalesced that with (escaped) HTML CDATA.
If you wanted to extend that, to support chunks of traditional HTML angle-bracket syntax inline with code (without using string literals), like in React, you could define a Racket reader to parse those chunks and transform them to parentheses syntax above.
I was building a bunch of html pages for an htmx frontend and a golang back end, and got really exhausted from using the builtin `html/template` library in golang. I kept trying to build reusable components, and it just didn’t work as I wanted it to.
I ended up doing this exact thing as mentioned in this blog post. https://github.com/DBarney/Glua
Granted this is just for me, and for prototyping. I put it on GitHub so I wouldn’t loose it.
Writing html got so much easier this way:
local function Component(data)
return div{"my name is", data.name}
end
return function(params)
html{
head{title="this is my page"},
body{
h1{"This is my small page I wrote"},
p{
"some content",
"more content"
},
div{"page has some dynamic values",params.count},
Component{name="daniel"}
}
}
end
Lua's LPeg library would also be a great option here. A couple of years ago I taught an interpreters class together with Roberto Ierusalimschy and we used LPeg for lexing and parsing. The end result was great. Even if you're not using PEGs for your implementation I would recommend spending 30 minutes and looking at LPeg.
While I’m surprised to see they aren’t writing about https://www.inf.puc-rio.br/~roberto/lpeg/ it looks like they’ve covered LPEG in a previous post, and their chosen method this time around is neat anyway!
While I can't speak for MicroLua, this can indeed be accomplished with vanilla Lua.
There are two things in play here:
First, the _ENV variable here is special.
It is implicitly used to lookup any identifier that has no visible binding, locally or in the surrounding scopes.
Thus a script just containing print("Hello world") is really _ENV.print("Hello world").
Usually _ENV is just implicitly defined to be the global environment (available as _G), but it can be overridden within a lexical scope to a custom value.
Second, the jump labels just make use of the alternate function call syntax of object:method(args), which is equivalent to object.method(object, args).
The whitespace is non-significant, which allows it to be written like that.
In combination with metatables you can use _ENV to track variable reads, calls etc. within a function, which you can (abuse) to create DSLs.
You can get an idea of whats possible by just tracking what we can intercept with _ENV.
Adding the following code after the pio_timer function, and running the script with Lua 5.4, already gives us quite a bit for just the first three lines of the function.
local loggerMetatable = {}
function loggerMetatable:__bnot()
print("Called bnot on " .. rawget(self, "name"))
end
function loggerMetatable:__call(...)
local name = rawget(self, "name")
print("Invoked " .. name .. " with " .. #table.pack(...) .. " args")
-- Return another logger table to visualize variable interactions
return setmetatable({ name = name .. "(...)" }, loggerMetatable)
end
function loggerMetatable:__index(key)
local name = rawget(self, "name")
print("Accessed key " .. key .. " on " .. name)
-- Return another logger table to visualize variable interactions
return setmetatable({ name = name .. "." .. key }, loggerMetatable)
end
local pio_env = setmetatable({ name = "_ENV" }, loggerMetatable)
pio_timer(pio_env)
It never fails to strike me as odd how polarizing Lua is. Some people (like me) find it really pleasant and fluid to use. Other people get hives from things like 1-based indexing and global-by-default. The intense complaints are just so silly, especially when the “better alternative” presented is typically Python, which is one of the worst languages I’ve ever used. Lua’s syntax and semantics are so much smaller and cleaner that it’s mindblowing how simple it makes rather complex and custom abstractions. When I use Lua I feel like I’m working on another level of abstraction than in other languages, almost a lisp, but with just the right amount of convenient / quality-of-life features that it makes everything trivial. Multiple inheritance, mixins, type constructors are all trivial classes, if you want them, or parts of them, or to do your own thing entirely, or you can choose none of it at all and merely build a very primitive C-composition scripting API for your library, and yes embedding languages inside it is therefore also trivial. I’ll never understand the mindset that leads people to hating it.
“But bro, Python has C libraries!!”
and?
Lua is built out of C semantics and has a beautiful C API
if I want native libraries, I’ll just take a little time to write a trivial wrapper for precisely as much functionality as I want from the library
Around the time when CoffeeScript was popular, there was also https://moonscript.org/ for Lua. Made by the very same prolific Leaf who authored this DSL tutorial.
Yup, and it's still in active production use (itch.io, for example).
There's even a fork of moonscript, called yuescript, which has a speedier parser, and includes macro's. Effectively, it's moonscript-2.0, and it's bundled directly into the dora-ssr game engine (the same dev is involved with both).
Of course, the "big dog" in the compile-to-lua space is fennel. The great thing with all of these options is that they don't have a runtime component- it's all lua semantics, all the way down. So you can mix & match from project to project, as your whim takes you - and all the while you're really just gaining more experience in building lua solutions (while not having to fight against some of lua's syntax)
Lua is a much cooler and more useful language than I think it, publicly, gets credit for. I know it’s widely used in many industries but so many times I’ve read an article like this and thought that it could be even more widely useful.
I’m currently writing a just for me static site generator in Fennel and it’s sad to see just how many powerful packages are abandoned. A lot of it is because they are mostly complete and part of why I like it is lack of churn but a last update on 18 months ago is very different from 10 years.
I want to host knitting patterns with my site generator and techniques like this make it super easy to build a little pattern dsl. It’s stuff like that which makes me wish it was more widespread.
You can learn about various opinions in this thread: https://news.ycombinator.com/item?id=42517102
so many times I’ve read an article like this and thought that it could be even more widely useful
Lua is not widely used because it doesn’t exist. Lua {5..1-4,JIT} do exist, but these are different languages between which you and others can easily port code sometimes. It’s sort of cool in a moment, but if (when) you live through a couple of versions, that just gets old. It’s not immediately visible, so you’ll see reviews from 5 stars to zero at all times.
Actually, Lua exists all over the place, in all kinds of amazing applications - but because Lua is an embeddable VM, you mostly don't know about it that much. Its not exposed because - when you use Lua professionally, you tend to bury it deep in the problem it solves for you.
> Lua is a much cooler and more useful language than I think it, publicly, gets credit for.
I think it’s gets about the right amount of credit. It isn’t obscure by any means, openresty is used quite a bit in odd places, so many games have used it. PyTorch (well what became it) was originally written in it (it was just torch but basically LuaTorch)
It doesn’t get more adoption because its warts and limitations really start to show once pushed into more into a “real” language. It doesn’t have the browser monopoly. It’s also quite slow, and LuaJIT doesn’t always magically fix that.
Lua without JIT is one of the fastest interpreted languages. In fact it might just be the fastest. LuaJIT is also one of the most advanced JIT compilers out there, frequently beating V8 and others.
There are very valid complaints about Lua. It lacks useful quality of life features. Some of its design decisions grate on users coming from other languages. Its standard library is barebones verging on anemic, and its ecosystem of libraries does not make up for it.
But after using Lua in many places for well over a decade, I’ve gotta say, this is the first time I’ve heard someone claim it’s slow. Even without JIT, just interpreter to interpreter, it’s consistently 5-10x faster than similar languages like Python or Ruby. Maybe you’re comparing it to AOT systems languages like C and C++? That’s obviously not fair. But if you put LuaJIT head to head with AOT systems languages you’ll find it’s within an order of magnitude, just like all the other high quality JITs.
Vanilla Lua traditionally doesn’t show any substantial difference to python or ruby. You’re probably talking about LuaJIT in interpreter mode (jit off).
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
> consistently 5-10x faster than similar languages like Python or Ruby
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
Python and Ruby have had steady increments of improvement over the last decade.
> Lua without JIT is one of the fastest interpreted languages.
Maybe I should say it is "still slow". Of course you can find examples of 5-10x over python performance (not that this is saying much), this is not the norm. There's just no way, PUC-Lua is still fundamentally dynamic hash tables for lookup with a big switch statement, there's a certain limit to how fast this can go. Let's just say there are plenty of game devs that have found that Lua quickly becomes too slow for the task it has been pressed into which then becomes an exercise of exporting more and more stuff to C++ or whatever. There is Luau of course, which sort of proves this point.
These interlanguage comparisons are mostly meaningless anyway. Python may be slow, but you do the heavy lifting in other ways and that ecosystem is huge. Since the Lua ecosystem is small though you are usually on your own. LuaJIT FFI is sorta cool, but wrapping can still be a PITA.
> Maybe you’re comparing it to AOT systems languages like C and C++? That’s obviously not fair.
Or C#/.NET/Java/Kotlin/JVM. Which you can often use as part of a plugin system.
> this is the first time I’ve heard someone claim it’s slow.
Quite ironic to say this, because I'm pretty sure this is why Mike Pall created LuaJIT in the first place. For example http://lua-users.org/lists/lua-l/2011-02/msg00742.html (but there are many other great in depth essays in the archive)
> LuaJIT is also one of the most advanced JIT compilers out there, frequently beating V8 and others.
No it's not these days. LuaJIT is awesome and may have been alien technology in 2005 but has a lot of missing functionality compared to the most advanced JIT implementations.
There were of course many examples of LuaJIT beating V8 on particular microbenchmarks through the years, and many of these are rehashed from over a decade ago and are outdated. I wouldn't say this means it "frequently" beats V8, especially in a practical sense.
Anyone that has much experience with LuaJIT and has stared at a few traces is aware of "NYI" - there is a ton of stuff that is not implemented in the JIT and means fallback to the interpreter, including even pedestrian uses of closures: https://github.com/tarantool/tarantool/wiki/LuaJIT-Not-Yet-I.... There is a ton of stuff that current V8 does to optimize real world larger applications that LuaJIT JIT will bail on.
LuaJIT does well when you write code in a particular style that exploits its strengths. It's not quite as amazing for general purpose use.
Elsewhere you suggest that a GC rewrite is actively being developed, but LuaJIT development has slowed tremendously, the mailing list is not very active and I wouldn't be holding my breath for anything major (this new GC has been talked about for well over a decade).
> LuaJIT doesn’t always magically fix that.
Not to mention there are some significant performance issues on aarch64. Love2D, a popular game framework, opted to not use JIT on macOS for this reason.
https://github.com/LuaJIT/LuaJIT/issues/285
This is strictly about how much memory LuaJIT can address on that platform. There’s no “significant performance issues”. If your workload fits inside 2 GiB (IIRC) it will be fast. LuaJIT is a world class JIT.
For a long time a new garbage collector has been an open issue for LuaJIT, which would fix this particular issue, and make the language even faster. Last I checked this was actively being worked on.
> This is strictly about how much memory LuaJIT can address on that platform
No it is not.
https://github.com/LuaJIT/LuaJIT/issues/285#issuecomment-197...
> If your workload fits inside 2 GiB (IIRC) it will be fast.
This has nothing to do with the issue being mentioned here. You are understating the severity of this issue and seem to be confusing it with a different issue that is no longer relevant in 2.1.
This is a nice example of one way to implement DSLs in a language that doesn't support them well.
IIUC, they're using Lua's language's function application syntax, combined with the ability to introduce a different namespace environment for function lookup, and then they're evaluating the tree of function calls at run time.
In languages that support doing this at compile time, through compile-time evaluation or compile-time syntax transformation, you can make run time faster. For example of syntax transformation, https://www.neilvandyke.org/racket/html-template/ shows an example of expanded syntax, where it's rendered some of the angle brackets and coalesced that with (escaped) HTML CDATA.
If you wanted to extend that, to support chunks of traditional HTML angle-bracket syntax inline with code (without using string literals), like in React, you could define a Racket reader to parse those chunks and transform them to parentheses syntax above.
> This is a nice example of one way to implement DSLs in a language that doesn't support them well.
It's not a coincidence that Lua supports such syntax for such use cases.
I love Lua for being able to do things like this.
I was building a bunch of html pages for an htmx frontend and a golang back end, and got really exhausted from using the builtin `html/template` library in golang. I kept trying to build reusable components, and it just didn’t work as I wanted it to.
I ended up doing this exact thing as mentioned in this blog post. https://github.com/DBarney/Glua Granted this is just for me, and for prototyping. I put it on GitHub so I wouldn’t loose it.
Writing html got so much easier this way:
Edit: formattingyou should try https://pkg.go.dev/maragu.dev/gomponents and do it all in go
Lua's LPeg library would also be a great option here. A couple of years ago I taught an interpreters class together with Roberto Ierusalimschy and we used LPeg for lexing and parsing. The end result was great. Even if you're not using PEGs for your implementation I would recommend spending 30 minutes and looking at LPeg.
Related:
Writing a DSL in Lua (2015) - https://news.ycombinator.com/item?id=22223542 - Feb 2020 (41 comments)
While I’m surprised to see they aren’t writing about https://www.inf.puc-rio.br/~roberto/lpeg/ it looks like they’ve covered LPEG in a previous post, and their chosen method this time around is neat anyway!
MicroLua for the RP2040 has a nice dsl for the PIO assembly with some special use of methods: https://github.com/MicroLua/MicroLua/blob/main/lib/pico/hard...
Was that really achieved with vanilla Lua or MicroLua has some language extensions?
While I can't speak for MicroLua, this can indeed be accomplished with vanilla Lua.
There are two things in play here:
First, the _ENV variable here is special.
It is implicitly used to lookup any identifier that has no visible binding, locally or in the surrounding scopes.
Thus a script just containing print("Hello world") is really _ENV.print("Hello world").
Usually _ENV is just implicitly defined to be the global environment (available as _G), but it can be overridden within a lexical scope to a custom value.
Second, the jump labels just make use of the alternate function call syntax of object:method(args), which is equivalent to object.method(object, args).
The whitespace is non-significant, which allows it to be written like that.
In combination with metatables you can use _ENV to track variable reads, calls etc. within a function, which you can (abuse) to create DSLs.
You can get an idea of whats possible by just tracking what we can intercept with _ENV. Adding the following code after the pio_timer function, and running the script with Lua 5.4, already gives us quite a bit for just the first three lines of the function.
local loggerMetatable = {} function loggerMetatable:__bnot() print("Called bnot on " .. rawget(self, "name")) end function loggerMetatable:__call(...) local name = rawget(self, "name") print("Invoked " .. name .. " with " .. #table.pack(...) .. " args") -- Return another logger table to visualize variable interactions return setmetatable({ name = name .. "(...)" }, loggerMetatable) end function loggerMetatable:__index(key) local name = rawget(self, "name") print("Accessed key " .. key .. " on " .. name) -- Return another logger table to visualize variable interactions return setmetatable({ name = name .. "." .. key }, loggerMetatable) end
local pio_env = setmetatable({ name = "_ENV" }, loggerMetatable) pio_timer(pio_env)
-- Output:
Accessed key public on _ENV
Accessed key start on _ENV
Invoked _ENV.public with 1 args
Accessed key set on _ENV.public(...)
Accessed key y on _ENV
Invoked _ENV.public(...).set with 3 args
Accessed key mov on _ENV
Accessed key y on _ENV
Accessed key y on _ENV
Called bnot on _ENV.y
Invoked _ENV.mov with 1 args
[...snip...]
It never fails to strike me as odd how polarizing Lua is. Some people (like me) find it really pleasant and fluid to use. Other people get hives from things like 1-based indexing and global-by-default. The intense complaints are just so silly, especially when the “better alternative” presented is typically Python, which is one of the worst languages I’ve ever used. Lua’s syntax and semantics are so much smaller and cleaner that it’s mindblowing how simple it makes rather complex and custom abstractions. When I use Lua I feel like I’m working on another level of abstraction than in other languages, almost a lisp, but with just the right amount of convenient / quality-of-life features that it makes everything trivial. Multiple inheritance, mixins, type constructors are all trivial classes, if you want them, or parts of them, or to do your own thing entirely, or you can choose none of it at all and merely build a very primitive C-composition scripting API for your library, and yes embedding languages inside it is therefore also trivial. I’ll never understand the mindset that leads people to hating it.
“But bro, Python has C libraries!!”
and?
Lua is built out of C semantics and has a beautiful C API
if I want native libraries, I’ll just take a little time to write a trivial wrapper for precisely as much functionality as I want from the library
I've always wanted to recreate a Python like syntax for Lua, this might be a good guide for that.
Around the time when CoffeeScript was popular, there was also https://moonscript.org/ for Lua. Made by the very same prolific Leaf who authored this DSL tutorial.
Yup, and it's still in active production use (itch.io, for example).
There's even a fork of moonscript, called yuescript, which has a speedier parser, and includes macro's. Effectively, it's moonscript-2.0, and it's bundled directly into the dora-ssr game engine (the same dev is involved with both).
Of course, the "big dog" in the compile-to-lua space is fennel. The great thing with all of these options is that they don't have a runtime component- it's all lua semantics, all the way down. So you can mix & match from project to project, as your whim takes you - and all the while you're really just gaining more experience in building lua solutions (while not having to fight against some of lua's syntax)
trailing closure syntax looks similar but is probably even a bit nicer. I just wish js would get this feature so we could stop using jsx.