Why

Even with the excellent Lua luv async library, uv.new_async is not truly multithreaded because of Neovim's event loop architecture (source).

Also, because Rust is awesome and you love it 🦀

How

Let's take a look at how Neovim attempts to load Lua modules by looking at the output when you attempt to load a missing module.

E5108: Error executing lua [string ":lua"]:1: module 'my_module' not found:
        no field package.preload['my_module']
        no file './my_module.lua'
        no file '/opt/homebrew/Cellar/luajit-openresty/2.1-20210510/share/luajit-2.1.0-beta3/my_module.lua'
        no file '/usr/local/share/lua/5.1/my_module.lua'
        no file '/usr/local/share/lua/5.1/my_module/init.lua'
        no file '/opt/homebrew/Cellar/luajit-openresty/2.1-20210510/share/lua/5.1/my_module.lua'
        no file '/opt/homebrew/Cellar/luajit-openresty/2.1-20210510/share/lua/5.1/my_module/init.lua'
        no file '/Users/mat/.cache/nvim/packer_hererocks/2.1.0-beta3/share/lua/5.1/my_module.lua'
        no file '/Users/mat/.cache/nvim/packer_hererocks/2.1.0-beta3/share/lua/5.1/my_module/init.lua'
        no file '/Users/mat/.cache/nvim/packer_hererocks/2.1.0-beta3/lib/luarocks/rocks-5.1/my_module.lua'
        no file '/Users/mat/.cache/nvim/packer_hererocks/2.1.0-beta3/lib/luarocks/rocks-5.1/my_module/init.lua'
        no file './my_module.so'
        no file '/usr/local/lib/lua/5.1/my_module.so'
        no file '/opt/homebrew/Cellar/luajit-openresty/2.1-20210510/lib/lua/5.1/my_module.so'
        no file '/usr/local/lib/lua/5.1/loadall.so'
        no file '/Users/mat/.cache/nvim/packer_hererocks/2.1.0-beta3/lib/lua/5.1/my_module.so'

You'll notice that it attempts to load my_module.so, which is a dynamically linked library, which makes sense, considering that Lua is just an embeddable language, implemented in C.

Any shared library that exposes the correct ABI (Application Binary Interface) can be loaded directly as a Lua module into the Lua runtime.

Enter Rust

The Rust crate mlua provides high-level native Rust bindings to Lua runtimes. Using mlua, we can compile a Rust project to a shared library exposing the proper ABI to be loaded as a Lua module.

Setting Things Up

To get started, run cargo new --lib my_plugin to create a new Rust library project called my_plugin. Then, in your Cargo.toml, add the following lines:

[lib]
crate-type = ["cdylib"]

[dependencies]
mlua = {version = "0.6", features = ["luajit", "vendored", "module", "macros", "send", "async"]}

For our use, it's important to add the features option and include, at a minimum, luajit, vendored, and module.

According to the README.md file for the mlua crate , if you are on macOS, you'll also need to create a .cargo/config file with the following contents:

[target.x86_64-apple-darwin]
rustflags = [
  "-C", "link-arg=-undefined",
  "-C", "link-arg=dynamic_lookup",
]

[target.aarch64-apple-darwin]
rustflags = [
  "-C", "link-arg=-undefined",
  "-C", "link-arg=dynamic_lookup",
]

Next, set up a Makefile to make some of the build tasks easier. Create a file called Makefile and add the following:

.PHONY: build
build:
    cargo build --release
    rm -f ./lua/libmy_module.so
    cp ./target/release/libmy_module.dylib ./lib/libmy_module.so
    # if your Rust project has dependencies,
    # you'll need to do this as well
    mkdir -p ./lua/deps/
    cp ./target/release/deps/*.rlib ./lua/deps/

This will create a make target called build such that when you run make build, it will build your Rust library, then copy the generated libmy_module.dylib, renaming it to libmy_module.so (since Neovim will not attempt to load dylib files) and places it in the ./lua/ directory so that it will be loaded into Neovim's Lua runtime path.

Additionally, the last 2 lines will copy and required rlib Rust libraries into ./lua/deps/, so that your Rust library will be able to find and load its dependencies.

A Basic Rust Lua Module

The file src/lib.rs, which should've been generated by the cargo new command above, is the main entry point to our Rust library, which will be loaded as a Lua module. In src/lib.rs we can write something as simple as the following:

use mlua::prelude::{Lua, LuaTable, LuaResult};

fn greet_people(lua: &Lua, names: Vec<String>) -> LuaResult<LuaTable> {
  let strings = lua.create_table()?;
  for (i, name) in names.into_iter().enumerate() {
      // i + 1 because Lua indexing starts at 1 instead of 0
      strings.raw_insert((i + 1).try_into().unwrap(), format!("Hello {}!", name))?;
  }

  Ok(strings)
}

fn my_module(lua: &Lua) -> LuaResult<LuaTable> {
  let exports = lua.create_table()?;
  exports.set("greet_people", lua.create_function(greet_people)?)?;
  Ok(exports)
}

This is equivalent to writing the following in Lua:

local M = {}

function M.greet_people(names)
  local strings = {}
  for _, name in pairs(names) do
    table.insert(strings, 'Hello ' + name + '!')
  end
  return strings
end

return M

Trying It Out

Once we have all this in place, run make build to compile your Rust code and copy the shared library to the ./lua/ directory. Then, we can add our plugin to Neovim's Lua runtime path using packadd or the excellent packer.nvim plugin manager:

-- install packer if it isn't already installed
local install_path = fn.stdpath('data')..'/site/pack/packer/start/packer.nvim'
if vim.fn.empty(fn.glob(install_path)) > 0 then
  vim.fn.system({'git', 'clone', 'https://github.com/wbthomason/packer.nvim', install_path})
  vim.cmd('packadd packer.nvim')
end

local packer = require('packer')

packer.startup(function(use)
  -- packer.nvim can manage itself
  use('wbthomason/packer.nvim')
  -- load your module from the local directory path
  use('~/git/my_module')
end)

Now, in Neovim, once we've run :PackerSync to install our new module, we can run

:lua print(vim.inspect(require('my_module').greet_people({ 'Mat', 'Jeff', 'Shannon' })))

And we should get some output like the following:

{
  'Hello Mat!',
  'Hello Jeff!',
  'Hello Shannon!'
}

A Real World Use Example

I wanted to be able to search Dash.app, a documentation aggregator app for MacOS, from Neovim with a Telescope picker, so I started writing a plugin to do so.

One feature I added was the ability to automatically filter the Dash search with a keyword based on the filetype of the buffer you currently have open. For example, searching "your query" from a javascript buffer would transform the query into javascript:your query, which filters Dash to only search the Javascript docset. This isn't very useful in Typescript files though, for example; in a Typescript file, you'd probably want to search javascript, typescript, and nodejs docsets, all at once.

I had originally implemented this in Lua, by running the three queries, one after the other. This, of course, had serious performance issues and wasn't very usable. I tried parallelizing it with Neovim's uv.new_async, and while it was slightly better, it didn't really give me the results I'd hoped for.

So, I reimplemented this with a Rust backend, which handled running all the queries in parallel, packaged them all up, and sent them back to a Lua client. An added bonus of this improved separation of concerns was that it made it relatively easy to pump data from the Rust backend into any Lua client, so the plugin now supports multiple fuzzy-finder plugins, not just Telescope.

You can check out my Rust plugin, Dash.nvim below.