The Code That Became a Value

stringify!(1);

It takes a Rust expression as input and turns it into a string.

stringify!(1+1);

No, it actually gives you something like "1 + 1".

let x = 1 + 1;
stringify!(x);

What does this macro receive and return?

Exactly. stringify! takes the tokens spelling 1 + 1 as input and returns new tokens spelling another expression—"1 + 1".

stringify!(relation edge(src: i32, dst: i32);)

Because stringify! runs before Rust checks whether the tokens inside its invocation form a Rust expression. At first, Rust only needs to tokenize the contents and find balanced delimiters.

Right. What is a compiler?

Roughly, but not quite. A compiler translates code written in one language into code in another language. The target might be another source language, assembly, or machine code.

So, conceptually, we can begin with:

Compiler : Code -> Code

Who said that a macro cannot be a compiler?

What kind of code does stringify! return?

macro and compiler answer different questions. Calling something a macro tells us its expansion mechanism. Calling it a compiler describes a semantic job: translating a program from one language or stage into another. A macro can implement a compiler stage; not every macro is usefully understood as a compiler.

One way to build a compiler is to chain a series of macro calls. Each expansion makes one small change to the syntax and produces the next stage. After several rounds of expansion, our query has become ordinary Rust. The Rust compiler can then continue from Rust to assembly and machine code.

In this course, we will use this idea to build a compiler for a database query language. It will compile a database query into Rust code.

At the end of database Conversation 1.2, we had a query \(q\), a database instance \(I\), and the result relation \(q(I)\). Earlier, we represented each relation instance in Rust as a HashSet.

What would a HashSet look like in assembly?

Before we learn how to write the whole database-query compiler, let us learn how a macro works.

Try to implement stringify!.

Almost. Rust already has a built-in macro named stringify!, so we will give ours the distinct name code_string!.

Before we write the function, look at the files Cargo uses in the runnable example:

examples/rust-01-stringify/
├── Cargo.toml
├── stringify-macro/
│   ├── Cargo.toml
│   └── src/
│       └── lib.rs
└── demo/
    ├── Cargo.toml
    └── src/
        └── main.rs

Rust requires the procedural macro and the code using it to be in different crates. The macro crate must already be compiled before rustc can run it while compiling the demo crate.

The two nested manifests describe those two crates. The top manifest only groups them so one command can build both; it is not a third program.

Exactly. I prepare that shell before the meeting. We will read the caller, then write the macro; the checked-in source is the recovery copy. Begin with main.rs.

What behavior is this caller asking code_string! to provide?

Now open stringify-macro/src/lib.rs. Rust fixes the compiler-facing type, so write the boundary first and leave its body open:

use proc_macro::TokenStream;

#[proc_macro]
pub fn code_string(input: TokenStream) -> TokenStream {

TokenStream stores code as an ordered sequence of identifiers, literals, punctuation, and delimited groups—not as evaluated values. It is called a stream because macro code reads that sequence from left to right, one item at a time. A delimited group contains another token stream.

#[proc_macro] registers the function as a function-like macro; pub exports that entry point to the demo crate. Rust requires the function at the root of the macro crate.

Raw proc_macro token trees are tedious. Procedural-macro code normally uses proc_macro2 for preprocessing code token, Syn for turning code into rust object, and Quote for converting rust object back into rust code tokens.

use proc_macro2::TokenStream as TokenStream2;

let input = TokenStream2::from(input);

No. proc_macro::TokenStream is rustc's boundary type; proc_macro2::TokenStream is the type used by the helper libraries. The conversion changes the representation, not the tokens.

Since we need text for output, we can convert rust code into text:

let written_input = input.to_string();

Yes. It produces x and something like 1 + 1 as text, without preserving comments or exact whitespace.

For code_string!(x), we must return the Rust string-literal expression "x", not the String stored in written_input. Add Syn's typed representation:

use proc_macro2::{Span, TokenStream as TokenStream2};
use syn::LitStr;

let output_literal = LitStr::new(&written_input, Span::call_site());

The macro must return Rust syntax, not a runtime string value. LitStr is a typed value representing valid Rust string-literal syntax and handles escaping. Span::call_site() attaches the invocation's source context for the compiler and its diagnostics.

Exactly. Quote emits that typed syntax as code:

use quote::quote;

quote! { #output_literal }

quote! treats its body as a Rust-code template. #output_literal inserts the syntax represented by that value. The result is a proc_macro2::TokenStream containing the string-literal expression "x".

The complete line also closes the compiler boundary:

quote! { #output_literal }.into()
}

Quote returns proc_macro2::TokenStream not proc_macro::TokenStream. into did the conversion.

Now the implementation exists. Run the project and compare the result with the target we wrote before it:

cargo run --manifest-path examples/rust-01-stringify/Cargo.toml --package stringify-demo

1 + 1
x
relation road(src: City, dst: City);
2

Exactly. The macro ran at compile time; main ran later at runtime.

To inspect the boundary, put the caret on code_string!(1 + 1) in examples/rust-01-stringify/demo/src/main.rs, and run rust-analyzer: Expand macro recursively at caret. What should the expansion window contain?

The complete path is now compact:

input tokens -> TokenStream2 -> String -> LitStr -> quoted output tokens

Can the macro ever observe the value assigned to x?

Are you ready to write a query compiler using macro?

The missing object is a typed syntax tree for the input. R.2 first builds that object for a small integer language: we define its enum-shaped AST, derive its token mapping, and interpret the resulting Rust value. The same method will later give relation declaration its source shape.