Work in progress

The Arithmetic That Ran Before the Program

Rust Conversation R.2 — From Our Syntax Object to Compile-Time Evaluation

Ada · 01

Our first macro can preserve this new piece of code as text:

code_string!(add(2, multiply(3, 4)))
Alice

It would produce the string-literal expression "add(2, multiply(3, 4))". The macro would still know nothing about the arithmetic.

Ada · 02

This time we want an integer evaluator. Its language will be deliberately small:

integer literal
add(expression, expression)
multiply(expression, expression)

Every accepted input must become one i64 literal. eval_integer! does not promise to evaluate an arbitrary Rust expression.

Alice

I understand: do not just print strings; compute the actual result.

Ada · 03
println("{}", eval_integer!(add(2, multiply(3, 4))));
Alice

14

Based on your description, the eval_integer macro needs to expand into code containing the literal 14.

Ada · 04

Can it expand into

2 + (3 * 4)

?

Alice

I would say no. Although they are equal, that is not the same code as 14.

Ada · 05

Yes. In the macro logic, you are not just creating code; you must also complete the arithmetic. To do that, we need some way to convert the Rust code into actual Rust values. We begin by designing a type that can hold these expressions. In compiler theory, this is usually called an abstract syntax tree.

Alice

I would use:

enum IntegerExpr {
    Literal { value: i64 },
    Call(IntegerCall),
}

struct IntegerCall {
    function: String,
    arguments: Vec<IntegerExpr>,
}
Ada · 06

Very close, but there is an important distinction between an AST type and an ordinary data type. An AST type must be defined either as an atomic AST type, such as the number 2, or as a struct or enum whose fields are themselves AST types.

Alice

Suppose I want to store the code 2. What is its atomic AST type?

Ada · 07

It is LitInt. You have already seen the AST type for a string: LitStr. Revise your definition.

Alice
enum IntegerExpr {
    Literal { value: LitInt },
    Call(IntegerCall),
}

struct IntegerCall {
    function: LitStr,
    arguments: Vec<IntegerExpr>,
}

How can we convert written code into this AST?

Ada · 08

Ask Syn to produce the type named on the left:

let expression: IntegerExpr = syn::parse2(input)?;
Alice

It does not compile:

the trait `Parse` is not implemented for `IntegerExpr`

The Rust type exists, but Syn does not yet know how to map the input into it. Do we have to write a parser ourselves?

Ada · 09

Not for these regular shapes. We use syn_derive, a helper crate whose procedural macros generate the mappings from our type definitions. Our project usually requests both directions:

#[derive(syn_derive::Parse, syn_derive::ToTokens)]

Parse maps tokens into our value. ToTokens lets a later compiler stage put that value back into code.

Alice

So while we build eval_integer!, another procedural macro generates part of its implementation. A macro inside our macro project!

Ada · 10

Exactly. syn_derive::Parse runs when this macro crate is compiled and generates ordinary Rust code. Later, eval_integer! calls that generated code when it receives tokens from its caller.

Alice

For a struct, the generated mapping can read its fields in order. But the input contains neither the word Literal nor the word Call. How can it choose an enum variant?

Ada · 11

It cannot infer that choice. We mark the first alternative with a rule:

enum IntegerExpr {
    #[parse(peek = LitInt)]
    Literal { value: LitInt },
    Call(IntegerCall),
}

#[parse(peek = LitInt)] asks whether the next token has the LitInt shape without consuming it. If it does, select Literal. If it does not, continue to Call.

Alice

Then I can simulate it:

input 2          -> peek sees LitInt -> choose Literal -> value consumes 2
input add(...)   -> next token is not LitInt -> continue to Call

So peek only chooses the alternative. The selected variant's fields still consume and store the input.

Runnable source Evolve the ordinary enum into a parseable syntax choice examples/rust-02-compile-time/compile-time-macros/src/integer.rs Open lines 7–14
#[derive(syn_derive::Parse, syn_derive::ToTokens)]enum IntegerExpr {    #[parse(peek = LitInt)]    Literal {        value: LitInt,    },    Call(IntegerCall),}
Ada · 12

Exactly. Call has no peek annotation because it is the final fallback. Its fields must still match the remaining input. For example, input beginning with + reaches Call but then fails because its first field expects an identifier.

This one-token distinction is enough because we deliberately designed the two forms to begin differently. If two alternatives began the same way, we would need a different or more detailed selection rule. That general theory is outside this tutorial.

Alice

The Call(IntegerCall) variant delegates the remaining tokens to IntegerCall. My current version begins with function: LitStr, but the input contains add, not "add". Are those the same syntax shape?

*Optional theory.* The general study of choosing a top-down grammar alternative with bounded lookahead is called LL(k) parsing. See Rosenkrantz and Stearns, *Properties of Deterministic Top-Down Grammars*90446-8) (1970).

Ada · 13

No. LitStr represents a quoted Rust string literal. A bare name such as add is an identifier, represented by Syn's Ident:

function: Ident,

After that field consumes add, the remaining input is (2, 3). Does Vec<IntegerExpr> say where that list begins or which token separates its elements?

Alice

No. Vec can store the expressions after they have been found, but it does not describe the written parentheses or commas.

Ada · 14

Then evolve that field into a syntax-aware, comma-separated list:

#[syn(parenthesized)]
paren_token: token::Paren,
#[syn(in = paren_token)]
#[parse(Punctuated::parse_terminated)]
arguments: Punctuated<IntegerExpr, Token![,]>,
parenthesized                         recognize (...) and keep its delimiter
in = paren_token                     read the next field from inside (...)
Punctuated<IntegerExpr, Token![,]>   keep expressions separated by commas
parse_terminated                     use Syn's existing list mapping

Unlike Vec, Punctuated retains the separators so ToTokens can write the same call shape back into code.

Alice

Let me trace add(2, 3) from left to right:

add       -> function: Ident
(2, 3)    -> paren_token opens the nested input
2, 3      -> arguments: Punctuated<IntegerExpr, Token![,]>

Each argument is mapped as another IntegerExpr. Now every written part has a destination.

Runnable source Map the pieces inside one call examples/rust-02-compile-time/compile-time-macros/src/integer.rs Open lines 18–26
#[derive(syn_derive::Parse, syn_derive::ToTokens)]struct IntegerCall {    function: Ident,    #[syn(parenthesized)]    paren_token: token::Paren,    #[syn(in = paren_token)]    #[parse(Punctuated::parse_terminated)]    arguments: Punctuated<IntegerExpr, Token![,]>,}
Ada · 15

Both IntegerExpr and IntegerCall now derive the two mappings. Try the line that failed before:

let expression: IntegerExpr = syn::parse2(input)?;
Alice

Now it works. parse2 invokes the generated Parse implementation, which selects a variant and fills each syntax-aware field. Tokens have become our IntegerExpr.

Ada · 16

Now the input is an IntegerExpr. How would you define its value from the two enum alternatives?

Alice

A literal produces the integer it represents. A call first evaluates its arguments, then applies the named operation:

Literal(n)                 -> n
Call add(left, right)      -> evaluate(left) + evaluate(right)
Call multiply(left, right) -> evaluate(left) * evaluate(right)

The call cases are recursive because their arguments are themselves IntegerExpr values.

Ada · 17

Write only that enum dispatch first.

Alice

value.base10_parse() is new. Does it turn the stored LitInt syntax into the ordinary integer value needed by evaluation, while the Call case delegates to a helper?

Runnable source Dispatch on our two AST alternatives examples/rust-02-compile-time/compile-time-macros/src/integer.rs Open lines 63–68
fn evaluate(expression: &IntegerExpr) -> syn::Result<i64> {    match expression {        IntegerExpr::Literal { value } => value.base10_parse(),        IntegerExpr::Call(call) => evaluate_call(call),    }}
Ada · 18

Yes. base10_parse() is Syn's conversion from an integer-literal syntax value to a Rust integer. For calls, the helper renders the stored Ident as text, pairs it with arguments.len(), and matches the supported name-and-arity combination. Which two combinations should have meaning?

Alice

("add", 2) should evaluate its two children and add them. ("multiply", 2) should evaluate its two children and multiply them. Any other name or argument count should be rejected.

So the derived mapping accepts the general shape of a call, while our evaluator decides which call names and arities have meaning.

Runnable source Give add and multiply their meaning examples/rust-02-compile-time/compile-time-macros/src/integer.rs Open lines 74–75
        ("add", 2) => Ok(evaluate(&call.arguments[0])? + evaluate(&call.arguments[1])?),        ("multiply", 2) => Ok(evaluate(&call.arguments[0])? * evaluate(&call.arguments[1])?),
Ada · 19

We now have the token-to-IntegerExpr mapping, the evaluator, and Quote from R.1. How should expand_expression connect them?

Alice
let expression: IntegerExpr = syn::parse2(input)?;
let value = evaluate(&expression)?;
Ok(quote! { #value })

Tokens become our IntegerExpr, ordinary Rust evaluates it to an i64, and Quote turns that value back into code.

Runnable source Map, evaluate, and emit examples/rust-02-compile-time/compile-time-macros/src/integer.rs Open lines 39–43
pub(crate) fn expand_expression(input: TokenStream) -> syn::Result<TokenStream> {    let expression: IntegerExpr = syn::parse2(input)?;    let value = evaluate(&expression)?;    Ok(quote! { #value })}
Ada · 20

With the notes folder open in VS Code, expand eval_integer!(add(2, multiply(3, 4))) using rust-analyzer: Expand macro recursively at caret.

Alice

The expansion is 14i64. The call-shaped arithmetic has disappeared; only the computed literal remains.

Ada · 21

Now run the caller:

cargo run --manifest-path examples/rust-02-compile-time/Cargo.toml --package compile-time-demo
14
Alice

The executable prints the value of the generated literal. The expansion—not only the printed result—is the evidence that the macro performed the arithmetic.