Draft

The Matches the Algebra Did Not Find

Conversation 3.1 — From a Bound Selection to an Index

The matches the algebra did not find

Ada · 01

Our textbook keeps its running examples in a small cinema database. Here are its three relations; call them $M$, $P$, and $L$.

Movies

TitleDirectorActor
filmABergmanactor1
filmABergmanactor2
filmBOtheractor3

Pariscope

TheaterTitleSchedule
ChampofilmA18:00
ChampofilmA21:00
RialtofilmB20:00

Location

TheaterAddressPhone
Champoaddr1phone1
Rialtoaddr2phone2

The book asks for the names and addresses of theaters featuring a Bergman film. Drop Bergman for now: which theaters show a film, and what are their addresses? Write the CQ. Use $t,d,c$ for a movie's Title, Director, and Actor; $h,s$ for a showing's Theater and Schedule; and $a,p$ for an Address and Phone.

Alice
theater_address(h, a) :-
    Movies(t, d, c),
    Pariscope(h, t, s),
    Location(h, a, p).

$t$ connects a movie to its showings, and $h$ connects a showing's theater to its address.

Ada · 02

Start with the first two atoms. Match Movies(t, d, c) against the first Movies tuple,

$$u=\langle \mathrm{Title}{:}\ \mathrm{filmA},\ \mathrm{Director}{:}\ \mathrm{Bergman},\ \mathrm{Actor}{:}\ \mathrm{actor1}\rangle . $$

What does the second atom then ask for?

Alice

The atom reads $u$'s Title into $t$, its Director into $d$, and its Actor into $c$:

$$\nu=\{t/\mathrm{filmA},\ d/\mathrm{Bergman},\ c/\mathrm{actor1}\}. $$

Then Pariscope(h, t, s) asks for the showings of filmA:

$$\sigma_{\mathrm{Title}=\nu(t)}(P)=\sigma_{\mathrm{Title}=\mathrm{filmA}}(P), $$

the two Champo showings.

Ada · 03

Good. Now pretend I am the computer. Where do I get those two showings?

Alice

From $P$, of course.

Hmm. The selection says which tuples to return. It does not say where they are. How does the computer find them?

Ada · 04

The simplest way: read every stored Pariscope tuple and test its Title.

Alice
  • (Champo,filmA,18:00): yes
  • (Champo,filmA,21:00): yes
  • (Rialto,filmB,20:00): no

Three tests, two matches.

Ada · 05

Could you have stopped after the first yes?

Alice

No. The second showing was still coming.

After the second yes, I would have lost nothing here. But I know that only because I tested the third tuple.

Ada · 06

Reading and testing every stored tuple is a scan. Does a scan return exactly $\sigma_{\mathrm{Title}=\mathrm{filmA}}(P)$?

Alice

Yes. Every tuple it keeps passes the test, and it tests every tuple.

But with a million showings, that is a million tests for one movie.

Ada · 07

Now take the second Movies tuple, (filmA,Bergman,actor2).

Alice

It gives $\nu=\{t/\mathrm{filmA},\ d/\mathrm{Bergman},\ c/\mathrm{actor2}\}$: filmA again. We already found its showings, so we can skip this one.

Ada · 08

Count the valuations of the first two atoms if we skip it.

Alice

actor1 gives two and actor3 gives one: three. But the first two atoms have five valuations, one for each tuple of $M\bowtie P$.

Oh. actor2 needs both showings too. I cannot skip the tuple. I only want to skip the search.

Ada · 09

Here is Pariscope again, with a second table beside it.

Pariscope

TheaterTitleSchedule
ChampofilmA18:00
ChampofilmA21:00
RialtofilmB20:00

Second table

TitlePariscope tuples
filmA(Champo,filmA,18:00), (Champo,filmA,21:00)
filmB(Rialto,filmB,20:00)
Alice

What is the second table? It holds the same showings, grouped by Title.

Ada · 10

It is an index on Title. You give it a Title value, called its search key. A lookup returns every stored tuple with that Title, without testing the others.

Alice

So when the second atom has $t$ bound, I look up $\nu(t)$. For actor2, I look up filmA and get both showings. No tests.

Ada · 11

Look up filmB.

Alice

One showing: (Rialto,filmB,20:00).

And filmC? There is no filmC row.

Ada · 12

What is $\sigma_{\mathrm{Title}=\mathrm{filmC}}(P)$?

Alice

Empty. So returning nothing is the right answer, not a failure.

Ada · 13

Suppose the index kept only one showing per Title:

TitlePariscope tuples
filmA(Champo,filmA,18:00)
filmB(Rialto,filmB,20:00)

Is it still an index on Title?

Alice

Its filmA lookup returns one showing, but $\sigma_{\mathrm{Title}=\mathrm{filmA}}(P)$ has two. So actor1 gets one valuation instead of two.

theater_address still finds Champo through the 18:00 showing. But a query that also asked for the schedule would lose 21:00. No: a lookup must return every tuple with that Title.

Ada · 14

Every tuple, and nothing else.

Alice

Does a search key have to pick out one tuple, like a key of the relation?

Ada · 15

Look at filmA.

Alice

Two tuples. So Title is a search key for this index, but not a key of Pariscope.

Ada · 16

Now delete the index. What happens to $P$? To $\sigma_{\mathrm{Title}=\mathrm{filmA}}(P)$?

Alice

Nothing happens to either. $P$ still has three tuples, and the selection still has two. Only my work changes: I would scan again.

So the index is not part of $P$. It sits beside it.

Ada · 17

Was it free?

Alice

No. Building it took one scan, and keeping it takes space. If a showing is added to Pariscope, the index must change too.

Ada · 18

The third atom, Location(h, a, p), will want an index too, and Pariscope might also need one on Theater.

Alice

Then does every index keep its own copy of the tuples? That seems wasteful.

Ada · 19

Keep one copy. Store the showings in an array, and let each index remember where its tuples are:

Stored Pariscope

Row IDTheaterTitleSchedule
0ChampofilmA18:00
1ChampofilmA21:00
2RialtofilmB20:00

Index on Title

TitleRow IDs
filmA0, 1
filmB2

Look up filmA.

Alice

Row IDs 0 and 1. That is not a showing yet.

I still have to read the array at 0 and 1 to get (Champo,filmA,18:00) and (Champo,filmA,21:00).

Ada · 20

Right. Now a lookup finds row IDs, and a fetch reads the stored tuple at a row ID.

Write the index on Theater.

Alice
TheaterRow IDs
Champo0, 1
Rialto2

Both indexes point into the same array. The showings are stored once.

Ada · 21

Is Row ID an attribute of Pariscope?

Alice

It is a column in the table, so yes?

Ada · 22

Store the same three showings in the reverse order.

Alice

Now (Rialto,filmB,20:00) is at 0, and the filmA showings are at 1 and 2. Pariscope is still the same set of three showings.

So a row ID is not an attribute. It records where this array put a tuple. And both indexes are now wrong: they still point to the old places.

Ada · 23

An index can instead keep the rest of each matching tuple:

Title(Theater, Schedule)
filmA(Champo,18:00), (Champo,21:00)
filmB(Rialto,20:00)

What does a lookup need now?

Alice

No fetch. The lookup already has Theater and Schedule, and the Title is the search key itself.

But this index copies the rest of every tuple, which is what I worried about. Which design is right?

Ada · 24

Both return the same tuples for every search key. Row IDs pay with a fetch after each lookup. Copies pay with space and with another update whenever a showing changes.

Alice

So the design changes the work, not the answer.

Ada · 25

Back to the first two atoms. Find all their valuations with the index on Title.

Alice

For each Movies tuple, the first atom binds $t$. I look up $\nu(t)$, fetch each row, and extend $\nu$:

Movies tuple$\nu(t)$Row IDsValuations
(filmA,Bergman,actor1)filmA0, 12
(filmA,Bergman,actor2)filmA0, 12
(filmB,Other,actor3)filmB21

Five valuations, one for each tuple of $M\bowtie P$.

Ada · 26

Count your work, and compare it with scanning Pariscope for each Movies tuple.

Alice

With the index: 3 lookups and 5 fetches, and no Title test on Pariscope. With scans: 3 scans of 3 tuples, so 9 tests.

Is 3 plus 5 less than 9?

Ada · 27

What would you be adding?

Alice

A lookup, a fetch, and a test. Those are different operations.

Then I should not add them. The index made 3 lookups and 5 fetches; the scans made 9 tests.

Ada · 28

In theater_address, $t$ sits in Pariscope's Title position, so the lookup is on Title. Now try a query where one variable sits in two different positions, over a fresh directed relation; SLC is Salt Lake City.

edge

Row IDsrcdst
0LoganOgden
1LoganTremonton
2OgdenSLC
3TremontonSLC
two_hop(x, z) :-
    edge(x, y),
    edge(y, z).

The first atom has matched row 0, giving the valuation $\nu=\{x/\mathrm{Logan},\ y/\mathrm{Ogden}\}$. What does the second atom look up?

Alice

$y$ was the dst of the first edge. So an index on dst, with search key Ogden.

Ada · 29

Read the second atom again.

Alice

edge(y, z). Here $y$ is in the first position, src. The second atom wants edges that start at Ogden.

So the index is on src, with search key $\nu(y)=\mathrm{Ogden}$. It returns row 2, (Ogden,SLC), and extends the valuation to $\nu\cup\{z/\mathrm{SLC}\}$.

Ada · 30

One variable, two stored attributes. What decided which one?

Alice

The atom being matched. The position of $y$ in that atom names the stored attribute. The variable alone says nothing about columns.

Ada · 31

Finish all four tuples of the first atom.

Alice
First atom row$\nu(y)$Lookup on srcAdds
0Ogden2$z/\mathrm{SLC}$
1Tremonton3$z/\mathrm{SLC}$
2SLCnone—
3SLCnone—

Four lookups, two fetches, and two valuations, one through Ogden and one through Tremonton. Both give (Logan,SLC), so two_hop has one answer.

Ada · 32

Now bind both positions:

return_trip(a) :-
    edge(a, b),
    edge(b, a).

After the first atom matches row 0, what does the second atom ask?

Alice

$\nu=\{a/\mathrm{Logan},\ b/\mathrm{Ogden}\}$, so the second atom is edge(Ogden, Logan). Both positions are known.

Nothing is left to find. I only ask whether (Ogden,Logan) is stored. It is not.

Ada · 33

And an atom with no bound position, like the first atom of two_hop?

Alice

Nothing to look up with. I scan.

So the bound positions decide: none bound, scan; some bound, look up on their stored attributes; all bound, test one tuple.

Ada · 34

Classify the three atoms of theater_address, matched in written order.

Alice

Movies(t, d, c): nothing is bound, so scan.

Pariscope(h, t, s): $t$ is bound in the second position, so look up Pariscope on Title with $\nu(t)$.

Location(h, a, p): $h$ is bound in the first position, so look up Location on Theater with $\nu(h)$.

With the right indexes, only the first atom scans. Should every relation keep an index on every attribute, then?

Ada · 35

Try a separate illustration: 100 Movies tuples, each with a different Title. Director A directs 80 of them, Bergman 2, and Director B 18. Movies has an index on Director. Compare

bergman_films(t) :-
    Movies(t, "Bergman", c).

director_a_films(t) :-
    Movies(t, "Director A", c).

How does each atom read Movies, and how much work does it take?

Alice

Each atom has a constant in the Director position, so each looks up the index on Director. A scan would test 100 tuples for either query.

For bergman_films: 1 lookup and 2 fetches. For director_a_films: 1 lookup and 80 fetches. The lookup avoids the 100 tests, but I still fetch 80 tuples, one row ID at a time.

Ada · 36

The scan reads the array in order. The 80 fetches may jump anywhere in it, and a large relation lives on many disk pages. Which would you choose?

Alice

It depends on how many tuples match and where they sit. The index finds the group; it does not make the group smaller.

And every index has to be built and kept up to date, even for queries that never use it.

Ada · 37

Scan, lookup with fetches, lookup with stored attributes, membership test. Did any of them change an answer in this conversation?

Alice

No. Only the index with one showing per Title did, and it did not return exactly the selection. The correct ones changed only the work: tests, lookups, fetches, space, and updates.

Ada · 38

Our index on Title has two search keys. Suppose it had a million. How would a lookup find the entry for filmA?

Alice

If it checked the search keys one by one, that would be a scan of the index. The index needs its own way to find an entry without checking every key.

Recap

The matches the algebra did not find

Matching a CQ's atoms binds variables, and a bound variable turns the next atom into a selection on its stored relation. In theater_address, matching Movies(t, d, c) binds $t$, and Pariscope(h, t, s) then needs

$$\sigma_{\mathrm{Title}=\nu(t)}(P)=\{\,v\in P\mid v(\mathrm{Title})=\nu(t)\,\}. $$

The selection states which stored tuples are required; it does not state how to find them. A scan tests every stored tuple and returns exactly the selection. An index on $A$ takes a search-key value $k$ and returns exactly the tuples of $\sigma_{A=k}(r)$, possibly none; a search key need not be a key of the relation. An index that returns fewer tuples loses valuations, even when a projection happens to hide the loss.

When index entries hold row IDs, a lookup finds row IDs and a fetch reads the stored tuples there. Row IDs belong to a representation: storing the same tuples in another order changes them without changing the relation. An index may instead keep the remaining attributes of each matching tuple, trading the fetch for space and additional updates.

For outer $I_1$, inner $I_2$, and a single shared attribute $A$, the indexed nested-loop join computes

$$I_1\bowtie I_2=\{\,u\bowtie v\mid u\in I_1,\ v\in\sigma_{A=u(A)}(I_2)\,\}. $$

Every outer tuple issues its own lookup, even when its search key repeats.

The atom being matched decides the access. Under a valuation $\nu$, positions holding constants or variables in the domain of $\nu$ are bound. No bound position means a scan, some mean a lookup on the attributes at those positions, and all mean a membership test. The stored attribute comes from the variable's position in that atom, not from the variable.

Access methods that return exactly the same tuples for every request are interchangeable without changing any answer. They differ in tests, lookups, fetches, space, and maintenance, which we count separately rather than add.