Draft
The Matches the Algebra Did Not Find
Conversation 3.1 — From a Bound Selection to an Index
The matches the algebra did not find
Our textbook keeps its running examples in a small cinema database. Here are its three relations; call them $M$, $P$, and $L$.
Movies
| Title | Director | Actor |
|---|---|---|
| filmA | Bergman | actor1 |
| filmA | Bergman | actor2 |
| filmB | Other | actor3 |
Pariscope
| Theater | Title | Schedule |
|---|---|---|
| Champo | filmA | 18:00 |
| Champo | filmA | 21:00 |
| Rialto | filmB | 20:00 |
Location
| Theater | Address | Phone |
|---|---|---|
| Champo | addr1 | phone1 |
| Rialto | addr2 | phone2 |
The book asks for the names and addresses of theaters featuring a Bergman film. Drop Bergman for now: which theaters show a film, and what are their addresses? Write the CQ. Use $t,d,c$ for a movie's Title, Director, and Actor; $h,s$ for a showing's Theater and Schedule; and $a,p$ for an Address and Phone.
theater_address(h, a) :-
Movies(t, d, c),
Pariscope(h, t, s),
Location(h, a, p).
$t$ connects a movie to its showings, and $h$ connects a showing's theater to its address.
Start with the first two atoms. Match Movies(t, d, c) against the first Movies tuple,
What does the second atom then ask for?
The atom reads $u$'s Title into $t$, its Director into $d$, and its Actor into $c$:
Then Pariscope(h, t, s) asks for the showings of filmA:
the two Champo showings.
Good. Now pretend I am the computer. Where do I get those two showings?
From $P$, of course.
Hmm. The selection says which tuples to return. It does not say where they are. How does the computer find them?
The simplest way: read every stored Pariscope tuple and test its Title.
(Champo,filmA,18:00): yes(Champo,filmA,21:00): yes(Rialto,filmB,20:00): no
Three tests, two matches.
Could you have stopped after the first yes?
No. The second showing was still coming.
After the second yes, I would have lost nothing here. But I know that only because I tested the third tuple.
Reading and testing every stored tuple is a scan. Does a scan return exactly $\sigma_{\mathrm{Title}=\mathrm{filmA}}(P)$?
Yes. Every tuple it keeps passes the test, and it tests every tuple.
But with a million showings, that is a million tests for one movie.
Now take the second Movies tuple, (filmA,Bergman,actor2).
It gives $\nu=\{t/\mathrm{filmA},\ d/\mathrm{Bergman},\ c/\mathrm{actor2}\}$: filmA again. We already found its showings, so we can skip this one.
Count the valuations of the first two atoms if we skip it.
actor1 gives two and actor3 gives one: three. But the first two atoms have five valuations, one for each tuple of $M\bowtie P$.
Oh. actor2 needs both showings too. I cannot skip the tuple. I only want to skip the search.
Here is Pariscope again, with a second table beside it.
Pariscope
| Theater | Title | Schedule |
|---|---|---|
| Champo | filmA | 18:00 |
| Champo | filmA | 21:00 |
| Rialto | filmB | 20:00 |
Second table
| Title | Pariscope tuples |
|---|---|
| filmA | (Champo,filmA,18:00), (Champo,filmA,21:00) |
| filmB | (Rialto,filmB,20:00) |
What is the second table? It holds the same showings, grouped by Title.
It is an index on Title. You give it a Title value, called its search key. A lookup returns every stored tuple with that Title, without testing the others.
So when the second atom has $t$ bound, I look up $\nu(t)$. For actor2, I look up filmA and get both showings. No tests.
Look up filmB.
One showing: (Rialto,filmB,20:00).
And filmC? There is no filmC row.
What is $\sigma_{\mathrm{Title}=\mathrm{filmC}}(P)$?
Empty. So returning nothing is the right answer, not a failure.
Suppose the index kept only one showing per Title:
| Title | Pariscope tuples |
|---|---|
| filmA | (Champo,filmA,18:00) |
| filmB | (Rialto,filmB,20:00) |
Is it still an index on Title?
Its filmA lookup returns one showing, but $\sigma_{\mathrm{Title}=\mathrm{filmA}}(P)$ has two. So actor1 gets one valuation instead of two.
theater_address still finds Champo through the 18:00 showing. But a query that also asked for the schedule would lose 21:00. No: a lookup must return every tuple with that Title.
Every tuple, and nothing else.
Does a search key have to pick out one tuple, like a key of the relation?
Look at filmA.
Two tuples. So Title is a search key for this index, but not a key of Pariscope.
Now delete the index. What happens to $P$? To $\sigma_{\mathrm{Title}=\mathrm{filmA}}(P)$?
Nothing happens to either. $P$ still has three tuples, and the selection still has two. Only my work changes: I would scan again.
So the index is not part of $P$. It sits beside it.
Was it free?
No. Building it took one scan, and keeping it takes space. If a showing is added to Pariscope, the index must change too.
The third atom, Location(h, a, p), will want an index too, and Pariscope might also need one on Theater.
Then does every index keep its own copy of the tuples? That seems wasteful.
Keep one copy. Store the showings in an array, and let each index remember where its tuples are:
Stored Pariscope
| Row ID | Theater | Title | Schedule |
|---|---|---|---|
| 0 | Champo | filmA | 18:00 |
| 1 | Champo | filmA | 21:00 |
| 2 | Rialto | filmB | 20:00 |
Index on Title
| Title | Row IDs |
|---|---|
| filmA | 0, 1 |
| filmB | 2 |
Look up filmA.
Row IDs 0 and 1. That is not a showing yet.
I still have to read the array at 0 and 1 to get (Champo,filmA,18:00) and (Champo,filmA,21:00).
Right. Now a lookup finds row IDs, and a fetch reads the stored tuple at a row ID.
Write the index on Theater.
| Theater | Row IDs |
|---|---|
| Champo | 0, 1 |
| Rialto | 2 |
Both indexes point into the same array. The showings are stored once.
Is Row ID an attribute of Pariscope?
It is a column in the table, so yes?
Store the same three showings in the reverse order.
Now (Rialto,filmB,20:00) is at 0, and the filmA showings are at 1 and 2. Pariscope is still the same set of three showings.
So a row ID is not an attribute. It records where this array put a tuple. And both indexes are now wrong: they still point to the old places.
An index can instead keep the rest of each matching tuple:
| Title | (Theater, Schedule) |
|---|---|
| filmA | (Champo,18:00), (Champo,21:00) |
| filmB | (Rialto,20:00) |
What does a lookup need now?
No fetch. The lookup already has Theater and Schedule, and the Title is the search key itself.
But this index copies the rest of every tuple, which is what I worried about. Which design is right?
Both return the same tuples for every search key. Row IDs pay with a fetch after each lookup. Copies pay with space and with another update whenever a showing changes.
So the design changes the work, not the answer.
Back to the first two atoms. Find all their valuations with the index on Title.
For each Movies tuple, the first atom binds $t$. I look up $\nu(t)$, fetch each row, and extend $\nu$:
| Movies tuple | $\nu(t)$ | Row IDs | Valuations |
|---|---|---|---|
(filmA,Bergman,actor1) | filmA | 0, 1 | 2 |
(filmA,Bergman,actor2) | filmA | 0, 1 | 2 |
(filmB,Other,actor3) | filmB | 2 | 1 |
Five valuations, one for each tuple of $M\bowtie P$.
Count your work, and compare it with scanning Pariscope for each Movies tuple.
With the index: 3 lookups and 5 fetches, and no Title test on Pariscope. With scans: 3 scans of 3 tuples, so 9 tests.
Is 3 plus 5 less than 9?
What would you be adding?
A lookup, a fetch, and a test. Those are different operations.
Then I should not add them. The index made 3 lookups and 5 fetches; the scans made 9 tests.
In theater_address, $t$ sits in Pariscope's Title position, so the lookup is on Title. Now try a query where one variable sits in two different positions, over a fresh directed relation; SLC is Salt Lake City.
edge
| Row ID | src | dst |
|---|---|---|
| 0 | Logan | Ogden |
| 1 | Logan | Tremonton |
| 2 | Ogden | SLC |
| 3 | Tremonton | SLC |
two_hop(x, z) :-
edge(x, y),
edge(y, z).
The first atom has matched row 0, giving the valuation $\nu=\{x/\mathrm{Logan},\ y/\mathrm{Ogden}\}$. What does the second atom look up?
$y$ was the dst of the first edge. So an index on dst, with search key Ogden.
Read the second atom again.
edge(y, z). Here $y$ is in the first position, src. The second atom wants edges that start at Ogden.
So the index is on src, with search key $\nu(y)=\mathrm{Ogden}$. It returns row 2, (Ogden,SLC), and extends the valuation to $\nu\cup\{z/\mathrm{SLC}\}$.
One variable, two stored attributes. What decided which one?
The atom being matched. The position of $y$ in that atom names the stored attribute. The variable alone says nothing about columns.
Finish all four tuples of the first atom.
| First atom row | $\nu(y)$ | Lookup on src | Adds |
|---|---|---|---|
| 0 | Ogden | 2 | $z/\mathrm{SLC}$ |
| 1 | Tremonton | 3 | $z/\mathrm{SLC}$ |
| 2 | SLC | none | — |
| 3 | SLC | none | — |
Four lookups, two fetches, and two valuations, one through Ogden and one through Tremonton. Both give (Logan,SLC), so two_hop has one answer.
Now bind both positions:
return_trip(a) :-
edge(a, b),
edge(b, a).
After the first atom matches row 0, what does the second atom ask?
$\nu=\{a/\mathrm{Logan},\ b/\mathrm{Ogden}\}$, so the second atom is edge(Ogden, Logan). Both positions are known.
Nothing is left to find. I only ask whether (Ogden,Logan) is stored. It is not.
And an atom with no bound position, like the first atom of two_hop?
Nothing to look up with. I scan.
So the bound positions decide: none bound, scan; some bound, look up on their stored attributes; all bound, test one tuple.
Classify the three atoms of theater_address, matched in written order.
Movies(t, d, c): nothing is bound, so scan.
Pariscope(h, t, s): $t$ is bound in the second position, so look up Pariscope on Title with $\nu(t)$.
Location(h, a, p): $h$ is bound in the first position, so look up Location on Theater with $\nu(h)$.
With the right indexes, only the first atom scans. Should every relation keep an index on every attribute, then?
Try a separate illustration: 100 Movies tuples, each with a different Title. Director A directs 80 of them, Bergman 2, and Director B 18. Movies has an index on Director. Compare
bergman_films(t) :-
Movies(t, "Bergman", c).
director_a_films(t) :-
Movies(t, "Director A", c).
How does each atom read Movies, and how much work does it take?
Each atom has a constant in the Director position, so each looks up the index on Director. A scan would test 100 tuples for either query.
For bergman_films: 1 lookup and 2 fetches. For director_a_films: 1 lookup and 80 fetches. The lookup avoids the 100 tests, but I still fetch 80 tuples, one row ID at a time.
The scan reads the array in order. The 80 fetches may jump anywhere in it, and a large relation lives on many disk pages. Which would you choose?
It depends on how many tuples match and where they sit. The index finds the group; it does not make the group smaller.
And every index has to be built and kept up to date, even for queries that never use it.
Scan, lookup with fetches, lookup with stored attributes, membership test. Did any of them change an answer in this conversation?
No. Only the index with one showing per Title did, and it did not return exactly the selection. The correct ones changed only the work: tests, lookups, fetches, space, and updates.
Our index on Title has two search keys. Suppose it had a million. How would a lookup find the entry for filmA?
If it checked the search keys one by one, that would be a scan of the index. The index needs its own way to find an entry without checking every key.
The matches the algebra did not find
Matching a CQ's atoms binds variables, and a bound variable turns the next atom into a selection on its stored relation. In theater_address, matching Movies(t, d, c) binds $t$, and Pariscope(h, t, s) then needs
The selection states which stored tuples are required; it does not state how to find them. A scan tests every stored tuple and returns exactly the selection. An index on $A$ takes a search-key value $k$ and returns exactly the tuples of $\sigma_{A=k}(r)$, possibly none; a search key need not be a key of the relation. An index that returns fewer tuples loses valuations, even when a projection happens to hide the loss.
When index entries hold row IDs, a lookup finds row IDs and a fetch reads the stored tuples there. Row IDs belong to a representation: storing the same tuples in another order changes them without changing the relation. An index may instead keep the remaining attributes of each matching tuple, trading the fetch for space and additional updates.
For outer $I_1$, inner $I_2$, and a single shared attribute $A$, the indexed nested-loop join computes
Every outer tuple issues its own lookup, even when its search key repeats.
The atom being matched decides the access. Under a valuation $\nu$, positions holding constants or variables in the domain of $\nu$ are bound. No bound position means a scan, some mean a lookup on the attributes at those positions, and all mean a membership test. The stored attribute comes from the variable's position in that atom, not from the variable.
Access methods that return exactly the same tuples for every request are interchangeable without changing any answer. They differ in tests, lookups, fetches, space, and maintenance, which we count separately rather than add.