L1 Cached Papers

"Cache Oblivious Search Tree via Binary Trees of Small Height"(2001) 论文解读

更新于 2025/9/10

"Cache Oblivious Search Tree via Binary Trees of Small Height" (2001): A Paper Walkthrough

更新于 2025/9/10

1. 前言

无独有偶今天冲浪的时候阅读到一篇文章,主要是关于“cache无关”的搜索树的,正好最近学习到的内容也与“cache无关”的程序设计有联系,于是决定深入看看这篇文章。

Aggarwal and Vitter I/O Model

在2001年代,时兴的IO模型就是Aggarwal and Vitter IO模型。它假设memory是有两个层级,分别为Memory 1和Memory 2. 其中Memory 2被视为lower Level memory,拥有一块可以被量化为MM大小的空间(如Figure 1所示)。

示例图片

Figure 1. Aggarwal & Vitter I/O Model

在两块Memory之间,可以通过一个大小为BB elements的block来进行沟通。该模型最主要的观点:

The cost of the computation in the I/O model is the number of blocks transferred

也就是说该模型认为CPU对于Memory 1的访问开销可以视为_免费_的,而总体IO的开销就等于两层memory之间blocks的传递次数。这种设想在blocks transfer主导运行开销的情况下是成立的。

其实这个模型在现代角度来看,Memory 2就可以视为磁盘Disk。换言之,在磁盘  data  size>>memory1  size磁盘\;data \; size >> memory_1 \; size的情况下,因为会有频繁的block换入换出到memory1memory_1,磁盘IO的时间远大于其他运算时间,这时候该模型的假设即可成立。

What is cache-oblivious?

cache-oblivious的中文直译是_“缓存无关的”_,但是任何程序都不可能离开cache的作用而运行,所以在这里正确的理解方式是:cache-oblivious不是“不受 cache 影响”,而是“不用知道 cache 的参数,也能自动把 cache 用好”。

cache-oblivious的算法/数据结构在 I/O 模型里分析代价(按块传输数计),但设计时不知道块大小BB和可用内存MM——分析对任意 BB、MM都成立,因此对内存层级的每一层都同时近似最优;实现里也不需要把这些硬件参数固定下来,移植性更好。

反过来,如果是cache-aware的算法,会显式的用到B、MB、M来调节 一个节点的degree、page size。

Why cache-oblivious?

那么为什么会有cache-oblivious的说法?有了上面的基础,我们就可以这样延伸:

  1. 在Disk-Memory的内存层级中,如果memory与disk的IO占据了运行时的主要开销,那么我们分析的重点就聚焦于这个阶段。
  2. 在Memory-Cache的内存层级中,如果cache与memory的IO成为了运行时的bottleneck,那么我们分析的重点就聚焦于这个阶段。

有了cache-oblivious的概念之后,我们分析的考量,就仅仅剩下了底层的size MM,和中间可搬运的单次数据大小BB,从而可以cover所有的memory hierarchy,不管是Disk-Memory还是Memory-Cache,也不管是在任何品牌、任何规格的CPU、Memory、Disk设备上,都能有一套通行的算法可以全自动最优化算法的细节策略。

cache-oblivious让我们的算法可迁移、可移植,而不是针对某一种规格的硬件特殊定制,到了下一次又要手动tune。

其实整体的思想有点类似cutlass的模板元编程思想,auto-tuning。在我们后面的实现中,也会借用模板元编程来实现cache-oblivious的数据结构与算法。

一些围绕cache-oblivious的实现(Before 2001)

  1. Frigo等人提出的cache-oblivious 矩阵转置、FFT、排序算法。
  2. Bender等人提出的cache-oblivious 搜索树,在效率上几乎能够与cache-aware的B-Tree持平。

2. 先验知识

2.1 Ω,  Θ,  O\Omega,\; \Theta,\; O

在这里要先讲一下这三个符号的含义,因为论文中大量出现,我们需要做一个梳理。

在现代计算机数据结构与算法的学习中,最常讨论的就是衡量复杂度的符号OO,但是它代表的是上界。也就是说最糟糕的情况。

还有另外两个符号,我们先介绍Ω\Omega,它代表的是下界,也就是说最好的情况:

假如我们有函数f(n)=2n2+3n+1f(n)=2n^2+3n+1,但是它明显比g(n)=0.5n2g(n)=0.5n^2大,Figure 2展示了他们的函数图像:

示例图片

Figure 2. f(n) vs g(n)

所以这个时候,我们就可以用Ω\Omega来代表它的下界,可以被形式化表示为:

f(n)=Ω(n2) f(n)=\Omega(n^2)

那么如果一个函数既有客观的上界OO,又有客观的下界Ω\Omega,比如Figure 3情况发生的时候:

示例图片

Figure 3. f(n)的上下界可以被夹逼在n方范围内

这个时候,可以形式化为: c1g(n)<f(n)<c2g(n)c_1 g(n) < f(n) < c_2 g(n)。也就是说 f(n)f(n)的上下界全都在n2n^2复杂度内。此时我们可以使用Θ\Theta符号来表示:

f(n)=Θ(n2)f(n) = \Theta(n^2)

所以在后文看到Θ\Theta符号的时候,心中自然有一个印象:该操作可以严格被夹逼在某复杂度之内,上下界都不超过。

2.2 van Emde Boas 树

van Emde Boas(下称vEB)提出的数据结构1,是一类专门面向integer key的字典/优先队列数据结构。它最有名的性质是:在 宇宙大小 为UU(键域是 [0,U−1][0, U-1])时,支持以下操作都在O(loglogU)O(loglogU) 时间完成:

  • member(x): 是否存在
  • insert(x) / delete(x): 插入/删除
  • min() / max(): 取最小/最大
  • predecessor(x) / successor(x): 前驱/后继

vEB树的设计思想,来自于递归分解宇宙值UU:

  1. 把键的二进制位拆分为“hi + lo”
  2. 把UU递归分解为U\sqrt{U}个簇,英文为cluster
  3. 每一个cluster大小为U\sqrt{U}
    1. cluster内部有一个summary:用来记录哪些子簇是非空的
    2. cluster[0,U−1]cluster[0, \sqrt{U}-1]: 每一个簇里面再递归放置同样的vEB结构。

可以自行搜索vEB树的算法复杂度分析,此处不再赘述。

2.3 van Emde Boas Layout

vEB布局2是一种把静态完全/近完全二叉树“挤进一个数组”的递归排布方式,目的是在不知道 cache 参数(B、MB、M)的情况下,仍然让“从根到叶”的访问、以及许多局部子树访问,自动拥有很强的局部性(这样做保证cache-oblivious友好)。

层0:                 1
层1:           2           3             ----- 上半树
层2:        4     5     6     7
层3:      8  9 10 11  12 13 14 15        ----- 下半树

现在一分为二,结构发现下半层的小树,和上班层的树的结构是一样的:

上半树 1       下半树   4
    2   3           8   9

那么就递归的放进来,注意下半树在放的时候,如果下半树的结构也很复杂,那么同样再把h2\frac{h}{2}切分为h4+h4\frac{h}{4} + \frac{h}{4}的形式,递归放入。

在这里,小树再切分,就变成了 4 8,9, 无可再切,于是放入:

[1,2,3,4,8,9]
 
// 依次放入:
[1, 2, 3,  4, 8, 9,  5, 10, 11,  6, 12, 13,  7, 14, 15]

那我们比较一下BFS, DFS会是什么情况呢:

// BFS
[ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 ]
// DFS
[ 1, 2, 4, 8, 9, 5, 10, 11, 3, 6, 12, 13, 7, 14, 15 ]

所以如果真要讨论局部性的话,vEB的layout局部性是最佳的,对于cache最友好。

为什么我们要关注layout,就是因为存储的layout会显著影响程序的performance3。

3. 文章工作

文章采用如下形式化语言来描述相应特性:

  • vv: 某个节点
  • TT: Tree,代指我们目前讨论的这棵树的本体。
  • d(v)d(v): 某个节点的深度depth。也就是它目前距离root节点的距离。
  • h(T)h(T): 树的高度。一般指最深的节点到root节点的距离。
  • ∣T∣|T|: 这棵树的节点个数。用绝对值符号来表示。
  • TvT_v: 以当前节点vv为root的一颗_子树_。
  • h(v)h(v): TvT_v子树的高度。

特别的,如果一棵树T1T_1可以通过T2T_2进行剪枝(剪枝子树)而得到,我们称T1T_1 coubd be embedded into T2T_2. 如Figure 4所示:

示例图片

Figure 4. height=4, size=10的树,可以视作它补全为height=5的完全树的剪枝

3.1 用数组序列化一棵树的不同layout

示例图片

Figure 5. 四种不同的Layout

  • 对于BFS layout,左右孩子分别在2i2i和2i+12i+1的位置:
[1, 2, 3, 4, 5]
// 1*2位置是2,1*2+1位置是3  这是1的左右孩子
// 2*2位置是4,2*2+1位置是5  这是2的左右孩子
  • 对于DFS layout,左右孩子分别在i+1i+1和i+2h(v)−1i+2^{h(v)-1}的位置。
[1,2,3,4,5,6,7,8,9,...]
// 1+1的位置是2,1+2^(4-1)的位置是9   这是1的左右孩子:2和9,其中这个4是以当前节点为子树的高度:4
// 2+1的位置是3,2+2^(3-1)的位置是6   这是2的左右孩子:3和6,其中这个3是以当前节点为子树的高度:3
  • 对于in-order layout(中序),左右孩子分别在i−2h(v)−2i-2^{h(v)-2}和i+2h(v)−2i+2^{h(v)-2}的位置,在这里就不列举了,因为不是我们讨论重点。

  • 对于vEB layout,我们在Section 2.3 讨论过它是怎么形成的,在这里也很容易懂:就是分簇+递归形成的。

Our solution is based on the fact that if we for a node in the tree unfold the recursion in the van Emde Boas layout until this node is the root of a bottom tree, then the unfolding will be the same for all nodes of the same depth.

对于一棵高度 H 的大树,作者选择某个节点,顺着 vEB 的规则往下展开递归,直到这个节点本身变成某一层递归调用中的“小树的根”,这就是“unfold until node is root of a bottom tree”。对于树中所有位于相同深度 d 的节点,递归展开到它变成某一层“小树的根”时,展开的形状和布局规则完全一样。

建立搜索表

作者要在隐式的 vEB 布局数组里,给出“当前访问节点在数组中的下标”。为此他们先准备一个按深度预计算的小表:

对于一个在深度dd的节点,当vEB展开到这一轮的时候,并且这一深度为dd的节点刚好成为底部小树的根的时候:

  1. 上方的顶层小树树的大小(即nodes个数),记为Top(d)Top(d),
  2. 这个节点下方小树的大小记为Bottom(d)Bottom(d),
  3. D(d)D(d)为corresponding top tree的深度。
  4. 而这样一个表需要建立起来,只需要O(logn)O(logn)大小,因此也只需要O(logn)O(logn)时间。

搜索时,需要维护两个内容:

  1. 以BFS编号表示的当前位置ii (根是1,左右孩子是2i2i和2i+12i+1)
  2. 路上每个深度jj已经走过的那个节点vEB在数组中的位置Pos[j]Pos[j].
  3. 因为ii的二进制从高到低 编码了“左(0)右(1)”的转向方式,所以ii的低log(T[d]+1)log(T[d]+1)位就能告知:在该轮递归的Top Tree下面,当前节点所在的第几个bottom tree。又因为T[d]T[d]恰好是2k−12^k-1的形式,所以“取这个kk的低位”就是一个按位与 (i  AND  T[d]i \; AND \; T[d])。于是位置公式就可以被形式化表示为:
Pos[d]=Pos[D[d]]+T[d]+(i  AND  T[d])×B[d]Pos[d] = Pos[D[d]] + T[d] + (i \; AND \; T[d]) \times B[d]

Holy shit, 完全读不懂,这简直是天书,让我们举一个例子,手动算一遍吧:

假设有这样一棵树。

示例图片

Figure 6. BFS Layout

计算Pos的公式是:Pos[d]=Pos[D[d]]+T[d]+(i  AND  T[d])×B[d]Pos[d] = Pos[D[d]] + T[d] + (i \; AND \; T[d]) \times B[d]

按深度维护的参数表

  1. d=2d=2,这个时候就是2,3这两个节点。他们在“上半树”的再切割中,成为了bottom tree的root。在第二轮递归切割里,top tree只有{1},所以T[2]=1T[2]=1, 每一个bottom tree只有自己(2,3),所以B[2]=1B[2]=1,这个top tree的深度(节点1到root(也是节点1))D[2]=1D[2]=1
  2. d=3d=3,我们来到了第三层,也就是{4,5,6,7}这四个节点。他们在第一轮切割中,就成了bottom树的树根,还记得我们的切割规则是:当成为小树树根的时候就停止,所以这一层的4个节点不参与第二轮切割,top tree是高度为2的头顶小树:{1,2,3},所以T[3]=3T[3]=3, 而每一个bottom tree的大小,又是3(4,8,9;5,10,11;…)所以B[3]=3B[3]=3,top根的深度:D[3]=1D[3]=1
  3. d=4d=4,我们来到了第四层,也就是{8,9,10,11,...,15}这8个节点。第一轮切割,他们成为了小树的底,他们在第二轮切割中,成为了以他们自己为小树的根,这个时候停止切割。top就是他们的父节点(比如4),T[4]=1T[4]=1, B[4]=1B[4]=1,top根的深度是D[4]=3D[4]=3

计算vEB layout的数组index

  • 节点2:d=2,i=2d=2, i=2, (d是深度,i是在BFS layout里面的index)
    • Pos[2]=Pos[1]+T[2]+(2  and  1)×B[2]=1+1+0×1=2Pos[2]=Pos[1]+T[2]+(2 \; and \; 1) \times B[2]=1+1+0\times1=2 那么它在vEB数组里面的位置就是2
    • 同理可得:3:Pos[2]=33: Pos[2]=3
  • 节点4:d=3,i=4d=3, i=4 (深度是3,它在BFS layout里面位于4的位置)
    • Pos[3]=Pos[1]+T[3]+(4  and  T[3])×B[3]=1+3+0×3=4Pos[3]=Pos[1]+T[3]+(4 \; and \; T[3]) \times B[3]=1+3+0 \times 3 = 4
    • 同理可得:5:Pos[3]=75: Pos[3]=7
  • 节点9:d=4,i=9d=4, i=9
    • Pos[4]=Pos[D[4]]+T[4]+(9  and  T[4])×B[4]=4+1+1=6Pos[4]=Pos[D[4]]+T[4]+(9 \; and \; T[4]) \times B[4]=4+1+1=6
    • 同理可得:8:Pos[4]=58: Pos[4]=5

所以理论上一份vEB layout应该长这样(在数组中):

[1, 2, 3, 4, 8, 9, 5, 10, 11, 6, 12, 13, 7, 14, 15]

3.2 layout对cache/memory的影响

根据Section 1中提到的 I/O模型,假设两层memory之间通过Block来传播数据。又假设在我们树状结构中,不管使用什么layout,一块block中最多可以存储的nodes数量假设有BB个。

如果是使用BFS layout,最高的log(B+1)log(B+1)的层可以可以装在2个blocks里面,而其后的每一块被读到的块里,路径上都只有一个节点。总的memory transfer次数是 Θ(log(nB))\Theta(log(\frac{n}{B})) 次。

相比较的,vEB layout只需要O(logBn)O(log_Bn)次memory transfer(注意这个OO是上界,在Section 2.1分析过)。值得注意的是,只有vEB layout的性能可以逼近B-Tree。

对于四种layout,一次区间查询的块传输数 = 两次查找的块传输数 + O(kB)O(\frac{k}{B})(kk 为输出元素个数,BB为每块可装的节点数)。

3.3 Search Trees of small height

上一节讲了静态完全二叉树在内存中的四种布局(尤其 vEB 布局)。这一节要把这些静态布局“用在动态树上”:维护一棵动态二叉搜索树,但始终把它嵌入到一棵高度仅为logn+O(1)log n + O(1)的静态完全树里,并把静态树按 vEB 布局放进数组。规模翻倍/减半时做一次全局重建。这样既简单(布局基本不变),又能得到和理论最优相匹配的 I/O 复杂度。

这样的设计有一个难点:既然要把树固定嵌在“静态完全树”的数组里,就不能用旋转这种“改指针就能搬子树”的再平衡之法,文章是通过用“重分布/重建子树”的方式来控制高度。

3.3.1 Insertion

记号与不变量:

目标高度的上界,取H≈lognH \approx log n, 在embedded完全树里,一个深度为d(v)d(v)的位置,其容量是:s(v)=2H−d(v)+1−1s(v)=2^{H-d(v)+1}-1。把真实子树大小∣Tv∣|T_v| 与容量比值定义为:

ρ(v)=∣Tv∣s(v)\rho(v) = \frac{|T_v|}{s(v)}

我们设计一串上界密度阈值 0<τ1<τ2<...<τH=10 < \tau_1 < \tau_2 < ... < \tau_H=1 (注意这些τi\tau_i是等差的),并且维持根满足 ρ(root)≤τ1\rho(root) \le \tau_1. 这样的设计就可以强制满足H=logn+O(1)H=logn + O(1)的性质。

插入步骤:

  1. 先按照普通Binary Search Tree的规则自顶向下找到插入的位置,生成新节点vv.
  1. 如果vv的深度到了 H+1H+1,这就意味着超出了规定的高度,这时就 自底向上 找 最近的祖先 ω\omega 使得 ρ(ω)≤τd(w)\rho(\omega) \le \tau_d(w), 对TwT_w做一次均匀重建:
    1. 中序扫描出有序数组,把中位数放在 ω\omega,
    2. 左右两半递归填回左右子树(注意:这一步不需要额外数组就能完成)(先把元素临时“push right”,再按序回填)。这样就能把TwT_w的密度给抹平掉,让高度回归之前的正常值
    3. 找ω\omega的规则是:这个祖先的密度ρ\rho算出来不超出阈值,就从这个不出问题的祖先开始重建。

举例:重建

光说还是很抽象,我们还是举一个例子:

假设我们现在就有一个H=3H=3的宿主完全二叉树。(最多三层,容量有1+2+4=7个节点)

每一个深度的容量 s(v)=2H−d(v)+1−1s(v)=2^{H-d(v)+1}-1:

  1. 深度1:s=7
  2. 深度2:s=3
  3. 深度3:s=1

然后我们设定上界阈值:τ1=0.8,τ2=0.9,τ3=1\tau_1=0.8, \tau_2=0.9, \tau_3=1.

假设我们现在直接暴力插入(就是退化为链表那种)

// 插入10
10			// rho(根)=1/7=0.14 < 0.8
    
 
// 插入20
  10
    \
    20      // rho(根)=2/7=0.28 < 0.8;  rho(node_20) = 1/3 = 0.66 < 0.8  合法
    
// 插入30
  10
    \
    20
      \
      30    // rho(根)=3/7 < 0.8; rho(node_20) = 2/3=0.67 < 0.8; rho(node_30)=1/1=1 <= tau_3
 
// 插入40
  10
    \
    20
      \
      30
        \
        40   ← 新插入,深度4 超高

这个时候40已经超出了高度3,让这棵树的高度变成了4.

  • ρ(d=3)=21=2\rho(d=3)=\frac{2}{1}=2
  • ρ(d=2)=33=1\rho(d=2)=\frac{3}{3}=1, 超出了τ2=0.9\tau_2=0.9
  • ρ(d=1)=47<0.8\rho(d=1)=\frac{4}{7} < 0.8, 没超出τ1\tau_1

因为没超出的就是d=1,所以我们要针对d=1d=1层进行重建:

  1. 此刻的中序序列是:{10,20,30,40}, 中位数我们选择20
  2. 左右各自一半填回:
    20
   /  \
 10   30
         \
         40

举例:插入50

那假设现在还要再插入50呢?

    20
   /  \
 10   30
         \
         40
           \
           50   ← 新插入,深度4
  • ρ(d=3)=21=2\rho(d=3)=\frac{2}{1}=2
  • ρ(d=2)=33=1\rho(d=2)=\frac{3}{3}=1, 超出了τ2=0.9\tau_2=0.9, 实际的元素有3个,理论上它应该存放的容量是 s(v)=23−2+1−1=3s(v) = 2^{3-2+1}-1=3
  • ρ(d=1)=57=0.71\rho(d=1)=\frac{5}{7}=0.71, 没超出τ1\tau_1

这个时候,没出问题的就依然是根 d=1d=1,所以我们针对根进行重建:

[10,20,30,40,50]就变成了:

     30
    /  \
  20    40
 /        \
10        50

举例:再插入60

再插入60的时候,连ρ(d=1)≈0.86>0.8\rho(d=1) \approx 0.86 > 0.8了,连祖先都满足不了阈值,只能对整棵树重建,提升树的高度。

小结

从上面的几个例子可以观察出来:

  1. 我们的树并不是要全部填满,才触发全局重建的,甚至没有填满,就得重建了。
  2. 树的容量以及是否需要重建,是由我们预先设定的参数 τi\tau_i 们决定的,在实战中需要好好定义这个参数。

为什么这样做能够长期稳定?

在 vv 处做了重分布后,任意后代 ω\omega的真实大小都会落在某个区间之间,这个区间是由密度和常量τi\tau_i决定的。也就是“大小=容量x密度”,常数级别的误差,这就会让每一层不至于太挤,也不至于太稀疏。

插入的摊还复杂度分析

所谓摊还,就是平均到每一个操作中。有一些操作会导致重建,有一些操作不会导致重建,但是对于重建我们并不能单纯的计算重建本身的复杂度,它的复杂度应该被均摊反还到之前没有触发重建的操作上去,这样才是公平的。

时间: O(HΔ)=O(log2n(1−τ1))O(\frac{H}{\Delta})=O(\frac{log^2n}{(1-\tau_1)})

块传输:O(logBn)+O(log2nB(1−τ1))O(log_Bn)+O(\frac{log^2n}{B(1-\tau_1)})

3.3.2 Deletion

Delete操作在不打乱vEB 静态布局的前提下,用“密度阈值 + 局部重建”来支持删除,同时继续保证区间查询、搜索的 I/O 上界。

单纯给结点打“已删除”标记、等到删了一半再全局重建,局部会变得很稀疏,导致区间查询无法给出最坏情况下的块传输上界(即:有时要跨很多块才能把一小段区间扫出来)。所以删除后必须及时再平衡,而不是无限期拖着标记位。

与之前我们设置的一连串上界阈值的操作类似,在这里文中也提出:设置一系列下界阈值 0<γH<γ2<gamma1<τ10 < \gamma_H < \gamma_2 < gamma_1 < \tau_1 注意哦,最左侧是上界阈值的最小值τ1\tau_1,

删除步骤:

  1. 先做标准 BST 删除:自顶向下找到要删的元素所在结点vv。若vv不是叶且有右子树,就找到“后继”并与之交换,反复直到叶子,再删它;若没有右子树,就对称地用“前驱”。
  2. 平衡:从被删除的叶子往上看,找到最低的祖先ω\omega 满足密度回到合法区间 γd(ω)<ρ(ω)<τd(ω)\gamma_{d(\omega)} < \rho(\omega) < \tau_{d(\omega)},然后对TwT_w整颗子树做一次均匀重建。具体步骤跟上面几乎一样。

举例:删除30

设定 τ1=0.8,γ1=0.4,γ2=0.35,γ3=0.30\tau_1=0.8, \gamma_1=0.4, \gamma_2=0.35, \gamma_3=0.30

      40            (深1, s=7)
    /    \
  20      60        (深2, s=3)
 /  \    /  \
10  30  50  70      (深3, s=1)

删除了30之后,∣T20∣=2|T_20|=2 (它自己+底下的10),密度ρ(20)=23>γ2\rho(20)=\frac{2}{3} > \gamma_2 不用重建。

删除10

      40            (深1, s=7)
    /    \
  20      60        (深2, s=3)
         /  \
        50  70      (深3, s=1)

现在ρ(20)=0.33<γ2\rho(20)=0.33 < \gamma_2, 这时候需要均匀重建。

找到父节点40,ρ(40)=57>γ1\rho(40)=\frac{5}{7} > \gamma_1, 没什么问题,我们对40这个节点进行重建。

[20,40,50,60,70]中位数是50,重建后:

      50            (深1, s=7)
    /    \
  40      60        (深2, s=3)
 /         \
20         70      (深3, s=1)

再往后,如果还要删除,就会触发全局减高。

摊还成本分析

  1. 摊还时间:O(log2nα)O(\frac{log^2n}{\alpha}), 其中 α∈min{γ1−γH,1−τ1}\alpha \in min\{ \gamma_1-\gamma_H, 1 - \tau_1 \}
  2. I/O 块传输:搜索还是logBnlog_B n,更新摊还 O(logBn+log2nBα)O(log_B n + \frac{log^2 n}{B \alpha})

3.3.3 Imporve Densities

Recap:上面我们梳理了文章提出的三种方案:嵌入静态完全树、vEB布局、基于密度阈值的重建方案。

接下来的问题就是:如何把最坏空间从 2n2n 压缩到 接近最优的 (1+ε)n(1+\varepsilon)n, 同时不让搜索与区间查询的IO上界变差。

不知B,MB, M,怎么压缩空间?

文章设一个使用空间NN,并且选出一个根密度阈值 γ1≤nN≤τ1\gamma_1 \le \frac{n}{N} \le \tau_1,一旦超出了,就换一个新的NN并且做全局重建,这样整体的空间 就被严格控制在(1+ε)n(1+\varepsilon)n量级。

那么对于NN值有两个条件:

  1. 如果N=2k−1N = 2^k - 1,直接用前面的单棵“宿主完全树”方案。
  2. 否则,把NN写成二进制和:N=∑bi2bi=2b1+2b2+2b3+...+2bnN = \sum_{b_i} 2^{b_i} = 2^{b_1} + 2^{b_2} + 2^{b_3} + ... + 2^{b_n}
    1. 为每一个bib_i建立一颗FiF_i, 只有一个根rir_i, 没有左孩子,右侧接一颗大小为 2bi−12^{b_i}-1的完全右子树CiC_i
    2. 把所有元素,切成连续段,分配到这样的FiF_i上界。
    3. 在内存中,按照r1,r2,...,rk,C1,C2,...,Ckr_1, r_2, ..., r_k, C_1, C_2, ..., C_k排列,每一个CiC_i都用vEB布局。
  3. 搜索的时候,先按照顺序比较r1,...,rkr_1, ..., r_k落在哪个段TiT_i,再在TiT_i里面自顶向下搜索。这样的IO时间是O(iB+logB(2ib−1))=O(logBN)O(\frac{i}{B}+log_B(2^b_i-1))=O(log_B N)
  4. 换N的频率,导致了全局重建的均摊时间是 O(1ε)O(\frac{1}{\varepsilon}), IO是 O(1εB)O(\frac{1}{\varepsilon B})

4. 实验

示例图片

Figure 7. 左图对比四种layout基于指针实现的性能,右图对比四种layout基于数组实现的性能

based on pointers的意思是基于指针实现的树。Fig 7中左侧的图就是横向对比集中数据结构基于指针实现的版本。在n比较小的时候, 因为整棵树都在cache里面,所以几种树其实没有太大性能差距(n<14n < 14 ),而在n增大时,BFS每层缺失一次,DFS每两层缺失一次,而vEB每Θ(logBn)\Theta(log_Bn)层一次, 这根论文中分析的结论几乎一致。

右侧图像是针对隐式实现的版本的实验分析。即:没有指针、纯依靠数组实现的版本。 在 cache 内:地址计算开销主导,BFS 最快、vEB 最慢(vEB 的寻址最复杂)。 而超出cache的时候,这时访存(IO操作)成为了bottleneck,高叉(d=8,16d=8,16)的cache-aware版本是最快的,因为比较适合cacheline的大小。vEB逐渐追赶上BFS最终反超。(大约在 n=21∼22n=21 \sim 22的时候)。最终仅比高叉方案慢约50%。 inorder组表现出来的曲线,尤其当nn是2的幂时有波动,疑因有限相联度让顶部节点映射到同一组cacheline。

示例图片

Figure 8. vEB和BFS的指针vs数组实现性能对比

从上面的第一个实现我们得出先验结论:vEB与BFS有可以研究的价值,其他两种layout效果都不太好。因此,作者在下面来直接横向对比vEB、BFS两种layout 的基于指针实现与基于数组实现的性能效果。

  • BFS:隐式版始终优于指针版,这说明“算地址”并不比“跟指针”慢。
  • vEB:cache内指针版更快,但在 cache 外隐式版翻盘,分析原因,去掉指针后同一块能装更多节点(B 变大)。
示例图片

Figure 9. vEB与非平衡树对比:插入与查找的性能

从Fig 9中大致可以观察出如下结论:若系统以搜索为主、仅偶尔更新(更新就是insert)的基调下,vEB 半动态在 n≈216n \approx 2^{16}时候更优(最低的曲线)。

示例图片

Figure 10. 超出主存时的性能对比

主存内:BFS最快;超出了主存的size,BFS 反而慢到5×5 \times开外(注意图中的纵轴并不是线性的)。为页大小优化的 1024 叉树在主存外最快, 但在主存内慢约2×2 \times (注意,纵轴不是线性的);反观vEB,一路都跟最优的一种架构打平,跨层级表现稳定。

作者结论

  1. 内存层级影响主导运行时间,哪怕树规模还远小于主存。
  2. cache-oblivious 的好处在实践中成立:vEB 对比 cache-aware 有竞争力,对比“未优化”方案几乎总是更好,且跨多级内存都稳健。
  3. 隐式布局(无指针)带来的省空间 / 更大扇出(更大的 B)对性能有显著贡献;动态方案既易实现、又有不错的时间表现。

5. Reflection

隐式且缓存无关的小高度搜索树达到了 B-树的 I/O 性能下界,为全范围查询的稳健性奠定了坚实基础。 基于 vEB 布局,其 I/O 代价可保持在 O(logBn+kB)O(log_B n + \frac{k}{B}), 这些最优化结论在 2025 年依然成立;不过作为“骨干”也可以由其他结构替换(如 cache-oblivious B-Tree)。 考虑到硬件在各层级(寄存器、L1/L2/L3 缓存、内存)的容量持续增加,以及现代的预取、分支预测、SIMD 等策略, 在某些纯内存场景里往往会比本文方案更快,我们将在后文讨论潜在改进。

从根到叶的访问中,vEB 布局具有清晰的 logBnlog_B n 复杂度(此处公式在原文中显示为乱码), 并且通常在 n≥thresholdn \ge threshold 时优于其他方案。与既有工作相比,小高度树不需要重量平衡, 避免了沉重或高频的更新,而是仅在必要时对树做一次重分布。需要注意的是,当 n≤thresholdn \le threshold 时 (原式同样存在乱码),隐式的基于数组的布局会因为索引计算成为主导成本而产生额外开销;随着 nn 增大, 这种开销与指针跳转之间形成权衡。数组化布局也意味着单个传输块 nn 的有效承载更大,从而一次 I/O 能取回更多数据。 只有当 nn 膨胀到使根节点违反规则时,才会触发整棵树的重建(H←H+1H ← H + 1);两次重建之间相隔 Θ(n)Θ(n) 级别的更新,因此摊到单次更新的成本很小。

在总结优点之后,也需要看到不足。正如Fig 9 所示, 尽管 vEB 布局在大规模插入对比中表现更好,但其性能抖动较为频繁,长期可能影响硬件与系统整体表现。 此外,可以清楚地看到:vEB 方案在较大缓存规模下表现不错,但在较小缓存下不如其他方案。 现代 CPU 的缓存与内存都在增大,这在一定程度上限制了其潜在适用面。

现代 I/O 模型通常涉及多级缓存(L1/L2/L3)、页缓存以及 NVMe 协议。 本文采用的两层 I/O 模型相对理想化,只分析两级之间的块传输,忽略了 cache line 之外的 TLB 与页大小等因素——这一点在 Cache-Conscious B+ Tree 中已有体现。 再者,本文主要在单线程语境下讨论数据结构;而现代系统常采用多线程访问,更关注无闩或增量更新, 以减少共享写带来的缓存失效与锁竞争,Bw-Tree 对此有专门讨论。

最后需要强调的是,2005 年提出的 cache-oblivious B-tree 保留了本文的核心优势——跨各内存层级的快速查找与范围扫描, 同时不需要周期性的子树重建;它更易运维、对并行读写更友好,并能从 CPU 缓存到 SSD 都无需手工调参而良好工作, 既保持紧凑、指针开销小与强局部性,又带来更平稳的性能与更低的工程成本。

Reference

Footnotes

  1. P. van Emde Boas. Preserving order in a forest in less than logarithmic time and linear space. Inf. Process. Lett., 6:80–82, 1977. ↩

  2. H. Prokop. Cache-oblivious algorithms. Master’s thesis, Massachusetts Institute of Technology, June 1999. ↩

  3. S. Chatterjee, V. V. Jain, A. R. Lebeck, S. Mundhra, and M. Thottethodi. Nonlinear array layouts for hierarchical memory systems. In Proceedings of the 1999 Conference on Supercomputing, ACM SIGARCH, pages 444–453.ACM Press, 1999. ↩

1. Introduction

As chance would have it, while browsing the web today I came across a paper on “cache-oblivious” search trees. What I’ve been learning lately also happens to be connected to cache-oblivious program design, so I decided to take a deep look at this paper.

Aggarwal and Vitter I/O Model

Around 2001, the prevailing I/O model was the Aggarwal and Vitter I/O model. It assumes that memory has two levels, Memory 1 and Memory 2. Memory 2 is regarded as the lower-level memory, with a space whose size can be quantified as MM (as shown in Figure 1).

Example image

Figure 1. Aggarwal & Vitter I/O Model

The two memories communicate by transferring blocks of BB elements. The model’s central point is:

The cost of the computation in the I/O model is the number of blocks transferred

In other words, the model treats the CPU’s accesses to Memory 1 as free, and the total I/O cost equals the number of blocks transferred between the two memory levels. This assumption holds whenever block transfers dominate the running cost.

In fact, from a modern perspective, Memory 2 in this model can simply be viewed as the disk. In other words, when disk  data  size>>memory1  sizedisk\;data \; size >> memory_1 \; size, blocks are frequently swapped in and out of memory1memory_1, so disk I/O time far exceeds all other computation time, and the model’s assumption holds.

What is cache-oblivious?

The literal Chinese translation of cache-oblivious reads as “cache-irrelevant”, but no program can run without the cache coming into play, so the right way to understand it here is: cache-oblivious does not mean “unaffected by the cache”, but rather “makes good use of the cache automatically, without needing to know the cache’s parameters”.

A cache-oblivious algorithm/data structure is analyzed in the I/O model (with cost counted as the number of block transfers), but it is designed without knowing the block size BB or the available memory MM. The analysis holds for any BB and MM, so it is simultaneously near-optimal at every level of the memory hierarchy; the implementation also doesn’t need to hard-code these hardware parameters, which makes it more portable.

Conversely, a cache-aware algorithm explicitly uses B,MB, M to tune things like a node’s degree and the page size.

Why cache-oblivious?

So why does the notion of cache-oblivious exist at all? With the groundwork above, we can extend the idea like this:

  1. At the Disk-Memory level of the memory hierarchy, if I/O between memory and disk accounts for most of the runtime cost, then our analysis focuses on that stage.
  2. At the Memory-Cache level of the memory hierarchy, if I/O between cache and memory becomes the runtime bottleneck, then our analysis focuses on that stage.

With the cache-oblivious concept, the only things left to consider in our analysis are the size MM of the lower level and the size BB of the data that can be moved between levels in a single transfer. As a result, we can cover every level of the memory hierarchy: whether it’s Disk-Memory or Memory-Cache, and regardless of the brand or specs of the CPU, memory, or disk, one universal algorithm can fully automatically optimize its detailed strategy.

cache-oblivious makes our algorithms transferable and portable, instead of being custom-tailored to one particular hardware spec and then having to be hand-tuned all over again the next time.

In fact, the overall idea is somewhat similar to the template metaprogramming philosophy of cutlass: auto-tuning. In our later implementation, we will also borrow template metaprogramming to implement cache-oblivious data structures and algorithms.

Some implementations built around cache-oblivious (Before 2001)

  1. The cache-oblivious matrix transpose, FFT, and sorting algorithms proposed by Frigo et al.
  2. The cache-oblivious search tree proposed by Bender et al., whose efficiency is nearly on par with the cache-aware B-Tree.

2. Preliminaries

2.1 Ω,  Θ,  O\Omega,\; \Theta,\; O

First, let’s go over what these three symbols mean. They appear all over the paper, so we need to sort them out.

In the study of modern data structures and algorithms, the most commonly discussed complexity notation is OO, but it represents an upper bound, i.e., the worst case.

There are two other symbols. Let’s first introduce Ω\Omega, which represents a lower bound, i.e., the best case:

Suppose we have the function f(n)=2n2+3n+1f(n)=2n^2+3n+1; it is clearly larger than g(n)=0.5n2g(n)=0.5n^2. Figure 2 shows their graphs:

Example image

Figure 2. f(n) vs g(n)

So in this case, we can use Ω\Omega to denote its lower bound, which can be written formally as:

f(n)=Ω(n2) f(n)=\Omega(n^2)

Now, what if a function has both a definite upper bound OO and a definite lower bound Ω\Omega, as in the situation shown in Figure 3:

Example image

Figure 3. The upper and lower bounds of f(n) can be squeezed within the n-squared range

In this case, it can be formalized as: c1g(n)<f(n)<c2g(n)c_1 g(n) < f(n) < c_2 g(n). That is, both the upper and lower bounds of f(n)f(n) are within n2n^2 complexity. We can then use the Θ\Theta notation:

f(n)=Θ(n2)f(n) = \Theta(n^2)

So whenever you see the Θ\Theta symbol later on, you should naturally have this picture in mind: the operation is tightly sandwiched within a certain complexity, with neither the upper nor the lower bound going beyond it.

2.2 van Emde Boas Tree

The data structure proposed by van Emde Boas (hereafter vEB)1 is a dictionary/priority-queue data structure designed specifically for integer keys. Its best-known property is that, with a universe size of UU (the key domain is [0,U−1][0, U-1]), it supports all of the following operations in O(loglogU)O(loglogU) time:

  • member(x): whether x exists
  • insert(x) / delete(x): insert/delete
  • min() / max(): get the minimum/maximum
  • predecessor(x) / successor(x): predecessor/successor

The design idea of the vEB tree comes from recursively decomposing the universe UU:

  1. Split the binary bits of the key into “hi + lo”
  2. Recursively decompose UU into U\sqrt{U} groups, called clusters
  3. Each cluster has size U\sqrt{U}
    1. There is a summary inside: it records which sub-clusters are non-empty
    2. cluster[0,U−1]cluster[0, \sqrt{U}-1]: each cluster recursively contains the same vEB structure.

You can look up the complexity analysis of the vEB tree on your own; I won’t go into it here.

2.3 van Emde Boas Layout

The vEB layout2 is a recursive arrangement that “squeezes” a static complete/nearly complete binary tree “into an array”. Its goal is that, without knowing the cache parameters (B,MB, M), root-to-leaf accesses, as well as many local subtree accesses, still automatically enjoy strong locality (which is what makes it cache-oblivious friendly).

Level 0:                 1
Level 1:           2           3             ----- top tree
Level 2:        4     5     6     7
Level 3:      8  9 10 11  12 13 14 15        ----- bottom trees

Now split it in two, and it turns out that the small trees in the bottom half have the same structure as the tree in the top half:

Top tree    1       Bottom tree    4
          2   3                  8   9

Then we lay them out recursively. Note that when laying out a bottom tree, if its structure is also complex, we likewise split h2\frac{h}{2} into h4+h4\frac{h}{4} + \frac{h}{4} and lay it out recursively.

Here, splitting the small tree again gives 4 and 8,9, which can’t be split any further, so we lay them out:

[1,2,3,4,8,9]
 
// Lay them out one after another:
[1, 2, 3,  4, 8, 9,  5, 10, 11,  6, 12, 13,  7, 14, 15]

So what would BFS and DFS look like by comparison?

// BFS
[ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 ]
// DFS
[ 1, 2, 4, 8, 9, 5, 10, 11, 3, 6, 12, 13, 7, 14, 15 ]

So if we really want to talk about locality, the vEB layout has the best locality and is the most cache-friendly.

The reason we care about layout is that the storage layout significantly affects a program’s performance3.

3. The Paper’s Work

The paper uses the following formal notation to describe the relevant properties:

  • vv: a node
  • TT: Tree, referring to the tree we are currently discussing.
  • d(v)d(v): the depth of a node, i.e., its distance from the root node.
  • h(T)h(T): the height of the tree, generally the distance from the deepest node to the root node.
  • ∣T∣|T|: the number of nodes in the tree, denoted with the absolute-value symbol.
  • TvT_v: a subtree rooted at the current node vv.
  • h(v)h(v): the height of the subtree TvT_v.

In particular, if a tree T1T_1 can be obtained from T2T_2 by pruning (removing subtrees), we say T1T_1 could be embedded into T2T_2, as shown in Figure 4:

Example image

Figure 4. A tree with height=4 and size=10 can be viewed as a pruned version of the complete tree of height=5 obtained by filling it out

3.1 Different Layouts for Serializing a Tree into an Array

Example image

Figure 5. Four different layouts

  • For the BFS layout, the left and right children are at positions 2i2i and 2i+12i+1, respectively:
[1, 2, 3, 4, 5]
// position 1*2 is 2, position 1*2+1 is 3  these are the left and right children of 1
// position 2*2 is 4, position 2*2+1 is 5  these are the left and right children of 2
  • For the DFS layout, the left and right children are at positions i+1i+1 and i+2h(v)−1i+2^{h(v)-1}, respectively.
[1,2,3,4,5,6,7,8,9,...]
// position 1+1 is 2, position 1+2^(4-1) is 9   these are the left and right children of 1: 2 and 9, where 4 is the height of the subtree rooted at the current node: 4
// position 2+1 is 3, position 2+2^(3-1) is 6   these are the left and right children of 2: 3 and 6, where 3 is the height of the subtree rooted at the current node: 3
  • For the in-order layout, the left and right children are at positions i−2h(v)−2i-2^{h(v)-2} and i+2h(v)−2i+2^{h(v)-2}, respectively. I won’t list an example here, since it isn’t the focus of our discussion.

  • For the vEB layout, we discussed how it is formed in Section 2.3, and it’s easy to understand here too: it is formed by clustering + recursion.

Our solution is based on the fact that if we for a node in the tree unfold the recursion in the van Emde Boas layout until this node is the root of a bottom tree, then the unfolding will be the same for all nodes of the same depth.

For a large tree of height H, the authors pick a node and unfold the recursion downward following the vEB rules until that node itself becomes the “root of a small tree” at some level of the recursive calls; this is what “unfold until node is root of a bottom tree” means. For all nodes in the tree at the same depth d, when the recursion is unfolded until the node becomes the “root of a small tree” at some level, the shape of the unfolding and the layout rules are exactly the same.

Building the lookup table

The authors want to determine, within the implicit vEB layout array, “the index in the array of the node currently being visited”. To do this, they first prepare a small table precomputed per depth:

For a node at depth dd, when the vEB unfolding reaches the round in which this node at depth dd has just become the root of a bottom tree:

  1. The size (i.e., number of nodes) of the top tree above it is denoted Top(d)Top(d),
  2. The size of the small tree below this node is denoted Bottom(d)Bottom(d),
  3. D(d)D(d) is the depth of the corresponding top tree.
  4. Building such a table only takes O(logn)O(logn) space, and therefore only O(logn)O(logn) time.

During a search, we need to maintain two things:

  1. The current position ii in BFS numbering (the root is 1, and the left and right children are 2i2i and 2i+12i+1)
  2. For each depth jj along the path, the position Pos[j]Pos[j] in the vEB array of the node already visited at that depth.
  3. Since the binary representation of ii, read from the high bits to the low bits, encodes the sequence of “left (0) / right (1)” turns, the low log(T[d]+1)log(T[d]+1) bits of ii tell us which bottom tree under the Top Tree of this round of recursion the current node lies in. And since T[d]T[d] is exactly of the form 2k−12^k-1, “taking the low kk bits” is just a bitwise AND (i  AND  T[d]i \; AND \; T[d]). The position formula can therefore be written formally as:
Pos[d]=Pos[D[d]]+T[d]+(i  AND  T[d])×B[d]Pos[d] = Pos[D[d]] + T[d] + (i \; AND \; T[d]) \times B[d]

Holy shit, I can’t make heads or tails of this; it’s pure gibberish. Let’s work through an example and compute it by hand:

Suppose we have a tree like this.

Example image

Figure 6. BFS Layout

The formula for computing Pos is: Pos[d]=Pos[D[d]]+T[d]+(i  AND  T[d])×B[d]Pos[d] = Pos[D[d]] + T[d] + (i \; AND \; T[d]) \times B[d]

The per-depth parameter table

  1. d=2d=2: these are nodes 2 and 3. In the second split of the “top half-tree”, they become the roots of bottom trees. In the second round of recursive splitting, the top tree is just {1}, so T[2]=1T[2]=1; each bottom tree contains only the node itself (2 or 3), so B[2]=1B[2]=1; the depth of this top tree (from node 1 to the root, which is also node 1) is D[2]=1D[2]=1
  2. d=3d=3: we’ve reached the third level, i.e., the four nodes {4,5,6,7}. They already became the roots of bottom trees in the first round of splitting. Remember that our splitting rule is: stop once a node becomes the root of a small tree, so the 4 nodes on this level do not take part in the second round of splitting. The top tree is the small tree of height 2 above them, {1,2,3}, so T[3]=3T[3]=3; each bottom tree again has size 3 (4,8,9; 5,10,11; …), so B[3]=3B[3]=3; the depth of the top tree’s root: D[3]=1D[3]=1
  3. d=4d=4: we’ve reached the fourth level, i.e., the 8 nodes {8,9,10,11,...,15}. In the first round of splitting, they end up at the bottom of the small trees; in the second round of splitting, each of them becomes the root of a small tree consisting of just itself, at which point the splitting stops. The top is their parent node (e.g., 4), so T[4]=1T[4]=1, B[4]=1B[4]=1, and the depth of the top root is D[4]=3D[4]=3

Computing the array index in the vEB layout

  • Node 2: d=2,i=2d=2, i=2 (d is the depth, i is the index in the BFS layout)
    • Pos[2]=Pos[1]+T[2]+(2  and  1)×B[2]=1+1+0×1=2Pos[2]=Pos[1]+T[2]+(2 \; and \; 1) \times B[2]=1+1+0\times1=2, so its position in the vEB array is 2
    • Similarly: 3:Pos[2]=33: Pos[2]=3
  • Node 4: d=3,i=4d=3, i=4 (the depth is 3, and it is at position 4 in the BFS layout)
    • Pos[3]=Pos[1]+T[3]+(4  and  T[3])×B[3]=1+3+0×3=4Pos[3]=Pos[1]+T[3]+(4 \; and \; T[3]) \times B[3]=1+3+0 \times 3 = 4
    • Similarly: 5:Pos[3]=75: Pos[3]=7
  • Node 9: d=4,i=9d=4, i=9
    • Pos[4]=Pos[D[4]]+T[4]+(9  and  T[4])×B[4]=4+1+1=6Pos[4]=Pos[D[4]]+T[4]+(9 \; and \; T[4]) \times B[4]=4+1+1=6
    • Similarly: 8:Pos[4]=58: Pos[4]=5

So in theory, a vEB layout should look like this (in the array):

[1, 2, 3, 4, 8, 9, 5, 10, 11, 6, 12, 13, 7, 14, 15]

3.2 The Impact of Layout on Cache/Memory

Following the I/O model described in Section 1, assume that data moves between the two memory levels in blocks. Further assume that in our tree structure, regardless of the layout used, a block can hold at most BB nodes.

With the BFS layout, the top log(B+1)log(B+1) levels fit in 2 blocks, but every block read after that contains only one node on the path. The total number of memory transfers is Θ(log(nB))\Theta(log(\frac{n}{B})).

In comparison, the vEB layout needs only O(logBn)O(log_Bn) memory transfers (note that this OO is an upper bound, as analyzed in Section 2.1). Notably, only the vEB layout’s performance can approach that of a B-Tree.

For all four layouts, the number of block transfers for a range query = the number of block transfers for two searches + O(kB)O(\frac{k}{B}) (kk is the number of output elements, and BB is the number of nodes a block can hold).

3.3 Search Trees of small height

The previous section covered four in-memory layouts of a static complete binary tree (especially the vEB layout). This section applies these static layouts “to dynamic trees”: maintain a dynamic binary search tree, but always embed it in a static complete tree of height only logn+O(1)log n + O(1), and store the static tree in an array using the vEB layout. When the size doubles/halves, do a global rebuild. This is both simple (the layout stays essentially unchanged) and achieves I/O complexity matching the theoretical optimum.

This design has one difficulty: since the tree is embedded in the array of a fixed “static complete tree”, we cannot use rotations, the rebalancing technique that “moves subtrees just by changing pointers”. Instead, the paper controls the height by redistributing/rebuilding subtrees.

3.3.1 Insertion

Notation and invariants:

For the upper bound on the target height, take H≈lognH \approx log n. In the embedded complete tree, a position at depth d(v)d(v) has capacity s(v)=2H−d(v)+1−1s(v)=2^{H-d(v)+1}-1. The ratio of the actual subtree size ∣Tv∣|T_v| to the capacity is defined as:

ρ(v)=∣Tv∣s(v)\rho(v) = \frac{|T_v|}{s(v)}

We define a sequence of upper density thresholds 0<τ1<τ2<...<τH=10 < \tau_1 < \tau_2 < ... < \tau_H=1 (note that these τi\tau_i form an arithmetic progression) and maintain that the root satisfies ρ(root)≤τ1\rho(root) \le \tau_1. This design enforces the property H=logn+O(1)H=logn + O(1).

Insertion steps:

  1. First, following the usual Binary Search Tree rules, find the insertion position top-down and create the new node vv.
  1. If the depth of vv reaches H+1H+1, the prescribed height has been exceeded. We then search bottom-up for the nearest ancestor ω\omega such that ρ(ω)≤τd(w)\rho(\omega) \le \tau_d(w), and do an even rebuild of TwT_w:
    1. Do an in-order scan to get a sorted array, and place the median at ω\omega;
    2. Recursively fill the left and right halves back into the left and right subtrees (note: this step can be done without an extra array) (first temporarily “push right” the elements, then fill them back in order). This evens out the density of TwT_w and brings the height back to its previous normal value
    3. The rule for finding ω\omega: if this ancestor’s computed density ρ\rho does not exceed the threshold, start rebuilding from this problem-free ancestor.

Example: rebuilding

Talk alone is still too abstract, so let’s go through an example:

Suppose we have a host complete binary tree with H=3H=3 (at most three levels, with a capacity of 1+2+4=7 nodes).

The capacity at each depth, s(v)=2H−d(v)+1−1s(v)=2^{H-d(v)+1}-1:

  1. Depth 1: s=7
  2. Depth 2: s=3
  3. Depth 3: s=1

Then we set the upper thresholds: τ1=0.8,τ2=0.9,τ3=1\tau_1=0.8, \tau_2=0.9, \tau_3=1.

Suppose we now just insert naively (the kind that degenerates into a linked list)

// Insert 10
10			// rho(root)=1/7=0.14 < 0.8
    
 
// Insert 20
  10
    \
    20      // rho(root)=2/7=0.28 < 0.8;  rho(node_20) = 1/3 = 0.66 < 0.8  valid
    
// Insert 30
  10
    \
    20
      \
      30    // rho(root)=3/7 < 0.8; rho(node_20) = 2/3=0.67 < 0.8; rho(node_30)=1/1=1 <= tau_3
 
// Insert 40
  10
    \
    20
      \
      30
        \
        40   ← newly inserted, depth 4, exceeds the height

At this point, 40 has gone beyond height 3, making the tree’s height 4.

  • ρ(d=3)=21=2\rho(d=3)=\frac{2}{1}=2
  • ρ(d=2)=33=1\rho(d=2)=\frac{3}{3}=1, exceeding τ2=0.9\tau_2=0.9
  • ρ(d=1)=47<0.8\rho(d=1)=\frac{4}{7} < 0.8, not exceeding τ1\tau_1

Since the one that doesn’t exceed its threshold is d=1, we rebuild at level d=1d=1:

  1. The in-order sequence at this point is {10,20,30,40}; we choose 20 as the median
  2. Fill the left and right halves back in:
    20
   /  \
 10   30
         \
         40

Example: inserting 50

Now suppose we want to insert 50 as well?

    20
   /  \
 10   30
         \
         40
           \
           50   ← newly inserted, depth 4
  • ρ(d=3)=21=2\rho(d=3)=\frac{2}{1}=2
  • ρ(d=2)=33=1\rho(d=2)=\frac{3}{3}=1, exceeding τ2=0.9\tau_2=0.9; there are actually 3 elements, and the capacity it should theoretically hold is s(v)=23−2+1−1=3s(v) = 2^{3-2+1}-1=3
  • ρ(d=1)=57=0.71\rho(d=1)=\frac{5}{7}=0.71, not exceeding τ1\tau_1

At this point, the one without a problem is still the root, d=1d=1, so we rebuild at the root:

[10,20,30,40,50] becomes:

     30
    /  \
  20    40
 /        \
10        50

Example: inserting 60 next

When we then insert 60, even ρ(d=1)≈0.86>0.8\rho(d=1) \approx 0.86 > 0.8; not even the ancestor can satisfy the threshold, so the only option is to rebuild the entire tree and increase its height.

Summary

From the examples above, we can observe that:

  1. Our tree doesn’t have to be completely full to trigger a global rebuild; it may need to be rebuilt even before it is full.
  2. The tree’s capacity, and whether a rebuild is needed, are determined by the parameters τi\tau_i that we set in advance; in practice, these parameters need to be defined carefully.

Why does this stay stable in the long run?

After a redistribution at vv, the actual size of any descendant ω\omega falls within some interval, which is determined by the density and the constants τi\tau_i. That is, “size = capacity x density”, up to a constant-factor error, which keeps every level from becoming either too crowded or too sparse.

Amortized complexity analysis of insertion

Amortization means spreading the cost evenly across every operation. Some operations trigger a rebuild and some don’t, but for a rebuild we can’t simply count the complexity of the rebuild itself; its cost should be amortized back onto the earlier operations that didn’t trigger a rebuild. Only that is fair.

Time: O(HΔ)=O(log2n(1−τ1))O(\frac{H}{\Delta})=O(\frac{log^2n}{(1-\tau_1)})

Block transfers: O(logBn)+O(log2nB(1−τ1))O(log_Bn)+O(\frac{log^2n}{B(1-\tau_1)})

3.3.2 Deletion

The Delete operation supports deletion using “density thresholds + local rebuilding” without disturbing the static vEB layout, while still guaranteeing the I/O upper bounds for range queries and searches.

If we simply mark nodes as “deleted” and wait until half of them have been deleted before doing a global rebuild, some regions become very sparse, and range queries can no longer be given a worst-case upper bound on block transfers (i.e., sometimes you have to cross many blocks just to scan a small range). So after a deletion we must rebalance promptly, rather than leaving deletion marks around indefinitely.

Similar to the sequence of upper thresholds we set earlier, the paper here proposes a series of lower thresholds 0<γH<γ2<gamma1<τ10 < \gamma_H < \gamma_2 < gamma_1 < \tau_1. Note that the bound at the far end is the smallest of the upper thresholds, τ1\tau_1.

Deletion steps:

  1. First do a standard BST deletion: find, top-down, the node vv that holds the element to delete. If vv is not a leaf and has a right subtree, find its “successor” and swap with it, repeating until reaching a leaf, then delete that leaf; if there is no right subtree, symmetrically use the “predecessor”.
  2. Rebalance: looking upward from the deleted leaf, find the lowest ancestor ω\omega whose density is back within the valid range γd(ω)<ρ(ω)<τd(ω)\gamma_{d(\omega)} < \rho(\omega) < \tau_{d(\omega)}, then do an even rebuild of the entire subtree TwT_w. The concrete steps are almost the same as above.

Example: deleting 30

Set τ1=0.8,γ1=0.4,γ2=0.35,γ3=0.30\tau_1=0.8, \gamma_1=0.4, \gamma_2=0.35, \gamma_3=0.30

      40            (depth 1, s=7)
    /    \
  20      60        (depth 2, s=3)
 /  \    /  \
10  30  50  70      (depth 3, s=1)

After deleting 30, ∣T20∣=2|T_20|=2 (itself + the 10 below it), and the density ρ(20)=23>γ2\rho(20)=\frac{2}{3} > \gamma_2, so no rebuild is needed.

Deleting 10

      40            (depth 1, s=7)
    /    \
  20      60        (depth 2, s=3)
         /  \
        50  70      (depth 3, s=1)

Now ρ(20)=0.33<γ2\rho(20)=0.33 < \gamma_2, so an even rebuild is needed.

We go up to the parent node 40: ρ(40)=57>γ1\rho(40)=\frac{5}{7} > \gamma_1, which is fine, so we rebuild at node 40.

The median of [20,40,50,60,70] is 50. After the rebuild:

      50            (depth 1, s=7)
    /    \
  40      60        (depth 2, s=3)
 /         \
20         70      (depth 3, s=1)

Going further, if we keep deleting, a global height reduction will be triggered.

Amortized cost analysis

  1. Amortized time: O(log2nα)O(\frac{log^2n}{\alpha}), where α∈min{γ1−γH,1−τ1}\alpha \in min\{ \gamma_1-\gamma_H, 1 - \tau_1 \}
  2. I/O block transfers: searches are still logBnlog_B n, and updates are amortized O(logBn+log2nBα)O(log_B n + \frac{log^2 n}{B \alpha})

3.3.3 Improve Densities

Recap: above, we went through the three techniques the paper proposes: embedding in a static complete tree, the vEB layout, and the density-threshold-based rebuilding scheme.

The next question is: how do we compress the worst-case space from 2n2n down to the near-optimal (1+ε)n(1+\varepsilon)n, without making the I/O upper bounds for searches and range queries any worse?

How do we compress space without knowing B,MB, M?

The paper sets a space usage NN and chooses a root density threshold γ1≤nN≤τ1\gamma_1 \le \frac{n}{N} \le \tau_1; once this is violated, it switches to a new NN and does a global rebuild. This way, the overall space is kept strictly on the order of (1+ε)n(1+\varepsilon)n.

There are then two cases for the value of NN:

  1. If N=2k−1N = 2^k - 1, directly use the earlier single “host complete tree” scheme.
  2. Otherwise, write NN as a sum of powers of two: N=∑bi2bi=2b1+2b2+2b3+...+2bnN = \sum_{b_i} 2^{b_i} = 2^{b_1} + 2^{b_2} + 2^{b_3} + ... + 2^{b_n}
    1. For each bib_i, build a tree FiF_i that has just a root rir_i with no left child, and a complete right subtree CiC_i of size 2bi−12^{b_i}-1 attached on the right
    2. Split all the elements into contiguous segments and distribute them onto these FiF_i.
    3. In memory, arrange them in the order r1,r2,...,rk,C1,C2,...,Ckr_1, r_2, ..., r_k, C_1, C_2, ..., C_k, with each CiC_i using the vEB layout.
  3. When searching, first compare against r1,...,rkr_1, ..., r_k in order to find which segment TiT_i the key falls into, then search top-down within TiT_i. The I/O cost of this is O(iB+logB(2ib−1))=O(logBN)O(\frac{i}{B}+log_B(2^b_i-1))=O(log_B N)
  4. Given how often N is changed, the amortized time of global rebuilds is O(1ε)O(\frac{1}{\varepsilon}), and the I/O cost is O(1εB)O(\frac{1}{\varepsilon B})

4. Experiments

Example image

Figure 7. Left: performance of the four layouts with pointer-based implementations; right: performance of the four layouts with array-based implementations

based on pointers means trees implemented with pointers. The left plot in Fig 7 compares the pointer-based versions of several data structures side by side. When n is small, the whole tree fits in the cache, so there isn’t much performance difference among the trees (n<14n < 14 ). As n grows, BFS incurs a miss at every level, DFS a miss every two levels, and vEB one every Θ(logBn)\Theta(log_Bn) levels, which is almost exactly what the paper’s analysis concludes.

The right plot is the experimental analysis of the implicit versions, i.e., the versions with no pointers that rely purely on arrays. Within the cache, address-computation overhead dominates: BFS is the fastest and vEB the slowest (vEB has the most complex addressing). Once the data exceeds the cache, memory accesses (I/O operations) become the bottleneck, and the high-degree (d=8,16d=8,16) cache-aware versions are the fastest, since they better match the cache line size. vEB gradually catches up with BFS and eventually overtakes it (at around n=21∼22n=21 \sim 22). In the end it is only about 50% slower than the high-degree schemes. The curve for the inorder group fluctuates, especially when nn is a power of 2, presumably because limited associativity maps the top nodes to the same cache set.

Example image

Figure 8. Performance of pointer-based vs. array-based implementations of vEB and BFS

From the first experiment above, we draw a preliminary conclusion: vEB and BFS are worth studying further, while the other two layouts don’t perform very well. So next, the authors directly compare the vEB and BFS layouts side by side in their pointer-based and array-based implementations.

  • BFS: the implicit version always beats the pointer version, which shows that “computing addresses” is no slower than “following pointers”.
  • vEB: within the cache the pointer version is faster, but outside the cache the implicit version turns the tables. The reason is that without pointers, the same block can hold more nodes (B gets larger).
Example image

Figure 9. vEB vs. an unbalanced tree: insertion and search performance

From Fig 9 we can roughly draw the following conclusion: if the system is mostly doing searches with only occasional updates (updates here meaning inserts), the semi-dynamic vEB is better around n≈216n \approx 2^{16} (the lowest curve).

Example image

Figure 10. Performance comparison beyond main memory

Within main memory, BFS is the fastest; once the size of main memory is exceeded, BFS instead becomes more than 5×5 \times slower (note that the vertical axis in the figure is not linear). The 1024-ary tree optimized for the page size is the fastest outside main memory, but about 2×2 \times slower within main memory (again, the vertical axis is not linear). vEB, by contrast, keeps pace with the best of the structures all the way, performing stably across levels.

Authors’ conclusions

  1. The memory hierarchy dominates running time, even when the tree is still far smaller than main memory.
  2. The benefits of cache-obliviousness hold in practice: vEB is competitive with cache-aware schemes, almost always better than “unoptimized” schemes, and robust across multiple memory levels.
  3. The space savings / larger fan-out (larger B) brought by the implicit (pointer-free) layout contribute significantly to performance; the dynamic scheme is both easy to implement and shows good time performance.

5. Reflection

The implicit, cache-oblivious search tree of small height achieves the B-tree’s lower bound on I/O performance, laying a solid foundation for robust range queries across the board. With the vEB layout, its I/O cost can be kept at O(logBn+kB)O(log_B n + \frac{k}{B}), and these optimality results still hold in 2025; however, the “backbone” could also be replaced by other structures (such as the cache-oblivious B-Tree). Given the steadily increasing capacity of hardware at every level (registers, L1/L2/L3 caches, main memory), as well as modern techniques such as prefetching, branch prediction, and SIMD, other approaches are often faster than this paper’s scheme in some purely in-memory scenarios; we will discuss potential improvements later.

For root-to-leaf accesses, the vEB layout has a clean logBnlog_B n complexity (the formula here appears garbled in the original), and it usually outperforms other schemes when n≥thresholdn \ge threshold. Compared with prior work, the small-height tree does not need weight balancing, avoiding heavy or frequent updates; instead, it redistributes the tree only when necessary. Note that when n≤thresholdn \le threshold (the original formula is likewise garbled), the implicit array-based layout incurs extra overhead because index computation becomes the dominant cost; as nn grows, this overhead turns into a trade-off against pointer chasing. The array-based layout also means that the effective capacity nn of a single transferred block is larger, so one I/O can fetch more data. A rebuild of the whole tree (H←H+1H ← H + 1) is triggered only when nn grows enough that the root violates the rule; consecutive rebuilds are separated by Θ(n)Θ(n) updates, so the cost amortized over each update is small.

Having summarized the strengths, we should also look at the shortcomings. As Fig 9 shows, although the vEB layout performs better in the large-scale insertion comparison, its performance jitters fairly often, which in the long run may affect the overall behavior of the hardware and the system. In addition, it is clear that the vEB scheme performs well at larger cache scales but falls behind other schemes with smaller caches. The caches and memory of modern CPUs keep growing, which to some extent limits its potential range of application.

Modern I/O models typically involve multi-level caches (L1/L2/L3), the page cache, and the NVMe protocol. The two-level I/O model used in this paper is fairly idealized: it only analyzes block transfers between two levels and ignores factors beyond the cache line, such as the TLB and page size—something already reflected in the Cache-Conscious B+ Tree. Furthermore, this paper mainly discusses the data structure in a single-threaded context, whereas modern systems commonly use multi-threaded access and care more about latch-free or incremental updates to reduce the cache invalidation and lock contention caused by shared writes; the Bw-Tree discusses this specifically.

Finally, it’s worth emphasizing that the cache-oblivious B-tree proposed in 2005 retains this paper’s core advantages—fast lookups and range scans across all levels of the memory hierarchy— while not requiring periodic subtree rebuilds. It is easier to operate, friendlier to concurrent reads and writes, and works well everywhere from CPU caches to SSDs without manual tuning; it stays compact, with low pointer overhead and strong locality, while also delivering smoother performance and lower engineering cost.

Reference

Footnotes

  1. P. van Emde Boas. Preserving order in a forest in less than logarithmic time and linear space. Inf. Process. Lett., 6:80–82, 1977. ↩

  2. H. Prokop. Cache-oblivious algorithms. Master’s thesis, Massachusetts Institute of Technology, June 1999. ↩

  3. S. Chatterjee, V. V. Jain, A. R. Lebeck, S. Mundhra, and M. Thottethodi. Nonlinear array layouts for hierarchical memory systems. In Proceedings of the 1999 Conference on Supercomputing, ACM SIGARCH, pages 444–453.ACM Press, 1999. ↩