Principles：flatbuffers逆向分析

Flatbuffers 简介

FlatBuffers 是一个开源的、跨平台的、高效的、提供了多种语言接口的序列化工具库，实现了与 Protocal Buffers 类似的序列化格式

FlatBuffers 通过 Scheme 文件定义数据结构，在官方网站 FlatBuffers: Tutorial 中可以找到如下的测试案例：

namespace MyGame.Sample;
enum Color:byte { Red = 0, Green, Blue = 2 }
union Equipment { Weapon } // Optionally add more tables.
 
struct Vec3 {
  x:float;
  y:float;
  z:float;
}
 
table Monster {
  pos:Vec3; // Struct.
  mana:short = 150;
  hp:short = 100;
  name:string;
  friendly:bool = false (deprecated);
  inventory:[ubyte];  // Vector of scalars.
  color:Color = Blue; // Enum.
  weapons:[Weapon];   // Vector of tables.
  equipped:Equipment; // Union.
  path:[Vec3];        // Vector of structs.
}
 
table Weapon {
  name:string;
  damage:short;
}
 
root_type Monster;

Table：每个字段都有一个名称，一个类型和一个可选的默认值（默认为 0 / NULL）
Structs：只包含标量或其他结构（没有默认值，其字段也不会被添加或者弃用）
Types：包括标量类型和非标量类型
- 标量类型的字段有默认值
- 非标量的字段 (string/vector/table/enums/structs) 如果没有值的话，默认值为 NULL
Enums：定义一系列命名常量，默认的第一个值是 “0”
Unions：一个 Unions 中可以放置多种类型，共同使用一个内存区域
- 可以声明一个 Unions 字段，该字段可以包含对这些类型中的任何一个的引用，即这块内存区域只能由其中一种类型使用
- 另外还会生成一个带有后缀 _type 的隐藏字段，该字段包含相应的枚举值，从而可以在运行时知道要将哪些类型转换为类型
Root Type：声明了序列化数据的根表

参考：深入浅出 FlatBuffers 之 Schema (halfrost.com)

Flatbuffers Table

Table 中每个字段都是可选 optional 的，这种机制可以令 Flatbuffers 前向和后向兼容

假设当前 schema 如下：

1	table { a:int; b:int; }

在后添加字段：

1	table { a:int; b:int; c:int; }

旧的 schema 读取新的数据结构会忽略新字段 c 的存在（不会浪费存储空间），新的 schema 读取旧的数据，将会取到 c 的默认值

在前添加字段：

1	table { c:int a:int; b:int; }

在前面添加新字段是不允许的，因为这会使 schema 新旧版本不兼容

指定字段 ID 序列：

1	table { c:int (id: 2); a:int (id: 0); b:int (id: 1); }

可以手动指定 ID 排序/分组，只要顺序与旧版本相同也是可以兼容的

删除字段：

1	table { a:int (deprecated); b:int; }

不能直接从 schema 中删除字段（否则会破坏源代码），可以将需要删除的字段标记为 deprecated
旧的 schema 读取新的数据结构会获得 a 的默认值（因为它不存在）
新的 schema 代码不能读取也不能写入 a，它们将忽略该字段

更改字段：

1	table { a:uint; b:uint; }

如果旧数据不包含任何负数，这将是安全的，如果包含了负数，这样改变会出现问题
不能修改字段名称与字段默认值（否则会破坏所有使用此版本 schema 的代码）

Flatbuffers Structs

Structs 只包含标量或其他结构，只要确定以后就不会进行任何更改

Structs 使用的内存少于 Table，并且访问速度更快，它们总是以串联方式存储在其父对象中，并且不使用虚表

Structs 不提供前向/后向兼容性，占用内存更小

Flatbuffers 内存布局

测试样例：

namespace Test.Sample;
 
table Monster {
  name:string;
  weapon:[Weapon];
}

table Weapon {
  key:string;
  value:int;
}
 
root_type Monster;

#include "flatbuffers.h"
#include "test_generated.h"
#include <iostream> // C++ header file for printing
#include <fstream> // C++ header file for file access
 
using namespace Test::Sample;

int main(){
    flatbuffers::FlatBufferBuilder builder(1024);
    auto name = builder.CreateString("name");
    auto key1 = builder.CreateString("aaaaaaaa");
    auto key2 = builder.CreateString("bbbbbbbb");
    auto key3 = builder.CreateString("cccccccc");
    auto key4 = builder.CreateString("dddddddd");
    auto value1 = 0x31;
    auto value2 = 0x32;
    auto value3 = 0x33;
    auto value4 = 0x34;

    auto weapon1 = CreateWeapon(builder,key1,value1);
    auto weapon2 = CreateWeapon(builder,key2,value2);
    auto weapon3 = CreateWeapon(builder,key3,value3);
    auto weapon4 = CreateWeapon(builder,key4,value4);
    std::vector<flatbuffers::Offset<Weapon>> weapons_vector;
    weapons_vector.push_back(weapon1);
    weapons_vector.push_back(weapon2);
    weapons_vector.push_back(weapon3);
    weapons_vector.push_back(weapon4);

    auto weapons = builder.CreateVector(weapons_vector);
    auto monster = CreateMonster(builder,name,weapons);

    builder.Finish(monster);
    uint8_t *buf = builder.GetBufferPointer();
    auto size = builder.GetSize();
    flatbuffers::Verifier verifier(buf, size);
    bool ok = verifier.VerifyBuffer<Monster>(nullptr);
    if (ok){
        auto monster_bk = GetMonster(buf);
        auto name_bk = monster_bk->name()->c_str();
        auto weapons_bk = monster_bk->weapon();
        auto key1_bk = weapons_bk->Get(0)->key()->str();
        auto key2_bk = weapons_bk->Get(1)->key()->str();
        auto key3_bk = weapons_bk->Get(2)->key()->str();
        auto key4_bk = weapons_bk->Get(3)->key()->str();
        auto value1_bk = weapons_bk->Get(0)->value();
        auto value2_bk = weapons_bk->Get(1)->value();
        auto value3_bk = weapons_bk->Get(2)->value();
        auto value4_bk = weapons_bk->Get(3)->value();
    }
}

在 CreateMonster 处打断点，然后 GDB 调试打印数据如下：

02:0010│  0x5555555702b0 ◂— 0x9c00000060 /* '`' */
03:0018│  0x5555555702b8 ◂— 0xa000000006
04:0020│  0x5555555702c0 ◂— 0x4
    ......
6c:0360│  0x555555570600 ◂— 0xc000800000000
6d:0368│  0x555555570608 ◂— 0xffffffbc00080004
6e:0370│  0x555555570610 ◂— 0x400000094
6f:0378│  0x555555570618 ◂— 0x3c00000004
70:0380│  0x555555570620 ◂— 0x1400000024 /* '$' */
71:0388│  0x555555570628 ◂— 0xffffffdc00000004
72:0390│  0x555555570630 ◂— 0x3400000034 /* '4' */
73:0398│  0x555555570638 ◂— 0x38ffffffe8
74:03a0│  0x555555570640 ◂— 0xfffffff400000033 /* '3' */
75:03a8│  0x555555570648 ◂— 0x320000003c /* '<' */
76:03b0│  0x555555570650 ◂— 0x80004000c0008
77:03b8│  0x555555570658 ◂— 0x3800000008
78:03c0│  0x555555570660 ◂— 0x800000031 /* '1' */
79:03c8│  0x555555570668 ◂— 'dddddddd'
7a:03d0│  0x555555570670 ◂— 0x800000000
7b:03d8│  0x555555570678 ◂— 'cccccccc'
7c:03e0│  0x555555570680 ◂— 0x800000000
7d:03e8│  0x555555570688 ◂— 'bbbbbbbb'
7e:03f0│  0x555555570690 ◂— 0x800000000
7f:03f8│  0x555555570698 ◂— 'aaaaaaaa'
80:0400│     0x5555555706a0 ◂— 0x400000000
81:0408│     0x5555555706a8 ◂— 0x656d616e /* 'name' */

首先在 Flatbuffers 的缓冲区中，数据是倒序排列的
从下往上看：在字符串之后，有4字节的 \x00 用于分割 data 和 size，又有4字节的空间用于指示字符串的大小
位于缓冲区最上方的控制信息是实时更新的

Flatbuffers 逆向分析

下面先展示一个 IDA 逆向分析出来的伪代码：

v3 = flatbuffers::AlignOf<unsigned long>();
flatbuffers::FlatBufferBuilderImpl<false>::FlatBufferBuilderImpl(&builder, 0x400uLL, 0LL, 0, v3);
name.o = flatbuffers::FlatBufferBuilderImpl<false>::CreateString<flatbuffers::Offset>(&builder, "name").o;
key1.o = flatbuffers::FlatBufferBuilderImpl<false>::CreateString<flatbuffers::Offset>(&builder, "aaaaaaaa").o;
key2.o = flatbuffers::FlatBufferBuilderImpl<false>::CreateString<flatbuffers::Offset>(&builder, "bbbbbbbb").o;
key3.o = flatbuffers::FlatBufferBuilderImpl<false>::CreateString<flatbuffers::Offset>(&builder, "cccccccc").o;
key4.o = flatbuffers::FlatBufferBuilderImpl<false>::CreateString<flatbuffers::Offset>(&builder, "dddddddd").o;
value1 = 49;
value2 = 50;
value3 = 51;
value4 = 52;
weapon1.o = Test::Sample::CreateWeapon(&builder, key1, 49).o;
weapon2.o = Test::Sample::CreateWeapon(&builder, key2, 50).o;
weapon3.o = Test::Sample::CreateWeapon(&builder, key3, 51).o;
weapon4.o = Test::Sample::CreateWeapon(&builder, key4, 52).o;
std::vector<flatbuffers::Offset<Test::Sample::Weapon>>::vector(&weapons_vector);
std::vector<flatbuffers::Offset<Test::Sample::Weapon>>::push_back(&weapons_vector, &weapon1);
std::vector<flatbuffers::Offset<Test::Sample::Weapon>>::push_back(&weapons_vector, &weapon2);
std::vector<flatbuffers::Offset<Test::Sample::Weapon>>::push_back(&weapons_vector, &weapon3);
std::vector<flatbuffers::Offset<Test::Sample::Weapon>>::push_back(&weapons_vector, &weapon4);
weapons.o = flatbuffers::FlatBufferBuilderImpl<false>::CreateVector<flatbuffers::Offset<Test::Sample::Weapon>,std::allocator<flatbuffers::Offset<Test::Sample::Weapon>>>(
              &builder,
              &weapons_vector).o;
monster.o = Test::Sample::CreateMonster(&builder, name, weapons).o;
flatbuffers::FlatBufferBuilderImpl<false>::Finish<Test::Sample::Monster>(&builder, monster, 0LL);
buf = flatbuffers::FlatBufferBuilderImpl<false>::GetBufferPointer(&builder);
size = flatbuffers::FlatBufferBuilderImpl<false>::GetSize(&builder);
flatbuffers::Verifier::Verifier(&verifier, buf, size, 0x40u, 0xF4240u, 1);
if ( flatbuffers::Verifier::VerifyBuffer<Test::Sample::Monster>(&verifier, 0LL) )
{
  monster_bk = Test::Sample::GetMonster(buf);
  v4 = Test::Sample::Monster::name(monster_bk);
  name_bk = flatbuffers::String::c_str(v4);
  weapons_bk = Test::Sample::Monster::weapon(monster_bk);
  v5 = flatbuffers::Vector<flatbuffers::Offset<Test::Sample::Weapon>,unsigned int>::Get(weapons_bk, 0);
  v6 = Test::Sample::Weapon::key(v5);
  flatbuffers::String::str[abi:cxx11](&key1_bk, v6);
  v7 = flatbuffers::Vector<flatbuffers::Offset<Test::Sample::Weapon>,unsigned int>::Get(weapons_bk, 1u);
  v8 = Test::Sample::Weapon::key(v7);
  flatbuffers::String::str[abi:cxx11](&key2_bk, v8);
  v9 = flatbuffers::Vector<flatbuffers::Offset<Test::Sample::Weapon>,unsigned int>::Get(weapons_bk, 2u);
  v10 = Test::Sample::Weapon::key(v9);
  flatbuffers::String::str[abi:cxx11](&key3_bk, v10);
  v11 = flatbuffers::Vector<flatbuffers::Offset<Test::Sample::Weapon>,unsigned int>::Get(weapons_bk, 3u);
  v12 = Test::Sample::Weapon::key(v11);
  flatbuffers::String::str[abi:cxx11](&key4_bk, v12);
  v13 = flatbuffers::Vector<flatbuffers::Offset<Test::Sample::Weapon>,unsigned int>::Get(weapons_bk, 0);
  value1_bk = Test::Sample::Weapon::value(v13);
  v14 = flatbuffers::Vector<flatbuffers::Offset<Test::Sample::Weapon>,unsigned int>::Get(weapons_bk, 1u);
  value2_bk = Test::Sample::Weapon::value(v14);
  v15 = flatbuffers::Vector<flatbuffers::Offset<Test::Sample::Weapon>,unsigned int>::Get(weapons_bk, 2u);
  value3_bk = Test::Sample::Weapon::value(v15);
  v16 = flatbuffers::Vector<flatbuffers::Offset<Test::Sample::Weapon>,unsigned int>::Get(weapons_bk, 3u);
  value4_bk = Test::Sample::Weapon::value(v16);
  std::string::~string(&key4_bk);
  std::string::~string(&key3_bk);
  std::string::~string(&key2_bk);
  std::string::~string(&key1_bk);
}
std::vector<flatbuffers::Offset<Test::Sample::Weapon>>::~vector(&weapons_vector);
flatbuffers::FlatBufferBuilderImpl<false>::~FlatBufferBuilderImpl(&builder);

这里我们重点分析反序列化的部分：

const flatbuffers::String *__cdecl Test::Sample::Monster::name(const Test::Sample::Monster *const this)
{
  return flatbuffers::Table::GetPointer<flatbuffers::String const*,unsigned int>(this, 4u);
}

程序的反序列化会大量使用 flatbuffers::Table::GetXXXX 的函数模板，从这些函数中可以看出一些 Scheme 文件的信息

const flatbuffers::String *__cdecl Test::Sample::Monster::name(const Test::Sample::Monster *const this)
{
  return flatbuffers::Table::GetPointer<flatbuffers::String const*,unsigned int>(this, 4u);
}

第二个参数的数字可以体现该条目在 Scheme 文件中的位置

int32_t __cdecl Test::Sample::Weapon::value(const Test::Sample::Weapon *const this)
{
  return flatbuffers::Table::GetField<int>(this, 6u, 0);
}

从传参个数可以判断该条目是否为标量类型（有3个参数即是标量类型，第3个参数为其默认值）

在逆向分析 flatbuffers 的过程中，Verifier 可以暴露大量信息（Verifier 是 flatbuffers 的检查器），它的内部结构如下：

if ( !flatbuffers::Verifier::Check(this, this->size_ > 0xB) )
  return 0;
if ( identifier )
{
  v4 = this->size_ > 7 && flatbuffers::BufferHasIdentifier(&this->buf_[start], identifier, 0);
  if ( !flatbuffers::Verifier::Check(this, v4) )
    return 0;
}
o = flatbuffers::Verifier::VerifyOffset<unsigned int,int>(this, start);
return flatbuffers::Verifier::Check(this, o != 0)
    && Test::Sample::Monster::Verify((const Test::Sample::Monster *const)&this->buf_[start + o], this);

最后一个函数 Verify 可以泄露有关 Scheme 的信息：

if ( flatbuffers::Table::VerifyTableStart(this, verifier)
  && flatbuffers::Table::VerifyOffset<unsigned int>(this, verifier, 4u) )
{
  v2 = Test::Sample::Monster::name(this);
  if ( flatbuffers::Verifier::VerifyString(verifier, v2)
    && flatbuffers::Table::VerifyOffset<unsigned int>(this, verifier, 6u) )
  {
    v3 = Test::Sample::Monster::weapon(this);
    if ( ZNK11flatbuffers8Verifier12VerifyVectorIJENS_6OffsetIN4Test6Sample6WeaponEEEjEEbPKNS_6VectorIT0_T1_EE(
           verifier,
           v3) )
    {
      v4 = Test::Sample::Monster::weapon(this);
      if ( flatbuffers::Verifier::VerifyVectorOfTables<Test::Sample::Weapon>(verifier, v4)
        && flatbuffers::Verifier::EndTable(verifier) )
      {
        return 1;
      }
    }
  }
}

VerifyString 用于分析 string 类型
VectorVerifyVectorOfTables 用于分析 Vector 类型

bool __cdecl Test::Sample::Weapon::Verify(const Test::Sample::Weapon *const this, flatbuffers::Verifier *verifier)
{
  const flatbuffers::String *v2; // rdx
  bool result; // al

  result = 0;
  if ( flatbuffers::Table::VerifyTableStart(this, verifier)
    && flatbuffers::Table::VerifyOffset<unsigned int>(this, verifier, 4u) )
  {
    v2 = Test::Sample::Weapon::key(this);
    if ( flatbuffers::Verifier::VerifyString(verifier, v2)
      && flatbuffers::Table::VerifyField<int>(this, verifier, 6u, 4uLL)
      && flatbuffers::Verifier::EndTable(verifier) )
    {
      return 1;
    }
  }
  return result;
}

VerifyField 的第4个参数表示该标量类型条目的大小