学习Swift源码（一）

Luminixus
1,826 阅读17分钟

2021-03-24

前言

学习框架最好的方式是学习它的源码。对于 Objective-C Runtime 源码，虽然代码不少但起码可以找到下手的地方。可面对 Swift 源码的时候，第一感觉就是这工程也忒大了，一时间竟无从下手。按照 startup 文档的指示，拉下来单单是相关工程就有十几个，看着就挺吓人的。不过这些相关工程要么是单元测试、要么是编译运行调试 Swift 的必要 ToolChain，由于机子也扛不下 70G 的最小构建磁盘空间，所以打算还是 Objective-C Runtime 那一套，干看代码。

截屏2021-03-27 下午10.20.49.png

一、划定范围

主工程显然是 swift 文件夹，swift 工程使用 llvm 通过 CMake 配置文件构建的，具体怎么构建的工程，CMake 文件怎么写的，这些方面不是我想学习的内容，首先给自己明确目标，划定好范围，个人比较感兴趣的是以下几个方面：

  • Swift 底层是怎么实现的；
  • Swift 的内存管理是怎么实现的；
  • Swift 基础类型框架是怎么搭建起来的；
  • Swift 的可选类型是怎么实现的；
  • Swift 是如何与 Objective-C Runtime 进行交互的；
  • Swift 的 DispatchQueue 框架是如何实现的；

二、突破口

前面提到过 swift 整个工程拉下来洋洋洒洒几百兆全是代码，docs 简单浏览了一下，没什么干货，帮不上什么忙，只能靠自己。前面其实有点眉目了，先把范围缩小到 swift 主工程，文件夹中包含的目录及文件如下。其中，include、lib、stdlib 看起来就像是比较多干货的地方，随便搜个比较感兴趣的话题 Enum，定位到 stdlib/public/runtime/Enum.cpp 看来是 C++ 实现的，看文件夹是个 runtime，Swift 也有 Runtime？带着问题继续往下探索。总之可以先确定 stdlib 是下手的点。

截屏2021-03-27 下午10.21.09.png

找到着力点就可以使劲了。很遗憾，找到 Enum.cpp 也一时间没看懂，又翻了一下关联的 Enum.h、EnumImpl.h，还是没闹明白，看来先得热热身。想到编程珠玑中提到的数据结构的关键性，看 Objective-C Runtime 源码的时候，也是从 Class、Object、SideTable、DenseMap 等数据结构入手的。先这么办吧，在 stdlib/include/llvm 中翻到个 ADT 文件夹，浏览一下其中的头文件，很多是依赖了 llvm/ADT、llvm/Support，再去看看 llvm/Support 文件夹中的头文件，只有几个依赖了 llvm/ADT。那么依赖关系基本确定了 llvm/Support 所处逻辑层更低，大多数情况下是 llvm/ADT 调用了 llvm/Support 的服务。

至此突破口敲定：先 llvm/Support，后 llvm/ADT。

三、硬刚源码

粗略浏览了一遍两个文件夹的内容，实际上这两个文件夹内容，在严格意义上并不在前面划定的范围内，因此决定只简要了解，不深入探究。

3.1 Support

对 Support 文件夹内容作简要分类：

  • Alignment：内存字节对齐；
  • Allocator ：内存分配相关；
  • Casting：类型转换；

3.1.1 内存对齐

这里涉及的代码是用于指导内存对齐，主要逻辑在 Alignment.h 中。其功能从以下几行代码可以大致了解，Align结构体包含一个占 1 字节的ShiftValue私有成员，可以通过调用explicit Align(uint64_t Value)构建结构体，其中传入的Value必须是 2 的整数次幂，将Value的 2 的对数保存到ShiftValue。

// Alignment.h

struct Align {
private:
  uint8_t ShiftValue = 0;
  ...

public:
  ...
  explicit Align(uint64_t Value) {
    assert(Value > 0 && "Value must not be 0");
    assert(llvm::isPowerOf2_64(Value) && "Alignment is not a power of 2");
    ShiftValue = Log2_64(Value);
    assert(ShiftValue < 64 && "Broken invariant");
  }
  ...

}

上述Align类型和内存按字节对齐的行为是一致的。例如，64 位的Integer类型，其对齐字节数是 8，对齐字节数的 2 的对数是 3，则值为 n * 23 的内存地址都满足对齐Integer类型的条件，因此Integer对应的Align的ShiftValue的值为 3。

3.1.2 内存分配

内存分配主要涉及三个文件，根据依赖关系，阅读的先后次序应为 MemAlloc.h -> AllocatorBase.h -> Allocator.h。MemAlloc.h 是若干个调用std::malloc实现内存分配的工具函数增加了闭包的安全检查和断言。

AllocatorBase.h 中主要定义了MallocAllocator类型，该类型满足 llvm 的Allocator协议，传入分配字节数以及对齐字节数，并请求分配内存，Allocator除了具有分配内存的功能，还具有释放内存功能，对内存的操作基本上是使用了 MemAlloc.h 中定义的服务，例如：allocate_buffer、deallocate_buffer。

// MallocAllocator.h

...

class MallocAllocator : public AllocatorBase<MallocAllocator> {
public:
  void Reset() {}

  LLVM_ATTRIBUTE_RETURNS_NONNULL void *Allocate(size_t Size, size_t Alignment) {
    return allocate_buffer(Size, Alignment);
  }

  // Pull in base class overloads.
  using AllocatorBase<MallocAllocator>::Allocate;

  void Deallocate(const void *Ptr, size_t Size, size_t Alignment) {
    deallocate_buffer(const_cast<void *>(Ptr), Size, Alignment);
  }

  // Pull in base class overloads.
  using AllocatorBase<MallocAllocator>::Deallocate;

  void PrintStats() const {}
};

...

Allocator.h 中主要定义了BumpPtrAllocator，该类型同样满足 llvm 的Allocator协议，以 4096 字节（SlabSize）为最小内存分配单位（Slab），以慢启动策略来连续分配内存。所谓慢启动是指，最开始分配内存的最小单位是 4096 字节，随着分配内存数量增多，当分配 Slab 数量到达一定的阈值（GrowthDelay）后，Slab 的大小将会翻倍，翻倍后分配内存的最小单位将增加到 2*SlabSize，以此类推。BumpPtrAllocator分配的内存不会一边分配一边释放，而是当确定不再使用时全部释放。BumpPtrAllocator分配内存的方式，和 Objective-C 的 AutoreleasePool 十分相似。

从以下BumpPtrAllocator的私有成员可以大致看出其实现框架，其核心逻辑是以下两个私有方法：

  • computeSlabSize：计算索引为SlabIdx的 Slab 的SlabSize，SlabSize的值一定满足 4096*2n 的约束，其中 n 为整数；
  • StartNewSlab：当探测到即将分配的内存空间超出当前 Slab 的范围后，需要调用该方法开辟新的 Slab，由于 Slab 尺寸随着分配内存空间增长而阶梯式指数型增长，显然计算新开辟的 Slab 尺寸通过computeSlabSize计算得出；
// Allocator.h

template <typename AllocatorT = MallocAllocator, size_t SlabSize = 4096,
          size_t SizeThreshold = SlabSize, size_t GrowthDelay = 128>
class BumpPtrAllocatorImpl
    : public AllocatorBase<BumpPtrAllocatorImpl<AllocatorT, SlabSize,
                                                SizeThreshold, GrowthDelay>> {

public:
  ...

private:
  /// The current pointer into the current slab.
  ///
  /// This points to the next free byte in the slab.
  char *CurPtr = nullptr;

  /// The end of the current slab.
  char *End = nullptr;

  /// The slabs allocated so far.
  SmallVector<void *, 4> Slabs;

  /// Custom-sized slabs allocated for too-large allocation requests.
  SmallVector<std::pair<void *, size_t>, 0> CustomSizedSlabs;

  /// How many bytes we've allocated.
  ///
  /// Used so that we can compute how much space was wasted.
  size_t BytesAllocated = 0;

  /// The number of bytes to put between allocations when running under
  /// a sanitizer.
  size_t RedZoneSize = 1;

  /// The allocator instance we use to get slabs of memory.
  AllocatorT Allocator;

  static size_t computeSlabSize(unsigned SlabIdx) {
    // Scale the actual allocated slab size based on the number of slabs
    // allocated. Every GrowthDelay slabs allocated, we double
    // the allocated size to reduce allocation frequency, but saturate at
    // multiplying the slab size by 2^30.
    return SlabSize *
           ((size_t)1 << std::min<size_t>(30, SlabIdx / GrowthDelay));
  }

  /// Allocate a new slab and move the bump pointers over into the new
  /// slab, modifying CurPtr and End.
  void StartNewSlab() {
    size_t AllocatedSlabSize = computeSlabSize(Slabs.size());

    void *NewSlab =
        Allocator.Allocate(AllocatedSlabSize, alignof(std::max_align_t));
    // We own the new slab and don't want anyone reading anything other than
    // pieces returned from this method.  So poison the whole slab.
    __asan_poison_memory_region(NewSlab, AllocatedSlabSize);

    Slabs.push_back(NewSlab);
    CurPtr = (char *)(NewSlab);
    End = ((char *)NewSlab) + AllocatedSlabSize;
  }
  /// Deallocate a sequence of slabs.
  void DeallocateSlabs(SmallVectorImpl<void *>::iterator I,
                       SmallVectorImpl<void *>::iterator E) {
    for (; I != E; ++I) {
      size_t AllocatedSlabSize =
          computeSlabSize(std::distance(Slabs.begin(), I));
      Allocator.Deallocate(*I, AllocatedSlabSize, alignof(std::max_align_t));
    }
  }

  /// Deallocate all memory for custom sized slabs.
  void DeallocateCustomSizedSlabs() {
    for (auto &PtrAndSize : CustomSizedSlabs) {
      void *Ptr = PtrAndSize.first;
      size_t Size = PtrAndSize.second;
      Allocator.Deallocate(Ptr, Size, alignof(std::max_align_t));
    }
  }

  template <typename T> friend class SpecificBumpPtrAllocator;
};

/// The standard BumpPtrAllocator which just uses the default template
/// parameters.
typedef BumpPtrAllocatorImpl<> BumpPtrAllocator;

注意：有内存分配的地方必然会有内存对齐的问题，不出意外，BumpPtrAllocator类型的实现，依赖了前面的Align类型。

基于BumpPtrAllocator，Allocator 文件还定义了SpecificBumpPtrAllocator类型。该类型分配内存用于为特定类型的元素分配内存空间。

// Allocator.h

...

/// A BumpPtrAllocator that allows only elements of a specific type to be
/// allocated.
///
/// This allows calling the destructor in DestroyAll() and when the allocator is
/// destroyed.
template <typename T> class SpecificBumpPtrAllocator {
  BumpPtrAllocator Allocator;
  
  ...

}

注意：看了这么多代码，实际上未必会直接应用到 Swift 的实现中。本文接下来的内容很可能也是如此，当作热身吧。

3.1.3 类型转换

Casting.h 文件中定义了若干用于简单类型转换的工具函数。其核心代码是isa_impl的doit方法， 传入需要类型转换的From类型的值或对象Val，返回目标转换类型To的类型。

//Casting.h

// Define a template that can be specialized by smart pointers to reflect the
// fact that they are automatically dereferenced, and are not involved with the
// template selection process...  the default implementation is a noop.
//
template<typename From> struct simplify_type {
  using SimpleType = From; // The real type this represents...

  // An accessor to get the real value...
  static SimpleType &getSimplifiedValue(From &Val) { return Val; }
};

template<typename From> struct simplify_type<const From> {
  using NonConstSimpleType = typename simplify_type<From>::SimpleType;
  using SimpleType =
      typename add_const_past_pointer<NonConstSimpleType>::type;
  using RetType =
      typename add_lvalue_reference_if_not_pointer<SimpleType>::type;

  static RetType getSimplifiedValue(const From& Val) {
    return simplify_type<From>::getSimplifiedValue(const_cast<From&>(Val));
  }
};

// The core of the implementation of isa<X> is here; To and From should be
// the names of classes.  This template can be specialized to customize the
// implementation of isa<> without rewriting it from scratch.
template <typename To, typename From, typename Enabler = void>
struct isa_impl {
  static inline bool doit(const From &Val) {
    return To::classof(&Val);
  }
};

...

至此，基本过完 llvm/Support 文件夹中的内容。相比 Objective-C Runtime 源码以 C 语言为主体的 C++ 代码，Swift 源码则是更加纯粹的 C++ 代码，所以阅读起来还是蛮吃力的。

3.2 ADT

开始之前还是先对内容进行简单分类：

  • 基础类型：None、Optional、iterator、iterator_range、STLExtras、Hashing；
  • 字符串：StringRef、StringSwitch、StringExtras；
  • 数组：ArrayRef；
  • 哈希表：DenseSet、DenseMap、DenseMapInfo；
  • 指针：PointerIntPair、PointerUnion；
  • Small：SmallStrSet、SmallString、SmallVector；

3.2.1 基础类型

STLExtras 扩展 STL 库提供了一些工具函数，Hashing 定义了一系列哈希函数，iterator 定义了通用的迭代器类型，iterator_range 定义了选定某个区间范围内的迭代器的类型，先不看。此外，SmallXXX 只是扩展了特定类型，并不是核心代码，也先不看。

None

None是为了支持可空类型Optional的而设定的表示null空的含义。

// None.h

inline namespace __swift { inline namespace __runtime {
namespace llvm {
/// A simple null object to allow implicit construction of Optional<T>
/// and similar types without having to spell out the specialization's name.
// (constant value 1 in an attempt to workaround MSVC build issue... )
enum class NoneType { None = 1 };
const NoneType None = NoneType::None;
}
}} // swift::runtime
Optional

Optional表示可能为空的类型，包含一个OptionStorage类型的私有成员。OptionStorage用联合体保存数据，数据要么是表示空的empty，要么是表示非空的value，用布尔类型的hasValue成员表示当前可空类型对象是否有值，核心方法是emplace，用于实现可空类型赋值。Optional通过map方法返回可空类型所保存的值，有值则返回该值，无值则返回None。

//Optional.h

/// Storage for any type.
template <typename T, bool = is_trivially_copyable<T>::value>
class OptionalStorage {
  union {
    char empty;
    T value;
  };
  bool hasVal;

  public:
  ...

  template <class... Args> void emplace(Args &&... args) {
    reset();
    ::new ((void *)std::addressof(value)) T(std::forward<Args>(args)...);
    hasVal = true;
  }

}

template <typename T> class OptionalStorage<T, true> {
  union {
    char empty;
    T value;
  };
  bool hasVal = false;

  public:
  ...

}

template <typename T> class Optional {
  optional_detail::OptionalStorage<T> Storage;

public:
  ...

  const T *getPointer() const { return &Storage.getValue(); }
  T *getPointer() { return &Storage.getValue(); }
  const T &getValue() const LLVM_LVALUE_FUNCTION { return Storage.getValue(); }
  T &getValue() LLVM_LVALUE_FUNCTION { return Storage.getValue(); }

  explicit operator bool() const { return hasValue(); }
  bool hasValue() const { return Storage.hasValue(); }
  const T *operator->() const { return getPointer(); }
  T *operator->() { return getPointer(); }

  ...

  /// Apply a function to the value if present; otherwise return None.
  template <class Function>
  auto map(const Function &F) const LLVM_LVALUE_FUNCTION
      -> Optional<decltype(F(getValue()))> {
    if (*this) return F(getValue());
    return None;
  }

  ...

}

3.2.2 指针

这里主要定义了两种由指针组成的复合数据类型。PointerIntPair是用于高效保存“指针+整数”的数据结构，不过有个很重要的限制条件：整数的取值范围很小，通常不能超过二进制3 bit。在 64 位机环境下，指针占 8 字节，且指针的内存地址必须按 8 字节对齐，也就是说指针的内存地址的最低 3 bit 必为全零。如此一来，则最低三位可以用于存储其他信息，取内存地址只需将 value 与0xFFFFFFFFFFFFFFF8作按位与运算，取整数则将 value 与0x07作按位与运算（代码未必是这样实现，但就是这个原理）。PointerIntPair实现原理实际是在PointerIntPairInfo的实现逻辑中。

// PointerIntPair

...
template <typename PointerTy, unsigned IntBits, typename IntType = unsigned,
          typename PtrTraits = PointerLikeTypeTraits<PointerTy>,
          typename Info = PointerIntPairInfo<PointerTy, IntBits, PtrTraits>>
class PointerIntPair {
  // Used by MSVC visualizer and generally helpful for debugging/visualizing.
  using InfoTy = Info;
  intptr_t Value = 0;

public:
  constexpr PointerIntPair() = default;

  PointerIntPair(PointerTy PtrVal, IntType IntVal) {
    setPointerAndInt(PtrVal, IntVal);
  }

  ...

  void setPointerAndInt(PointerTy PtrVal, IntType IntVal) LLVM_LVALUE_FUNCTION {
    Value = Info::updateInt(Info::updatePointer(0, PtrVal),
                            static_cast<intptr_t>(IntVal));
  }

}

...

template <typename PointerT, unsigned IntBits, typename PtrTraits>
struct PointerIntPairInfo {
  static_assert(PtrTraits::NumLowBitsAvailable <
                    std::numeric_limits<uintptr_t>::digits,
                "cannot use a pointer type that has all bits free");
  static_assert(IntBits <= PtrTraits::NumLowBitsAvailable,
                "PointerIntPair with integer size too large for pointer");
  enum MaskAndShiftConstants : uintptr_t {
    /// PointerBitMask - The bits that come from the pointer.
    PointerBitMask =
        ~(uintptr_t)(((intptr_t)1 << PtrTraits::NumLowBitsAvailable) - 1),

    /// IntShift - The number of low bits that we reserve for other uses, and
    /// keep zero.
    IntShift = (uintptr_t)PtrTraits::NumLowBitsAvailable - IntBits,

    /// IntMask - This is the unshifted mask for valid bits of the int type.
    IntMask = (uintptr_t)(((intptr_t)1 << IntBits) - 1),

    // ShiftedIntMask - This is the bits for the integer shifted in place.
    ShiftedIntMask = (uintptr_t)(IntMask << IntShift)
  };

  static PointerT getPointer(intptr_t Value) {
    return PtrTraits::getFromVoidPointer(
        reinterpret_cast<void *>(Value & PointerBitMask));
  }

  static intptr_t getInt(intptr_t Value) {
    return (Value >> IntShift) & IntMask;
  }

  static intptr_t updatePointer(intptr_t OrigValue, PointerT Ptr) {
    intptr_t PtrWord =
        reinterpret_cast<intptr_t>(PtrTraits::getAsVoidPointer(Ptr));
    assert((PtrWord & ~PointerBitMask) == 0 &&
           "Pointer is not sufficiently aligned");
    // Preserve all low bits, just update the pointer.
    return PtrWord | (OrigValue & ~PointerBitMask);
  }

  static intptr_t updateInt(intptr_t OrigValue, intptr_t Int) {
    intptr_t IntWord = static_cast<intptr_t>(Int);
    assert((IntWord & ~IntMask) == 0 && "Integer too large for field");

    // Preserve all bits other than the ones we are updating.
    return (OrigValue & ~ShiftedIntMask) | IntWord << IntShift;
  }
};

...

PointerUnion是用于保存指针的类似于联合体的数据类型，依赖了PointerIntPair的特征，其功能在PointerUnion的注释中已经描述得比较清楚了。

// PointerUnion.h

...
/// A discriminated union of two or more pointer types, with the discriminator
/// in the low bit of the pointer.
///
/// This implementation is extremely efficient in space due to leveraging the
/// low bits of the pointer, while exposing a natural and type-safe API.
///
/// Common use patterns would be something like this:
///    PointerUnion<int*, float*> P;
///    P = (int*)0;
///    printf("%d %d", P.is<int*>(), P.is<float*>());  // prints "1 0"
///    X = P.get<int*>();     // ok.
///    Y = P.get<float*>();   // runtime assertion failure.
///    Z = P.get<double*>();  // compile time failure.
///    P = (float*)0;
///    Y = P.get<float*>();   // ok.
///    X = P.get<int*>();     // runtime assertion failure.
template <typename... PTs>
class PointerUnion
    : public pointer_union_detail::PointerUnionMembers<
          PointerUnion<PTs...>,
          PointerIntPair<
              void *, pointer_union_detail::bitsRequired(sizeof...(PTs)), int,
              pointer_union_detail::PointerUnionUIntTraits<PTs...>>,
          0, PTs...> {
  ...
}

...

3.2.3 字符串

StringRef定义了字符串类型，其核心数据是指向连续分配的字符数组的指针。其中const char *Data = nullptr指向字符串起始内存地址，size_t Length = 0记录字符串长度。因此，StringRef本身是基本不干内存分配的活的，它只是只是字符串的引用。唯一包含内存分配相关代码是拷贝操作，

// String.h

...

  /// StringRef - Represent a constant reference to a string, i.e. a character
  /// array and a length, which need not be null terminated.
  ///
  /// This class does not own the string data, it is expected to be used in
  /// situations where the character data resides in some other buffer, whose
  /// lifetime extends past that of the StringRef. For this reason, it is not in
  /// general safe to store a StringRef.
  class LLVM_GSL_POINTER StringRef {
  public:
    static constexpr size_t npos = ~size_t(0);

    using iterator = const char *;
    using const_iterator = const char *;
    using size_type = size_t;

  private:
    /// The start of the string, in an external buffer.
    const char *Data = nullptr;

    /// The length of the string.
    size_t Length = 0;
    ...

  public:
    ...
    
    /// str - Get the contents as an std::string.
    LLVM_NODISCARD
    std::string str() const {
      if (!Data) return std::string();
      return std::string(Data, Length);
    }

    ...

    // copy - Allocate copy in Allocator and return StringRef to it.
    template <typename Allocator>
    LLVM_NODISCARD StringRef copy(Allocator &A) const {
      // Don't request a length 0 copy from the allocator.
      if (empty())
        return StringRef();
      char *S = A.template Allocate<char>(Length);
      std::copy(begin(), end(), S);
      return StringRef(S, Length);
    }

  ...
  }

...

3.2.4 数组

ArrayRef和StringRef的定义十分类似，同样只是指向一块连续分配的内存缓冲，内存缓冲保存的所有元素的类型相同，因此可以简单地通过指针偏移操作ArrayRef及其元素。在ArrayRef的基础上还扩展出MutableArrayRef，不过需要注意的是，即使是 mutable 的MutableArrayRef，也没有直接操作内存缓冲，做数据 append 或 remove 或 reallocate 之类的操作。

//ArrayRef.h

...

  /// ArrayRef - Represent a constant reference to an array (0 or more elements
  /// consecutively in memory), i.e. a start pointer and a length.  It allows
  /// various APIs to take consecutive elements easily and conveniently.
  ///
  /// This class does not own the underlying data, it is expected to be used in
  /// situations where the data resides in some other buffer, whose lifetime
  /// extends past that of the ArrayRef. For this reason, it is not in general
  /// safe to store an ArrayRef.
  ///
  /// This is intended to be trivially copyable, so it should be passed by
  /// value.
  template<typename T>
  class LLVM_GSL_POINTER LLVM_NODISCARD ArrayRef {
  public:
    using iterator = const T *;
    using const_iterator = const T *;
    using size_type = size_t;
    using reverse_iterator = std::reverse_iterator<iterator>;

  private:
    /// The start of the array, in an external buffer.
    const T *Data = nullptr;

    /// The number of elements.
    size_type Length = 0;

  public:
    ...
    /// front - Get the first element.
    const T &front() const {
      assert(!empty());
      return Data[0];
    }

    /// back - Get the last element.
    const T &back() const {
      assert(!empty());
      return Data[Length-1];
    }
    ...

    const T &operator[](size_t Index) const {
      assert(Index < Length && "Invalid index!");
      return Data[Index];
    }
    ...
  }
  ...

  template<typename T>
  class LLVM_NODISCARD MutableArrayRef : public ArrayRef<T> {
  public:
    using iterator = T *;
    using reverse_iterator = std::reverse_iterator<iterator>;
     
    ...
  }
...

3.2.5 哈希表

首先确定依赖关系，阅读顺序是 DenseMapInfo -> DenseMap -> DenseSet。

基础

DenseMapInfo只需要看下面几行代码，这其实就是哈希表的几个核心因素。类型名称带 Dense，所以这里定义的哈希表不太可能采用拉链法之类的方式解决冲突。因此需要区分空 Bucket 以及先前被移除的 Bucket，先前被移除的 Bucket 会被哈希表插上墓碑（TombStone）。注意：保存不同数据类型的哈希表需要采用不同的DenseMapInfo模板设置空值、TombStone 值、哈希函数和判等函数；

  • getEmptyKey：定义用于标记哈希表 Bucket 为空的值；
  • getTombstoneKey：定义用于标记哈希表 Bucket 已变成 TombStone 的值；
  • getHashValue：定义哈希函数；
  • isEqual：定义判等函数；
//DenseMapInfo.h

...
template <> struct DenseMapInfo<hash_code> {
  static inline hash_code getEmptyKey() { return hash_code(-1); }
  static inline hash_code getTombstoneKey() { return hash_code(-2); }
  static unsigned getHashValue(hash_code val) { return val; }
  static bool isEqual(hash_code LHS, hash_code RHS) { return LHS == RHS; }
};
...
定义

DenseMap是用于保存键值对的哈希表。DenseMap键值对的定义如下所示：

// We extend a pair to allow users to override the bucket type with their own
// implementation without requiring two members.
template <typename KeyT, typename ValueT>
struct DenseMapPair : public std::pair<KeyT, ValueT> {
  using std::pair<KeyT, ValueT>::pair;

  KeyT &getFirst() { return std::pair<KeyT, ValueT>::first; }
  const KeyT &getFirst() const { return std::pair<KeyT, ValueT>::first; }
  ValueT &getSecond() { return std::pair<KeyT, ValueT>::second; }
  const ValueT &getSecond() const { return std::pair<KeyT, ValueT>::second; }
};

DenseMap类型定义以及核心成员如下面代码所示，键值对在DenseMap中记为Bucket。

template <typename KeyT, typename ValueT,
          typename KeyInfoT = DenseMapInfo<KeyT>,
          typename Bucket = llvm::detail::DenseMapPair<KeyT, ValueT>,
          bool IsConst = false>
class DenseMapIterator;

template <typename DerivedT, typename KeyT, typename ValueT, typename KeyInfoT,
          typename BucketT>
class DenseMapBase {
  template <typename T>
  using const_arg_type_t = typename const_pointer_or_const_ref<T>::type;

public:
  using size_type = unsigned;
  using key_type = KeyT;
  using mapped_type = ValueT;
  using value_type = BucketT;

  using iterator = DenseMapIterator<KeyT, ValueT, KeyInfoT, BucketT>;
  using const_iterator =
      DenseMapIterator<KeyT, ValueT, KeyInfoT, BucketT, true>;
  
  ...
}
查找

该部分代码可以提取一些关键信息。

  • 从unsigned BucketNo = getHashValue(Val) & (NumBuckets-1);可以看出哈希函数的用处；
  • 从if (LLVM_LIKELY(KeyInfoT::isEqual(Val, ThisBucket->getFirst())))可以看出判等函数的用处；
  • 从BucketNo += ProbeAmt++;可以看出DenseMap采用的冲突解决策略是线性探测；

注意：后面会提到由于NumBuckets总是满足 NumBuckets = 2N 的约束条件，因此& (NumBuckets-1)相当于取模运算% NumBuckets 。

  /// LookupBucketFor - Lookup the appropriate bucket for Val, returning it in
  /// FoundBucket.  If the bucket contains the key and a value, this returns
  /// true, otherwise it returns a bucket with an empty marker or tombstone and
  /// returns false.
  template<typename LookupKeyT>
  bool LookupBucketFor(const LookupKeyT &Val,
                       const BucketT *&FoundBucket) const {
    const BucketT *BucketsPtr = getBuckets();
    const unsigned NumBuckets = getNumBuckets();

    if (NumBuckets == 0) {
      FoundBucket = nullptr;
      return false;
    }

    // FoundTombstone - Keep track of whether we find a tombstone while probing.
    const BucketT *FoundTombstone = nullptr;
    const KeyT EmptyKey = getEmptyKey();
    const KeyT TombstoneKey = getTombstoneKey();
    assert(!KeyInfoT::isEqual(Val, EmptyKey) &&
           !KeyInfoT::isEqual(Val, TombstoneKey) &&
           "Empty/Tombstone value shouldn't be inserted into map!");

    unsigned BucketNo = getHashValue(Val) & (NumBuckets-1);
    unsigned ProbeAmt = 1;
    while (true) {
      const BucketT *ThisBucket = BucketsPtr + BucketNo;
      // Found Val's bucket?  If so, return it.
      if (LLVM_LIKELY(KeyInfoT::isEqual(Val, ThisBucket->getFirst()))) {
        FoundBucket = ThisBucket;
        return true;
      }

      // If we found an empty bucket, the key doesn't exist in the set.
      // Insert it and return the default value.
      if (LLVM_LIKELY(KeyInfoT::isEqual(ThisBucket->getFirst(), EmptyKey))) {
        // If we've already seen a tombstone while probing, fill it in instead
        // of the empty bucket we eventually probed to.
        FoundBucket = FoundTombstone ? FoundTombstone : ThisBucket;
        return false;
      }

      // If this is a tombstone, remember it.  If Val ends up not in the map, we
      // prefer to return it than something that would require more probing.
      if (KeyInfoT::isEqual(ThisBucket->getFirst(), TombstoneKey) &&
          !FoundTombstone)
        FoundTombstone = ThisBucket;  // Remember the first tombstone found.

      // Otherwise, it's a hash collision or a tombstone, continue quadratic
      // probing.
      BucketNo += ProbeAmt++;
      BucketNo &= (NumBuckets-1);
    }
  }
插入

哈希表 Bucket 插入还需要考虑哈希表扩容及重哈希的问题。DenseMap满足NewNumEntries * 4 >= NumBuckets * 3)或者NumBuckets-(NewNumEntries+getNumTombstones() <= NumBuckets/8时，需要调用grow方法对哈希表进行扩容，其中的逻辑是：

  • 随着空 Bucket 数量的减少，哈希表发生碰撞的效率会逐渐增大；
  • TombStone 可以理解为可利用性打折扣的空 Bucket，因此 TombStone 数量增多也会增大碰撞概率；

填入 Bucket 时，需要判断目标 Bucket 是空 Bucket 还是 TombStone，如果是 TombStone 则需要记录哈希表 TombStone 数量减少。

为什么需要 TombStone？假设没有 TombStone，把没有存储值的 Bucket 全视为 100% 可用，当哈希表擦除某个 Bucket 的内容，则当查找另外一个碰撞检测链途经该 Bucket 的 Bucket 时，会终止于该 Bucket，得到查找结果为空的错误结果。TombStone 在某些情况下是可以填入值的：查找某个关键字的 Bucket 确定为空，则可以在查找的碰撞路径上的首个TombStone 中，填入该关键字对应的 Bucket，仅在这种情况下对 TombStone 填入内容，才不会影响其他 Bucket 的查找结果。

  template <typename LookupKeyT>
  BucketT *InsertIntoBucketImpl(const KeyT &Key, const LookupKeyT &Lookup,
                                BucketT *TheBucket) {
    // If the load of the hash table is more than 3/4, or if fewer than 1/8 of
    // the buckets are empty (meaning that many are filled with tombstones),
    // grow the table.
    //
    // The later case is tricky.  For example, if we had one empty bucket with
    // tons of tombstones, failing lookups (e.g. for insertion) would have to
    // probe almost the entire table until it found the empty bucket.  If the
    // table completely filled with tombstones, no lookup would ever succeed,
    // causing infinite loops in lookup.
    unsigned NewNumEntries = getNumEntries() + 1;
    unsigned NumBuckets = getNumBuckets();
    if (LLVM_UNLIKELY(NewNumEntries * 4 >= NumBuckets * 3)) {
      this->grow(NumBuckets * 2);
      LookupBucketFor(Lookup, TheBucket);
      NumBuckets = getNumBuckets();
    } else if (LLVM_UNLIKELY(NumBuckets-(NewNumEntries+getNumTombstones()) <=
                             NumBuckets/8)) {
      this->grow(NumBuckets);
      LookupBucketFor(Lookup, TheBucket);
    }
    assert(TheBucket);

    // Only update the state after we've grown our bucket space appropriately
    // so that when growing buckets we have self-consistent entry count.
    incrementNumEntries();

    // If we are writing over a tombstone, remember this.
    const KeyT EmptyKey = getEmptyKey();
    if (!KeyInfoT::isEqual(TheBucket->getFirst(), EmptyKey))
      decrementNumTombstones();

    return TheBucket;
  }
扩容

从前面插入的代码可以发现，当插入 Bucket 时，由于 TombStone 数量饱和触发的扩容使用的是扩容前的NumBuckets，也就是说哈希表的内存缓冲区空间是不变的，实际上是为了清空旧缓冲中累积的 TombStone。扩容实际上是开辟新的内存缓冲，并触发重哈希，最后释放旧的内存缓冲，以达到清理 TombStone 或者扩展哈希表空 Bucket 数量的目的。

  void grow(unsigned AtLeast) {
    unsigned OldNumBuckets = NumBuckets;
    BucketT *OldBuckets = Buckets;

    allocateBuckets(std::max<unsigned>(64, static_cast<unsigned>(NextPowerOf2(AtLeast-1))));
    assert(Buckets);
    if (!OldBuckets) {
      this->BaseT::initEmpty();
      return;
    }

    this->moveFromOldBuckets(OldBuckets, OldBuckets+OldNumBuckets);

    // Free the old table.
    deallocate_buffer(OldBuckets, sizeof(BucketT) * OldNumBuckets,
                      alignof(BucketT));
  }

  bool allocateBuckets(unsigned Num) {
    NumBuckets = Num;
    if (NumBuckets == 0) {
      Buckets = nullptr;
      return false;
    }

    Buckets = static_cast<BucketT *>(
        allocate_buffer(sizeof(BucketT) * NumBuckets, alignof(BucketT)));
    return true;
  }
重哈希

重哈希就是重新对哈希表中的所有 Bucket 进行逐个插入，是从旧的哈希表内存缓冲中插入到新分配的内存缓冲中。在哈希表扩容后需要对哈希表进行重哈希。

  void moveFromOldBuckets(BucketT *OldBucketsBegin, BucketT *OldBucketsEnd) {
    initEmpty();

    // Insert all the old elements.
    const KeyT EmptyKey = getEmptyKey();
    const KeyT TombstoneKey = getTombstoneKey();
    for (BucketT *B = OldBucketsBegin, *E = OldBucketsEnd; B != E; ++B) {
      if (!KeyInfoT::isEqual(B->getFirst(), EmptyKey) &&
          !KeyInfoT::isEqual(B->getFirst(), TombstoneKey)) {
        // Insert the key/value into the new table.
        BucketT *DestBucket;
        bool FoundVal = LookupBucketFor(B->getFirst(), DestBucket);
        (void)FoundVal; // silence warning.
        assert(!FoundVal && "Key already in new map?");
        DestBucket->getFirst() = std::move(B->getFirst());
        ::new (&DestBucket->getSecond()) ValueT(std::move(B->getSecond()));
        incrementNumEntries();

        // Free the value.
        B->getSecond().~ValueT();
      }
      B->getFirst().~KeyT();
    }
删除

删除键值对的代码实现比较简单。

  bool erase(const KeyT &Val) {
    BucketT *TheBucket;
    if (!LookupBucketFor(Val, TheBucket))
      return false; // not in map.

    TheBucket->getSecond().~ValueT();
    TheBucket->getFirst() = getTombstoneKey();
    decrementNumEntries();
    incrementNumTombstones();
    return true;
  }

四、总结

总体简单浏览了 lllvm/Support 和 llvm/ADT 中代码，目前来看，其在数据结构层面上采用的套路和 Objective-C Runtime 中的套路还是蛮相似的，不过 Swift 工程中采用的 C++ 语法貌似整体要高级得多，阅读起来比较吃力。不像 Objective-C Runtime 可以依稀看出核心代码是从 C 语言工程逐步迁移过来的。下一步是从 stdlib/public/runtime 以及 include/swift/Runtime 文件夹寻找更贴近前面所划定范围的突破口。

avatar
文章
阅读
粉丝