HDFS之Fsimage 和 Edits 解析NameNode被格式化之后，将在/opt/module/hadoop-3

NameNode被格式化之后，将在/opt/module/hadoop-3.1.3/data/tmp/dfs/name/current目录中产生如下文件

-rw-rw-r--. 1 muyi muyi      42 11月 15 09:58 edits_0000000000000000296-0000000000000000297
-rw-rw-r--. 1 muyi muyi    1205 11月 15 10:58 edits_0000000000000000298-0000000000000000313
-rw-rw-r--. 1 muyi muyi     175 11月 15 11:58 edits_0000000000000000314-0000000000000000317
-rw-rw-r--. 1 muyi muyi    1300 11月 15 12:58 edits_0000000000000000318-0000000000000000336
-rw-rw-r--. 1 muyi muyi 1048576 11月 15 12:58 edits_inprogress_0000000000000000337
-rw-rw-r--. 1 muyi muyi    3309 11月 15 11:58 fsimage_0000000000000000317
-rw-rw-r--. 1 muyi muyi      62 11月 15 11:58 fsimage_0000000000000000317.md5
-rw-rw-r--. 1 muyi muyi    2911 11月 15 12:58 fsimage_0000000000000000336
-rw-rw-r--. 1 muyi muyi      62 11月 15 12:58 fsimage_0000000000000000336.md5
-rw-rw-r--. 1 muyi muyi       4 11月 20 10:27 seen_txid
-rw-rw-r--. 1 muyi muyi     219 11月 20 10:27 VERSION

Fsimage文件:HDFS文件系统元数据的一个永久性的检查点,其中包含HDFS文件系统的所有目录和文件inode的序列化信息
Edits文件:存放HDFS文件系统的所有更新操作的路径,文件系统客户端执行的所有写操作首先会被记录到Edits文件中
seen_txid文件保存的是一个数字,就是最后一个edits_的数字
每次NameNode启动的时候都会将Fsimage文件读入内存,加载Edits里面的更新操作,保证内容中的元数据信息是最新的、同步的,可以看成NameNode启动的时候就将Fsimage和Edits文件进行了合并

oiv查看Fsimage文件

查看oiv和oev命令

[muyi@hadoop102 current]$ hdfs

oev                  apply the offline edits viewer to an edits file
oiv                  apply the offline fsimage viewer to an fsimage

基本语法

hdfs oiv -p 文件类型 -i 镜像文件 -o 转换后文件输出路径

实操

[muyi@hadoop102 current]$ hdfs oiv -p XML -i fsimage_0000000000000000336 -o /opt/module/hadoop-3.1.3/fsimage.xml
2024-11-21 09:08:05,568 INFO offlineImageViewer.FSImageHandler: Loading 3 strings
[muyi@hadoop102 current]$ pwd
/opt/module/hadoop-3.1.3/data/dfs/name/current


[muyi@hadoop102 current]$ cat /opt/module/hadoop-3.1.3/fsimage.xml 
<?xml version="1.0"?>
<fsimage><version><layoutVersion>-64</layoutVersion><onDiskVersion>1</onDiskVersion><oivRevision>ba631c436b806728f8ec2f54ab1e289526c90579</oivRevision></version>
<NameSection><namespaceId>1409295519</namespaceId><genstampV1>1000</genstampV1><genstampV2>1033</genstampV2><genstampV1Limit>0</genstampV1Limit><lastAllocatedBlockId>1073741856</lastAllocatedBlockId><txid>336</txid></NameSection>
<ErasureCodingSection>
<erasureCodingPolicy>
<policyId>1</policyId><policyName>RS-6-3-1024k</policyName><cellSize>1048576</cellSize><policyState>DISABLED</policyState><ecSchema>
<codecName>rs</codecName><dataUnits>6</dataUnits><parityUnits>3</parityUnits></ecSchema>
</erasureCodingPolicy>

<erasureCodingPolicy>
<policyId>2</policyId><policyName>RS-3-2-1024k</policyName><cellSize>1048576</cellSize><policyState>DISABLED</policyState><ecSchema>
<codecName>rs</codecName><dataUnits>3</dataUnits><parityUnits>2</parityUnits></ecSchema>
</erasureCodingPolicy>

<erasureCodingPolicy>
<policyId>3</policyId><policyName>RS-LEGACY-6-3-1024k</policyName><cellSize>1048576</cellSize><policyState>DISABLED</policyState><ecSchema>
<codecName>rs-legacy</codecName><dataUnits>6</dataUnits><parityUnits>3</parityUnits></ecSchema>
</erasureCodingPolicy>

<erasureCodingPolicy>
<policyId>4</policyId><policyName>XOR-2-1-1024k</policyName><cellSize>1048576</cellSize><policyState>DISABLED</policyState><ecSchema>
<codecName>xor</codecName><dataUnits>2</dataUnits><parityUnits>1</parityUnits></ecSchema>
</erasureCodingPolicy>

<erasureCodingPolicy>
<policyId>5</policyId><policyName>RS-10-4-1024k</policyName><cellSize>1048576</cellSize><policyState>DISABLED</policyState><ecSchema>
<codecName>rs</codecName><dataUnits>10</dataUnits><parityUnits>4</parityUnits></ecSchema>
</erasureCodingPolicy>

</ErasureCodingSection>

<INodeSection><lastInodeId>16459</lastInodeId><numInodes>34</numInodes><inode><id>16385</id><type>DIRECTORY</type><name></name><mtime>1731645794758</mtime><permission>muyi:supergroup:0755</permission><nsquota>9223372036854775807</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16386</id><type>DIRECTORY</type><name>tmp</name><mtime>1731462394582</mtime><permission>muyi:supergroup:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16387</id><type>DIRECTORY</type><name>hadoop-yarn</name><mtime>1731459248976</mtime><permission>muyi:supergroup:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16388</id><type>DIRECTORY</type><name>staging</name><mtime>1731459844346</mtime><permission>muyi:supergroup:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16389</id><type>DIRECTORY</type><name>history</name><mtime>1731459249003</mtime><permission>muyi:supergroup:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16390</id><type>DIRECTORY</type><name>done</name><mtime>1731459911391</mtime><permission>muyi:supergroup:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16391</id><type>DIRECTORY</type><name>done_intermediate</name><mtime>1731459848433</mtime><permission>muyi:supergroup:1777</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16392</id><type>DIRECTORY</type><name>input</name><mtime>1731645632131</mtime><permission>muyi:supergroup:0755</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16393</id><type>FILE</type><name>word.txt</name><replication>3</replication><mtime>1731459549016</mtime><atime>1731459548053</atime><preferredBlockSize>134217728</preferredBlockSize><permission>muyi:supergroup:0644</permission><blocks><block><id>1073741825</id><genstamp>1001</genstamp><numBytes>39</numBytes></block>
</blocks>
<storagePolicyId>0</storagePolicyId></inode>
<inode><id>16396</id><type>DIRECTORY</type><name>muyi</name><mtime>1731459844346</mtime><permission>muyi:supergroup:0700</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16397</id><type>DIRECTORY</type><name>.staging</name><mtime>1731462407513</mtime><permission>muyi:supergroup:0700</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16403</id><type>DIRECTORY</type><name>muyi</name><mtime>1731462425115</mtime><permission>muyi:supergroup:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16416</id><type>FILE</type><name>job_1731459138840_0001-1731459845836-muyi-word+count-1731459859323-1-1-SUCCEEDED-default-1731459850919.jhist</name><replication>3</replication><mtime>1731459858618</mtime><atime>1731459858593</atime><preferredBlockSize>134217728</preferredBlockSize><permission>muyi:supergroup:0770</permission><blocks><block><id>1073741834</id><genstamp>1010</genstamp><numBytes>22325</numBytes></block>
</blocks>
<storagePolicyId>0</storagePolicyId></inode>
<inode><id>16417</id><type>FILE</type><name>job_1731459138840_0001_conf.xml</name><replication>3</replication><mtime>1731459858652</mtime><atime>1731459858627</atime><preferredBlockSize>134217728</preferredBlockSize><permission>muyi:supergroup:0770</permission><blocks><block><id>1073741835</id><genstamp>1011</genstamp><numBytes>214461</numBytes></block>
</blocks>
<storagePolicyId>0</storagePolicyId></inode>
<inode><id>16418</id><type>DIRECTORY</type><name>2024</name><mtime>1731459911391</mtime><permission>muyi:supergroup:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16419</id><type>DIRECTORY</type><name>11</name><mtime>1731459911392</mtime><permission>muyi:supergroup:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16420</id><type>DIRECTORY</type><name>13</name><mtime>1731459911392</mtime><permission>muyi:supergroup:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16421</id><type>DIRECTORY</type><name>000000</name><mtime>1731462425115</mtime><permission>muyi:supergroup:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16427</id><type>DIRECTORY</type><name>logs</name><mtime>1731462394609</mtime><permission>muyi:muyi:1777</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16428</id><type>DIRECTORY</type><name>muyi</name><mtime>1731462394612</mtime><permission>muyi:muyi:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16429</id><type>DIRECTORY</type><name>logs-tfile</name><mtime>1731462394615</mtime><permission>muyi:muyi:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16430</id><type>DIRECTORY</type><name>application_1731462174093_0001</name><mtime>1731462414103</mtime><permission>muyi:muyi:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16443</id><type>FILE</type><name>job_1731462174093_0001-1731462394118-muyi-word+count-1731462407141-1-1-SUCCEEDED-default-1731462397902.jhist</name><replication>3</replication><mtime>1731462406437</mtime><atime>1731462406411</atime><preferredBlockSize>134217728</preferredBlockSize><permission>muyi:supergroup:0770</permission><blocks><block><id>1073741844</id><genstamp>1020</genstamp><numBytes>22310</numBytes></block>
</blocks>
<storagePolicyId>0</storagePolicyId></inode>
<inode><id>16444</id><type>FILE</type><name>job_1731462174093_0001_conf.xml</name><replication>3</replication><mtime>1731462406471</mtime><atime>1731462406446</atime><preferredBlockSize>134217728</preferredBlockSize><permission>muyi:supergroup:0770</permission><blocks><block><id>1073741845</id><genstamp>1021</genstamp><numBytes>214640</numBytes></block>
</blocks>
<storagePolicyId>0</storagePolicyId></inode>
<inode><id>16445</id><type>FILE</type><name>hadoop102_44526</name><replication>3</replication><mtime>1731462413658</mtime><atime>1731462413572</atime><preferredBlockSize>134217728</preferredBlockSize><permission>muyi:muyi:0640</permission><blocks><block><id>1073741846</id><genstamp>1022</genstamp><numBytes>34415</numBytes></block>
</blocks>
<storagePolicyId>0</storagePolicyId></inode>
<inode><id>16446</id><type>FILE</type><name>hadoop104_40584</name><replication>3</replication><mtime>1731462414099</mtime><atime>1731462413987</atime><preferredBlockSize>134217728</preferredBlockSize><permission>muyi:muyi:0640</permission><blocks><block><id>1073741847</id><genstamp>1023</genstamp><numBytes>31988</numBytes></block>
</blocks>
<storagePolicyId>0</storagePolicyId></inode>
<inode><id>16447</id><type>FILE</type><name>hadoop103_33465</name><replication>3</replication><mtime>1731462414095</mtime><atime>1731462413998</atime><preferredBlockSize>134217728</preferredBlockSize><permission>muyi:muyi:0640</permission><blocks><block><id>1073741848</id><genstamp>1024</genstamp><numBytes>67067</numBytes></block>
</blocks>
<storagePolicyId>0</storagePolicyId></inode>
<inode><id>16450</id><type>FILE</type><name>weiguo.txt</name><replication>3</replication><mtime>1731627055620</mtime><atime>1731627055425</atime><preferredBlockSize>134217728</preferredBlockSize><permission>muyi:supergroup:0644</permission><blocks><block><id>1073741850</id><genstamp>1026</genstamp><numBytes>7</numBytes></block>
</blocks>
<storagePolicyId>0</storagePolicyId></inode>
<inode><id>16452</id><type>DIRECTORY</type><name>jinguo</name><mtime>1731629269263</mtime><permission>muyi:supergroup:0755</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16453</id><type>FILE</type><name>shuguo.txt</name><replication>10</replication><mtime>1731628622413</mtime><atime>1731628622248</atime><preferredBlockSize>134217728</preferredBlockSize><permission>muyi:supergroup:0644</permission><blocks><block><id>1073741852</id><genstamp>1029</genstamp><numBytes>14</numBytes></block>
</blocks>
<storagePolicyId>0</storagePolicyId></inode>
<inode><id>16454</id><type>DIRECTORY</type><name>xiyou</name><mtime>1731638168297</mtime><permission>muyi:supergroup:0755</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16455</id><type>DIRECTORY</type><name>huaguoshan</name><mtime>1731639298152</mtime><permission>muyi:supergroup:0755</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16457</id><type>FILE</type><name>sunwukong.txt</name><replication>3</replication><mtime>1731638731641</mtime><atime>1731643038251</atime><preferredBlockSize>134217728</preferredBlockSize><permission>muyi:supergroup:0644</permission><blocks><block><id>1073741853</id><genstamp>1030</genstamp><numBytes>9</numBytes></block>
</blocks>
<storagePolicyId>0</storagePolicyId></inode>
<inode><id>16458</id><type>FILE</type><name>sunwukong2.txt</name><replication>1</replication><mtime>1731639298279</mtime><atime>1731643038649</atime><preferredBlockSize>134217728</preferredBlockSize><permission>muyi:supergroup:0644</permission><blocks><block><id>1073741854</id><genstamp>1031</genstamp><numBytes>9</numBytes></block>
</blocks>
<storagePolicyId>0</storagePolicyId></inode>
</INodeSection>
<INodeReferenceSection></INodeReferenceSection><SnapshotSection><snapshotCounter>0</snapshotCounter><numSnapshots>0</numSnapshots></SnapshotSection>
<INodeDirectorySection><directory><parent>16385</parent><child>16392</child><child>16452</child><child>16386</child><child>16393</child><child>16454</child></directory>
<directory><parent>16386</parent><child>16387</child><child>16427</child></directory>
<directory><parent>16387</parent><child>16388</child></directory>
<directory><parent>16388</parent><child>16389</child><child>16396</child></directory>
<directory><parent>16389</parent><child>16390</child><child>16391</child></directory>
<directory><parent>16390</parent><child>16418</child></directory>
<directory><parent>16391</parent><child>16403</child></directory>
<directory><parent>16396</parent><child>16397</child></directory>
<directory><parent>16418</parent><child>16419</child></directory>
<directory><parent>16419</parent><child>16420</child></directory>
<directory><parent>16420</parent><child>16421</child></directory>
<directory><parent>16421</parent><child>16416</child><child>16417</child><child>16443</child><child>16444</child></directory>
<directory><parent>16427</parent><child>16428</child></directory>
<directory><parent>16428</parent><child>16429</child></directory>
<directory><parent>16429</parent><child>16430</child></directory>
<directory><parent>16430</parent><child>16445</child><child>16447</child><child>16446</child></directory>
<directory><parent>16452</parent><child>16453</child><child>16450</child></directory>
<directory><parent>16454</parent><child>16455</child></directory>
<directory><parent>16455</parent><child>16457</child><child>16458</child></directory>
</INodeDirectorySection>
<FileUnderConstructionSection></FileUnderConstructionSection>
<SecretManagerSection><currentId>0</currentId><tokenSequenceNumber>0</tokenSequenceNumber><numDelegationKeys>0</numDelegationKeys><numTokens>0</numTokens></SecretManagerSection><CacheManagerSection><nextDirectiveId>1</nextDirectiveId><numDirectives>0</numDirectives><numPools>0</numPools></CacheManagerSection>
</fsimage>

将显示的xml文件内容拷贝到Windows系统中创建的xml文件中,并格式化,部分显示结果如下:

<inode>  
    <id>16403</id>  
    <type>DIRECTORY</type>  
    <name>muyi</name>  
    <mtime>1731462425115</mtime>  
    <permission>muyi:supergroup:0770</permission>  
    <nsquota>-1</nsquota>  
    <dsquota>-1</dsquota>  
</inode>
<inode>  
    <id>16416</id>  
    <type>FILE</type>  
    <name>  
    job_1731459138840_0001-1731459845836-muyi-word+count-1731459859323-1-1-SUCCEEDED-default-1731459850919.jhist  
    </name>  
    <replication>3</replication>  
    <mtime>1731459858618</mtime>  
    <atime>1731459858593</atime>  
    <preferredBlockSize>134217728</preferredBlockSize>  
    <permission>muyi:supergroup:0770</permission>  
    <blocks>  
    <block>  
    <id>1073741834</id>  
    <genstamp>1010</genstamp>  
    <numBytes>22325</numBytes>  
    </block>  
    </blocks>  
    <storagePolicyId>0</storagePolicyId>  
</inode>

思考：可以看出，Fsimage 中没有记录块所对应 DataNode，为什么？

在集群启动后，要求 DataNode 上报数据块信息，并间隔一段时间后再次上报。

oev查看Edits文件

基本语法

hdfs oev -p 文件类型 -i 编辑日志 -o 转换或文件输出路径

实例实操

[muyi@hadoop102 current]$ hdfs oev -p XML -i edits_0000000000000000337-0000000000000000337 -o /opt/module/hadoop-3.1.3/edits.xml
[muyi@hadoop102 current]$ cat /opt/module/hadoop-3.1.3/edits.xml 
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<EDITS>
  <EDITS_VERSION>-64</EDITS_VERSION>
  <RECORD>
    <OPCODE>OP_START_LOG_SEGMENT</OPCODE>
    <DATA>
      <TXID>337</TXID>
    </DATA>
  </RECORD>
</EDITS>

思考：NameNode 如何确定下次开机启动的时候合并哪些 Edits？

NameNode在确定下次开机启动时合并哪些Edits时，主要依赖于其内部的记录机制和Secondary NameNode（或Backup Node，在新版本的Hadoop中）的辅助。以下是详细的解释：

一、NameNode的内部记录机制

FsImage与Edits的关系：

FsImage：HDFS的最新状态文件，它记录了文件系统的目录结构和数据块关系等元数据。

Edits：自FsImage创建后的namespace操作日志，它记录了文件系统状态的变化。

合并原理：

当NameNode启动时，它会读取FsImage，并根据Edits中的记录，将FsImage更新到最新状态。

为了确保数据的一致性，NameNode需要合并上次停机前正在写入的Edits，即edits_inprogress_xxx。

seen_txid的作用：

seen_txid记录了FsImage中已包含的最新的transaction ID（事务ID）。

NameNode会根据seen_txid的值，确定需要合并的Edits范围。

二、Secondary NameNode（或Backup Node）的辅助

Secondary NameNode的作用（在新版本Hadoop之前）：

Secondary NameNode是一台独立的机器，它定期从NameNode上获取元数据。

当准备获取元数据时，它会通知NameNode暂停写入Edits文件，并将之后的log记录写入一个名为edits.new的文件。

Secondary NameNode获取到元数据后，会在本机将edits文件和FsImage文件进行合并，创建出一个新的FsImage文件，并发送回NameNode。

NameNode收到新的FsImage文件后，会用它覆盖掉原来的FsImage文件，并删除edits文件，将edits.new重命名为edits。

Backup Node的作用（在新版本Hadoop中）：

Backup Node提供了与Secondary NameNode类似的功能，但更加高效。

它在内存中维护了一份从NameNode同步过来的FsImage，并实时接收NameNode的Edits日志流，将它们持久化到硬盘。

Backup Node将收到的Edits文件和内存中的FsImage文件进行合并，创建一份元数据备份。

这样，在NameNode启动时，就不需要再花费大量时间进行FsImage和Edits的合并操作了。

三、合并策略的配置

合并周期：

可以通过配置dfs.namenode.checkpoint.period来设置Secondary NameNode（或Backup Node）的合并周期，默认为1小时。

操作次数检查：

可以通过配置dfs.namenode.checkpoint.check.period来设置每分钟检查一次操作次数。

当操作次数达到dfs.namenode.checkpoint.txns配置的值（默认为100万次）时，会触发一次合并操作。

综上所述，NameNode通过内部的记录机制（如seen_txid）和Secondary NameNode（或Backup Node）的辅助来确定下次开机启动时合并哪些Edits。这种机制确保了HDFS文件系统元数据的一致性和安全性。