Golang语言情怀--第100期 区块链技术-以太坊公链NFT图片去中心化存储-第1节：IPFS存储方案
IPFS节点部署：Golang语言情怀--第94期 区块链技术-centos安装部署IPFS
IPFS node deployment: IPFS原理详解 IPFS EXPLORATION 以下内容调研截止到2021/11/5日
The following studies are available as of 2021/11/5. IPFS是一种内容可寻址、点对点、分布式文件系统。IPFS采用内容-地址寻址技术，即通过文件内容进行检索而不是通过文件的网络地址。简单来说，就是对文件内容进行hash运算，将hash值作为文件名保存在本地数据库中，所以，只要文件内容不变，则文件名也保持不变。 IPFS is a content search, point-to-point, distributed file system. The IPFS uses content-to-site search techniques, i.e. a web address that retrieves through the contents of a document rather than through it. Simply put, the content of a document hash is calculated and the hash value is saved as a file name in a local database, so the file name remains the same as long as the contents of the document remain unchanged. 多个运行IPFS程序的节点构成IPFS存储网络，存储在IPFS网络中的数据被划分成多个块，存放在不同节点中。当节点请求网络中的数据时，会在节点本地缓存该文件。即每个节点都保存其下载过的文件的缓存，用来保证即使某一个存有该资源的节点推出IPFS网络，该资源仍然可以被其他节点访问。当用户把文件上传到IPFS节点存储时，节点会将文件分块后进行存储，每个文件块以Merkle有向无环图（如图1所示）的形式组织，而Merkle有向无环图的根哈希则用来表示该文件。同时采用分布式Hash表（如图2所示）实现通过hash值到文件内容的定位。 Multiple IPFS program nodes form the IPFS storage network, where the data stored in the IPFS network are divided into blocks and stored in different nodes. When node requests data in the network, the file is cached locally at the node. That is, each node keeps a cache of the files it downloads to ensure that the IPFS network can be accessed by another node even if one of the node where the resource is stored is rolled out. When the user uploads the document to the IPFS node, the node will store the file in blocks, organized in the form of a Merkel with no ring map (as shown in figure 1) and Merkle with a Guernhashi with no ring map. At the same time, the location of the file via the hash value (as shown in figure 2) is achieved by using the distribution Hash table. 本地表中存储文件的K/V值。正如我们在前面所提到的，每个文件在保存到 IPFS 网络中都可能进行分片，即把大的文件分成小的碎片，每个碎片有自己的哈希，根据碎片的哈希生成对应的 Link，以碎片在文件中出现的顺序，使用这些 Link 生成连接数组，使用连接数组生成最终的顶层 Object 对象，以此来表示文件。每一个分块都可以作为Object对象进行访问。 The K/V value of the files stored in the local table. As we mentioned earlier, each file may be divided into small fragments in the IPFS network, which divides the large file into small fragments, each with its own Hashi, which produces the corresponding Link in the order in which the fragments appear in the document, using the Link to generate connection arrays, using the connecting arrays to generate the final top Object object as an indication of the file. Each fragment can be accessed as an Object object. 图1 Merkle有向无环图 Figure 1 Merkle with no rings 图2 IPFS存储对象数据结构 Figure 2 Structure of IPFS stored object data 图3 分布式hash表 Figure 3 Distribution of hash tables 当存储小文件时，会直接将该文件存到DHT（分布式hash表）上。存储大文件时，会存储文件的根hash和对应节点的ID。DHT中存储三种类型的记录。 When you store small files, you store them directly on the DHT (distributed hash table). When you store large files, you store the root hash of the file and the ID of the corresponding node. Three types of records are stored in the DHT. 图4 K桶 Figure 4 K cylinders 如果一个节点的ID，前面所有位数相同，从倒数第n位开始不同，这样的节点只有2(i-1)个，与基础节点的距离范围为[2(i-1), 2i）；对于0000110而言，这样的节点归为“k-bucket i”； If the ID of a node is identical for all places in the front, from the nth of the penultimate place, such node is only 2 (i-1) and the range of distance from the base node is [2 (i-1), 2i); for 000110, such node is classified as “k-bucketi”; 图5 M节点维护的路由表 Figure 5 Roadways maintained at node M Kademlia协议对每个桶内维护的节点数设置了一个上限，一旦桶内节点数超过，便根据一定的淘汰算法进行更新，一般上限设置为20。 The Kademlia agreement sets a ceiling on the number of nodes to be maintained in each barrel, and once the number of nodes in the barrel is exceeded, it is updated on the basis of a certain phase-out algorithm, with a general ceiling of 20. 搜索时间复杂度为O(log2n)，n是指网络的规模。 Search time complexity is O (log2n), n refers to the size of the network. 当节点查到文件根hash时，同时也拿到了link数组，发起广播，询问谁有数组中的hash块，请发给我。 When the root of the document hash is found at the node, the link array is also taken, and the broadcast is launched, asking who has hash in the array, please send it to me. （1）新节点加入情况： (1) Accession to new nodes: 当有新节点N加入时，需要通过访问网络中的任一节点S ，以S作为中介加入网络，具体来说： When a new node N joins, it is necessary to join the network with S as an intermediary through access to any one of the networks, specifically: （2）节点离线： (2) Node offline: 节点离线在Kademlia协议中无需做特殊处理，如果某个节点离线，那么其离线事件最终会反馈到网络节点的路由表中，将其从路由表中剔除即可。 Node delineation does not require special treatment in the Kademlia protocol, and if a node is delineated, the offline event will eventually feed back to the route list of the network node and remove it from the route list. （3）数据冗余存储： (3) Data redundancy storage: 并不能保证在任一时刻目标节点N均一定存在或者在线，因此Kad网络规定：任一条目，依据其key的具体取值，该条目将被复制并存放在节点ID距离key值最近(即当前距离目标节点N最近)的k个节点当中；之所以要将重复保存k份，这完全是考虑到整个Kad系统稳定性而引入的冗余； There is no guarantee that the target node N will always exist or be online at any given time, so the Kad network provides that any entry, based on its specific key extraction, will be copied and stored in the k-node of the nearest ID distance (i.e., currently the nearest N from the target node N) to the nearest node of the node of the node of the node of the node, and that the k-part will be duplicated solely because of the redundancy introduced in view of the stability of the whole Kad system; 运行IPFS的节点，既是客户端又是服务器。客户端通过发送文件名到服务器，请求下载文件，服务器会根据文件名到分布式Hash表中查找对应的文件，查找成功后将文件发送给客户端，当文件下载完成后，客户端通过对文件内容进行hash运算，将hash值和文件名作比较就可以确定文件的完整性。 Runs the IPFS node, both as a client and as a server. By sending a filename to the server, requesting to download the file, the server will search for the corresponding file according to the filename to the distributed Hash table, and, when a file is downloaded, send it to the client. The client will determine the integrity of the file by having the ash calculation of its contents and comparing the ash value with the filename. IPFS采用Kademlia协议实现节点路由及内容查询。当查询一个内容时，先计算hash值，再将该hash值与节点ID进行异或，得到距离，根据距离去对应的K桶中查找，若查找不到，询问谁是距离最近的节点，得到最近的节点，在该节点上再次执行该算法，直到找到对应节点。 IPFS uses the Kademlia protocol to achieve node routing and content querying. When you search for a content, you calculate the hash value and then you separate it from the node ID or get a distance from the node ID. If you can't find it, ask who is the nearest node, get the nearest node, and execute the algorithm again on that node until the corresponding node is found. 找到对应节点后，再次查询该DHT表，找到该节点对应的ip地址等信息定位到该节点，再查询该节点的本地数据库，得到要查找的文件内容。 Once the corresponding node is found, the DHT table is again checked, information such as the pip address of the node is located at the node, and the local database of the node is searched for and the contents of the file to be found are found. 通过在GitHub网站中EOS仓库中提问得出EOS并未实现基于IPFS的存储。以下内容摘自EOS存储白皮书及网上博客资料。 The question in the EOS warehouse on GitHub does not make IPFS-based storage possible. The following is taken from the EOS white paper and online blog material. EOS存储的核心是IPFS，它提供了一个任何人都可以托管文件的去中心化网络，这些文件可以通过地址远程访问。区块生产者实际代表了21个超级节点，每个超级节点需要拥有支持高吞吐EOS交易量的数据中心，可以在全球范围提供文件托管服务，而且只要有至少一个超级节点在线，用户的文件就是可以访问的。 The core of EOS storage is IPFS, which provides a decentralised network where anyone can host documents, which can be accessed remotely through addresses. Block producers actually represent 21 supernodes, each of which requires a data centre that supports high eOS transactions, can provide document hosting services on a global scale, and user documents are accessible as long as at least one supernode is online. EOS设计了一套文件系统智能合约，发行了一种 token 叫TOK。它允许每个EOS客户端可以定义一个本地home目录，用于存放IPFS文件链接。链接内容主要包括本地home目录路径、文件名（文件内容的hash值）、文件大小。 EOS designed a document system smart contract and issued a token took. It allows each EOS client to define a local home directory to store IPFS file links. The links include, inter alia, the local home directory path, file name (hash value of file content), file size. 图6 目录结构 Figure 6 Contents structure 用户上传文件时，将文件链接打包成交易信息，签名后广播给区块生产者，然后用户通过EOS存储软件定义的标准化REST应用程序接口将文件上传到其中一个区块生产者。区块生产者会验证文件和文件名匹配，然后将交易广播发送到整个区块链系统，其他的区块生产会通过IPFS网络复制那个文件。这样用户就成功的上传了一个文件，同时在的home目录下保存了该文件的链接。和使用EOS资源类似，当用户需要存东西的时候，系统就会锁定一部分的TOK，当然如果用户不需要存东西了，系统就会解锁TOK。 When a user uploads a file, the file link is packaged into the transaction information, the signature is broadcast to the block producer, and the user uploads the file to one of the block producers through the standardized REST application interface defined by the EOS storage software. Block producers verify the matching of the file and send the transaction radio to the entire block chain system, and the other block production reproduces the file via the IPFS network. The user then successfully uploads a file and saves the link under the home directory. 图7 上传流程 Figure 7 Uploading process 文件上传成功后，当客户端需要下载文件时，只需要将文件名发给生产者服务器，然后，服务器通过IPFS检索到对应的文件发送给客户端即可。一般来说，IPFS文件系统中的文件都是只读的，因为文件内容的细微修改就会导致客户端校验失败。 When the file uploads successfully, when the client needs to download the file, it simply needs to send the filename to the producer server, and then the server retrieves the corresponding file through the IPFS and sends it to the client. Generally, the files in the IPFS filesystem are read-only, because minor changes in the content of the document lead to a failure in the client's end-checking. 长安链暂未使用IPFS进行存储。长安链现已实现使用LevelDB、RocksDB、MySQL进行存储。同时长安链会将区块的历史数据归档在链下数据库中进行存储，采用MySQL数据库进行存储，存储状态数据和非状态数据。 The long-singing chain is not stored using IPFS for the time being. The long-single chain is now stored using LevelDB, RocksDB, MySQL. At the same time, the long-single chain will archive the historical data of blocks into a database under the chain, using MySQL database for storage and storage of state and non-state data. 长安链采用MySQL数据库实现链下存储，IPFS存储格式后续会进行支持。 The long-chain is stored below the chain using the MySQL database, which will be supported in follow-up to the IPFS storage format. 长安链暂时还没有实现基于IPFS的存储系统，但长安链采用了基于IPFS的组网方式。 While long-chain storage systems based on IPFS have not yet been achieved, long-chaining uses an IPFS-based network approach. 长安链基于IPFS提供的DHT，实现TLS的证书认证服务。 The long-chain is based on the DHT provided by IPFS to achieve TLS certification services. 长安链支持自动发现、自动连接的组网方式，默认在线的每个节点都可以作为种子节点为其他节点提供网络发现服务，每个种子节点都会记录网内节点地址信息。当有新节点连接到某个种子节点时，新节点会向该种子节点查询网内其他可连接节点的地址，拿到其他节点地址后，新节点会主动尝试与这些节点建立连接；另外，种子节点在接受了新节点链接后，会通过网络发现服务将该新节点的地址通知给其他在线的种子节点，其他节点在获得该新节点地址后，也会主动尝试与该新节点建立连接。 The long-term safety chain supports the automatic discovery and automatic connection of a network, and each node of the default online can provide a web discovery service for other nodes as a seed node, and each seed node records information about the node in the network. When a new node is connected to a seed node, the new node will search for the address of the other node in the network and get access to the other node node, the new node will voluntarily attempt to connect to the node; in addition, the new node will be notified to the other node online through the web node when the new node link is accepted, and the other node will attempt to connect to the new node once the new node is available. EOS、Filecoin（github上点赞数1.9K）等。 EOS, Filecoin (github with 1.9K) et cetera. IPFS是一个对标HTTP协议的内容寻址协议，底层采用leveldb作为数据库。 IPFS is a site-based protocol for the content of the HTTP Protocol, with a base-level use of leveldb as a database. 若采用MySQL实现存储量弹性扩展，首先是查询性能较K/V数据库较低，二是SQL语句过于灵活，难以控制。 If MySQL is used to achieve storage elastic expansion, first, the query performance is lower than that of the K/V database, and second, the SQL statement is too flexible to be controlled. HDFS:中心化、距离的度量是物理距离。一个HDFS集群是由一个NameNode和若干个DataNode组成的。其中NameNode作为主服务器，管理文件系统的命名空间和客户端对文件的访问操作；集群中的DataNode管理存储的数据。 HDFS: Centralization, the measure of distance is physical distance. A HDFS cluster is made up of a name Node and several DataNode. Name Node is the main server that manages file system naming space and client access to files; DataNode in the cluster manages stored data. 本期就到这里，下期再见。 That's it for the current period. I'll see you next time. 同学们，兴趣是最好的老师；只争朝夕，不负韶华！加油！ Ladies and gentlemen, interest is the best teacher; 参考资料： References: Go语言中文文档 Go Chinese Document http://www.golang.ltd/ Go语言在线编辑器： Go Language Online Editor: http://OL.Golang.LTD/ LollipopGo社区(筹建中) Lollipopgo community () https://www.LollipopGo.Com/
注册有任何问题请添加
微信：MVIP619
拉你进入群
打开微信扫一扫 添加客服 进入交流群
1.本站遵循行业规范，任何转载的稿件都会明确标注作者和来源；2.本站的原创文章，请转载时务必注明文章作者和来源，不尊重原创的行为我们将追究责任；3.作者投稿可能会经我们编辑修改或补充。
