Memory Map Technology

Answer 1

Memory map was a technique that maps the contents of a file to the virtual address space of a process. There are the following aspects to the memory map technology of txt-files:

* * 1. Concept Understanding **
1. * * Basic Principles **
- A one-to-one correspondence is established between the location of the txt-file on the disk and an area of the same size in the process logical address space. After the map, an object similar to an array type (such as mmap.mmap () object, etc.) is obtained. The contents of the file can be changed like an array.
- In the windows system, the memory-map file provided a memory management method that allowed applications to access the txt-file on the disk through the memory pointer, just like how they accessed the file loaded into memory. This process allowed all or part of the contents of the disk file to be associated with a certain area of the virtual address space of the process, so that the file could be directly accessed without performing file I/O operations and without the need to buffer the file contents. This feature was suitable for managing large-sized txts.
2. * * Compared to traditional reading methods **
- The traditional way of reading txt-files, for example, using the streamreader in C#to read it out at once, then using the memorystream to store it in memory, and then using the streamreader to read it out line by line, although the speed was faster when processing a certain size of the file (such as 16M text, about 8 seconds), it might be insufficient for processing large files. The Memory Map technology could reduce the buffer operation of data in memory when processing large files, improving efficiency.

* * 2. Operation Method **
1. * * Operation example in C#**
- You can use methods related to the MemoryMappedfile class, such as the Create From file method to open a txtfile, and then use the Create View Accessory method to extract the corresponding content according to the position of the text. You can extract it according to the number of words (for documents with English or numerical symbols, it can save resources). However, extracting by blocks might destroy the entire line of content in the document. The solution was to determine the position of the line break near the block according to the line break, and divide the large file into blocks reasonably. You can also combine the Task () thread group to assign a task to each block for related processing. During the operation, you can choose the action without return value or the Fuc with parameters with return value to operate. In addition, you can also use the streamreader to operate on the opened txt-file. The operation process involves the management of the file stream, the location of the target line and the current line, the determination of the file size, and the creation of the memory map view stream. At the same time, when reading the content, it also involves operations such as the conversion of strings and byte-arrays.

* * 3. Memory-Mapped I/O Characteristics **
1. * * Strengths **
- When dealing with txts, many I/O-intensive operations could be completed faster because there was no need to copy data between the core space and the user space. And in some cases, such as when the sqlite-library processes txts (if it is treated like a database file), using memory-mapped I/O may require less ram because it shares pages with the operating system's page buffer and doesn't have to always maintain a copy of its own working page.
2. * * Disadvantage **
- There may be a risk of a program crash, such as the inability of sqlit to capture and handle I/O errors on memory-map files. Moreover, the operating system must have a unified buffer to work properly. Not all operating systems met this condition. In some operating systems that claimed to have this function but had a flawed implementation, it could cause file corruption. In addition, memory-mapped I/O does not always improve performance. In some cases, it may reduce performance. For example, in the windows system, the memory map file's interception function is limited. When performing some operations such as reducing the file size, it may silently fail and leave unused space. It may also cause incorrect database corruption reports (under certain versions of sqrite).
Read more exciting novels for free