Introduction to File System in NodeJS

File systems have been integral to computing since its inception. Initially, they were physical arrangements of data storage, evolving from punched cards to magnetic tapes and disks. With the advent of computers, hierarchical file systems emerged, enabling structured data storage and retrieval.
Today, modern file systems span various storage mediums, from traditional hard disk drives to solid-state drives and cloud storage solutions.

File Management in JS

Client-side JavaScript operates within web browsers and has limited access to the file system due to security restrictions. It primarily interacts with files through mechanisms like the File API for handling user-uploaded files and web storage for local data persistence.
In contrast, Node.js is built on Chrome's V8 JavaScript engine and operates outside the browser environment. It provides extensive file system capabilities through the `fs` module, allowing developers to manipulate files, directories, and file descriptors directly on the server-side.

Node.js Philosophy on File Systems: Node.js embraces a philosophy of asynchronous, non-blocking I/O, making it well-suited for handling file operations efficiently.

Its event-driven architecture enables concurrent file processing without blocking the execution thread, enhancing scalability and responsiveness.

Additionally, Node.js emphasizes modularity and simplicity, offering a rich ecosystem of npm modules for diverse file management tasks, from basic file I/O to advanced file system operations.

In the up coming sections, we'll explore various ways of using node's fs API.

Understanding File Stats

Files on any system are stored alongside a set of details that describe them, and in the nodeJS, these details are accessed through the `stat` method exposed by the `fs` module.

In this short guide, we'll be exploring some mainly the `stat` and related methods to access and read file-related details.

fs.stat()

We can defined the `stat` method as a Asynchronous method that retrieves file stats for a given file path.

Let's jump straight some an example.

- Create a directory with 2 files: index.js and index.txt (throw some random text in here.)

// index.js

const fs = require('fs');

fs.stat('./index.txt', (err, stats) => {
  if (err) {
    console.error('Error retrieving file stats:', err);
    return;
  }
  console.log('File Stats:', stats);
});

When you run the code above you should something similar to what we have below.

Let's see what each of these properties reperesents:

• ➢`dev`: Represents the device ID of the file system where the file resides. Each file system mounted on the system is assigned a unique device ID.
  
• ➢`mode`: Specifies the file's mode or permission bits, indicating the file's access permissions for the owner, group, and others. It consists of a 12-bit value that defines read, write, and execute permissions.
  
• ➢`nlink`: Indicates the number of hard links to the file. Hard links are additional directory entries that reference the same inode as the original file.
  
• ➢`uid`: Represents the numeric user ID (UID) of the file's owner. It identifies the user who owns the file.
  
• ➢`gid`: Denotes the numeric group ID (GID) of the file's owner group. It identifies the group to which the file belongs.
  
• ➢`rdev`: Specifies the device ID for special files, such as character or block devices. For regular files, this property is typically 0.
  
• ➢`blksize`: Specifies the optimal block size for I/O operations on the file. It indicates the preferred size (in bytes) for reading and writing data.
  
• ➢`ino`: Represents the inode number, which uniquely identifies the file within its file system. Inodes store metadata and pointers to the data blocks of a file.
  
• ➢`size`: Indicates the size of the file in bytes. It represents the total number of bytes allocated to store the file's contents on disk.
  
• ➢`blocks`: Specifies the number of 512-byte blocks allocated for the file's data storage. It indicates the actual disk space used by the file.
  
• ➢`atimeMs`: The timestamp when the file was last accessed (in milliseconds since the epoch).
  
• ➢`mtimeMs`: The timestamp when the file was last modified (in milliseconds since the epoch).
  
• ➢`ctimeMs`: The timestamp when the file's status was last changed (in milliseconds since the epoch).
  
• ➢`birthtimeMs`: The timestamp when the file was created (in milliseconds since the epoch).
  
• ➢`atime`: The timestamp when the file was last accessed (Date object).
  
• ➢`mtime`: The timestamp when the file was last modified (Date object).
  
• ➢`ctime`: The timestamp when the file's status was last changed (Date object).
  
• ➢`birthtime`: The timestamp when the file was created (Date object).
  

There are stat object also provides some very important methods that help determine what information is available on a file, permissions and more. We'll discuss some of them later on.

NodeJS also provides an equivalent method for synchronous operations, which blocks the event loop until the file stats are ready.

we might need to synchronously retrieve file stats in certain scenarios, such as during initialization or when processing files sequentially. Here, fs.statSync() allows us to block the event loop until file stats are retrieved, ensuring synchronous execution.


Let's refactor the example above to use the `statSync`.

// index.js

const fs = require('fs');

try {
    const stats = fs.statSync('./index.txt');
    console.log('File Stats:', stats);
} catch (error) {
    console.error('Error retrieving file stats:', err);
    return;
}

As expected we get the same stat object as above in the log.

A promise-based equivalent was also added in node version 10, that let's us use the promise pattern.

Let's have a quick example.

// index.js

const fs = require('fs').promises;
  
fs.stat('./play.js')
.then(stats => {
    console.log('File Stats:', stats);
})
.catch(error => {
    console.error('Error retrieving file stats:', error);
});

With `fs.promises.stat()`, we can retrieve file stats in a cleaner and more readable asynchronous syntax, enhancing the maintainability of our code.

fs.lstat()

The fs.lstat() method is similar to fs.stat() in that it retrieves file stats. However, it has one crucial difference: it distinguishes symbolic links from their targets.

When you use fs.stat() on a symbolic link, it returns the stats of the target file or directory that the symbolic link points to. In contrast, fs.lstat() specifically provides the stats of the symbolic link itself, rather than following the link to its target.

Interpreting File Stats:

Sometimes it's very handy to be able to determine if the path provided is a directory or file and also if we have the required permissions to access the direct file (or directory).

Let's see how to use stat to pull these information out.

// index.js

const fs = require('fs').promises;
  
fs.lstat('./play.js')
.then(stats => {
    console.log('File Permissions:', stats.mode);
})
.catch(error => {
    console.error('Error retrieving file stats:', error);
});

NB: Don't use this for determining file access privileges, use the `fs.access` as shown below:

// index.js
const fs = require('fs');

const {constants, access} = fs;
const Path_to_file = './index.txt';

// Check if the file exists in the current directory.
access(Path_to_file, constants.F_OK, (err) => {
  console.log(`${Path_to_file} ${err ? 'does not exist' : 'exists'}`);
});

// Check if the file is readable.
access(Path_to_file, constants.R_OK, (err) => {
  console.log(`${Path_to_file} ${err ? 'is not readable' : 'is readable'}`);
});

// Check if the file is writable.
access(Path_to_file, constants.W_OK, (err) => {
  console.log(`${Path_to_file} ${err ? 'is not writable' : 'is writable'}`);
});

// Check if the file is readable and writable.
access(Path_to_file, constants.R_OK & constants.W_OK, (err) => {
  console.log(`${Path_to_file} ${err ? 'is not' : 'is'} readable and writable`);
});

We can also discerns the nature of the entity, distinguishing between files, directories, symbolic links, and more.

Here is a simple example.

// index.js

const fs = require('fs').promises;
  
fs.lstat('./index.txt')
.then(stats => {
    console.log('Is Directory:', stats.isDirectory());
    console.log('Is File:', stats.isFile());
    console.log('Is Symbolic Link:', stats.isSymbolicLink());

})
.catch(error => {
    console.error('Error retrieving file stats:', error);
});

Understanding and interpreting file stats allow us to make informed decisions and implement functionalities such as file size validation, permission management, version control, file type detection etc...

There are some other methods that aren't covered here but you can read about them from the official page.

Working with Paths and Descriptors

In this section, we'll explore the intricacies of working with file paths and descriptors in NodeJS, exploring techniques for effective file navigation, manipulation, and interaction.

Understanding File Paths

File paths specify the location of a file or directory in the file system hierarchy. Understanding how to work with file paths is crucial for navigating the file system, resolving relative paths, and ensuring platform compatibility.

Assume you're consuming an api and the api responds with files of different types. How do you handle such a scenario dynamically?
We can extract information out of these files using some of the mothods we'll discuss below.

1. __dirname: The `__dirname` variable contains the absolute path of the directory containing the currently executing script.

2. dirname(): The extracts the directory name from a given file path, which represents the file name.

3. path.basename(): The `basename()` method extracts the last portion of a file path, which represents the file name.

4. path.extname(): The `extname()` method extracts the file extension from a file path.

Let's have an example to illustrates the above.

// index.js

const path = require('path');

const file = 'play.txt'; 

// __dirname
console.log('__dirname:', __dirname);

const file_path = __dirname + '/' + file;

// dirname
const fileDir = path.dirname(file_path);
console.log('File\'s Parant Directory Name:', fileDir);

// basename
const fileName = path.basename(file_path);
console.log('File Name:', fileName);

// extname
const fileExt = path.extname(file_path);
console.log('File Extension:', fileExt);

NB: While `__dirname` specifically refers to the directory name of the current module, `path.dirname()` is a more general-purpose function for extracting directory names from file paths.

Let's continue with more methods, but this time for path manipulation.

path.resolve():

• ➢path.resolve() resolves a sequence of paths or path segments into an absolute path.
• ➢It starts from the rightmost argument and resolves each subsequent segment relative to the previous one.
• ➢If the resolved path is relative, it is resolved relative to the current working directory.
• ➢If no arguments are provided, it returns the current working directory.
• ➢It is commonly used to resolve file paths based on the current module's location or to resolve paths relative to a known root directory.

// index.js

const path = require('path');

console.log(path.resolve('/dir1', 'dir11', 'file1')); // Output: /dir1/dir11/file1
console.log(path.resolve('dir1', 'dir11', 'file1')); // Output: /Users/donaldteghen/Desktop/play/dir1/dir11/file1
console.log(path.resolve(__dirname, 'file1.txt')); // Resolves file1.txt relative to the current module's directory (/Users/donaldteghen/Desktop/play/file1.txt)

path.join():

• ➢path.join() joins multiple path segments into a single path using the platform-specific separator (e.g., '/' on Unix-like systems and '' on Windows).
• ➢It resolves relative paths and normalizes resulting paths.
• ➢Unlike path.resolve(), it does not resolve the resulting path to an absolute path based on the current working directory; instead, it concatenates the provided path segments.
• ➢It is commonly used to construct file paths in a platform-independent manner.

// index.js

const path = require('path');

console.log(path.join('/dir1', 'dir11', 'file1')); // Output: /dir1/dir11/file1
console.log(path.join('dir1', 'dir11', 'file1')); // Output: dir1/dir11/file1
console.log(path.join(__dirname, 'file1.txt')); // Joins file.txt to the current module's directory (/Users/donaldteghen/Desktop/play/file1.txt)

path.normalize():

• ➢path.normalize() normalizes a given path by resolving `..` and `.` segments and removing any redundant separators or special characters.
• ➢It converts backslashes to forward slashes on Windows systems.
• ➢It resolves any symbolic links and converts relative paths to absolute paths.
• ➢It is useful for standardizing paths obtained from user input or generated programmatically to ensure consistency across different platforms.

// index.js

const path = require('path');

console.log(path.normalize('/foo/bar/../baz')); // Output: /foo/baz
console.log(path.normalize('C:\\temp\\..\\file.txt')); // Output: C:/file.txt (on Windows)

Working with File Descriptors

File descriptors are unique identifiers assigned by the operating system to files opened by a process. In Node.js, file descriptors are represented as integers and can be used to perform various file operations directly.

To get a file descriptor to work with, we use the fs.open method.

Using fs.open():

The fs.open() method in Node.js is used to open a file and obtain its file descriptor, allowing for low-level file operations such as reading, writing, and seeking.
Once a file is opened and its file descriptor obtained, data can be read from the file using methods like fs.read(), providing fine-grained control over the reading process.

Similarly, data can be written to a file using its file descriptor with methods like fs.write(), enabling direct manipulation of file contents.

Also, the fs.open() method provides more control over file opening than higher-level methods like fs.readFile() or fs.createReadStream(), allowing for advanced file operations.

Let's have an all complete example.

// index.js

const fs = require('fs');

const options = {
    file: './play.txt',
    mode: 'a+'
};

// Open a file in a read/write mode (a+)
fs.open(options.file, options.mode, (err, fd) => {
    if (err) {
        console.error('Error opening file:', err);
        return;
    }
    console.log('File opened successfully. File Descriptor:', fd);

    // Performing file operations using the obtained file descriptor
    console.log('*********************************************** Writing data ************************************************');
    // Write data to the file descriptor
    const dataToWrite = 'This is random data to be writen to file.txt via a unique descriptor.';
    fs.write(fd, dataToWrite, (err, bytesWritten) => {
        if (err) {
            console.error('Error writing to file:', err);
            closeFile(fd);
            return;
        }
        console.log(`${bytesWritten} Bytes successfully written`);

        console.log('*********************************************** Reading data ************************************************');
        // Read data from the file descriptor
        const buffer = Buffer.alloc(1024); // Allocate a buffer for reading data
        fs.read(fd, buffer, 0, buffer.length, 0, (err, bytesRead, data) => {
            if (err) {
                console.error('Error reading from file:', err);
                closeFile(fd);
                return;
            }
            console.log('Bytes read from file:', bytesRead);
            console.log('Data read from file:', data.toString());
            closeFile(fd);    
        });
    });    
});



/**
 * 
 * @param {number} _fd 
 * @returns 
 */
function closeFile (_fd) {
    if (!_fd) {
        console.log('A file descriptor is required!')
        return
    }
    fs.close(_fd, (err) => {
        if (err) {
            console.log(`File with descriptor ${_fd} couldn't be closed due to:`, err);
            return
        }
        console.log(`File with descriptor ${_fd} closed successfully!`)
    });
}

Breakdown:

• ➢we use fs.open() to open a file named file.txt for appending mode ('a+').
  Upon successful opening of the file, the callback function receives the file descriptor (fd), which can be used for subsequent file operations.
• ➢we use fs.write() to write data to a file descriptor (fd). The callback function receives the number of bytes written (bytesWritten) and the data written to the file, allowing for confirmation of the write operation.
• ➢we use fs.read() to read data from a file descriptor (fd) into a buffer. The callback function receives the number of bytes read (bytesRead) and the data read from the file, which can then be processed as needed.
• ➢we use fs.close() to close the file descriptor (fd) once we're done with the file operations. This ensures proper cleanup and frees up system resources associated with the file descriptor.

Conclusion

To recap, we've covered various methods for interacting with file stats, from retrieving file stats asynchronously to synchronously blocking the event loop, and from utilizing promise-based syntax to distinguishing symbolic links.
We also saw how we can manipulate files using file descriptors and the fine-gained control it provides for that purpose.

It's evident that understanding and leveraging these concepts are essential for effective file management and manipulation in Node.js applications.

In the next part, we'll continue with reading and writing files using simpler but less fine-grained control alternatives.