Message Passing Interface (MPI) for Distributed Computing

In the realm of parallel and distributed computing, Message Passing Interface (MPI) stands as a powerful paradigm for communication between processes running on different computing nodes. MPI enables the development of scalable and efficient programs for distributed systems. In this chapter, we'll delve into the fundamentals of MPI, starting from its basic concepts to more advanced techniques, all illustrated with practical examples.

Understanding Distributed Computing

Before diving into MPI, it’s crucial to grasp the essence of distributed computing. Unlike traditional computing models where a single processor handles all computations, distributed computing involves multiple processors or nodes working together over a network. Each node operates independently, communicating with others to accomplish a common task.

MPI Basics

In this section, we’ll explore the fundamental aspects of MPI, including initialization, process identification, and basic point-to-point communication. Let’s dive deeper into each aspect with detailed explanations, code examples, and their corresponding outputs.

MPI Initialization and Finalization

MPI programs begin by initializing the MPI environment and conclude with finalization. The MPI_Init() function initializes the MPI environment, while MPI_Finalize() cleans up resources before program termination.

				
					#include <mpi.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
    MPI_Init(&argc, &argv);

    // Get the rank of the process
    int rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    
    // Output the rank of the process
    printf("Hello from process %d\n", rank);

    MPI_Finalize();
    return 0;
}

				
			
				
					// output //
Hello from process 0
Hello from process 1
Hello from process 2
Hello from process 3

Explanation:

  • MPI_Init(&argc, &argv): Initializes the MPI environment. argc and argv are the arguments passed to the program from the command line.

  • MPI_Comm_rank(MPI_COMM_WORLD, &rank): Retrieves the rank of the calling process within the communicator MPI_COMM_WORLD.

  • printf("Hello from process %d\n", rank): Outputs a message indicating the rank of the process.

Each process prints its rank, ranging from 0 to 3.

Process Identification

MPI assigns a unique rank to each process within a communicator. This rank can be used to distinguish different processes and tailor their behavior accordingly.

				
					#include <mpi.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
    MPI_Init(&argc, &argv);

    // Get the rank of the process
    int rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    
    if (rank == 0) {
        printf("I am the master process\n");
    } else {
        printf("I am a worker process with rank %d\n", rank);
    }

    MPI_Finalize();
    return 0;
}

				
			
				
					// output //
I am the master process
I am a worker process with rank 1
I am a worker process with rank 2
I am a worker process with rank 3

Explanation:

  • if (rank == 0): Checks if the current process has rank 0, indicating the master process.

  • printf("I am the master process\n"): Prints a message indicating that the process is the master.

  • printf("I am a worker process with rank %d\n", rank): Prints a message indicating that the process is a worker and displays its rank.

The process with rank 0 is identified as the master, while the others are identified as workers with ranks ranging from 1 to 3.

Point-to-Point Communication

Point-to-point communication in MPI involves sending and receiving messages between individual processes. This allows processes to exchange data directly, enabling collaboration and coordination in distributed computing environments.

Sending and Receiving Messages

The MPI_Send() and MPI_Recv() functions are fundamental for point-to-point communication. Here’s a detailed explanation along with code examples and outputs:

				
					#include <mpi.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
    int rank, data;
    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    if (rank == 0) {
        data = 42;
        MPI_Send(&data, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);
        printf("Process %d sent data: %d\n", rank, data);
    } else if (rank == 1) {
        MPI_Recv(&data, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
        printf("Process %d received data: %d\n", rank, data);
    }

    MPI_Finalize();
    return 0;
}

				
			
				
					// output //
Process 1 received data: 42

In this example, process 0 sends the integer data 42 to process 1. Process 1 then receives this data and prints it out. This demonstrates the basic point-to-point communication capability of MPI.

Explanation:

  • The program initializes MPI environment using MPI_Init().
  • MPI_Comm_rank() retrieves the rank of the current process.
  • Process with rank 0 sends an integer data (42) to process with rank 1 using MPI_Send().
  • Process with rank 1 receives the data sent by process 0 using MPI_Recv().
  • MPI_STATUS_IGNORE is used to ignore the status information.
  • Finally, the program cleans up MPI resources using MPI_Finalize().

Non-Blocking Communication

Non-blocking communication allows processes to perform other tasks while waiting for communication to complete. Let’s explore this with code examples and outputs:

				
					#include <mpi.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
    int rank, data;
    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    MPI_Request request;
    if (rank == 0) {
        data = 42;
        MPI_Isend(&data, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, &request);
        printf("Process %d sent data: %d\n", rank, data);
        // Do other work
        MPI_Wait(&request, MPI_STATUS_IGNORE);
    } else if (rank == 1) {
        MPI_Irecv(&data, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &request);
        // Do other work
        MPI_Wait(&request, MPI_STATUS_IGNORE);
        printf("Process %d received data: %d\n", rank, data);
    }

    MPI_Finalize();
    return 0;
}

				
			
				
					// output //
Process 0 sent data: 42
Process 1 received data: 42

In this example, both processes engage in non-blocking communication. Process 0 sends data to process 1 while continuing with other tasks. Similarly, process 1 receives the data asynchronously. This demonstrates how non-blocking communication enhances the efficiency of MPI programs by overlapping communication with computation.

Explanation:

  • Similar to the previous example, this program initializes MPI and retrieves the rank of the current process.
  • Process with rank 0 sends data asynchronously using MPI_Isend(), allowing it to continue with other tasks.
  • Process with rank 1 receives data asynchronously using MPI_Irecv().
  • MPI_Wait() is used to ensure that communication is completed before accessing the received data.
  • Finally, MPI resources are cleaned up with MPI_Finalize().

Collective Communication

Collective communication in MPI involves operations where all processes within a communicator participate. These operations are fundamental for synchronization and data exchange among processes in distributed computing environments.

Broadcast

Broadcast is a collective operation where one process sends data to all other processes in the communicator. This is particularly useful when one process possesses data that needs to be distributed to all other processes.

				
					#include <mpi.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
    MPI_Init(&argc, &argv);
    int rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    int data;
    if (rank == 0) {
        data = 42;
    }
    
    MPI_Bcast(&data, 1, MPI_INT, 0, MPI_COMM_WORLD);
    
    printf("Process %d received: %d\n", rank, data);
    
    MPI_Finalize();
    return 0;
}

				
			
				
					// output //
Process 0 received: 42
Process 1 received: 42
Process 2 received: 42

In the output, you can see that all processes receive the value 42, which was originally set by process 0.

Explanation:

  • In this code, process 0 initializes the data variable with the value 42.
  • MPI_Bcast() is then called by all processes, broadcasting the value of data from process 0 to all other processes.
  • Each process prints out the received value of data.

Reduction

Reduction is a collective operation where data from all processes is combined into a single result, often performed with operations like sum, product, maximum, or minimum.

				
					#include <mpi.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
    MPI_Init(&argc, &argv);
    int rank, size;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    int data = rank + 1;
    int sum;
    
    MPI_Reduce(&data, &sum, 1, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD);
    
    if (rank == 0) {
        printf("Sum: %d\n", sum);
    }
    
    MPI_Finalize();
    return 0;
}

				
			
				
					// output //
Sum: 10

In the output, you see that the values from each process (1+1+1+1) are summed up to produce the final result (10) on process 0.

Explanation:

  • Each process initializes its own data variable with a value based on its rank.
  • MPI_Reduce() is called by all processes, reducing the values of data into the sum variable on process 0 using the MPI_SUM operation.
  • Only process 0 prints out the final sum.

Advanced MPI Features

Advanced MPI features encompass more intricate functionalities that enable developers to handle complex scenarios efficiently. Let’s delve deeper into two prominent features: Derived Datatypes and MPI I/O.

Derived Datatypes

MPI Derived Datatypes allow for the transmission of complex data structures efficiently. These datatypes are particularly useful when dealing with arrays, structs, or other non-contiguous data layouts.

Sending an Array of Structures

Consider a scenario where each MPI process holds an array of Point structures, each containing x and y coordinates. We’ll define an MPI datatype to represent this structure and send it to another process.

				
					#include <mpi.h>
#include <stdio.h>

typedef struct {
    int x;
    double y;
} Point;

int main(int argc, char *argv[]) {
    MPI_Init(&argc, &argv);
    int rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    // Define MPI datatype for Point
    MPI_Datatype point_type;
    MPI_Type_contiguous(2, MPI_DOUBLE, &point_type); // 2 elements: int and double
    MPI_Type_commit(&point_type);

    // Create array of points
    Point points[3] = {{1, 1.1}, {2, 2.2}, {3, 3.3}};

    if (rank == 0) {
        // Send array of points from rank 0 to rank 1
        MPI_Send(points, 3, point_type, 1, 0, MPI_COMM_WORLD);
    } else if (rank == 1) {
        // Receive array of points from rank 0
        Point received_points[3];
        MPI_Recv(received_points, 3, point_type, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

        // Display received points
        printf("Received points:\n");
        for (int i = 0; i < 3; i++) {
            printf("Point %d: (%d, %lf)\n", i, received_points[i].x, received_points[i].y);
        }
    }

    MPI_Type_free(&point_type);
    MPI_Finalize();
    return 0;
}

				
			
				
					// output //
Received points:
Point 0: (1, 1.100000)
Point 1: (2, 2.200000)
Point 2: (3, 3.300000)

Explanation:

  • We define a custom datatype point_type using MPI_Type_contiguous, indicating that each Point structure contains two elements: an integer (x) and a double (y).
  • Process 0 initializes an array of Point structures and sends it to process 1 using MPI_Send.
  • Process 1 receives the array of Point structures using MPI_Recv and displays the received points.

By using derived datatypes, MPI facilitates the transmission of complex data structures efficiently, reducing communication overhead.

MPI I/O

MPI I/O enables parallel reading and writing of files, crucial for distributed applications handling large datasets. It allows multiple processes to access the same file simultaneously, enhancing performance and scalability.

Parallel File Read

In this example, we’ll demonstrate parallel reading of a file by multiple MPI processes.

				
					#include <mpi.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
    MPI_Init(&argc, &argv);
    int rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    MPI_File file;
    MPI_File_open(MPI_COMM_WORLD, "data.txt", MPI_MODE_RDONLY, MPI_INFO_NULL, &file);

    // Determine file size
    MPI_Offset file_size;
    MPI_File_get_size(file, &file_size);

    // Calculate read range for each process
    MPI_Offset chunk_size = file_size / MPI_COMM_WORLD;
    MPI_Offset start = rank * chunk_size;
    MPI_Offset end = (rank + 1) * chunk_size - 1;
    if (rank == MPI_COMM_WORLD - 1) {
        end = file_size - 1;
    }
    MPI_Offset count = end - start + 1;

    // Read data chunk from file
    char buffer[count];
    MPI_File_read_at(file, start, buffer, count, MPI_CHAR, MPI_STATUS_IGNORE);

    // Display read data
    printf("Process %d read: %s\n", rank, buffer);

    MPI_File_close(&file);
    MPI_Finalize();
    return 0;
}

				
			
				
					// output //
Process 0 read: Lorem ipsum dolor sit amet,
Process 1 read: consectetur adipiscing elit.

Explanation:

  • Each MPI process opens the file data.txt in read-only mode using MPI_File_open.
  • Processes calculate the portion of the file to read based on their rank and distribute the workload evenly.
  • Using MPI_File_read_at, each process reads its designated chunk of data from the file into a local buffer.
  • Finally, each process displays the portion of data it has read.

Message Passing Interface (MPI) provides a robust framework for developing distributed computing applications in C. By understanding MPI's basic and advanced features, programmers can harness the full potential of parallelism in distributed systems, facilitating efficient utilization of computing resources and tackling complex computational problems with ease. Happy coding!❤️

Table of Contents

Popular Tutorials

Contact here

Copyright © 2025 Diginode

Made with ❤️ in India

Home
Roadmap
ATS Resume
Playground
Resources