Parallel programming models
(Shared memory vs distributed memory)

Parallel programming is a paradigm in which tasks are broken down into smaller parts that can be executed simultaneously, thereby improving computational efficiency. This chapter focuses on two primary parallel programming models: Shared Memory and Distributed Memory.

Shared Memory Model

In the Shared Memory Model, multiple processors or threads share a single address space, allowing them to directly access the same data. This model simplifies programming since communication between threads is implicit.

Basic Explanation: Imagine a group of workers in a factory. They all work in the same space and can access the same tools and materials without explicitly sharing them. Similarly, in shared memory parallel programming, threads or processes can access shared variables or data structures without needing to pass them explicitly.

Advanced Explanation: In shared memory systems, threads or processes communicate by reading and writing to shared memory locations. Synchronization mechanisms like mutexes, semaphores, and barriers ensure that data integrity is maintained and race conditions are avoided.

				
					#include <stdio.h>
#include <omp.h>

int main() {
    int sum = 0;
    #pragma omp parallel for reduction(+:sum)
    for (int i = 0; i < 10; ++i) {
        sum += i;
    }
    printf("Sum: %d\n", sum);
    return 0;
}

				
			
				
					// output //
Sum: 45

Explanation of Code:

  • In this OpenMP example, the pragma omp parallel for directive parallelizes the loop.
  • The reduction(+:sum) clause ensures that each thread has a private copy of the sum variable, and at the end of the parallel region, the partial sums are combined into the final sum.

Distributed Memory Model

In the Distributed Memory Model, each processor has its own local memory, and communication between processors is achieved explicitly through message passing. This model is typically used in clusters or distributed computing environments.

Basic Explanation: Imagine different households working together on a project. Each household has its own set of tools and resources (local memory), and they communicate by explicitly passing messages or sharing information when needed. Similarly, in distributed memory parallel programming, processes communicate by explicitly sending and receiving messages.

Advanced Explanation: In distributed memory systems, processes execute independently but can communicate by exchanging messages through communication libraries like MPI (Message Passing Interface) or OpenSHMEM. This model allows for scalable parallelism but requires careful management of data movement and synchronization.

				
					#include <stdio.h>
#include <mpi.h>

int main(int argc, char** argv) {
    MPI_Init(&argc, &argv);
    int rank, size;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    int data;
    if (rank == 0) {
        data = 100;
        MPI_Send(&data, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);
    } else if (rank == 1) {
        MPI_Recv(&data, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
        printf("Received data: %d\n", data);
    }
    MPI_Finalize();
    return 0;
}

				
			
				
					// output //
Received data: 100

Explanation of Code:

  • This MPI example demonstrates message passing between two processes.
  • Process 0 sends the integer value 100 to Process 1 using MPI_Send, and Process 1 receives the data using MPI_Recv.

Key Takeaways

  • Shared Memory model: Simplicity, implicit communication, lock mechanisms for synchronization.
  • Distributed Memory model: Scalability, explicit message passing, communication protocols like MPI.

Choice of model depends on system architecture, scalability requirements, and programming complexity considerations.

By mastering both models and their intricacies, programmers can design parallel applications that harness the full potential of modern computing architectures, achieving high performance and efficiency.

In conclusion, both Shared Memory and Distributed Memory models offer parallel programming capabilities with their respective advantages and challenges. Shared Memory is simpler to program and is suitable for multi-core systems, while Distributed Memory allows for scalable parallelism across clusters but requires explicit management of communication and data synchronization. Choosing the appropriate model depends on the specific requirements and architecture of the parallel computing system. By understanding these models, programmers can effectively leverage parallelism to improve the performance of their applications.Happy coding!❤️

Table of Contents

Home
Roadmap
ATS Resume
Playground
Resources