Parallel and distributed computing explain how large systems gain performance and availability. PSC questions often ask speedup, Amdahl law, Flynn taxonomy, clusters, distributed system challenges, high-speed networks and architecture styles.
Engineering Definitions
Parallel computing
Standard definition: Simultaneous use of multiple processing elements to solve a problem faster.
Exam meaning: Multiple processors/cores प्रयोग गरेर computation छिटो गर्ने method।
Distributed computing
Standard definition: Computing across networked independent machines that coordinate by message passing.
Exam meaning: Networked machines ले message passing गरेर काम बाँड्ने computing।
Scalability
Standard definition: Ability of a system to handle increased load by adding resources.
Exam meaning: Resources थप्दा load handle गर्न सक्ने क्षमता।
Software architecture
Standard definition: High-level structure of software components, connectors and design decisions.
Exam meaning: System components र interactions को high-level design।
Concept Teaching
Parallel computing focuses on speed; distributed computing also handles location, failure and coordination. High-speed networks reduce communication bottleneck. Software architecture organizes components so systems can scale, evolve and remain maintainable.
Parallel Computing Concepts
Parallelism is limited by serial portions and communication overhead.
- Task parallelism runs different tasks concurrently.
- Data parallelism applies same operation to data partitions.
- Shared-memory systems communicate through common memory.
- Message-passing systems communicate explicitly.
- Synchronization overhead can reduce speedup.
- Load balancing distributes work evenly.
Flynn Taxonomy
Classic classification of computer architectures.
| Class | Meaning | Example idea |
|---|---|---|
| SISD | Single instruction single data | Traditional sequential processor |
| SIMD | Single instruction multiple data | Vector/GPU-style data parallelism |
| MISD | Multiple instruction single data | Rare/specialized |
| MIMD | Multiple instruction multiple data | Multicore/cluster systems |
Distributed Systems and Networks
Distributed systems face partial failure and latency.
- Message passing replaces shared memory assumption.
- Network latency is much slower than local memory.
- Partial failure means one node/link can fail while others run.
- Replication improves availability.
- Consensus coordinates agreement but costs time.
- High-speed networks improve bandwidth and reduce latency for clusters/data centers.
Software Architecture Styles
Architecture style shapes quality attributes.
| Style | Use | Tradeoff |
|---|---|---|
| Layered | Separation by responsibility | Can add overhead |
| Client-server | Central service access | Server bottleneck risk |
| Microservices | Independent services | Operational complexity |
| Event-driven | Asynchronous events | Harder tracing |
| Pipe-filter | Data processing pipeline | Good composition |
| Repository | Shared data store | Data coupling risk |
Engineering Mechanism
- Decompose work into tasks/data partitions.
- Assign work to cores/nodes.
- Communicate shared state through memory or messages.
- Synchronize where dependencies exist.
- Monitor speedup and bottlenecks.
- Choose architecture style based on quality attributes.
- Scale vertically or horizontally as needed.
Diagrams / Models To Draw
- Draw shared memory vs message passing.
- Draw Flynn taxonomy grid.
- Draw cluster with high-speed interconnect.
- Draw layered architecture.
- Draw microservices communicating through API/events.
Formulas, Algorithms and Rules
- Speedup S = T1/Tp.
- Efficiency E = S/p.
- Amdahl law: speedup <= 1/(s + (1-s)/p).
- Scalability depends on computation/communication ratio.
- Latency and bandwidth both affect network performance.
| Concept | Purpose | Exam distinction |
|---|---|---|
| Parallel | Performance via simultaneous processing | Shared or distributed memory |
| Distributed | Networked independent nodes | Partial failure |
| Speedup | Performance gain | Limited by serial fraction |
| High-speed network | Low latency/high bandwidth | Still not local memory |
| Architecture style | System organization | Quality tradeoffs |
| Microservices | Independent deployable services | Operational overhead |
Exam Point
- Use Amdahl law for speedup limit.
- Differentiate parallel and distributed computing.
- Flynn taxonomy is high-yield MCQ.
- Scalability is not automatic; bottlenecks matter.
- Architecture style should be tied to quality attributes.
Worked Example
If 20% of a program is serial, maximum speedup even with infinite processors is 1/0.2 = 5 by Amdahl law. This explains why parallelization must reduce serial bottlenecks, not just add processors.
Subjective Answer Pattern
- Define parallel and distributed computing.
- Explain taxonomy and speedup.
- Discuss distributed challenges.
- Explain high-speed network role.
- Compare software architecture styles.
- Conclude with scalability tradeoffs.
Common Engineering Mistakes
- Assuming doubling processors always doubles speed.
- Ignoring communication overhead.
- Confusing latency and bandwidth.
- Treating microservices as always better.
- Ignoring partial failure in distributed systems.
MCQ Revision
- Amdahl law limits what?
- SIMD means what?
- Efficiency formula?
- What is partial failure?
- Which architecture uses events?
- What is horizontal scaling?
Final Summary
- Parallel computing seeks speedup through concurrency.
- Distributed computing coordinates networked nodes.
- Speedup is limited by serial work and communication.
- High-speed networks reduce but do not remove communication cost.
- Software architecture organizes systems for quality attributes.