Feedforward in a perceptron is the process where inputs are multiplied by weights, summed, and passed through an activation function to produce an output. Backpropagation is the learning mechanism that adjusts these weights by calculating the error between predicted and actual values, then propagating this error backward using gradients.
A single-layer perceptron can only solve linearly separable problems. It fails with non-linear problems like XOR because it cannot create complex decision boundaries. This limitation led to the development of Multi-Layer Perceptrons (MLPs), which introduce hidden layers and non-linear activation functions to model complex relationships.
Recurrent Neural Networks (RNNs) are designed to handle sequential data like text, audio, or time series. They maintain a memory of previous inputs through loops within the network, allowing them to process data with temporal dependencies. However, they suffer from vanishing gradients in long sequences.
Long Short-Term Memory (LSTM) networks improve RNNs by introducing gates that control the flow of information, allowing them to remember important data for longer periods. Despite this, LSTMs are still sequential in nature and difficult to parallelize, making them inefficient on large datasets.
Transformers revolutionized deep learning by eliminating recurrence and using self-attention mechanisms to process entire sequences in parallel. This allows them to scale efficiently and learn relationships between distant words or elements in a sequence, leading to breakthroughs in models like BERT and GPT.