2 min read · Dec 7, 2023
--
1. Epoch:
Definition:
- An epoch is one complete pass through the entire training dataset.
Explanation:
- Imagine you have a deck of cards, and you go through the entire deck one time, that’s one epoch. In neural network training, it means the model has seen and learned from every example in your training dataset once.
Use Case:
- Training a neural network for multiple epochs helps the model learn from the data iteratively, refining its weights and improving performance.
2. Batch:
Definition:
- A batch is a subset of the training dataset used in one iteration of the optimization algorithm.
Explanation:
- Continuing with the deck of cards analogy, if you shuffle your deck and pick, say, 10 cards to look at and learn from, that’s a batch. You repeat this process until you’ve gone through the entire deck.
Use Case:
- Batching is practical for efficient computation. It allows you to update your model’s weights more frequently, and it can take advantage of parallelism in modern hardware.
3. Iteration:
Definition:
- An iteration is one update of the model’s weights, typically after processing one batch.
Explanation:
- Going back to the card example, if you go through 10 cards, learn from them, and then adjust your understanding (weights), that’s one iteration. You repeat this process until you’ve done it for all batches in an epoch.
Use Case:
- Iterations are the fundamental steps in training. Each iteration involves computing the gradients, updating weights, and moving the model towards better performance.
Relationships:
- In each epoch, you go through several batches, and in each batch, you perform multiple iterations.
- The number of iterations in an epoch depends on the size of your dataset and the batch size. For example, if you have 1000 examples and use a batch size of 100, you’d have 10 iterations per epoch.
Key Considerations:
- Too Many Epochs: Training for too many epochs can lead to overfitting, where the model memorizes the training data instead of learning general patterns.
- Batch Size: Smaller batch sizes introduce more noise but allow for more frequent updates. Larger batch sizes provide more stable updates but might require more memory.
- Computational Efficiency: Batching and iterations help make the training process computationally efficient, especially when dealing with large datasets.
In summary, an epoch is a complete pass through the dataset, a batch is a subset of the dataset processed in one go, and an iteration is one update of the model’s weights after processing one batch. These concepts are crucial for understanding and optimizing the training process in neural networks.
