Lecture 01
Image classification with Vision Transformer
Vision Transformers (ViT) are a type of neural network architecture that has gained popularity in recent years for image classification tasks. Unlike traditional convolutional neural networks (CNNs), which use a series of convolutional layers to extract features from images, ViT relies on self-attention mechanisms to capture global image context.
Here are the general steps for performing image classification with ViT:
Preprocessing: Resize the input images to a fixed size and normalize the pixel values to a common range (e.g., [0,1] or [-1,1]).
Feature extraction: Split the resized image into fixed-size patches, and linearly project each patch into a high-dimensional feature space.
Position encoding: Add a learnable positional embedding to each patch to account for the spatial location of the patch within the image.
Transformer layers: Stack several Transformer layers to capture global image context using self-attention mechanisms. Each Transformer layer consists of a multi-head self-attention module followed by a feedforward neural network.
Classification head: Append a classification head on top of the Transformer layers to predict the class label of the input image.
Training: Train the ViT model using a standard supervised learning objective, such as cross-entropy loss.
Inference: Given a new image, pass it through the trained ViT model to obtain the predicted class label.
Overall, ViT has shown promising results on various image classification benchmarks and can be a viable alternative to traditional CNN-based approaches.
This example implements the Vision Transformer (ViT) model by Alexey Dosovitskiy et al. for image classification and demonstrates it on the CIFAR-100 dataset. The ViT model applies the Transformer architecture with self-attention to sequences of image patches, without using convolution layers.
This example requires TensorFlow 2.4 or higher, as well as TensorFlow Addons, which can be installed using the following command:
