PyTorch Tutorial: Extracting Features from ResNet by Excluding the Last FC Layer

Understanding ResNets and FC Layers:

• ResNets (Residual Networks): A powerful convolutional neural network (CNN) architecture known for its ability to learn deep representations by leveraging skip connections. These connections help alleviate the vanishing gradient problem that can hinder training in very deep networks.
• Fully-Connected (FC) Layers: In CNNs, FC layers are typically placed at the end of the network to transform the extracted features into the final output (e.g., class probabilities for image classification).

Removing the Last FC Layer:

There are two primary approaches to achieve this in PyTorch:

1. Using torch.nn.Sequential:

   • This method is suitable when you want to create a new model that excludes the last FC layer.
   • Steps:
     1. Load the pre-trained ResNet model using torchvision.models.
     2. Iterate through the model's child modules (layers) using model.children().
     3. Create a new nn.Sequential module, including all child modules except the last one (the FC layer).
     4. Assign the new model to a variable.

   import torch
   from torchvision import models
   
   # Load the pre-trained ResNet model (e.g., ResNet18)
   model = models.resnet18(pretrained=True)
   
   # Create a new model excluding the last FC layer
   new_model = torch.nn.Sequential(*list(model.children())[:-1])
   

2. Using Model Slicing:

   • This approach modifies the original model in-place, removing the last FC layer. It's generally not recommended for pre-trained models as it alters their weights.
   • Steps:
     1. Load the pre-trained ResNet model.
     2. Access the module index of the last FC layer (often named fc). You can print the model structure to identify its index.
     3. Use list slicing to create a new model that includes all modules except the one at the last index.

   # Not recommended for pre-trained models due to weight modification
   last_layer_index = len(list(model.children())) - 1
   new_model = model[:last_layer_index]
   

Important Considerations:

• new_model = torch.nn.Sequential(OrderedDict([*(list(model.named_children())[:-1])]))
  

Choosing the Right Approach:

• If you intend to create a new model with a custom FC layer, use nn.Sequential.
• If you're just experimenting and don't mind modifying the original model (not recommended for pre-trained models), use model slicing with caution.

By following these steps and considering the key points, you can effectively remove the last FC layer from your ResNet model in PyTorch.

Removing Last FC Layer with nn.Sequential (Preserving Layer Names - Optional):

import torch
from torchvision import models

# Load the pre-trained ResNet model (e.g., ResNet18)
model = models.resnet18(pretrained=True)

# Create a new model excluding the last FC layer with preserved layer names
new_model = torch.nn.Sequential(OrderedDict([*(list(model.named_children())[:-1])]))

# Print the new model structure to verify layer names are preserved
print(new_model)

Removing Last FC Layer with Model Slicing (Not Recommended for Pre-Trained Models):

# Load the pre-trained ResNet model (e.g., ResNet18)
model = models.resnet18(pretrained=True)

# **Not recommended for pre-trained models due to weight modification**
last_layer_index = len(list(model.children())) - 1

# Create a new model excluding the last FC layer (modifies original model)
new_model = model[:last_layer_index]

# Print the new model structure to verify the last FC layer is removed
print(new_model)

import torch
from torchvision import models

# Load the pre-trained ResNet model (e.g., ResNet18)
model = models.resnet18(pretrained=True)

# Create a new model excluding the last FC layer using nn.Sequential
new_model = torch.nn.Sequential(*list(model.children())[:-1])

# Define a new FC layer with your desired output size
num_features_in = new_model[-1].out_features  # Get output size of the last layer
num_classes = 10  # Example: 10 class classification

new_fc = torch.nn.Linear(num_features_in, num_classes)

# Add the new FC layer to the new model (modify the model)
new_model.add_module('fc', new_fc)

# Print the new model structure to verify the new FC layer is added
print(new_model)

Remember to choose the approach that best suits your needs and be mindful of modifying pre-trained models directly.

Using torch.nn.ModuleDict (Similar to nn.Sequential but More Flexible):

This approach offers more flexibility than nn.Sequential when you need to modify the remaining modules in the model:

import torch
from torchvision import models

# Load the pre-trained ResNet model (e.g., ResNet18)
model = models.resnet18(pretrained=True)

# Create a new model excluding the last FC layer using ModuleDict
new_model_dict = torch.nn.ModuleDict({name: module for name, module in model.named_children()[:-1]})
new_model = torch.nn.Sequential(*new_model_dict.values())

# Print the new model structure
print(new_model)

Looping and Conditional Inclusion:

This method iterates through the model's modules and includes all except the FC layer:

import torch
from torchvision import models

# Load the pre-trained ResNet model (e.g., ResNet18)
model = models.resnet18(pretrained=True)

new_model_layers = []
for name, module in model.named_children():
    if not name == 'fc':  # Exclude layer named 'fc'
        new_model_layers.append(module)

new_model = torch.nn.Sequential(*new_model_layers)

# Print the new model structure
print(new_model)

model.state_dict() and Selective Loading:

This approach involves saving the model state and then loading it selectively, excluding the FC layer weights:

import torch
from torchvision import models

# Load the pre-trained ResNet model (e.g., ResNet18)
model = models.resnet18(pretrained=True)

# Save the model state
state_dict = model.state_dict()

# Exclude the FC layer weights from the state_dict
del state_dict['fc.weight']
del state_dict['fc.bias']

# Create a new model instance
new_model = models.resnet18(pretrained=False)

# Load the modified state_dict into the new model
new_model.load_state_dict(state_dict)

# Print the new model structure
print(new_model)

• Use nn.Sequential or nn.ModuleDict when you want a clear separation between the original and modified models.
• Use loop-based inclusion for more control over which layers to keep.
• model.state_dict() with selective loading offers a different approach but might be less intuitive for beginners.

Remember to adapt these methods based on your specific needs and the structure of your ResNet model.