Scenario 2: EC2 Instance Boot or Initialization Failure

(This is a Free Sample. Mindmaps and Podcast Discussions are a paid feature) 

Mind Map

Podcast Discussion

Problem:

• A newly launched EC2 instance fails to boot.
  
  

• The AWS Console shows "initialization failure."
  
  

• Errors are visible in the console output.
  
  

Step-by-Step Troubleshooting:

1. Check Console Output:
   
   

   • Access the EC2 Console Output or Serial Console.
     
     

   • Look for kernel panic messages.
     
     

   • Check for cloud-init failures.
     
     

   • Identify block device or mount errors.
     
     

2. Validate User Data Script:
   
   

   • Ensure the script starts with #!/bin/bash or the correct interpreter.
     
     

   • Check for commands like yum update or apt-get upgrade that might hang.
     
     

   • Verify that IAM roles assigned to the instance have the correct permissions.
     
     

   • Add set -ex at the start of the script to enable detailed error logging.
     
     

3. Confirm Block Device Mapping:
   
   

   • Check that the root volume is correctly mapped (e.g., /dev/xvda or /dev/sda1 depending on instance type).
     
     

   • Verify that the EBS snapshot used is valid.
     
     

   • Ensure additional volumes are correctly attached if used.
     
     

4. Verify Instance Type Compatibility:
   
   

   • Confirm that the selected AMI supports the instance type.
     
     

   • Nitro-based instances require NVMe-compatible AMIs.
     
     

   • ARM-based AMIs only work on Graviton instances (e.g., t4g, m6g).
     
     

   • Try launching the instance with a different instance type if in doubt.
     
     

5. Perform a Test Launch:
   
   

   • Launch the same AMI without user data to isolate script issues.
     
     

   • Launch using a minimal security group and subnet settings.
     
     

   • Try a different AMI to see if the problem is specific to the original AMI.
     
     

6. Rescue Procedure:
   
   

   • Stop the instance and detach the root EBS volume.
     
     

   • Attach the volume to another healthy instance as a secondary volume.
     
     

   • Mount the volume and check for issues in files like /etc/fstab.
     
     

   • Roll back any recent changes or updates made to the system.
     
     

   • Detach from the rescue instance and reattach to the original instance as root.
     
     

   • Start the original instance.
     
     

Key AWS Terms:

• AMI: Amazon Machine Image, a template for launching EC2 instances.
  
  

• cloud-init: Initialization tool that runs user data scripts on first boot.
  
  

• Block Device Mapping: Defines which EBS volumes attach to the instance and how.
  
  

• Nitro/Xen: Different AWS hypervisors, Nitro supports newer instance types.
  
  

• Serial Console: Direct console access for troubleshooting boot issues.
  
  

Interview Insight:

• When an EC2 instance fails to initialize, I always check the console output for cloud-init errors or kernel panics.
  
  

• If user data scripts are involved, I validate and test them separately.
  
  

• For deeper issues, I detach the root volume, fix errors using a rescue instance, and reattach to minimize downtime.