Booting Cisco 3560E switches with IOS 15.2 does not work

2021-09-01

TL;DR: You can not boot IOS 15.2 from the 3560X switches on 3560E switches.

The why and what happens if you try it

I have a couple of 3560E switches. Sadly they only run IOS 15.0. I wanted to have IPv4 address families over OSPFv3. This is not supported in IOS 15.0. But it is in IOS 15.2. And the IOS 15.2 image for the successor model to the 3560E, the 3560X, is called c3560e-universalk9-mz.152-4.E10.bin. And the IOS 15.0 firmware for the 3560X switches is the same file as for the 3560. So I tried to boot the 15.2(4)-E10 image. The switch itself boots the image but hits a malloc error during the boot process, crashes and reboots. So no OSPFv3 with IPv4 address family on those switches.

Here is the output of the boot process:

<...snip...>
POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed

POST: PortASIC Port Loopback Tests : Begin
POST: PortASIC Port Loopback Tests : End, Status Passed

POST: EMAC Loopback Tests : Begin
POST: EMAC Loopback Tests : End, Status Passed

Waiting for Port download...Complete
SYSTEM INIT: INSUFFICIENT MEMORY TO BOOT THE IMAGE!



%Software-forced reload


 00:01:01 UTC Mon Jan 2 2006: Unexpected exception to CPUvector 2000, PC = 36828B0
-Traceback= 0x36828B0z 0x2AFD98Cz 0x2B05F5Cz 0x3678A24z 0x367DF60z 0x2B2A9E0z 0x31D2AE0z 0x3118FDCz 0x3206400z 0x3209A2Cz 0x2AE5D30z 0x2AE5EFCz 0x2AE604Cz 0x6D921Cz 0x6D9454z 0x3683BA0z

Writing crashinfo to flash:/crashinfo_ext/crashinfo_ext_4

=== Flushing messages (00:01:03 UTC Mon Jan 2 2006) ===

Buffered messages:
Queued messages:
*Jan  2 00:01:03.929: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Mar  1 00:00:07.012: Read env variable - LICENSE_BOOT_LEVEL =
*Jan  2 00:00:03.456: %IOS_LICENSE_IMAGE_APPLICATION-6-LICENSE_LEVEL: Module name = c3560e Next reboot level = ipservices and License = ipservices
*Jan  2 00:01:01.329: %SYS-2-MALLOCFAIL: Memory allocation of 60000 bytes failed from 0x3678A20, alignment 0  <<< This is the relevant part.
Pool: Processor  Free: 38104  Cause: Not enough free memory
Alternate Pool: None  Free: 0  Cause: No Alternate pool
 -Process= "Init", ipl= 0, pid= 3
-Traceback= 6C9738z 2AFD8F8z 2B05F5Cz 3678A24z 367DF60z 2B2A9E0z 31D2AE0z 3118FDCz 3206400z 3209A2Cz 2AE5D30z 2AE5EFCz 2AE604Cz 6D921Cz 6D9454z 3683BA0z
Cisco IOS Software, C3560E Software (C3560E-UNIVERSALK9-M), Version 15.2(4)E10, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Tue 31-Mar-20 21:44 by prod_rel_team

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!
SRR0 = 0x031BB534  SRR1 = 0x00029230  SRR2 = 0x036828B0  SRR3 = 0x00029230
ESR = 0x00000000  DEAR = 0x00000000  TSR = 0x84000000  DBSR = 0x10000000

CPU Register Context:
Vector = 0x00002000  PC = 0x036828B0  MSR = 0x00029230  CR = 0x35000053
LR = 0x0368284C  CTR = 0x006B4068  XER = 0xE0000075
R0 = 0x0368284C  R1 = 0x06357288  R2 = 0x00000000  R3 = 0x04F897A8
R4 = 0x00000000  R5 = 0x00000000  R6 = 0x06357258  R7 = 0x05850000
R8 = 0x00029230  R9 = 0x05850000  R10 = 0x00008000  R11 = 0x00000000
R12 = 0x35000059  R13 = 0x079817A8  R14 = 0x053AFDD8  R15 = 0x00000000
R16 = 0x038B2550  R17 = 0x00000004  R18 = 0x00000020  R19 = 0x05850000
R20 = 0x00000000  R21 = 0x00000000  R22 = 0x0000EA60  R23 = 0x00000000
R24 = 0x00000000  R25 = 0x053AFDD8  R26 = 0x03678A20  R27 = 0x0000EA60
R28 = 0x00000000  R29 = 0x05EE80C0  R30 = 0x05BA0978  R31 = 0x00000000

Stack trace:
PC = 0x036828B0, SP = 0x06357288
Frame 00: SP = 0x06357298    PC = 0x0368284C
Frame 01: SP = 0x063572C8    PC = 0x02AFD98C
Frame 02: SP = 0x06357360    PC = 0x02B05F5C
Frame 03: SP = 0x06357378    PC = 0x03678A24
Frame 04: SP = 0x06357390    PC = 0x0367DF60
Frame 05: SP = 0x063573C0    PC = 0x02B2A9E0
Frame 06: SP = 0x063573E8    PC = 0x031D2AE0
Frame 07: SP = 0x06357400    PC = 0x03118FDC
Frame 08: SP = 0x06357418    PC = 0x03206400
Frame 09: SP = 0x06357458    PC = 0x03209A2C
Frame 10: SP = 0x06357468    PC = 0x02AE5D30
Frame 11: SP = 0x06357488    PC = 0x02AE5EFC
Frame 12: SP = 0x063574A0    PC = 0x02AE604C
Frame 13: SP = 0x063574D0    PC = 0x006D921C
Frame 14: SP = 0x063575D8    PC = 0x006D9454
Frame 15: SP = 0x063575E0    PC = 0x03683BA0

<...switch reboots...>

The other interesting problem is now: How to reset the switch? Normaly the switch would try the next image on disk if the first one fails, but this image works "good enough" for that mechanic not to kick in. This means that the switch is stuck in an infinite boot loop.

Resetting the switch to the old image

Disconnect the power, connect to the console port, press and hold the Mode button. Then plug the power cord back in. After 10-20 seconds release the Mode button. After a short time the switch: prompt should appear and you are in the bootloader.

To boot the old image you have to initialize the filesystem (flash_init), find the image with dir flash: and delete it with delete flash:/c3560e-universalk9-mz.152-4.E10.bin. After that boot the switch with boot. The switch tries to boot the image that is in the config file, fails and tries the first available image on the device which is probably your old working image. If no working image is on the flash have fun with xmodem or tftp.


Using driver version 1 for media type 2
Base ethernet MAC Address: 08:1f:f3:39:8c:80
Xmodem file system is available.
The password-recovery mechanism is enabled.

The system has been interrupted prior to initializing the
flash filesystem.  The following commands will initialize
the flash filesystem, and finish loading the operating
system software:

    flash_init
    boot

switch: ?
           ? -- Present list of available commands
        boot -- Load and boot an executable image
         cat -- Concatenate (type) file(s)
        copy -- Copy a file
      delete -- Delete file(s)
         dir -- List files in directories
  flash_init -- Initialize flash filesystem(s)
      format -- Format a filesystem
        fsck -- Check filesystem consistency
        help -- Present list of available commands
      memory -- Present memory heap utilization information
       mkdir -- Create dir(s)
        more -- Concatenate (display) file(s)
      rename -- Rename a file
       reset -- Reset the system
       rmdir -- Delete empty dir(s)
         set -- Set or display environment variables
      set_bs -- Set attributes on a boot sector filesystem
   set_param -- Set system parameters in flash
       sleep -- Pause (sleep) for a specified number of seconds
        type -- Concatenate (type) file(s)
       unset -- Unset one or more environment variables
     version -- Display boot loader version

switch: flash_init
Initializing Flash...
mifs[2]: 10 files, 1 directories
mifs[2]: Total bytes     :    2097152
mifs[2]: Bytes used      :     614400
mifs[2]: Bytes available :    1482752
mifs[2]: mifs fsck took 2 seconds.
mifs[3]: 3 files, 1 directories
mifs[3]: Total bytes     :    4194304
mifs[3]: Bytes used      :     949248
mifs[3]: Bytes available :    3245056
mifs[3]: mifs fsck took 2 seconds.
mifs[4]: 5 files, 1 directories
mifs[4]: Total bytes     :     524288
mifs[4]: Bytes used      :       9216
mifs[4]: Bytes available :     515072
mifs[4]: mifs fsck took 0 seconds.
mifs[5]: 5 files, 1 directories
mifs[5]: Total bytes     :     524288
mifs[5]: Bytes used      :       9216
mifs[5]: Bytes available :     515072
mifs[5]: mifs fsck took 1 seconds.
 -- MORE --
mifs[6]: 15 files, 3 directories
mifs[6]: Total bytes     :   57671680
mifs[6]: Bytes used      :   49069056
mifs[6]: Bytes available :    8602624
mifs[6]: mifs fsck took 26 seconds.
...done Initializing Flash.

switch: dir

List of filesystems currently registered:

                  bs[0]: (read-only)
               flash[6]: (read-write)
              xmodem[7]: (read-only)
                null[8]: (read-write)

switch: dir flash:
Directory of flash:/

    2  -rwx  20310016  <date>               c3560e-universalk9-mz.150-2.SE11.bin
    3  drwx  512       <date>               crashinfo_ext
    8  -rwx  1560      <date>               express_setup.debug
    9  -rwx  916       <date>               vlan.dat
   10  -rwx  64        <date>               ztp.py
   11  -rwx  26771456  <date>               c3560e-universalk9-mz.152-4.E10.bin
   12  -rwx  1920      <date>               private-config.text
   13  -rwx  5144      <date>               multiple-fs
   14  -rwx  6223      <date>               config.text
   15  drwx  512       <date>               crashinfo

8602624 bytes available (49069056 bytes used)

switch: delete flash:c3560e-universalk9-mz.152-4.E10.bin
Are you sure you want to delete "flash:c3560e-universalk9-mz.152-4.E10.bin" (y/n)?y
File "flash:c3560e-universalk9-mz.152-4.E10.bin" deleted

switch: boot
Loading "flash:c3560e-universalk9-mz.152-4.E10.bin"...flash:c3560e-universalk9-mz.152-4.E10.bin: no such file or directory

Error loading "flash:c3560e-universalk9-mz.152-4.E10.bin"

Interrupt within 5 seconds to abort boot process.
Loading "flash:/c3560e-universalk9-mz.150-2.SE11.bin"...@@@@@@@@@@@@@@@<... switch booting image as usual ...>

swagspace.org

The why and what happens if you try it

Resetting the switch to the old image