Updating a HP SE326M1 Server

2018-10-08

I have an old HP SE326M1 server. According to rumors on the internet those servers are a OEM version of the DL180G6 with 25 2.5" drives and were used in Microsoft datacenters.

Anyway, I wanted to use this server again. One issue I had with this server was that the combination of the SAS backplane and the RAID controller have an issue. When using the cli tool from within Linux to configure the RAID controller all disks are shown in slot 0. This means that you can not create or modify arrays because the controller (at least from within the Linux cli tool) can not distinguish the different drives. A long time ago I heard that a firmware upgrade was required.

So I set out to find the firmware upgrade. Somewhen I stumbled over a SPP from HP. This is the "Server Pack for Proliant" which contains the firmware upgrades for the HP server. Normally these SPPs require a active support contract. But I found one on the HP website which did not. If can't find that SPP you may ask someone else who has access to one e.g. someone who is working for a company with HP servers. I downloaded this file after creating an account to download this firmware pack. (WTF HP!? why do I need an account to download a firmware package?). After downloading a 5.6GB iso file (WTF HP!?) with only 3MB/s (WTF HP!?) which took about 30 minutes I tried what every mentally sane Linux guys would do: use dd to copy the image to a usb stick. This did not boot at all. So I went to long way , booted up windows, copied the 5.6GB file to a USB stick, mounted the iso on windows and used the hpusbkey.exe tool to build a usb stick with that tool. The tool copied all the files and then just crashed. Anyways I tried booting it anyway. But after booting that I just got the message that the boot failed and that pressing any key would reboot the system. Okay, this did not work.

The next place to try out was iLo2. This is HPs remote management tool. So after several failed attempts to configure an IP address either static or via DHCP I was close to throwing the server out of the window. Then I reseted the iLo config completely and suddenly it accepted IPs via DHCP.

Now there was the next problem: When connecting to the ilo you get a SSL_BAD_HMAC_ERROR, which means that something is broken with SSL/TLS. A workaround is to change some settings of firefox in about:config. The required key is called security.tls.version.fallback-limit. If you set this to 1firefox is talking with ilo2. But be carefull, this enables fallbacks from newer TLS versions to older ones which makes some MITM attacks which involve falling back to older SSL/TLS versions possible.

Upgrading iLo2 to the most recent firmware version was working very well. The firmware can be found for example (here)[http://pingtool.org/latest-hp-ilo-firmwares/]. This page also describes how to get the bin file extracted from the .scexe file HP gives you. Somewhere in the Administration tab you can find a firmware upgrade section. Just upload the firmware there and wait a minute for the iLo to reboot.

Now starts the funny part. iLo2 uses Java applets for remote media mounting and for the remote console. All major browsers removed the support for Java applets. Even the current firefox extended support release got rid of it by now. Someone on (reddit)[reddit.com] mentioned a browser called PaleMoon which still supports it but I haven't tryed that yet. The solution that I ended up with was using Windows 7 and the Internet Explorer 11. This works because iLo2 also has an implementation specificly for IE. This way I was able to mount the SPP iso image on the server and managed to boot from that. But this takes ages. First it needs ~1 minute to boot. Then you have to select if you want to run it in automatic or interactive mode. I chose the automatic mode by accident which worked out. Then it copies things to memory. This took 10-15 minutes for me. Then I reached some kind of GUI with a blue background. At least I had a mouse now. After an other 15 minutes it loaded something which looked like a browser window. At least it pretended to be waiting to transfer some data from 127.0.0.1. After an other 10 minutes the page loaded and started analysing the system. Then it showed a page and was scanning for the firmware upgrades and applied some of them. After a sudden reboot the backplane and raid controller firmware were up-to-date.

installing the hpucacli or ssacli utilies

hpacucli which was later renamed to hpssacli which is now known as ssacli is a tool to access the HP RAID controllers from within the operating system. Sadly this is not in the official repositories but HP has a repository for debian which is maintained for the current (stretch) release.

Furter details on how to install this can be found here. If you have issues with installing take a look at the comment section of this page. I will keep it short and add the commands I used here.

echo -e "deb http://downloads.linux.hpe.com/SDR/repo/mcp/ stretch/current non-free" > /etc/apt/sources.list.d/proliant.sources.list
apt install dirmngr
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C208ADDE26C2B797 # this is adding the required key from a repo server.
apt update
apt install ssacli