HPDL385

Differences between revisions 2 and 3
⇤ ← Revision 2 as of 2005-06-30 14:53:44 →
Size: 10365
Editor: c-65-96-209-153
Comment: iLO ssh bug, MySQL glibc bug
← Revision 3 as of 2005-07-06 23:20:39 → ⇥
Size: 11771
Editor: c-65-96-209-153
Comment: hpsa6i script
Deletions are marked like this. Additions are marked like this.
Line 86: Line 86:
     * I have written/adapted a small script for monitoring the drives, I called it {{{hpsa6i.sh}}}
    * {{{
#!/bin/sh

##Monitoring script RAID on hp smart array controller 6i
emaillist=raid@xxx.com

host=`hostname`
/bin/echo -e "set target controller slot=0\nshow status\narray A physicaldrive all show status\narray A logicaldrive all show status" | hpacucli > /tmp/raid-current
curdiff=`/usr/bin/diff -u /usr/share/hpsa6i/raid-good /tmp/raid-current`
currentinfo=`cat /tmp/raid-current`
rm /tmp/raid-current

if [ "$curdiff" != "" ] ; then
# Send email alert
        /usr/lib/sendmail -f devnull@$host -t << HPsa6iRAIDEOF
To: $emaillist
From: root@$host
Subject: Raid may be broken on $host
Errors-To: devnull
Raid may be broken on $host.

====> A diff between production and current is:

$curdiff

====> Full Raid Current Info:

$currentinfo

HPsa6iRAIDEOF

fi
}}}
   * You'll want to customize that script as appropriate including
    * Email address
    * slot #
    * Array #
   * Create the {{{/usr/share/hpsa6i}}} directory
   * Create the initial ''raid-okay'' state file:
    * {{{
cd /usr/share/hpsa6i
/bin/echo -e "set target controller slot=0\nshow status\narray A physicaldrive all show status\narray A logicaldrive all show status" | hpacucli > raid-good
}}}
   * Add to root's cronjob
    * {{{
*/30 * * * * /usr/local/bin/hpsa6i.sh
}}}

HPDL385

HP DL385

This page was created to aid people trying to configure a HP DL385 under Ubuntu Linux. This page was originally based on the Hoary release, however I'm sure future releases will work equally well on this machine. Over all Hoary runs excellently on this machine, and for the most part, installs 100% painfree. That's not to say there's not quite a bit to know about your new 64bit machine. I also try to include other information related to HP machines that non-HP customers may be surprised by. This page may additionally be helpful to people installing ubuntu on other HP hardware, or people trying to install on other 64-bit Opteron hardware.

Hardware Description

  • Dual Opteron (250/2.4GHz)
  • Smart Array Controller 6i
    • Raid 0/1+0/5
    • 64M read cache onboard, non-battery backed
    • Optional battery-backed writeback cache (128M, giving total of 192M cache)
    • Online migration of stripe size, or raid levels
    • Online expansion of raid containers
      • I don't think sophisticated actions like this are supported without the battery backed cache
  • 6 Hotswap bays
  • Rack Mountable
  • ATI Rage XL
  • 2 Broadcom tygon3 Gigabit Ethernet
  • 1 iLO board
    • Integrated Lights out Management
    • 1 Ethernet port
    • SSH access
    • HTTP/HTTPS access
    • Allows remote console
      • Uses video bios to access console
      • Text modes are free
      • Graphics requires advanced license
      • Works under any OS
      • Can be accessed via SSH, or via Java applet
    • Allows remote power cycling
    • the iLO appears to be a little mini-linux box

General HP Info

Initial Ubuntu Installation

  • Initial Ubuntu installation should go smoothly, the drive controller and network controllers should be detected properly
  • Recommend doing a server installation
  • We used LVM for all of the partitions except root and swap

  • Would recommend installation of ia32-libs if you plan on running any 32-bit JVMs or other 32-bit packages

Kernel

  • Install package linux-amd64-k8-smp

  • After you have rebooted, remove linux-amd64-generic and linux-image-amd64-generic packages

  • After you have removed those packages, you should be able to aptitude update, aptitude upgrade your system without getting extra kernels you don't want

Raid Controller Tools & Monitoring

  • You'll need something to monitor for raid drive failure. The kernel module will not report any nastyness that may be happening with your drives. There's a number of different methods of monitoring this. I've elected to install the HP ACU CLI (Array Configuration Utility - Command Line Interface) and write a quick monitoring script to watch for drive failure. I've elected to do it this way rather than using some of the more sophisticated HP utilties that they provide because I have been burned by closed-source hardware tools in the past (FWIW, not by HP).

  • Visit HP's site and download the HP ACU CLI for Redhat Enterprise Server 4. It'll be an rpm. The filename will look something like hpacucli-7.30-9.linux.rpm. You'll have to convert this to a TGZ using alien, install the package manually, install some scarry 32bit libraries, and update some of the scripts to get this to thing to work.

  • {{{alien -t hpacucli-7.30-9.linux.rpm

mkdir tmp cd tmp tar -xzvf ../hpacucli-7.30.tgz cd opt mv compaq /opt cd usr/sbin mv hpacucli /usr/sbin cd ../../.. rm -rf tmp }}}

  • You'll need the ia32-libs package installed

  • At this point you should be getting this error:

/opt/compaq/hpacucli/bld/.hpacucli: error while loading shared libraries: libstdc++-libc6.2-2.so.3: cannot open shared object file: No such file or directory

{{{ wget http://gr.archive.ubuntu.com/ubuntu/pool/universe/g/gcc-2.95/libstdc++2.10-glibc2.2_2.95.4-22_i386.deb alien -t libstdc++2.10-glibc2.2_2.95.4-22_i386.deb mkdir tmp tar -xzvf libstdc++2.10-glibc2.2-2.95.4.tgz cd tmp/usr/lib mv * /usr/lib32 cd ../../.. rm -rf tmp }}}

  • Modify the /usr/sbin/hpacucli script and modify the LD_LIBRARY_PATH and put '/usr/lib32:' infront of what's currently there.. EG: export LD_LIBRARY_PATH=/usr/lib32:$HPACUCLI_BIN_INSTALLATION_DIR

  • hpacucli should work now. The cli isn't super, and doesn't include tab completion, or command history.. But it's functional

RAID Monitoring Tools

  • Now that the CLI is installed, it can be used to monitor the state of the RAID containers and drives

  • I have written/adapted a small script for monitoring the drives, I called it hpsa6i.sh 

    • ##Monitoring script RAID on hp smart array controller 6i
emaillist=raid@xxx.com

host=`hostname`
/bin/echo -e "set target controller slot=0\nshow status\narray A physicaldrive all show status\narray A logicaldrive all show status" | hpacucli > /tmp/raid-current
curdiff=`/usr/bin/diff -u /usr/share/hpsa6i/raid-good /tmp/raid-current`
currentinfo=`cat /tmp/raid-current`
rm /tmp/raid-current

if [ "$curdiff" != "" ] ; then
# Send email alert
        /usr/lib/sendmail -f devnull@$host -t << HPsa6iRAIDEOF
To: $emaillist
From: root@$host     
Subject: Raid may be broken on $host 
Errors-To: devnull
Raid may be broken on $host. 

====> A diff between production and current is:

$curdiff

====> Full Raid Current Info:

$currentinfo

HPsa6iRAIDEOF

fi
  • You'll want to customize that script as appropriate including
    • Email address
    • slot #
    • Array #

  • Create the /usr/share/hpsa6i directory

  • Create the initial raid-okay state file: 

    • cd /usr/share/hpsa6i
/bin/echo -e "set target controller slot=0\nshow status\narray A physicaldrive all show status\narray A logicaldrive all show status" | hpacucli > raid-good
  • Add to root's cronjob 
    • */30 * * * * /usr/local/bin/hpsa6i.sh

iLO

  • HP has a slick little monitoring board built into the DL385 (and many of their other servers)
  • I believe it's a mini little machine (running linux I believe) that has the capability of controlling the machine remotely, including remote console access
  • It has the following features
    • Ethernet access
    • Simulated console access (hooks into the bios to see what's on the screen. TEXT modes are free, graphics modes need a special license)
      • a serial console is probably more reliable than it's text remote console.. but it does work
    • ssh access
    • https / java clients
    • Remote power cycle, diagnostics
  • There is a bug in the iLO ssh server that will cause instability with Ubuntu Hoary
    • Ubuntu Hoary, is configured to send over the LANG environment variable whenever you log into a remote machine. the iLO ssh server crashes if you send over any environment variables when you establish a connection.

      • This has been reported to HP and has a case number of 3211317810, presumably they will have to issue an iLO firmware update to resolve the issue

      • You can work around this issue by modifying your /etc/ssh/ssh_config and removing the SendEnv LANG LC_* line

Java - 64bit/32bit JVMs

  • I have very little milage on the Java JVMs on these machines at this point. I will update this page if we discover any issues during our burnin period for these servers.
  • We install a potpourri of JVMs to be able to easily swap between them
  • we create a symlink /usr/local/java to point to one of the java packages below, then have add /usr/local/java to everybody's path
    • This makes it easy to swap the entire system between JVMs

IBM JVM 1.4.2 AMD64 (64-bit)

IBM JVM 1.4.2 i386 (32-bit)

Blackdown 1.4.2 AMD64 (64-bit)

Sun 1.5.0 AMD64 (64-bit)

Sun 1.4.2 i386 (32-bit)

!MySQL - 64bit

  • I have very little milage on MySQL on these machines at this point. I will update this page if we discover any issues during our burnin period for these servers.

  • Install package mysql-server

    • We run the MySQL stock binary, not the ubuntu binary. I am sure many of the ubuntu people may be confused by this, but we have been very successfully doing this on our 32-bit Debian machines for some time. This has the advantage that when we run into problems with MySQL, we can eliminate the distro as part of the problem. MySQL 32-bit binaries are statically linked, however their 64-bit binaries do not appear to be. This may end up changing our strategy on running MySQL standard binary, but I will update here based on our experiences.

      • http://dev.mysql.com/downloads/mysql/4.0.html

      • We are running MySQL 4.0 because of some issues with some incompatible changes in the MySQL 4.1 release
      • Download Linux (AMD64 / Intel EM64T, glibc-2.3, dynamic, gcc)
        • I have installed the ICC libraries and MySQL ICC... and it seemed to work, but I chickened out from acctually using it
      • unpack the tar.gz to a temporary directory
      • {{{ /etc/init.d/mysql shutdown

cd /usr/sbin mv mysqld mysqld.ubuntu mv /tmp/mysql....../bin/mysqld . cd /etc ln -s mysql/my.cnf my.cnf /etc/init.d/mysql mysql -u root -p status }}}

  • When you run status you should see 4.0.24-standard-log or something similar, and not 4.0.23_Debian-3ubuntu2-log
  • MySQL 64bit is not stable on hoary, and will fail regression at this time due to a glibc bug:
  • Another issue is present in the MySQL 64bit running under hoary, where the MySQLd will report an out of memory condition... with gigabytes of memory remaining.
    • This issue does not ocour under the 32-bit version
    • I will be trying glibc 2.3.4 to see if it resolves this issue
    • 64-bit MySQL is not stable under hoary, and we will be running 32-bit MySQL until this issue has a resolution

Other HP Utilities

  • I don't have any of the other HP utilties for linux installed other than the ACU CLI. Undoubtedly many of them work and are useful. Some of them are even compiled for x86-64 (YAY!). Anybody who has any feedback of what they use them for, may feel free to comment on them here. I got burned pretty badly by some MegaRaid2 monitoring utilities that crashed our kernels, so I now shy away from closed-source-scarry-vendor tools.

Server Burnin

HPDL385 (last edited 2009-08-24 12:54:03 by unibz)