Here’s a weird one for you. We were trying to install 6.1.0.3 on a Solaris 10 system to do some tests. The Portal install would fail after about 10 minutes. In the /tmp/wpinstalllog.txt file, it was clear that the problem was due to a failure in the internal WebSphere Application Server install. (When you install Portal, the Portal installer will kick off it’s own silent install of WAS).
The first thing to do when debugging a WAS install problem is to look at the logs in ~/waslogs . These indicated the following problem:
CWUPI0033E:
There is insufficient free disk space on the system:
/opt/WebSphere/AppServer:
Required: 1403 MB
Available: 0 MB
/var/tmp/:
Required: 1403 MB
Available: 0 MB
/opt/.ibm/.nif:
Required: 2 MB
Available: 0 MB
Please ensure that there is enough free disk space
on all required filesystems and restart the installation.
If /var/tmp/ , /opt/WebSphere/AppServer
and /opt/.ibm/.nif are on the same partition,
then the amount of space required is the sum of the space
required on /var/tmp/ , /opt/WebSphere/AppServer and
/opt/.ibm/.nif.
My system had heaps of space on it! Surely the installer wouldn’t even run if there was 0 MB free! The method that the installer used to determine how much disk space was free was failing. But how does the installer figure out how much disk space is free? After lots of poking and prodding around I stumbled on dtrace. I had heard of it before, but never had the opportunity to use it. Dtrace is a mechanism to instrument and probe the tiniest little interactions on a Solaris/BSD/OSX machine. Being so powerful, it has a steep learning curve. This collection of handy dtrace oneliners was really helpful.
I kicked off the WAS install portion of the Portal install and ran this dtrace command in another window.
dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }' -o trace.log
It captured each file interaction that occurred when running the install. Luckily the WAS install failed after about 30 seconds, so there wasn’t too much data to wade through.
Here is the dtrace log (trace.log from the command above). Something called gushellsupport.sh is calling df (standard unix disk free command). This must be how the installer determines how much disk space is free. The column on the left is the pid of the install process (which is java) . The library files on the far right are what is being called by each executable; the next column over to the left.
0 44056 open64:entry gushellsupport.s /var/tmp/ismp003/gushellsupport.sh
0 43668 open:entry df /var/ld/ld.config
0 43668 open:entry df /lib/libcmd.so.1
0 43668 open:entry df /lib/libc.so.1
0 43668 open:entry df /usr/dt/lib/nls/msg/C/SUNW_OST_OSCMD.cat
0 43668 open:entry df /usr/lib/locale/C/LC_MESSAGES/SUNW_OST_OSCMD.mo
0 43668 open:entry df /var/ld/ld.config
0 43668 open:entry df /lib/libcmd.so.1
0 43668 open:entry df /lib/libc.so.1
0 43668 open:entry df /etc/mnttab
0 43668 open:entry df /usr/dt/lib/nls/msg/C/SUNW_OST_OSCMD.cat
0 43668 open:entry df /usr/lib/locale/C/LC_MESSAGES/SUNW_OST_OSCMD.mo
This script, gushellsupport.sh, is owned by InstallShield so I can’t publish the contents of it. But it has a diskcheck function in it that relies on ‘/usr/xpg4/bin/df’ which I didn’t have installed. Solaris has many different versions of the same tools that are left behind for backwards compatibility. When installing this system initially, I used the “Core System Support” option in the Solaris install to build a lean, quick machine. Unfortunately it didn’t come with this legacy version of df.
df belongs in a package called SUNWxcu4. To install it, mount your Solaris CD and go to the directory Solaris_10/Product/ . In there, copy the subdirectory ‘SUNWxcu4’ to /var/spool/pkg and run
pkgadd SUNWxcu4
If you rerun the install again it’ll work since gushellsupport.sh is calling the correct version of df. Talk about obscure huh?
