To improve the performance of applications running MPI jobs on compute nodes, most services are disabled by default in compute node images. To see what adjustments are being made, view the /etc/opt/sgi/conf.d/80-compute-distro-services script.

If you wish to change anything in this script, SGI suggests that you copy the existing script to .local and adjust it there. Perform the following commands:

# cd /var/lib/systemimager/images/compute-image-name
# cp etc/opt/sgi/conf.d/80-compute-distro-services 80-compute-distro-services.local
# vi etc/opt/sgi/conf.d/80-compute-distro-services.local

At this point, the configuration framework will execute the .local version, and skip the other. For more information on making adjustments to configuration framework files, see “SGI Altix ICE System Configuration Framework”.

Use the cimage command to push the changed image out to the leader nodes.

You can use the crepo command to manage software repositories, such as, SGI Foundation, SMC for Altix ICE, SGI Performance Suite, and the Linux distribution(s) you are using on your system. You also use the crepo command to manage any custom repositories you create yourself.

The configure-cluster command calls the crepo command when it prompts you for media and then makes it available. You can also use the crepo command to add additional media.

Each repository has associated with it a name, directory, update URL, selection status, and suggested package lists. The update URL is used by the sync-repo-updates command. The directory is where the actual yum repository exists, and is located in one of these locations, as follows:

The repository information is determined from the media itself when adding media supplied by SGI, Linux distribution media (SLES, RHEL, and so on.), and any other YaST-compatible media. For customer-supplied repositories, the information must be provided to the crepo command when adding the repository.

Repositories can be selected and unselected. Usually, SMC for Altix ICE commands ignore unselected repositories. One notable exception is that sync-repo-updates always operates on all repositories.

The crepo command constructs default RPM lists based on the suggested package lists. The RPM lists can be used by the cinstallman command when creating a new image. These RPM lists are only generated if a single distribution is selected and can be found in /etc/opt/sgi/rpmlists; they match the form generated- *.rpmlist. The crepo command will tell you when it updates or removes generated rpmlists. For example:

# crepo --select SUSE-Linux-Enterprise-Server-10-SP3
Updating: /etc/opt/sgi/rpmlists/generated-compute-sles10sp3.rpmlist
Updating: /etc/opt/sgi/rpmlists/generated-service-sles10sp3.rpmlist

When generating the RPM lists, the crepo command combines the a list of distribution RPMs with suggested RPMs from every other selected repository. The distribution RPM lists are usually read from the /opt/sgi/share/rpmlists/distro directory. For example, the compute node RPM list for sles11sp1 is /opt/sgi/share/rpmlists/distro/compute-distro-sles11sp1.rpmlist . The suggested RPMs for non-distribution repositories are read from the /var/opt/sgi/sgi-repodata directory. For example, the rpmlist for SLES 11 SP1 compute nodes is read from /var/opt/sgi/sgi-repodata/SMC-for-ICE 1.0-for-Linux-sles11/smc-ice-compute.rpmlist .

The suggested rpmlists can be overridden by creating an override rpmlist in the /etc/opt/sgi/rpmlists/override/ directory. For example, to change the default SMC for ICE 1.0 suggested rpmlist, a file /etc/opt/sgi/rpmlists/override/SMC-for-ICE-1.0 -for-Linux-sles11/smc-ice-compute.rpmlist can be created.

The following example shows the contents of the /etc/opt/sgi/rpmlists directory after the crepo command has created the suggested RPM lists. Change directory (cd) to the /etc/opt/sgi/rpmlists directory. Use the ls command to see a list of rpms, as follows:

admin distro]# ls
compute-distro-centos5.4.rpmlist  lead-distro-sles11sp1.rpmlist
compute-distro-rhel5.4.rpmlist    service-distro-rhel5.4.rpmlist
compute-distro-rhel5.5.rpmlist    service-distro-rhel5.5.rpmlist
compute-distro-rhel6.0.rpmlist    service-distro-rhel6.0.rpmlist
compute-distro-sles10sp3.rpmlist  service-distro-sles10sp3.rpmlist
compute-distro-sles11sp1.rpmlist  service-distro-sles11sp1.rpmlist
lead-distro-rhel6.0.rpmlist

Specifically, SMC for Altix ICE software looks for /etc/opt/sgi/rpmlists/generate-*.rpmlist and creates an image for each rpmlist that matches.

It also determines the default image to use for each node type by hard-coding "$nodeType-$distro" as the type, where distro is the admin node's distro and nodeType is compute, service, leader, and so on. The default image can be overridden by specifying a global cattr attribute named image_default_$nodeType; for example, image_default_service. Use cattr -h, for information about the cattr command.

The following example shows the contents of the /etc/opt/sgi/rpmlists directory after the crepo command has created the suggested RPM lists. The files with -distro- in the name are the base Linux distro RPMs that SGI recommends.

Change directory (cd) to /etc/opt/sgi/rpmlists. Use the ls command to see a list of rpms, as follows:

admin:/etc/opt/sgi/rpmlists # ls
compute-minimal-sles11sp1.rpmlist  generated-lead-rhel6.0.rpmlist
generated-compute-rhel6.0.rpmlist  generated-service-rhel6.0.rpmlist

For more information on rpmlist customization information, see “Creating Compute and Service Node Images Using the cinstallman Command”.

For a crepo command usage statement, perform the following:

admin:~ # crepo --h
crepo Usage:
Operations:
--help                : print his usage message

--add {path/URL}      : add SGI/SMC media to the system repositories
       --custom {name}: Optional.Use with -add to add custom repo under
                        /tftpboot Repo must pre-exist for this case.

--del {product}       : delete an add-on product and associated /tftpboot repo

--select {product}    : mark the product as selected

--show                : show available add-on products

--show-distro         : like show, but only reports distro media like sles10sp2

--show-updateurls     : Show the update sources associated add-on products

--reexport            : re-export all repositories with yume.  Use if there
                        was a yume export problem previously.

--unselect {product}  : mark the product as not selected


Flags:

Note for --add: If the pathname is local to the machine, it can be an
ISO file or mounted media.  If a network path is used -- such as an nfs
path or a URL -- the path must point to an ISO file.  The argument to
--add may be a comma delimited list to specify multiple source media.

Use --add for SGI/SMC media, to make the repos and rpms available. If the
supplied SGI/SMC media has suggested rpms from SMC node types, those
suggested rpms will be integrated with the default rpmlists for leader,
service, and compute nodes.  You can use create-default-sgi-images to
re-create the default images including new suggested packages or you can
just browse the updated versions in /etc/opt/sgi/rpmlists.

Use --add with --custom to register your own custom repository.  This will
ensure that, by default, the custom repository is available to yume and
mksiimage commands.  It is assumed you will maintain your own default package
lists, perhaps using the sgi default package lists in /etc/opt/sgi/rpmlists
or /opt/sgi/share/rpmlists as a starting point.  The directory and rpms within
must pre-exist.  This script will create the yum metadata for it.
Example:
crepo --add /tftpboot/myrepo --custom my-custom-name

The cinstallman command is a wrapper tool for several SMC for Altix ICE operations that previously ran separately. You can use the cinstallman command to perform the following:

When compute images are created for the first time, a bittorrent tarball is also created. When images are pushed to rack leaders for the first time, bittorrent is used to transport the tarball snapshot of the image. However, as you make adjustments to your compute image, those changes do not automatically generate a new bittorrent tarball. We handle that situation by always doing a follow-up rsync of the compute image after transporting the tarball. However, as your compute image begins to diverge from the bittorrent tarball snapshot, it becomes less and less efficient to transport a given compute node image that is new to a given rack leader.

You no longer need to use yum, yume , or mksiimage commands directly for most common operations. Compute images are automatically configured in such a way as to make them available to the cimage command.

For a cinstallman command usage statement, perform the following:

admin:~ # cinstallman --help
cinstallman Usage:

cinstallman is a tool that manages:
 - image creation (as a wrapper to mksiimage)
 - node package updates (as a wrapper to yume)
 - image package updates (yume within a chroot to the image)

This is a convenience tool and not all operations for the commands that are
wrapped are provided.  The most common operations are collected here for
ease of use.

For operations that take the --node parameter, the node can be an aggregation
of nodes like cimage and cpower can take.  Depending on the situation,
non-managed or offline nodes are skipped.

The tool retrieves the registered repositories from crepo so that they
need not be specified on the command line.

Operations:
--help                  : print his usage message
--create-image          : create a new systemimager image
                          By default, requires --rpmlist and --image
                          Optional flags  below:
       --clone          : Clone existing image, requires --source, --image.
                          Doesn't require --rpmlist.
       --recreate       : Like --del-image then --add-image, but preserves any
                          node associations.
                          Requires --image and --rpmlist
       --repos {list}   : A comma-seperated list of repositories to use.
       --use-existing   : register an already existing image, doesn't
                          require --rpmlist
       --image {image}  : Specify the image to operate on
       --rpmlist {path} : Provide the rpmlist to use when creating images
       --source {image} : Specify a source image to operate on (for clone)

--del-image             : delete the image, may use with --del-nodes
       --image {image}  : Specify the image to operate on

--show-images           : List available images (similar to mksiimage -L)

--show-nodes            : Show non-compute nodes (similar to mksimachine -L)

--update-image          : update packages in image to latest packages available
                          in repos, Requires --image
       --image {image}  : Specify the image to operate on

--refresh-image         : Refresh the given image to include all packages
                          in the supplied rpmlist.  Use after registering
                          new media with crepo that has new suggested rpms.
       --image {image}  : Specify the node or nodes to operate on
       --rpmlist {path} : rpmlist containing packes to be sure are included

--yum-image             : Perform yum operations to supplied image, via yume
                          Requires --image, trailing arguments passed to yume
       --image {image}  : Specify the image to operate on

--update-node           : Update supplied node to latest pkgs avail in
                          repos, requires --node
       --node {node}    : Specify the node or nodes to operate on

--refresh-node          : Refresh the given node to include all packages
                          in the supplied rpmlist.  Use after registering
                          new media with crepo that has new suggested rpms.
       --node {node}    : Specify the node or nodes to operate on
       --rpmlist {path} : rpmlist containing packes to be sure are included

--yum-node              : Perform yum operations to nodes, via yume.  Requires
                          --node.  Trailing arguments passed to yume
       --node {node}    : Specify the node or nodes to operate on

--assign-image          : Assign image to node.  Requires --node, --image
       --node {node}    : Specify the node or nodes to operate on
       --image {image}  : Specify the image to operate on

--next-boot {image|disk}: node action next boot: boot from disk or
                          reinstall/reimage?  Requires --node

--refresh-bt            : Refresh the bittorrent tarball and torrent file
                          Requires --image
       --image {image}  : Specify the image to operate on

In the following example, the --refresh-node operation is used to ensure the online managed service nodes include all the packages in the list. You could use this if you updated your rpmlist to include new packages or if you recently added new media with the crepo command and want running nodes to have the newly updated packages. A similar --refresh-image operation exists for images.

# cinstallman --refresh-node --node service\* --rpmlist
/etc/opt/sgi/rpmlists/service-sles11.rpmlist

This section discusses how to manage various nodes on your SGI Altix ICE system. It describes how to configure the various nodes, including the compute and service nodes. It describes how to augment software packages. Many tasks having to do with package management have multiple valid methods to use.

For information on installing patches and updates, see “Installing SMC for Altix ICE Patches and Updating SGI Altix ICE Systems ” in Chapter 2.

Creating Compute Node Custom Images

You can add per-host compute node customization to the compute node images. You do this by adding scripts either to the /opt/sgi/share/per-host-customization/global/ directory or the /opt/sgi/share/per-host-customization/mynewimage/ directory on the system admin controller.

---

Note: When creating custom images for compute nodes, make sure you clone the original SGI images. This provides the original images intact that you can fall back to if necessary. The folowing example is based on SLES.

---

Scripts in the global directory apply to all compute nodes images. Scripts under the image name apply only to the image in question. The scripts are cycled through once per host when being installed on the rack leader controllers. They receive one input argument, which is the full path (on the rack leader controller) to the per-host base directory, for example, /var/lib/sgi/mynewimage/i2n11. There is a README file at /opt/sgi/share/per-host-customization/README on the system admin controller, as follows:

This directory contains compute node image customization scripts which are executed as part of the install-image operations on the leader nodes when pulling over a new compute node image. After the image has been pulled over, and the per-host-customization dir has been rsynced, the per-host /etc and /var directories are populated, then the scripts in this directory are cycled through once per-host. This allows the scripts to source the node specific network and cluster management settings, and set node specific settings. Scripts in the global directory are iterated through first, then if a directory exists that matches the image name, those scripts are iterated through next. You can use the scripts in the global directory as examples.

An example global script, /opt/sgi/share/per-host-customization/global/sgi-fstab is, as follows:

#!/bin/sh # # Copyright (c) 2007,2008 Silicon Graphics, Inc. # All rights reserved. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # # Set up the compute node's /etc/fstab file. # # Modify per your sites requirements. # # This script is excecuted once per-host as part of the install-image operation # run on the leader nodes, which is called from cimage on the admin node. # The full path to the per-host iru+slot directory is passed in as $1, # e.g. /var/lib/sgi/per-host/<imagename>/i2n11. # # sanity checks . /opt/sgi/share/per-host-customization/global/sanity.sh iruslot=$1 os=( $(/opt/oscar/scripts/distro-query -i ${iruslot} | sed -n '/^compat /s/^compat.*: //p') ) compatdistro=${os[0]}${os[1]} if [ ${compatdistro} = "sles10" -o ${compatdistro} = "sles11" ]; then # # SLES 10 compatible # cat <<EOF >${iruslot}/etc/fstab # <file system> <mount point> <type> <options> <dump> <pass> tmpfs /tmp tmpfs size=150m 0 0 EOF elif [ ${compatdistro} = "rhel5" ]; then # # RHEL 5 compatible # # # RHEL expects several subsys directories to be present under /var/run # and /var/lock, hence no tmpfs mounts for them # cat <<EOF >${iruslot}/etc/fstab # <file system> <mount point> <type> <options> <dump> <pass> tmpfs /tmp tmpfs size=150m 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 EOF else echo -e "\t$(basename ${0}): Unhandled OS. Doing nothing" fi

% cattr set kernel_extra_params-compute-sles11sp1 cgroup_disable=memory

# cimage --push-rack mynewimage r1

Compute Node Per-Host Customization for Additional Network Interfaces

---

Note: The following example is only for systems running SLES.

---

Per compute-node customization may be useful for configuring additional network interfaces that are on some, but not all, compute nodes. An example of how to configure network interfaces on individual compute nodes is the /opt/sgi/share/per-host-customization/mynewimage/mycustomization script, that follows:

Copyright (c) 2008 Silicon Graphics, Inc. # All rights reserved. # # do node specific setup # # This script is excecuted once per-host as part of the install-image operation # run on the leader nodes, which is called from cimage on the admin node. # The full path to the per-host iru+slot directory is passed in as $ARGV[0], # e.g. /var/lib/sgi/per-host/<imagename>/i2n11. # use lib "/usr/lib/systemconfig","/opt/sgi/share/per-host-customization/global"; use sanity; sanity_checks(); $blade_path = $node = $ARGV[0]; $node =~ s/.*\///; sub i0n4 { my $ifcfg="etc/sysconfig/network/ifcfg-eth2"; open(IFCFG, ">$blade_path/$ifcfg") or die "$0: can't open $blade_path/$ifcfg"; print IFCFG<<EOF BOOTPROTO='static' IPADDR='10.20.0.1' NETMASK='255.255.0.0' STARTMODE='onboot' WIRELESS='no' EOF ; close(IFCFG); } @nodes = ("i0n4"); foreach $n (@nodes) { if ( $n eq $node ) { eval $n; } }

Pushing mynewimage to rack 1 causes the eth2 interface of compute node r1i0n4 to be configured with IP address 10.20.0.1 when the node is brought up with mynewimage . Push the image, as follows:

# cimage --push-rack mynewimage r1

Customizing Software Images

---

Note: Procedures in this section describe how to work with service node and compute node images. Always use a cloned image. If you are adjusting an RPM list, use your own copy of the RPM list.

---

The service and compute node images are created during the configure-cluster operation (or during your upgrade from a prior release). This process uses an RPM list to generate a root on the fly.

You can clone a compute node image, or create a new one based on an RPM list. For service nodes, SGI does not support a clone operation. For compute images, you can either clone the image and work on a copy or you can always make a new compute node image from the SGI supplied default RPM list.

Procedure 3-1. Creating a Simple Compute Node Image Clone

---

Note: Always work from a clone image, see “Customizing Software Images”.

---

To create a simple compute node image clone from the system admin controller, perform the following steps:

1. To clone the compute node image, perform the following:

   # cinstallman --create-image --clone --source compute-sles11 --image compute-sles11-new

   
   

2. To see the images and kernels in the list, perform the following:

   # cimage --list-images image: compute-sles11 kernel: 2.6.27.19-5-smp image: compute-sles11-new kernel: 2.6.27.19-5-smp

   
   

3. To push the compute node image out to the rack, perform the following:

   # cimage --push-rack compute-sles11-new r\*

   
   

4. To change the compute nodes to use the cloned image/kernel pair, perform the following:

   # cimage --set compute-sles11-new 2.6.27.19-5-smp "r*i*n*"

   
   

Procedure 3-2. Manually Adding a Package to a Compute Node Image

To manually add a package to a compute node image, perform the steps:

---

Note: Use the cinstallman command to install packages into images when the package you are adding is in a repository. This example shows a quick way to manually add a package for compute nodes when you do not want the package to be in a custom repository. For information on the cinstallman command, see “cinstallman Command”.

---

1. Make a clone of the compute node image, as described in “Customizing Software Images”.

2. ---

   Note: This example shows SLES11.

   ---

   
   Determine what images and kernels you have available now, as follows:

   # cimage --list-images image: compute-sles11 kernel: 2.6.27.19-5-smp image: compute-sles11-new kernel: 2.6.27.19-5-smp

   
   

3. From the system admin controller, change directory to the images directory, as follows:

   # cd /var/lib/systemimager/images/

   
   

4. From the system admin controller, copy the RPMs your wish to add, as follows, where compute-sles11-new is your own compute node image, as follows:

   # cp /tmp/newrpm.rpm compute-sles11-new/tmp

   
   

5. The new RPMs now reside in /tmp directory in the image named compute-sles11-new. To install them into your new compute node image, perform the following commands:

   # chroot compute-sles11-new bash

   
   

   And then perform the following:

   # rpm -Uvh /tmp/newrpm.rpm

   
   

   At this point, the image has been updated with the RPM.

   ---

   Note: Remove the RPMs or ISO images before pushing an image or the RPM/ISO will be pushed multiple times for each image slowing down the push and even filling up the root of the RLC (leader node). # rm /tmp/newrpm.rpm

   ---

   
   

6. The image on the system admin controller is updated. However, you still need to push the changes out. Ensure there are no nodes currently using the image and then run this command:

   # cimage --push-rack compute-sles11-new r\*

   
   

   This will push the updates to the rack lead controllers and the changes will be seen by the compute nodes the next time they start up. For information on how to ensure the image is associated with a given node, see the cimage --set command and the example in Procedure 3-1.

Procedure 3-3. Manually Adding a Package to the Service Node Image

To manually add a package to the service node image, perform the following steps:

---

Note: Use the cinstallman command to install packages into images when the package you are adding is in a repository. This example shows a quick way to manually add a package for compute nodes when you do not want the package to be in a custom repository. For information on the cinstallman command, see “cinstallman Command”.

---

1. Use the cinstallman command to create your own version of the service node image. See “cinstallman Command”.

2. Change directory to the images directory, as follows:

   # cd /var/lib/systemimager/images/

   
   

3. From the system admin controller, copy the RPMs your wish to add, as follows, where my-service-image is your own service node image:

   # cp /tmp/newrpm.rpm my-service-image/tmp

   
   

4. The new RPMs now reside in /tmp directory in the image named my-service-image. To install them into your new service node image, perform the following commands:

   # chroot my-service-image bash

   
   And then perform the following:

   # rpm -Uvh /tmp/newrpm.rpm

   
   

   At this point, the image has been updated with the RPM. Please note, that unlike compute node images, changes made to a service node image will not be seen by service nodes until they are reinstalled with the image. If you wish to install the package on running systems, you can copy the RPM to the running system and use the RPM from there.

The cimage command allows you to list, modify, and set software images on the compute nodes in your system.

For a help statement, perform the following command:

admin:~ # cimage --help
cimage is a program for managing compute node root images in SMC for ICE.

Usage: cimage OPTION ...

Options
 --help                                 Usage and help text.
 --debug                                Output additional debug information.
 --list-images                          List images and their kernels.
 --list-nodes NODE                      List node(s) and what they are set to.
 --set [OPTION] IMAGE KERNEL NODE       Set node(s) to image and kernel.
    --nfs                               Use NFS roots (default).
    --tmpfs                             Use tmpfs roots.
 --set-default [OPTION] IMAGE KERNEL    Set default image, kernel, rootfs type.
    --nfs                               Use NFS roots (default).
    --tmpfs                             Use tmpfs roots.
 --show-default                         Show default image, kernel, rootfs type.
 --add-db IMAGE                         Add image and its kernels to the db.
 --del-db IMAGE                         Delete image and its kernels from db.
 --update-db IMAGE                      Short-cut for --del-db, then --add-db.
 --push-rack [OPTIONS] IMAGE RACK       Push or update image on rack(s).
    --force                             Bypass the booted nodes check, deletes.
    --update-only                       Skip files newer in dest, no delete.
    --quiet                             Turn off diagnostic information.
 --del-rack IMAGE RACK                  Delete an image from rack(s).
 --clone-image OIMAGE NIMAGE            Clone an existing image to a new image.
 --del-image [OPTIONS] IMAGE            Delete an existing image entirely.
    --quiet                             Turn off diagnostic information.

RACK arguments take the format 'rX'
NODE arguments take the format 'rXiYnZ'
ROOTFS argument can be either 'nfs' or 'tmpfs'

X, Y, Z can be single digits, a [start-end] range, or * for all matches.

EXAMPLES

Example 3-1. cimage Command Examples

The following examples walk you through some typical cimage command operations.

To list the available images and their associated kernels, perform the following:

# cimage --list-images image: compute-sles11 kernel: 2.6.27.19-5-carlsbad kernel: 2.6.27.19-5-default image: compute-sles11-1_7 kernel: 2.6.27.19-5-default

To list the compute nodes in rack 1 and the image and kernel they are set to boot, perform the following:

# cimage --list-nodes r1 r1i0n0: compute-sles11 2.6.27.19-5-default nfs r1i0n8: compute-sles11 2.6.27.19-5-default nfs

The cimage command also shows the root filesystem type (nfs or tmpfs)

To set the r1i0n0 compute node to boot the 2.6.27.19-5-smp kernel from the compute-sles11 image, perform the following: :

# cimage --set compute-sles11 2.6.27.19-5-smp r1i0n0

To list the nodes in rack 1 to see the changes set in the example above, perform the following:

# cimage --list-nodes r1 r1i0n0: compute-sles11 2.6.27.19-5-smp r1i0n1: compute-sles11 2.6.27.19-5-smp r1i0n2: compute-sles11 2.6.27.19-5-smp [...snip...]

To set all nodes in all racks to boot the 2.6.27.19-5-smp kernel from the compute-sles11 image, perform the following:

# cimage --set compute-sles11 2.6.27.19-5-smp r*i*n*

To set two ranges of nodes to boot the 2.6.27.19-5-smp kernel, perform the following:

# cimage --set compute-sles11 2.6.27.19-5-smp r1i[0-2]n[5-6] r1i[2-3]n[0-4]

To clone the compute-sles11 image to a new image (so that you can modify it) , perform the following:

# cinstallman --create-image --clone --source compute-sles11 --image mynewimage Cloning compute-sles11 to mynewimage ... done

The clone process adds the image and its kernels to the database.

---

Note: If you have made changes to the compute node image and are pushing that image out to leader nodes, it is a good practice to use the cinstallman --refresh-bt --image { image} command to refresh the bittorrent tarball and torrent file for a compute node image. This avoids duplication by rsync when the image is pushed out to the leader nodes. For more information, see the cinstallman -h usage statement or “cinstallman Command”.

---

To change to the cloned image created in the example, above, copy the needed rpms into the /var/lib/systemimager/images/mynewimage/tmp directory, use the chroot command to enter the directory and then install the rpms, perform the following:

# cp *.rpm /var/lib/systemimager/images/mynewimage/tmp # chroot /var/lib/systemimager/images/mynewimage/ bash # rpm -Uvh /tmp/*.rpm

If you make changes to the kernels in the image, you need to refresh the kernel database entries for your image, To do this, perform the following:

# cimage --update-db mynewimage

If you did not make changes to the kernels in the cloned image created in the example above, you can omit this step.

To push new software images out to the compute blades in a rack or set of racks, perform the following:

# cimage --push-rack mynewimage r* r1lead: install-image: mynewimage r1lead: install-image: mynewimage done.

To list images in the database the kernels they contain, perform the following:

# cimage --list-images image: compute-sles11 kernel: 2.6.16.60-0.7-carlsbad kernel: 2.6.16.60-0.7-smp image: mynewimage kernel: 2.6.16.60-0.7-carlsbad kernel: 2.6.16.60-0.7-smp

To set some compute nodes to boot an image, perform the following:

# cimage --set mynewimage 2.6.16.60-0.7-smp r1i3n*

You need to reboot the compute nodes to run the new images.

Completely remove an image you no longer use, both from system admin controller and all compute nodes in all racks, perform the following:

# cimage --del-image mynewimage r1lead: delete-image: mynewimage r1lead: delete-image: mynewimage done.

The packages that make up SMC for Altix ICE, SGI Foundation, and the Linux distribution media, and any other media or custom repositories you have added reside in repositories. The cinstallman command looks up the list of all repositories and provides that list to the commands it calls out for its operation such as yume.

---

Note: Always work with copies of software images.

---

The cinstallman command can update packages within systemimager images. You may also use cinstallman to install a single package within an image.

However, cinstallman and the commands it calls only works with the configured repositories. So if you are installing your own RPM, you will need that package to be part of an existing repository. You may use the crepo command to create a custom repository into which you can collect custom packages.

---

Note: The yum command maintains a cache of the package metadata. If you just recently changed the repositories, yum caches for the nodes or images you are working with may be out of date. In that case, you can issue the yum command "clean all" with --yum-node and --yum-image. The cinstallman command --update-node and --update-image options do this for you.

---

# cinstallman --yum-image --image my-service-sles11 install zlib-devel

You can perform a similar operation for compute node images. Note the following:

---

Note: These instructions only apply to managed service nodes and leader nodes. They do not apply to compute nodes.

---

You can use the yum command to install a package on a service node. From the admin node, you can issue a command similar to the following:

# cinstallman --yum-node --node service0 install zlib-devel

---

Note: To get all service nodes, replace service0 with service\*.

---

For more information on the cinstallman command, see “cinstallman Command”.

You can create service node and compute node images using the cinstallman command. This generates a root directory for images, automatically.

Fresh installations of SMC for Altix ICE create these images during the configure-cluster installation step (see “Installing SMC for Altix ICE Admin Node Software ” in Chapter 2).

The RPM lists that drive which packages get installed in the images are listed in files located in /etc/opt/sgi/rpmlists. For example, /etc/opt/sgi/rpmlists/compute-sles11.rpmlist (see “crepo Command”). You should NOT edit the default lists. These default files are recreated by the crepo command when repositories are added or removed. Therefore, you should only use the default RPM lists as a model for your own.

---

Note: The procedure below uses SLES.

---

To create a service node image using the cinstallman command, perform the following steps:

To create a compute node image using the cinstallman command, perform the following steps:

If you have a non-default service node image you wish to install on a service node, you have two choices, as follows:

The following example shows how to associate a custom image at discover time:

# discover --service 2,image=my-service-node-image

# cinstallman --assign-image --node service2 --image my-service-node-image
# cinstallman --next-boot image --node service2

When you reboot the node, it will reinstall itself.

For more information on the discover command, see “discover Command” in Chapter 2. For more information on the cinstallman command, see “cinstallman Command”.

To retrieve a service node image from a running service node, perform the following steps:

This section describes how to maintain packages specific to your site and have them available to the crepo command (see “crepo Command”).

SGI suggests putting site-specific packages in a separate location. They should not reside in the same location as SGI or Novell supplied packages.

To set up a custom repository for your custom packages, perform the following steps:

All node types that are part of an SGI Altix ICE system can have configuration settings adjusted by the configuration framework. There is some overlap between the per-host customization instructions and the configuration framework instructions. Each approach plays a role in configuring your system. The major differences between the two methods are, as follows:

This framework exists to make it easy to adjust configuration items. There are SGI-supplied scripts already present. You can add more scripts as you wish. You can also exclude scripts from running without purging the script if you decide a certain script should not be run. The following set of questions in bold and bulleted answers describes how to use the system configuration framework.

How does the system configuration framwork operate?

These files could be added, for example, to a running service node, or to an already created service or compute image. Remember that images destined for compute nodes need to be pushed with the cimage command after being altered. For more information, see “cimage Command”.

From where is the framework called?

How do I adjust my system configuration?

Why do scripts need to tolerate files that do not exist but should?

How does the yum plugin work?

How does yume work?

SMC for ICE contains a cluster configuration repository/update framework. This framework generates and distributes configuration updates to admin, service, and leader nodes in the cluster. Some of the configuration files managed by this framework include C3 conserver, DNS, Ganglia, hosts files, and NTP.

When an event occurs that requires these files to be updated, the framework executes on the admin node. The admin node stores the updated configuration framework in a special cached location and updates the appropriate nodes with their new configuration files.

In addition to the updates happening as required, the configuration file repository is consulted when a admin, service, or leader node boots. This happens shortly after networking is started. Any configuration files that are new or updated are transferred at this early stage so that the node is fully configured by the time the node is fully operational.

There are no hooks for customer configuration in the configuration repository at this time.

This update framework is tied in with the /etc/opt/sgi/conf.d configuration framework to provide a full configuration solution. As mentioned earlier, customers are encouraged to create /etc/opt/sgi/conf.d scripts to do cluster configuration.

The cnodes command provides information about the types of nodes in your system. For help information, perform the following:

[admin ~]# cnodes --help
Options:
 --all                  all compute, leader and service nodes, and switches
 --compute              all compute nodes
 --leader               all leader nodes
 --service              all service nodes
 --switch               all switch nodes
 --online               modifier: nodes marked online
 --offline              modifier: nodes marked offline
 --managed              modifier: managed nodes
 --unmanaged            modifier: unmanaged nodes
 --smc-for-ice-names    modifier: return SMC-for-ICE node names instead of hostnames

Note: default modifiers are 'online' and 'managed' unless otherwise specified.

EXAMPLES

[admin ~]# cnodes --all
r1i0n0
r1i0n1
r1lead
service0

[admin ~]# cnodes --compute
r1i0n0
r1i0n1

[admin ~]#  cnodes --service
service0

By default, SMC for Altix ICE software associates one software distribution (distro) with all the images and nodes in the system. For example, if RHEL 6 is used for the admin node then, by default, RHEL 6 is used for the compute blades, leader nodes, and service nodes.

However, SMC for Altix ICE software allows support for multiple distros for compute nodes and service nodes. This means that the nodes and images for service and compute nodes need not match the Linux distribution running on admin/leader nodes.

The following information is intended to make it easier for you to see which media goes with which distributions.

The crepo command, described in “crepo Command”, is the starting point for multi-distro support.

Here is an example of the commands you might run in order to create a RHEL 5.5 service and compute node image:

The cpower command is, as follows:

cpower [<option> ...] [<target_type>] [<action>] <target>

The <option> argument can be one or more of the following:

The <target> argument is one of the following:

The <action> argument is one of the following:

The target must always be specified except when the --system option is used. Wildcards may be used, but be careful not to accidentally power off or reboot the leader nodes. If wildcard use affects any leader node, the command fails with an error.

IPMI-style Commands

The default operation for the cpower command is to operate on nodes and to provide you the status of these nodes, as follows:

# cpower r1i*n*

This command is equivalent to the following:

# cpower --node --status r1i*n*

This command issues an ipmitool power off command to all of the nodes specified by the wildcard, as follows:

# cpower --off r2i*n*

The default is to apply to a node.

The following commands behave exactly as you would expect as if you were using ipmitool, and have no special extra logic for ordering:

# cpower --up r1i*n*

# cpower --reset r1i*n*

# cpower --cycle r1i*n*

# cpower --identify 5 r1i*n*

---

Note: --up is a synonym for --on and --down is a synonym for --off.

---

The cpower command contains more logic when you go up to higher levels of abstraction, for example, using --iru, --rack, and --system. These higher level domain specifiers tell the command to be smart about how to order various of the actions that you give on the command line.

The --iru option tells the command to use correct ordering with IRU power commands. In this case, it firsts connect to the CMC on each IRU in rack 1 to issue the power on command, which turns on power to the IRU chassis (this is not the equivalent ipmitool command). Then it powers up the compute nodes in the IRU. Powering things down is the opposite, with the power to the IRU being turned off after power to the blades. IRU targets are specified as follows: r3i2 for rack 3, IRU 2.

# cpower --iru --up r1i* 

The --rack option ensures power commands to the leader node are down in the correct order relative to compute nodes within a rack. First, it powers up the leader node and waits for it to boot up (if it is not already up). Then it will do the functional equivalent of a cpower --iru --up r4i* on each of the IRUs contained in the rack, including applying power to each IRU chassis. Using the --down option is the opposite, and also turns off the leader node (after doing a shutdown) after all the IRUs are powered down. To avoid including leader nodes in a power command for a rack, use the --noleader option. Rack targets are specified, as follows: r4 for rack 4. Here is an example:

# cpower --rack --up r4

Commands with the --system option ensures that power up commands are applied first to service nodes, then to leader nodes, then to IRUs and compute blades, in just the same way. Likewise, compute blades are powered down before IRUs, leader nodes, and service nodes, in that order. To avoid including service nodes in a system-domain command, use the --noservice option. Note that you must not specify a target with --system option, since it applies to the Altix ICE system.

---

Note: The --shutdown --off combination of actions were deprecated in a previous release. Use the --halt option in its place.

---

You need to configure the order in which service nodes are booted up and shut down as part of the overall system power management process. This is done by setting a boot_order for each service node. Use the cadmin command to set the boot order for a service node, for example:

# cadmin --set-boot-order --node service0 2

The cpower --system --boot command boots up service nodes with a lower boot order, first. It then boots up service nodes with a higher boot order. The reverse is true when shutting down the system with cpower. For example, if service1 has a boot order of 3 and service2 has a boot order of 5, service1 is booted completely, and then service2 is booted, afterwards. During shutdown, service2 is shut down completely before service1 is shutdown.

There is a special meaning to a service node having a boot order of zero. This value causes the cpower --system command to skip that service node completely for both start up and shutdown (although not for status queries). Negative values for the service node boot order setting are not permitted.

---

Note: The IPMI power commands necessary to enable a system to boot (either with a power reset, or a power on) may be sent to a node. The --halt option, halts the target node and then powers it off.

---

The --halt options works on node, IRU, or rack domain levels. It will shut down nodes (in the correct order if you use the --iru or --rack options), and then just leave them as they are, power still applied. Using both these actions results in nodes being halted, then powered off. This is particularly useful when powering off a rack, since otherwise, the leaders may be shutdown before there is a chance to power off the compute blades. Here is an example:

# cpower --halt --rack r1

To boot up systems that have not already been booted, perform the following:

# cpower --boot  r1i2n*

Again, the command boots up nodes in the right orders if you specify the --iru or --rack options and the appropriate target. Otherwise, there is no guarantee that, for example, the command will attempt to power on the leader node before compute nodes in the same rack.

To reboot all of the nodes specified, or boot them if they are already shut down, perform the following:

# cpower --reboot --iru r3i3

The --iru or --rack options ensure proper ordering if you use them. In this case, the command will make sure that power is supplied to the chassis for rack 3, IRU 3, and then the all the compute nodes in that IRU will be rebooted.

EXAMPLES

Example 3-3. cpower Command Examples

To boot compute blade r1i0n8, perform the following:

# cpower --boot r1i0n8

To boot a number of compute blades at the same time, perform the following:

# cpower --boot --rack r1

---

Note: The --boot option will only boot those nodes that have not already booted.

---

To shut down service node 0, perform the following:

# cpower --halt service0

To shutdown and switch off everything in rack 3, perform the following:

# cpower --halt --rack r3

---

Note: This command will shutdown and then power off all of the computer nodes in parallel, then shutdown and power off the leader node. Use the --noleader option if you want the leader node to remain booted up.

---

To shutdown the entire system, including all service nodes and all leader nodes, but not the admin node, and not turn the power off to anything, perform the following:

# cpower --halt --system

To shutdown all the compute nodes, but not the service nodes, leader nodes, perform the following:

# cpower --halt --system --noleader --noservice

---

Note: The only way to shut down the system admin controller (admin node) is to perform the operation manually.

---

The pdsh(1) command is the parallel shell utility. The pdcp(1) command is the parallel copy/fetch utility. The SMC for Altix ICE software populates some dshgroups files for the various node types. On the admin node, SMC for Altix ICE software populates the leader and service groups files, which contain the list of online nodes in each of those groups.

On the leader node, software populates the compute group for all the online compute nodes in that group.

On the service node, software populates the compute group which contains all the online compute nodes in the whole system.

For more information, see the pdsh(1) and pdcp(1) man pages.

EXAMPLES

From the admin node, to run the hostname command on all the leader nodes, perform the following:

# pdsh -g leader hostname

To run the hostname command on all the compute nodes in the system, via the leader nodes, perform the following:

# pdsh -g leader pdsh -g compute hostname

To run the hostname command on just r1lead and r2lead, perform the following:

# pdsh -w r1lead,r2lead hostname

When you used the configure-cluster command, it guided you through setting up NTP on the admin node. The NTP client on the system admin controller should point to the house network time server. The NTP server provides NTP service to system components so that nodes can consult it when they are booted. The system admin controller sends NTP broadcasts to some networks to keep the nodes in sync after they have booted.

NTP client on the rack leader controller gets time from the system admin controller when it is booted and then stays in sync by connecting to the admin node for time. The NTP server on the leader node provides NTP service to Altix ICE components so that compute nodes can sync their time when they are booted. The rack leader controller sends NTP broadcasts to some networks to keep the compute nodes in sync after they have booted.

The BMC controllers on managed service nodes, compute nodes, and leader nodes are also kept in sync with NTP. Note that you may need the latest BMC firmware for the BMCs to sync with NTP properly. The NTP server information for BMCs is provided by special options stored in the DHCP server configuration file.

The NTP client on managed service nodes ( for a definition of managed, see “discover Command” in Chapter 2) sets its time at initial booting from the system admin controller. It listens to NTP broadcasts from the system admin controller to stay in sync. It does not provide any NTP service.

The NTP Client on the compute node sets its time at initial booting from the rack leader controller. It listens to NTP broadcasts from the rack leader controller to stay in sync.

Sometime, especially during initial deployment of an Altix ICE system when system components are being installed and configured for the first time, NTP is not available to serve time to system components.

A non-modified NTP server, running for the first time, takes quite some time before it offers service. This means the leader and service nodes may fail to get time from the system admin controller as they come on-line. Compute nodes may also fail to get time from the leader when they first come up. This situation usually only happens at first deployment. After the ntp servers have a chance to create their drift files, ntp servers offer time with far less delay on subsequent reboots.

The following work arounds are in place for situations when NTP can not serve the time:

This framework is licensed. It cannot be used without the appropriate license.

This section describes some terminology associated with the firmware management, as follows:

This section describes the steps you need to take to update a set of nodes in your cluster with a new BIOS level, as follows:

The fwmgr command is the command line interface (CLI) to the firmware update infrastructure.

For a usage statement, enter fwmgr --help. The fwmgr command has several sub-commands, each of which can be called with the --help option for usage information.

You can use the fwmgr command to perform the following:

---

Note: It is currently not necessary to run the fwmgrd command (firmware manager daemon) to use the CLI.

---

This fwmgrd daemon is installed and enabled by default in SGIMC 1.3 on SGI Altix ICE systems, only. This daemon provides the services needed for the SGI Management Center graphical user interface to communicate with the firmware management infrastructure. This daemon needs to be running in order to access firmware management from the graphical user interface.

Even if you intend to only use the CLI, it is recommended that the fwmgrd daemon be left running and available.

By default, the fwmgrd log file is located at:

/var/log/fwmgrd.log

View this log for important messages during flashing operations from the SGI Management Center graphical interface.

The first release of the Firmware Management framework only supports SGI Altix ICE firmware, released as RPMs. This includes: sgi-ice-blade-bios , sgi-ice-blade-ib, sgi-ice-blade-zoar , sgi-ice-cmc, and sgi-ice-ib-switch . This includes the Altix ICE compute nodes but does not yet include other managed node types.

SGI intends to expand this firmware management framework to support additional node types in Altix ICE and SGI Rackable cluster hardware in later releases.

---

Note: SGI Altix ICE integrated InfiniBand switches are supported but only on SGI Altix ICE 8400 series systems or later. Some integrated InfiniBand switch parts in the SGI Altix ICE 8200 series systems will not flash properly with this framework.

---