Update of /cvsroot/ssic-linux/openssi/docs/fedora
In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv16109/fedora
Added Files:
README.X-Windows README.drbd README.ltsp README.lustre
README.network-bonding README.nfs-server README.ntp
README.upgrade rc-design-notes
Log Message:
Reorganization of openssi/docs to be common across all distro branches
--- NEW FILE: rc-design-notes ---
Note that this design document does not accurately describe the current
implementation of clusterwide RC handling. Instead, it describes what
clusterwide RC handling should ultimately look like. Do not be alarmed
by differences between this document and the current implementation.
---
August 21, 2003
Design notes for cluster RC handling
written by
David Zafman
Design by David Zafman & Bruce Walker
Background
I will describe the basic mechanism for how RC scripts, booting,
shutdown and run-level changes occur. Although I'm basing this on the
Redhat distribution, much of it is the same in other distributions.
If an application package wants to be started during system boot
it can install an RC script in /etc/rc.d/init.d (a symbolic link exists
pointing /etc/init.d at /etc/rc.d/init.d for backward compatibility).
An RC script generally responds to the following commands for argument 1:
start stop status restart reload condrestart reload. The script may support
other package specific commands. Many packages which comprise one or more
daemons can easily use the /etc/init.d/functions to make writing the script
easier. After installing its script in /etc/rc.d/init.d the package
installation executes "chkconfig --add svc_name". More on what chkconfig does
later.
When /sbin/init (process 1) enters a new run level it executes
/etc/rc.d/rc with argument 1 being the new run-level. Part of booting is just
changing to run levels 1, 2, 3, 4, or 5. Shutdown is simply going to runlevels
0 (halt) or 6 (reboot). The runlevel changes are control by the contents of
/etc/rc.d/rc?.d directories. Symbolic links for /etc/rc?.d (? = 0 - 6) exists
for backward compatibility. Inside /ec/rc.d/rc?.d are symbolic links to the
actual RC scripts in /etc/rc.d/init.d. The naming of these scripts control
the sequence and whether a service is enabled or disabled for a particular
run-level. Symbolic links that begin with "S" are executed with "start" and
scripts that begin with "K" are executed with "stop". When entering a runlevel
the K scripts are executed first then the S scripts. They are executed in
shell globing order, which is why they are named S##service_name and
K##servicce_name. The numeric part controls the sequencing.
The chkconfig command completely automates the creation of symbolic
links in the /etc/rc.d/rc?.d directories. It only works if the RC script has
a comment line with the following format:
# chkconfig: 345 05 95
The first field "345" specifies the default runlevels the service should be
started on. The second field specifies the numeric sequencing for starting
it. And the third field specifies the numeric sequencing for stopping. If
the service named "test" had the chkconfig line above and chkconfig --add test
were executed, it would create the following symbolic links:
/etc/rc.d/rc0.d/K95test -> ../init.d/test
/etc/rc.d/rc1.d/K95test -> ../init.d/test
/etc/rc.d/rc2.d/K95test -> ../init.d/test
/etc/rc.d/rc3.d/S05test -> ../init.d/test
/etc/rc.d/rc4.d/S05test -> ../init.d/test
/etc/rc.d/rc5.d/S05test -> ../init.d/test
/etc/rc.d/rc6.d/K95test -> ../init.d/test
From this we see that the "test" service is enabled in runlevels 3, 4
and 5. Although this is the default as specified by the RC script itself the
administrator can modify this at any time. Also, if the chkconfig line has
"-" instead of run-levels in the first field all run-levels will get stop
scripts. It might seem as though this is a stupid configuration, but although
the service doesn't start on boot, it will be stopped on shutdown, if the
administrator starts it manually. Also, many services can be installed which
might be the kind that open up remote access to files or services, and they
should not enabled by default.
Now that we understand how machines boot, shutdown and change runlevels,
lets look at what happens when a particular services is started and stopped.
First of all the /etc/rc.d/rc script looks at /var/lock/subsys/SVC_NAME it
see if a service is running. This means that the RC script should create
this file after performing whatever "start" action it wants to. It must
remove the file after performing whatever "stop" action it wants to. It
uses the presence of the lock file to control "condrestart" activity. The
conditional restart means only restart if it is currently running otherwise
do nothing. During shutdown or really any runlevel change only the K scripts
are executed for services that are actually running. In the same vein when
executing S scripts, if the /var/lock/subsys/SVC_NAME already exists, then the
start is skipped. Most likely for backward compatibility /etc/rc.d/rc also
looks for "/var/lock/subsys/SVC_NAME.init".
On a running system the contents of /var/lock/subsys indicates all the
services that are currently running. This may or may not have any relation to
the contents for /etc/rc.d/rc3.d if the machines is currently at runlevel 3 for
example. An administrator can manually start and stop services at will. Also,
during boot he can enter interactive start-up by type "I" and selectively
disable services that would otherwise have been started.
-----------------------------------------------------------------------------
Features:
Boot
Can boot the cluster to any run-level. For now the initnode boots
first then other nodes join.
shutdown
Can shutdown the entire cluster. Services on all nodes are stopped.
clusternode_shutdown
Can shutdown a single cluster node. Only services on that node are
stopped.
telinit N
The cluster can change run-levels.
redhat-config-services
This GUI tools must be clusterized and provide the ability to edit
new cluster parameters.
Interactive start-up
If services are disabled using this feature on the initnode, those
services do not try to start on dependent nodes. Booting dependent nodes do
not have interactive start-up capability. You can either stop the service,
or disable it (remove from node list), then boot the node. A stopped services
does not start on failover.
Existing RC scripts
Non-cluster aware RC scripts can be installed. They default to running
on the initnode only. They even get started on the new init node during
failover by default. Most scripts which are used this way do not need
modification, although some CDSLs may be required.
/sbin/service
This script is cluster-aware and allows start, stop, restart,
condrestart and any other service specific requests to occur on a cluster-wide
basis. Also, --status-all option will be cluster-wide as well as the
--full-restart options.
-----------------------------------------------------------------------------
Limitations:
Can't directly execute /etc/init.d/SERVICE start
An error is output and it does an exit 1. This may cause some
non-cluster-aware programs, scripts, and RPMs to fail, since they don't use
the /sbin/service script.
RPM installations will require special handling
Manually stopping and then starting individual services, will change the
counter files in /cluster/var/lock/subsys. This will possibly cause
incorrect node join semantics in terms of service start order.
Fortunately, restart and conrestart will not exhibit this behavior as designed.
They don't manipulate the /cluster/var/lock/subsys directory even
though the RC script MIGHT delete and re-create the
/cluster/node*/var/lock/subsys files.
-----------------------------------------------------------------------------
Implementation notes:
/cluster/node{nodenum} is a symbolic link to /cluster/node1
SSI installation must modify the root filesystem for this.
This allows the base kernel to boot on node 1 using the SSI modified
root. The value of 1 is really the initial install node, which current is
assumed to be 1, but in the future this could be made more flexible.
/var/lock is made a CDSL to /cluster/node#/var/lock
SSI installation must modify the root filesystem for this.
/var/run is made a CDSL to /cluster/node#/var/run
SSI installation must modify the root filesystem for this.
The directories console, named, netreport, radvd, saslauthd, sudo might
need to be sym linked back to the root. We must determine if these are global
or node specific items. If an unknown service puts stuff here, it would be
local to a node.
Other root files
Depending on the service other root file must be made into CDSLs.
For example /var/lib/random-seed must be made node specific.
New way to specify nodes (classes)
We are creatng a new uniform way to specify sets of node. In the future
this may become even more flexible. In the initial release the following are
the allowable node classes:
none - no nodes
initnode - the current node running init (process 1)
all - all currently UP nodes in the cluster
node=X,Y,Z - the subset of nodes X, Y, and Z that are UP in the cluster
/etc/sysconfig files
Some files in this directory will be changed into a CDSL to
/cluster/node#/etc/sysconfig/XXXX. This must be determined on a per-service
basis. Most "initnode" and "all" node services have global sysconfig. Some
services which are device specific for example, need a symbolic link here.
A service like "rawdevices" is node specific. So its /etc/sysconfig/rawdevices
configuration file is turned into a CDSL. The xinetd service MUST be
the same on all nodes, since right now we don't want to add node specifications
to the redhat-config-services for xinetd services. Its configuration file
is left global.
SSI installation must modify the root filesystem for this.
New directory tree /cluster/nodetemplate
This directory tree contains all the files needed to create an initial
version of a nodes /cluster/node# tree. During "addnode" the following is
performed:
"cp -a /cluster/nodetemplate /cluster/node#" When a file in /etc/sysconfig,
for example, is converted into a CDSL, a file in /cluster/nodetemplate may or
may not need to be installed. Depending on the service, we might not put a file
in /cluster/nodetemplate, if the service had a missing configuration file, for
example, on its initial install. Also, we could include in the SSI package
the "initial" version of some of these configuration files for the
/cluster/nodetemplate directory.
/etc/init.d/functions modifications
The killproc function and any code using pidof must work locally only.
Since an unmodified RC script may use these functions and a pid file might not
be present the pidof command will read /proc which will be looking at the
entire cluster.
Also the first thing this script should do is check for SSI kernel and
if an environment variable say RC_SSI is NOT set, it should output an error and
fail. This prevents an administrator from executing /etc/init.d/$service_name
directly without getting the clusterization of the /sbin/service script.
New command "onclass"
This has onall type funcationality, but it makes it easier to
implement shell scripts which have class designations for nodes. The initial
classes are described above. If we wanted later to add something like "all-5,7"
to exclude 5 and 7 from all, we would only need to expand the onclass command.
So the /etc/rc.d/rc script described below can simply pass the class argument
if specified for a given service and pass it to the onclass command to start
the service for example:
onclass $class /etc/rc.d/init.d/$subsys start
This command is also used by the redhat-config-services. If the class is
"none" the command will exit successfully without doing anything. Also,
if node 5 is not UP and the class is "node=5" the command will exit
succcessfully without doing anything.
An optional argument -L restrict the action to the local node only.
This way the onclass command can completely parse the node_class argument then
determine if the local node is included in that class. This would be used for
the rc.nodeup to determine if the joining node needs to start the service, for
example.
New command "onsvc"
This has onall type functionality, but it makes it easier to implement
shell scripts which need to run things on the set of nodes on which a service
is currently running. The set of nodes are determined by looking at nodes
that are UP and which the file /cluster/node*/var/lock/subsys/SVC_NAME
exists. An example, would be to get status of a service which is running on
some set of nodes in the cluster the command "onsvc $service
/etc/init.d/$service status". For the most part an administrator would not have
to use the onsvc command, but it is available. The onsvc command is how the
/sbin/service script will actually get the status, stop, reload a service in
the cluster.
An optional argument -L restrict the action to the local node only.
This way the onsvc command can look for the lock file only for the local node
This would be used clusternode_shutdown to determine if the leaving node needs
to stop a service.
/etc/rc.d/rc script modifications
This script looks at K??* and S??* files just as in the base, however,
it also checks to see if an SSI kernel is running changes its behavior as
follows:
This script will keep a counter in a file and create files in
the directory /cluster/var/lock/subsys with names L???$service_name
where the ??? is the current value of the counter. The service_name is the
standard service name as in the standard /var/lock/subsys directory.
When trying to "start" services in the cluster this script will do
the following. It will check /cluster/var/lock/subsys/L???$service_name
file instead of the local file, to make sure the service is not already running
in the cluster. It will parse the /etc/rc.d/rc.nodeinfo and retrieve the
node_class field for the service in question. If the service is not in the
file a warning will be issued and the script skipped. After issuing the
onclass command, it will create the file
/cluster/var/lock/subsys/L???$service_name. It will then bump its
counter.
When trying to perform the "stop" operations on services this script
will do the following. It will check /cluster/var/lock/subsys to make
sure the service is running. If it is, it will issue an
onsvc $service_name /etc/rc.d/init.d/$service_name stop
Afterward it will remove the /cluster/var/lock/subsys file.
The /sbin/services script is modified to behave must as /etc/rc.d/rc does for
"start" and "stop". All other request are assumed to be for running services
and use the same mechanism as "stop".
/etc/rc.d/rc.nodeup
This script must join a node to an existing cluster. Currently,
even simultaneous booting requries this to work, since dependent nodes wait
for the initnode to finish booting before they can join. This script must
go through /cluster/var/lock/subsys/L???$service_name files in sorted
order just as /etc/rc.d/rc does the S* and K* files in sorted order. The
service_name is then determined and based on the node_class in
/etc/rc.d/rc.nodeinfo, the service may or may not be started. Instead of having
the script having to determine if the node class include the current node, we
should have an onclass local option to restrict and exec to the local node
only. This way onclass is always called and it may be a no-op if the action
should not be performed.
redhat-config-services
This GUI is modified to allow you to specify what nodes a service
should run on. Initially this might be "all", "none", or specific nodes
"1,2,3". Also this interface does start/stop/restart of a service on all
appropriate nodes, just as during boot or run-level change.
When changing an xinetd service, a "reload" must be performed on all
nodes after saving changes.
/etc/rc.d/ssiconfig
This data file specifies the default behavior of non cluster-aware
sevices in the cluster. Entries will consist of lines having the following
format:
service_name type default_class default_failover_flag
The service_name matches the subsystem name (the file name in /etc/rc.d/init.d).
The type can be S for single cluster node, or M for multiple nodes allowed.
The default_class will be shipped as "none", "initnode", or "all". Hardware
specific services will default to "none", since the administrator must select
the node or node(s) to run the service on. Other services which we want to
run on every node will be set to "all". The default_failover_flag
can be Y for yes or N for no to starting the service again on initnode failover.
This file is the SSI enhanced portion of chkconfig information. In a
cluster-aware RC script this information is in a comment line following
"ssiconfig:". We don't want to be in the business of maintaining this
information. An RC script can over-ride the contents of this file. This file
is just for non-cluster-aware services that are not operating per the defaults.
We don't bother to put in entries which are purely the default, for example:
cron S initnode Y
As you can see from this the default for non-cluster aware services, it to
have the service running once in the cluster on the initnode at all times.
/etc/rc.d/rc.nodeinfo
This data files drives the behavior of enabled sevices in the cluster.
Entries have the following format:
service_name node_class failover_flag
When chkconfig is run the default_class and default_failover_flag specified
is put into this file. First, if the RC script has a valid ssiconfig comment
line the information is retrieved from there. If not, the information comes
from /etc/rc.d/ssiconfig file. Otherwise, the values "initnode" and "Y" or
put in for node_class and failover_flag, respectively.
If chkconfig is used to delete a service, the line is removed from this file.
The service_name matches the subsystem name (the file name in /etc/rc.d/init.d).
The node_class can be modified by using chkconfig (see below). The
failover_flag indicates whether the service should be activated with "start"
on init node failover.
chkconfig
We will add new options to output what nodes a given service will
be run on. This is not the default in case anyone is relying on the standard
behavior of showing just the run-levels. We will have this command allow you
to change what nodes a service will be run on. It will be in the same format as
the onclass command accepts and it will be written to "/etc/rc.d/rc.nodeinfo".
As indicated above the chkconfig command will have new options to set the
class of node the service runs on. For now the failover flag can not be set.
Also, as indicated above when a service is --add'ed an entry in rc.nodeinfo
is created based on the defaults. Again the default would be to look in the RC
script's ssiconfig line, then in /etc/rc.d/ssiconfig file, otherwise use
node_class = "initnode", failover_flag = "Y". Initially, we might not suport
the ssiconfig line in the RC script. We should add two options
--ssi and --node <class>. The first option modifies the --list option and
display the current node_class and failover flag selection in
/etc/rc.d/rc.nodeinfo information if present. The usage is
chkconfig --ssi --list [name]. The --node <class> allows the user to
modify the /etc/rc.d/rc.nodeinfo for a given services the usage is:
chkconfig --node <class> <name>
INSTALLATION
A new /etc/rc.d/ssiconfig will be installed by the chkconfig RPM. As
part of post installation, if there is no /etc/rc.d/rc.nodeinfo all currently
enabled services should be determined and "chkconfig --add $subsys" should be
run for each. The standard behavior of chkconfig should be to add the default
rc.nodeinfo line if it is missing for an added service even when the symbolic
links are already there.
ntsysv
This curses based configuration tool should let you edit node
specification parameters, much like redhat-config-services will be made to.
tksysv
This command is mentioned in chkconfig documentation, but my Redhat
installed machines don't have it nor its documentation.
/etc/init.d/halt
This script must be clusterized. It is used to both halt and reboot
a machine. Unfortunately, it also does some final clean-up on the node. Most
of these activities must be done on all nodes. We should break some of
these operations out so that they can be done using "onall" on all nodes for
cluster shutdown and locally for individual node shutdown.
The finally halt/reboot, but be sent to all nodes as a kernel message
possibly with CLMS monitoring disabled.
ssiadmin command
The administrator might have to use a new command to successfully
install an RPM. If an RPM directly calls /etc/init.d/$service_name start/stop
they will fail. We must document that to upgrade a package after SSI is
installed:
ssiadmin $service_name rpm -Uv RPM-NAME.rpm
The ssiadmin command will record the /cluster/lock/subsys/$service_name
state. Then execute /sbin/service $service_name stop to stop service on all
nodes. Then it will set the environment variable RC_SSI and execute rpm.
Afterward it will issue an /sbin/service $service_name start only if the global
state that was recorded first showed that the service was running. The
administator must figure out the name to specify ("service_name") for any rpm
which is also a Linux service.
Local /proc option
We need an inheritable task flag which causes the readdir of /proc
to work completely locally. This partially need for performance and for
correctness. This will speed up the pidof program extensively used by
RC scripts in /etc/init.d/functions shell functions. However, if an RC
script did its own thing like call killall command itself, or used ps to find
its daemon, it would not operate properly. The worst case is you stop a
service for clusternode_shutdown and the service stops on all nodes.
clusternode_shutdown
Need a clusternode_shutdown command based on "shutdown" but initiates a
specific node going down. It changes the node state to "SHUTDOWN" in CLMS
and eventually either signals init or just stops services locally on the
node being shutdown. This ends when the halt script is run, but it somehow
knows not to halt/reboot all the nodes.
Similiar options to shutdown. New required option -N # to specify
which node is to be shutdown. Additional option -A which is the application
notification time (app grace). The app grace defaults to 20 seconds if not
specified.
Let's change shutdown.c to check if it is invoked as
clusternode_shutdown and modify its behavior. It should have a usage as below.
Also, after sending warnings and getting ready to shutdown, it changes
the state of the node to SHUTDOWN and sleeps for app_grace period. Afterward it
changes the node state to GOINGDOWN and execs a shell script much as init would
during shutdown. This script similiar to /etc/rc.d/rc, but it specifies the
local flag to onsvc for stop and onclass for start. We need special handling
of /etc/rc.d/rc6.d/S*reboot which is a sym link to /etc/init.d/halt. This
script must either be run with special options or we detect the
clustenode_shutdown case and run a different script or have different behavior.
The /etc/init.d/halt script as it is today will bring the entire cluster down.
Usage: clusternode_shutdown [-Akrh] [-t secs] -N # time [warning message]
-N: Specify the node to take down
-A: Specify how long applications get warning
-k: don't really shutdown, only warn.
-r: reboot after shutdown.
-h: halt after shutdown.
-t secs: delay between SIGTERM and SIGKILL for init.
------------------------------------------------------------------------------
Future items:
Have a richer set of node classes which can be specified
regular
compute
user defined class
all-but-initnode
n1 or n2 or n3 (to allow non-initnode failover)
Cluster-aware RC scripts
Need to design and implement features which can be used by a
cluster-aware script. For example, a script might want to do things on every
node, but differentiate between nodes joining and initial start. It might
also want to be informed of nodes going down. It might want to do special
processing for failover. It must have an ssiconfig comment line in its script.
I envision a special node_class value going into /etc/rc.d/rc.nodeinfo from the
ssiconfig line that mark it cluster-aware and control where it runs things in
the cluster itself. This must affect how the GUI and chkconfig react to this
kind of service.
Boot
Allow a set of nodes to boot together. The RC scripts should still
work since booting is the same as a run-level change, executing the
/etc/rc.d/rc script.
This feature requires enhancements to the init command.
redhat-config-services
This program could also be made to immediately start/stop services on
specific nodes not just cluster-wide. This might be in the form of a
reconfigure button and/or "node start", "node stop" or "node restart" buttons.
The reconfigure button would cause the current node settings to take effect.
To use it you could modify a services node list parameter and press the
reconfigure button. The Service would be started or stopped as appropriate
on whatever nodes are required. The "node XX" feature would allow you to
independent of the current node settings, start/stop/restart a service
only a particular node.
Other /sbin/init improvements
Don't allow run-level changes while nodes are joining or leaving (doing
clusternode_shutdown). Delay transitioning to SHUTDOWN state if there is
a run-level change in progress. Must do all node transitions coordinated
through init since it changes the node states so that these restrictions are
maintained. Only init (process 1) can declare nodes fully up or down with the
exception of going back to UP from the SHUTDOWN state. Only init (process 1)
can put a node into SHUTDOWN or GOINGDOWN state.
clusternode_shutdown
Ultimately, clusternode_shutdown will use the process 1 to perform
various functions, so that it can coordinate run-level changes,
clusternode_shutdown, and node up and down transitions. Also, init will be
asked to perform node state changes.
--- NEW FILE: README.nfs-server ---
Notes on setting up an NFS server in an OpenSSI cluster.
Requirements:
HA-LVS must be enabled
The clustername should correspond to the CVIP IP address
Limitations:
NFS server only runs on the init node
A limited failover capability exists
-------------------
Server set-up steps:
1. See /usr/share/doc/openssi-tools-1.0.0/README.ipvs for setting up HA-LVS
and the Cluster Virtual IP (CVIP) address
2. The clustername must resolve to the CVIP you configure, either in your
DNS server or in /etc/hosts. If you need to change the clustername,
then edit /etc/clustername and reboot your cluster.
3. The NFS server service name is now "SSInfs." When OpenSSI is installed
the SSInfs service is enabled if the "nfs" service was enabled. If you
must enable NFS server after OpenSSI is installed you would need to do the
following: /sbin/chkconfig SSInfs on. The state of the regular "nfs"
service doesn't matter because it will automatically be skipped.
4. Review standard RedHat NFS server configuration documentation
This step involves creating a list of exported filesystems and
starting the nfs service.
See the following Red Hat documentation:
https://www.redhat.com/docs/manuals/linux/RHL-9-Manual/custom-guide/s1-nfs-export.html
---------
When NFS mounting from a client, use the clustername, NOT the nodename of any
particular node. Obviously, the clustername must resolve to the CVIP in the
client's /etc/hosts or DNS server.
For example, if the clustername of the cluster is 'cluster1':
$ mount cluster1:/home /mnt
--------
Failover
Limitation
1. lockd/statd failover not supported yet
2. Only failover chard mounts on initnode
You can export any chard CFS filesystem, but it is only supported
for filesystems which exists on the initnode and failover to the new initnode.
This generally has a node option in /etc/fstab like "node=1:2." The
/etc/export entries MUST have a unique fsid=X option specified or they will not
be exported. The value of fsid can simply be fsid=1 for the first entry,
fsid=2 for the second and so on.
Sample /etc/fstab entry
LABEL=/opt /opt ext3 rw,chard,node=1:2 1 2
Sample /etc/export entry
/opt *(rw,sync,no_root_squash,fsid=1)
--- NEW FILE: README.upgrade ---
Upgrading from OpenSSI 1.1.0 for Fedora Core 2
==============================================
Download the latest OpenSSI release and untar it (tar xf <filename>).
Make sure all nodes with local boot devices are booted into the cluster.
Run the upgrade script and reboot the cluster.
You can ignore the warnings about 'V3 DSA signature'. They're harmless.
Unfortunately, there is no clean upgrade path if you want to upgrade
from any version of OpenSSI for Red Hat 9.
--- NEW FILE: README.ltsp ---
This document describes what has to be done for successfully running LTSP on
SSI Cluster so that the LTSP client connections are load-balanced across
nodes in the cluster using HA-LVS. If the LTSP terminals need not run X-windows
or one doesn't want XDMCP connection loadbalancing to happen in the SSI
cluster, then just performing steps 1, 3, 5 and 6, in that order, will suffice.
1. Install SSI cluster
2. Follow README.X-windows and setup X to run on all nodes of the cluster.
Note: Make sure that the /etc/X11/XF86Config has proper entries in the
Screen, Monitor and Device sections to support all the nodes in the
cluster. This is needed in order for the gdm/X to run on all nodes in
the cluster.
3. Follow README.ipvs and setup ha-lvs on the cluster.
4. Run the following command on all nodes:
setport_weight --start-port=177 --end-port=177 --weight=1
To make this run automatically on each node when it boots, add the
above line to the start() section of /etc/init.d/ha-lvs.
5. Follow README.nfs and setup NFS server on SSI cluster
6. Follow instructions in http://www.ltsp.org/documentation/ltsp-3.0-4-en.html
and install and configure LTSP on the root node of SSI cluster.
--- NEW FILE: README.X-Windows ---
* Using a PS/2 mouse with X:
We are currently using a FC1 kernel on FC2 until we have finished
our port to the 2.6 kernel. Unfortunately, the way the kernels
handle PS/2 mice is incompatible between kernels and you need
to make the following changes so that X will work on both kernels.
In order to make X work with the FC1 kernel, modify the
/etc/X11/xorg.conf file. Change the following lines:
Option "Protocol" "IMPS/2"
Option "Device" "/dev/input/mice"
to:
Option "Protocol" "PS/2"
Option "Device" "/dev/psaux"
and reboot.
In order to make X work with the FC2 kernel with the modified
xorg.conf, while running the FC2 kernel, execute the following
commands:
rm /dev/psaux
ln -s input/mice /dev/psaux
and reboot.
I haven't been able to test, but I believe USB mice should
work without having to change anything.
* Running X on more than one node:
First edit /etc/inittab to add a U to the runlevel for prefdm. This
tells the SSI-enhanced init to start prefdm on every node:
# Run xdm in runlevel 5
-x:5:respawn:/etc/X11/prefdm -nodaemon
+x:5U:respawn:/etc/X11/prefdm -nodaemon
Each instance of X creates unix domain sockets in /tmp. To prevent
namespace collisions between them, each node must have its own
/tmp. So we must replace /tmp with a context-dependent symlink
(CDSL). Assuming bash and that node 1 is your initnode:
# mklocalfile /tmp
# rm -rf /cluster/node[1-9]*/tmp/.font-unix
# mkglobalfile /tmp/.font-unix
# rm /cluster/node1/tmp/.font-unix
# mv /cluster/tmp/.font-unix /cluster/node1/tmp
# ln -s /cluster/node1/tmp/.font-unix /cluster/tmp/.
# mklocalfile /var/gdm
# for i in /cluster/node*/var/log ; do mkdir -p $i/gdm ; done
Now each node effectively has its own /tmp and all nodes will use the
X fontserver on node 1 (replace node1 in the above commands if you have
a different initnode).
It's also a good idea to apply the following patch to /etc/console.perms:
--- /etc/security/console.perms.dist 2003-02-10 21:43:43.000000000
+0100
+++ /etc/security/console.perms 2003-11-10 15:02:01.000000000 +0100
@@ -15,7 +15,7 @@
# man 5 console.perms
# file classes -- these are regular expressions
-<console>=tty[0-9][0-9]* vc/[0-9][0-9]* :[0-9]\.[0-9] :[0-9]
+<console>=tty[0-9][0-9]* ([0-9]+/)?vc/[0-9][0-9]* :[0-9]\.[0-9] :[0-9]
<xconsole>=:[0-9]\.[0-9] :[0-9]
# device classes -- these are shell-style globs
--- NEW FILE: README.network-bonding ---
Network Bonding in OpenSSI.
-----------------------------
This document describes the current status of support for network
bonding in OpenSSI. It describes the steps required to get bonding
working. This process is currently manual requiring changes to the
ramdisk so that the bonding interface is associated with ICS.
More information on Network Bonding is available in the linux
source tree at:
Documentation/networking/bonding.txt
Note:
All steps outlined below need to hand-crafted right now. This is
a temporary solution before bonding is neatly integrated with SSI.
PART I: HAND-CODED SETUP
-------------------------
Steps 2-4 refer to changes inside the linuxrc. Following the instructions,
you will find a snipet of an example linuxrc. Have a look at it and make the
corresponding changes in your linuxrc.
1. Copy over bonding module and the ifenslave binary into the ramdisk
bonding.o
/sbin/ifenslave
2. Install the bonding module with the desired mode
For e.g.
insmod /lib/bonding.o mode=1 miimon=100
Possible values for mode are [from the bonding-howto]:
0 Round-robin policy: Transmit in a sequential order from the
first available slave through the last. This mode provides
load balancing and fault tolerance.
1 Active-backup policy: Only one slave in the bond is active. A
different slave becomes active if, and only if, the active slave
fails. The bond's MAC address is externally visible on only
one port (network adapter) to avoid confusing the switch.
This mode provides fault tolerance.
2 XOR policy: Transmit based on [(source MAC address XOR'd with
destination MAC address) modula slave count]. This selects the
same slave for each destination MAC address. This mode provides
load balancing and fault tolerance.
3 Broadcast policy: transmits everything on all slave interfaces.
This mode provides fault tolerance.
3. Configure the bonding interface using ifconfig.
4. Use ifenslave to enslave desired interfaces to the bonding interface.
Following is the relevant portion of an example linuxrc that has the
changes described above:
---<<snip>>>-------------------------------------------------------------------
for iface in `ifconfig -a | grep HWaddr|
sed 's/\(eth[0-9]*\).*HWaddr \(.*\)/\1-\2/'|sed 's/\(bond[0-9]*\).*HWaddr
\(.*\)/\1-\2/'`
do
ifdev=`echo $iface | cut -f 1 -d -`
ifaddr=`echo $iface | cut -f 2 -d -`
echo
rec=`tail +2 /etc/boottab | grep $ifaddr`
if [ $? -eq 0 ]; then
ifdev=bond0
ifaddr=`echo $rec | cut -f 3 -d / | cut -f 1 -d:`
ifmask=`echo $rec | cut -f 3 -d / | cut -f 2 -d:`
echo "Configuring $ifdev: $ifaddr/$ifmask"
ifconfig $ifdev $ifaddr netmask $ifmask
nodenum=`echo $rec | cut -f 1 -d /`
if [ $nodenum -gt 0 ]
then
# set variables used for cluster IP address
dev=$ifdev
addr=$ifaddr:$ifmask
node=$nodenum
nicfound=1
fi
fi
done
/bin/ifenslave bond0 eth1
/bin/ifenslave bond0 eth2
-----<<snip>>-------------------------------------------------------------------
--- NEW FILE: README.drbd ---
NOTICE: This procedure has not been verified for the 1.1.0 release.
DRBD-HowTO
--------------
The drbd-HowTO describes the following:
I. Failover of root file system using DRBD
II. Failover of non-root file systems using DRBD
I. Failover of root file system using DRBD
-------------------------------------------
These are the steps involved in getting drbd root failover on OpenSSI.
These steps are for a fresh install. Towards the ends of the section,
you can find steps to convert an existing openSSI cluster into
a drbd-enabled root-failover cluster.
1. Install base linux (Fedora Core 2). Make sure you partition your disk
such that it has a an extra partition with atleast 128 Mb for the
drbd index later on.( For e.g., in addition to /root,/boot and /swp there
will an extra partition for drbd of atleast size 128 MB)
2. Install the latest version OpenSSI available from http://www.openssi.org.
The current latest version is openssi-1.1.0. Make sure you
enable Root-Failover during installation.
3. Install openssi-enabled drbd.
The tarball is available from
http://www.openssi.org/contrib/drbd.tar
The tarball contains the drbd package (drbd.code) and sample files
for linuxrc and SSIfailover needed in steps below. Needs to be built
against the kernel-source currently (I am working on a rpm):
Rpm for openssi-kernel source is avalaible at
http://prdownloads.sourceforge.net/ssic-linux/kernel-ssi-source-*.i686.rpm
Find the version that corresponds to the openSSI release you are using.
4. Configure drbd.conf in /etc/drbd.conf. Include the correct hostname,
disk partitions and nodenum. The "device=" parameter should
be your root and meta-disk the partition that was left free during
installation during step 1.
5. Unpack ramdisk and mount it loopback(for example on /mnt/ramdisk/).
Copy over the following :
a. drbd.o
cp $LIBDIR/drbd.o /mnt/ramdisk/lib/
b. drbd utilities: drbdsetup, drbdadmn
cp /sbin/drbdsetup /mnt/ramdisk/bin/
cp /sbin/drbdadmn /mnt/ramdisk/bin/
c. drbd.conf
cp /etc/drbd.conf /mnt/ramdisk/etc/
d. Use mknod to create /mnt/ramdisk/dev/nbd/0
mkdir /mnt/ramdisk/dev/nbd
cd /mnt/ramdisk/dev/nbd
mknod 0 b 43 0
e. Similarly, use mknod to create the device files for root and meta-disk
f. Copy the mount command
cp -a /bin/mount /mnt/ramdisk/bin/
6. Handcraft linuxrc in ramdisk (your particular linuxrc might look
slightly different):
a. insmod lib/drbd.o
Corresponding patch for linuxrc will look like:
<----snip---->
echo "Loading mii.o module"
insmod /lib/mii.o
echo "Loading 8390.o module"
insmod /lib/8390.o
echo "Loading e100.o module"
insmod /lib/e100.o
echo "Loading tulip.o module"
insmod /lib/tulip.o
+ echo "Loading drbd.o module"
+ insmod /lib/drbd.o
<----snip---->
b. Initialize drbd, make init node the primary node.
and then mount /dev/nbd/0 on /sysroot
Corresponding patch for steps b,c,d in linuxrc will look like:
<----snip---->
if [ $? -ne "0" ]
then
echo Creating root device
- echo mkrootdev /dev/root | nash --quiet --force
- mount -o defaults --ro -t ext3 /dev/root /sysroot
+ for i in `drbdadm -s /bin/drbdsetup sh-devices`
+ do
+ drbdadm -s /bin/drbdsetup up $i
+ done
+ drbdadm -s /bin/drbdsetup wait_connect all
+ drbdadm -s /bin/drbdsetup primary all
+ mount -o defaults,chard --ro -t ext3 /dev/nbd/0 /sysroot
if [ $? -ne "0" ]
then
echo "ERROR: Mounting root file system failed."
echo "Unable to continue. Halting."
- halt -L -f
- exit 1 # NOT REACHED
+# Use for debugging and potentially forcing drbd mount primary
+ bash
+# halt -L -f
+# exit 1 # NOT REACHED
fi
fi
<----snip---->
7. Pack ramdisk and copy it into /boot. Make sure you retain
a copy of the old ramdisk so that you can boot up with the older
ramdisk if needed.
8. Change /etc/init.d/SSIfailOver to start drbd and enable failover for
/dev/nbd/0
Corresponding patch for SSIfailover will look like:
<----snip---->
return 0
fi
- DEVICE=`/sbin/findfs $FSDEV`
+# DEVICE=`/sbin/findfs $FSDEV`
+ /etc/init.d/drbd start
+ DEVICE=/dev/nbd/0
cfs_setroot $TYPE $DEVICE
action $"Enabled root failover: " /bin/true
return 0
<----snip---->
9. In /etc/fstab, replace the UUID=... with /dev/nbd/0
10. Shut down the cluster and boot only the first node.
The root partition should be picked up by the drbd-enabled ramdisk.
You are now running with the root as a drbd primary.
11.Once the first node completes booting up, boot up the second node.
When the second node comes up, use command
drbdsetup /dev/nbd/0 wait_sync
This should sync node2 with the first node. This might take a little
time while the partitions are being synced. It took me about 15 minutes.
Use
cat /proc/drbd
to check progress of the sync. The complete sync will show "ld:Consistent"
Once the sync is complete, node2 is ready for failover.
12. Now if the first node crashes, the second node should fail over.
Converting an existing openssi-cluster into a drbd-enabled root failover
-------------------------------------------------------------------------
Converting an existing openssi-cluster into a drbd-enabled root failover
cluster is only possible at the moment if you have an extra partition
that is atleast 128Mb for the index partition. The only alternative
is to use resize2fs to convert an existing root partition so that there
is space for the 128Mb index partition. This as can be guessed is going
to be tricky to do over a running and in-use root filesystem.
If the cluster does have an extra 128 Mb space for the index partition, then
you can convert the existing cluster into a root-failover cluster. And then
start from step 3 above.
There are basically three steps involved in changing a non-failover cluster
to a failover cluster.
1. Run ssi-chnode to turn secondary node into a takeover CLMS master
2. Add the chard option to the root line in /etc/fstab
e.g. UUID=$UUID / ext3 chard,defaults,node=1:2 1 1
3. Run mkinitrd to create a new mkinitrd
4. Follow How-to to engineer ramdisk to do drbd-failover.
4. Run ssi-ksync to propogate new mkinitrd to all nodes.
Note:
According to my experience, root fail-over through drbd has not always been
extremely reliable, especially when the cluster had multiple fail-overs.
In some cases I have had to recover things manually. This might be due to
the fact that the base-drbd code is still in flux and has not been extensively
tested in situations that reflect root-like usage.
II. Failover of non-root file systems
--------------------------------------
1. Download and install the OpenSSI enabled drbd and the patched mount
2. Edit /etc/drbd.conf to indicate the resources that need to
to be failed over. If drbd is being layered over a device with
an existing filesystem, use a separate meta-device.
3. Edit /etc/rc.d/ssiconfig and add the line
drbd M all Y
4. Enable drbd to start up at boot
chkconfig --add drbd
5. If drbd is being layered over an existing filesystem, skip this step.
Otherwise create the filesystem.
insmod drbd
drbdadm up r1 # where r1 is the name of the resource in drbd.conf
drbdadm primary r1
mkfs -t ext3 /dev/nbd/1 # where /dev/nbd/1 is the drbd
# device name in drbd.conf
6. Edit /etc/fstab to enable failover
/dev/nbd/1 <mntpt> ext3 default,chard,node=1:2 0 0
The node= are the numbers of the nodes which are in drbd.conf
7. Now start drbd
service drbd start
8. Mount the filesystems
mount -a
--- NEW FILE: README.ntp ---
This is written for RedHat.
The README assumes that ntp was configured on the original
installation node and that ssi has been installed.
NOTE: I am trying these things out as I write this, for the most
part, but your mileage may vary.
NOTE: Once you perform the following procedures, the Red Hat
"dateconfig" GUI cannot be used to change the ntp settings.
NOTE: All shell commands are for bash.
First, make sure ntp is off.
/sbin/service ntpd off
Second, we may need to use ntp authentication, so generate the ntp
keys file.
/usr/sbin/ntp-genkeys -f
Third, each node will need an individual "drift" file. Edit your
ntp.conf and make sure there is a driftfile line like the following:
driftfile /var/lib/ntp/drift
If the directory /var/lib/ntp already exists:
mkdir -p /cluster/node\{nodenum\}/var/lib
mv /var/lib/ntp /cluster/node\{nodenum\}/var/lib
Make /cluster/node<node>/var/lib/ directories for all the nodes in the cluster.
( cd /cluster; for d in node*; do
mkdir -p $d/var/lib/ntp; chown ntp:ntp $d/var/lib/ntp;
chmod 0755 $d/var/lib/ntp; done )
Make a context-symlink in /var/lib for the ntp directory:
ln -s /cluster/node\{nodenum\}/var/lib/ntp /var/lib/ntp
*** Configuration #1: All nodes have access to an external timeserver
This is a very straightforward configuration. All nodes use the same
configuration files with the exception of the drift file we have
already set up.
Edit your /etc/ntp.conf file:
Assuming your ICS network is 192.168.0.0/255.255.255.0, add the
following lines to /cluster/etc/ntp.conf (The key can be any
one of the keys generated in /etc/ntp/keys by ntp-genkeys.):
restrict 192.168.0.0 mask 255.255.255.0
broadcast 192.168.0.255 key 1
broadcastclient
trustedkey 1
requestkey 1
This allows all the machines on the ICS network for to use the
machine as a broadcast server and all nodes in the cluster to
use the other's idea of the time. The "requestkey" line is
needed so a script can modify the server configuration on the fly.
If the following lines are present in the ntp.conf, they should be
deleted or commented out:
server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 10
This lines allow ntpd to use the local clock as a time source
that ntp clients can synchronize to. We want only one node in
the cluster to have this setting at any given time: the
current CLMS master. A script named ssi-ntpsetrefclk will
be run from /etc/rc.d/rc and /etc/rc.d/rc.sysrecover to set
the local clock as a reference clock.
Add the ICS addresses for all the nodes to
/cluster/etc/ntp/step-tickers. (Any external servers should already be there.)
Start ntp
/sbin/service ntpd start
Assuming your networking is setup correctly, you should see a series
of messages similar to the following:
(node 1)
ntpd: Synchronizing with time server: [ OK ]
Starting ntpd: [ OK ]
(node 2)
ntpd: Synchronizing with time server: [ OK ]
Starting ntpd: [ OK ]
*** Configuration #2: A limited number of nodes has access to the external
time server. The nodes with the external access should synchronize to
the external clocks; all other nodes should synchronize with those
nodes via the ICS network.
For the nodes in communication with the external network, you need one
ntp.conf file; for the nodes that are only on the external network you
need another. This means we need another set of context-symlinks.
mv /etc/ntp.conf /cluster/etc
ln -s /cluster/node\{nodenum\}/etc/ntp.conf /etc/ntp.conf
Set up /cluster/etc/ntp.conf as for #1. For the step-tickers file, add
the ICS addresses of only those nodes that can talk to the external
servers. This ntp.conf file is used only by the nodes with access to
the external time servers. (Note: all possible CLMS masters must have
external access for things to work properly.)
Copy the /cluster/etc/ntp.conf to /cluster/etc/ntp.conf.dep and remove
the lines referring to external servers and the "broadcast" line and
the "requestkey" line. (The "broadcastclient" line is still required.)
This ntp.conf will be used by nodes without external access. (These
nodes are ntp clients only.)
For each node that has access to an external server:
ln -s /cluster/etc/ntp.conf /cluster/node<number>/etc/ntp.conf
For example node 1:
ln -s /cluster/etc/ntp.conf /cluster/node1/etc/ntp.conf
For all nodes that do not have access to the external servers:
ln -s /cluster/etc/ntp.conf.dep /cluster/node<number>/etc/ntp.conf
Also, assuming that any node now added to the cluster will not have
access to the external network, make the following symlink as well.
ln -s /cluster/etc/ntp.conf.dep /cluster/nodetemplate/etc/ntp.conf
All new nodes will get this symlink when they are added.
RedHat passes the servers in the step-tickers file to the ntpdate
command and uses it to set the time on boot. The ntp-server on the
CLMS master will take approximately 1 minute to stabilize. During this
window, nodes joining the cluster will get the following error:
ntpd: Synchronizing with time server: [FAILED]
The time on the node should have been set within a second by via the
ssi-timesync script and ntpd should begin to correct the time once the
server on the CLMS master node becomes ready.
*** Configuration #3: A cluster without access to any external ntp
timeservers. All nodes should synchronize with the CLMS master via the
ICS network.
As #2 above, except there are no external servers listed in the
ntp.conf or step-tickers files. The ntp.conf should be used by all the
potential master nodes in the cluster; the step-tickers file should
contain all the potential master nodes in ther cluster. The
ntp.conf.dep should be used by all non-master nodes.
The ntp.conf I used follows.
# Prohibit general access to this service.
restrict default ignore
# External servers
restrict 192.168.0.10 mask 255.255.255.255 nomodify notrap noquery
server 192.168.0.10 # linuxbuild1
restrict 192.168.0.11 mask 255.255.255.255 nomodify notrap noquery
server 192.168.0.11 # kahuna
restrict 192.168.0.0 mask 255.255.255.0
broadcast 192.168.0.255 key 1
broadcastclient
trustedkey 1
requestkey 1
# Permit all access over the loopback interface. This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
# -- CLIENT NETWORK -------
# Permit systems on this network to synchronize with this
# time service. Do not permit those systems to modify the
# configuration of this service. Also, do not use those
# systems as peers for synchronization.
# restrict 192.168.1.0 mask 255.255.255.0 notrust nomodify notrap
# --- OUR TIMESERVERS -----
# or remove the default restrict line
# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
# restrict mytrustedtimeserverip mask 255.255.255.255 nomodify notrap noquery
# server mytrustedtimeserverip
# --- NTP MULTICASTCLIENT ---
#multicastclient # listen on default 224.0.1.1
# restrict 224.0.1.1 mask 255.255.255.255 notrust nomodify notrap
# restrict 192.168.1.0 mask 255.255.255.0 notrust nomodify notrap
# --- GENERAL CONFIGURATION ---
#
# Undisciplined Local Clock. This is a fake driver intended for backup
# and when no outside source of synchronized time is available. The
# default stratum is usually 3, but in this case we elect to use stratum
# 0. Since the server line does not have the prefer keyword, this driver
# is never used for synchronization, unless no other other
# synchronization source is available. In case the local host is
# controlled by some external source, such as an external oscillator or
# another protocol, the prefer keyword would cause the local host to
# disregard all other synchronization sources, unless the kernel
# modifications are in use and declare an unsynchronized condition.
#
#server 127.127.1.0 # local clock
#fudge 127.127.1.0 stratum 10
#
# Drift file. Put this in a directory which the daemon can write to.
# No symbolic links allowed, either, since the daemon updates the file
# by creating a temporary in the same directory and then rename()'ing
# it to the file.
#
driftfile /var/lib/ntp/drift
broadcastdelay 0.008
#
# Authentication delay. If you use, or plan to use someday, the
# authentication facility you should make the programs in the auth_stuff
# directory and figure out what this number should be on your machine.
#
authenticate yes
#
# Keys file. If you want to diddle your server at run time, make a
# keys file (mode 600 for sure) and define the key number to be
# used for making requests.
#
# PLEASE DO NOT USE THE DEFAULT VALUES HERE. Pick your own, or remote
# systems might be able to reset your clock at will. Note also that
# ntpd is started with a -A flag, disabling authentication, that
# will have to be removed as well.
#
keys /etc/ntp/keys
--- NEW FILE: README.lustre ---
Model
The OpenSSI support for Lustre currently extends to enabling
OpenSSI nodes to act as Lustre clients. This means that nodes
within a SSI cluster can mount a Lustre file through a remote
Lustre server that is external to the cluster.
Each node performs the mount locally. So, if individual nodes
need access to the Lustre filesystem, they need to mount it
individualy.
Getting it to work.
Here are the steps to enable Lustre on OpenSSI.
1. Download and install the OpenSSI 1.1.1 release from
http://www.openssi.org. The website has detailed instructions
on how to install OpenSSI.
This kernel already includes the Lustre patches for a given
version of Lustre. The currently supported version is 1.0.4.
2. Boot up with the new OpenSSI kernel. Add new nodes if any.
Now that you have the OpenSSI cluster up and running, here are the
remaining steps:
3. Install the lustre-ssi RPMs.
Make sure you install at least the lustre-modules and lustre-lite
RPMs. These contain the kernel modules and Lustre utilities
such as lconf respectively.
Note: The Lustre RPMs in lustre-fc2-1.1.1.i686.tar assume that you have
installed the kernel-ssi RPM from the 1.1.1 release and thus the
lustre_modules RPM installs the modules in
/lib/modules/2.4.22-1.2199.nptl_ssi_5develsmp/.
4. Once the installation is complete, you can use
your existing configuration files with lconf just the way
you were using it on rh9.
Notes:
1. If you need to play around with the Lustre source, make sure you download
the OpenSSI source tree for OpenSSI-1.1.1 since you will require that for
compiling the Lustre source tree.
If you do that, the installation procedure for Lustre is pretty much the
same as on the base. Make and install the Lustre package.
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
|
