From: jon Date: November 28 2008 9:51am Subject: svn commit - mysqldoc@docsrva: r12688 - in trunk: refman-4.1 refman-5.0 refman-5.1 List-Archive: http://lists.mysql.com/commits/60131 Message-Id: <200811280951.mAS9pptS022932@docsrva.mysql.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Author: jstephens Date: 2008-11-28 10:51:51 +0100 (Fri, 28 Nov 2008) New Revision: 12688 Log: Fixes Docs Bug #40596 (dated SCI docs) Modified: trunk/refman-4.1/mysql-cluster-interconnects.xml trunk/refman-5.0/mysql-cluster-interconnects.xml trunk/refman-5.1/mysql-cluster-interconnects.xml Modified: trunk/refman-4.1/mysql-cluster-interconnects.xml =================================================================== --- trunk/refman-4.1/mysql-cluster-interconnects.xml 2008-11-28 03:28:32 UTC (rev 12687) +++ trunk/refman-4.1/mysql-cluster-interconnects.xml 2008-11-28 09:51:51 UTC (rev 12688) Changed blocks: 1, Lines Added: 40, Lines Deleted: 466; 18578 bytes @@ -111,486 +111,60 @@ SCI (Scalable Coherent Interface) - + MySQL Cluster - - In this section, we show how to adapt a cluster configured for - normal TCP/IP communication to use SCI Sockets instead. This - documentation is based on SCI Sockets version 2.3.0 as of 01 - October 2004. - + + [js] Update following para to mention Windows when 6.4 becomes + available. + - - - Prerequisites - - - Any machines with which you wish to use SCI Sockets must be - equipped with SCI cards. - - - - - No special builds (other than the -max builds) - are needed for SCI Sockets because it uses normal TCP/IP socket - calls which are already available in MySQL Cluster. However, SCI - Sockets are currently supported only on the Linux 2.4 and 2.6 - kernels. For other operating systems, you can use SCI - Transporters, but this requires that the server be built using - . + It is possible employing Scalable Coherent Interface (SCI) + technology to achieve a significant increase in connection speeds + and throughput between MySQL Cluster data and SQL nodes. To use + SCI, it is necessary to obtain and install Dolphin SCI network + cards and to use the drivers and other software supplied by + Dolphin. You can get information on obtaining these, from + Dolphin Interconnect + Solutions. SCI SuperSocket or SCI Transporter support is + available for 32-bit and 64-bit Linux, Solaris, and other + platforms. See the Dolphin documentation referenced later in this + section for more detailed information regarding platforms + supported for SCI. - - MySQL Cluster - SCI software requirements - - - - There are essentially four requirements for SCI Sockets: - - - - - - - Building the SCI Socket libraries. - - - - - - Installation of the SCI Socket kernel libraries. - - - - - - Installation of one or two configuration files. - - - - - - The SCI Socket kernel library must be enabled either for the - entire machine or for the shell where the MySQL Cluster - processes are started. - - - - - - - This process needs to be repeated for each machine in the cluster - where you plan to use SCI Sockets for inter-node communication. - - - - Two packages need to be retrieved to get SCI Sockets working: - - - - - - - The source code package containing the DIS support libraries - for the SCI Sockets libraries. - - - - - - The source code package for the SCI Socket libraries - themselves. - - - - - - - Currently, these are available only in source code format. The - latest versions of these packages at the time of this writing were - available as (respectively) - DIS_GPL_2_5_0_SEP_10_2004.tar.gz and - SCI_SOCKET_2_3_0_OKT_01_2004.tar.gz. You - should be able to find these (or possibly newer versions) at - . - - - - - Package Installation - - - MySQL Cluster - SCI software installation - - - - Once you have obtained the library packages, the next step is to - unpack them into appropriate directories, with the SCI Sockets - library unpacked into a directory below the DIS code. Next, you - need to build the libraries. This example shows the commands - used on Linux/x86 to perform this task: - - - - - -shell> tar xzf DIS_GPL_2_5_0_SEP_10_2004.tar.gz -shell> cd DIS_GPL_2_5_0_SEP_10_2004/src/ -shell> tar xzf ../../SCI_SOCKET_2_3_0_OKT_01_2004.tar.gz -shell> cd ../adm/bin/Linux_pkgs -shell> ./make_PSB_66_release - - - - It is possible to build these libraries for some 64-bit procesors. - To build the libraries for Opteron CPUs using the 64-bit - extensions, run make_PSB_66_X86_64_release - rather than make_PSB_66_release. If the build - is made on an Itanium machine, you should use - make_PSB_66_IA64_release. The X86-64 variant - should work for Intel EM64T architectures but this has not yet (to - our knowledge) been tested. - - - - Once the build process is complete, the compiled libraries will be - found in a zipped tar file with a name along the lines of - DIS-<operating-system>-time-date. - It is now time to install the package in the proper place. In this - example we will place the installation in - /opt/DIS. You most likely need to run the - following as the system root user.) - - - -shell> cp DIS_Linux_2.4.20-8_181004.tar.gz /opt/ -shell> cd /opt -shell> tar xzf DIS_Linux_2.4.20-8_181004.tar.gz -shell> mv DIS_Linux_2.4.20-8_181004 DIS - - - - - Network Configuration - - - MySQL Cluster - network configuration (SCI) - - - - MySQL Cluster - SCI network configuration - - - - Now that all the libraries and binaries are in their proper - place, we need to ensure that the SCI cards have proper node IDs - within the SCI address space. - - - - - - It is also necessary to decide on the network structure before - proceeding. There are three types of network structures which can - be used in this context: - - - - - - - A simple one-dimensional ring - - - - - - One or more SCI switches with one ring per switch port - - - - - - A two- or three-dimensional torus. - - - - - - - Each of these topologies has its own method for providing node - IDs. We discuss each of them in brief. - - - - A simple ring uses node IDs which are non-zero multiples of 4: 4, - 8, 12,... - - - - The next possibility uses SCI switches. An SCI switch has 8 ports, - each of which can support a ring. It is necessary to make sure - that different rings use different node ID spaces. In a typical - configuration, the first port uses node IDs below 64 (4 − - 60), the next 64 node IDs (68 − 124) are assigned to the - next port, and so on, with node IDs 452 − 508 being assigned - to the eighth port. - - - - Two- and three-dimensional torus network structures take into - account where each node is located in each dimension, incrementing - by 4 for each node in the first dimension, by 64 in the second - dimension, and (where applicable) by 1024 in the third dimension. - See - Dolphin's - Web site for more thorough documentation. - - - - In our testing we have used switches, although most large cluster - installations use 2- or 3-dimensional torus structures. The - advantage provided by switches is that, with dual SCI cards and - dual switches, it is possible to build with relative ease a - redundant network where the average failover time on the SCI - network is on the order of 100 microseconds. This is supported by - the SCI transporter in MySQL Cluster and is also under development - for the SCI Socket implementation. - - - - Failover for the 2D/3D torus is also possible but requires sending - out new routing indexes to all nodes. However, this requires only - 100 milliseconds or so to complete and should be acceptable for - most high-availability cases. - - - - By placing cluster data nodes properly within the switched - architecture, it is possible to use 2 switches to build a - structure whereby 16 computers can be interconnected and no single - failure can hinder more than one of them. With 32 computers and 2 - switches it is possible to configure the cluster in such a manner - that no single failure can cause the loss of more than two nodes; - in this case, it is also possible to know which pair of nodes is - affected. Thus, by placing the two nodes in separate node groups, - it is possible to build a safe MySQL Cluster - installation. - - - - To set the node ID for an SCI card use the following command in - the /opt/DIS/sbin directory. In this example, - refers to the number of the SCI card (this - is always 1 if there is only 1 card in the machine); refers to adapter 0; and 68 is the - node ID: - - - -shell> ./sciconfig -c 1 -a 0 -n 68 - - - - If you have multiple SCI cards in the same machine, you can - determine which card has which slot by issuing the following - command (again we assume that the current working directory is - /opt/DIS/sbin): - - - -shell> ./sciconfig -c 1 -gsn - - - - This will give you the SCI card's serial number. Then repeat this - procedure with , and so on, for each card in - the machine. Once you have matched each card with a slot, you can - set node IDs for all cards. - - - - After the necessary libraries and binaries are installed, and the - SCI node IDs are set, the next step is to set up the mapping from - hostnames (or IP addresses) to SCI node IDs. This is done in the - SCI sockets configuration file, which should be saved as - /etc/sci/scisock.conf. In this file, each SCI - node ID is mapped through the proper SCI card to the hostname or - IP address that it is to communicate with. Here is a very simple - example of such a configuration file: - - - -#host #nodeId -alpha 8 -beta 12 -192.168.10.20 16 - - - - It is also possible to limit the configuration so that it applies - only to a subset of the available ports for these hosts. An - additional configuration file - /etc/sci/scisock_opt.conf can be used to - accomplish this, as shown here: - - - -#-key -type -values -EnablePortsByDefault yes -EnablePort tcp 2200 -DisablePort tcp 2201 -EnablePortRange tcp 2202 2219 -DisablePortRange tcp 2220 2231 - - - - - Driver Installation - - - MySQL Cluster - SCI drivers - - - - With the configuration files in place, the drivers can be - installed. - - - - - - First, the low-level drivers and then the SCI socket driver need - to be installed: - - - -shell> cd DIS/sbin/ -shell> ./drv-install add PSB66 -shell> ./scisocket-install add - - - - If desired, the installation can be checked by invoking a script - which verifies that all nodes in the SCI socket configuration - files are accessible: - - - -shell> cd /opt/DIS/sbin/ -shell> ./status.sh - - - - If you discover an error and need to change the SCI socket - configuration, it is necessary to use - ksocketconfig to accomplish this task: - - -shell> cd /opt/DIS/util -shell> ./ksocketconfig -f - - - For more information about ksocketconfig, - consult the documentation available from - . - - - - - Testing the Setup - - - To ensure that SCI sockets are actually being used, you can - employ the latency_bench test program. Using - this utility's server component, clients can connect to the - server to test the latency of the connection. Determining - whether SCI is enabled should be fairly simple from observing - the latency. - - - - - Before using latency_bench, it is necessary - to set the LD_PRELOAD environment variable as - shown later in this section. + Prior to MySQL 4.1.24, there were issues with building MySQL + Cluster with SCI support (see Bug #25470), but these have been + resolved due to work contributed by Dolphin. SCI Sockets are now + correctly supported for MySQL Cluster hosts running recent + versions of Linux using the -max builds, and + versions of MySQL Cluster with SCI Transporter support can be + built using either of compile-amd64-max-sci + or compile-pentium64-max-sci. Both of these + build scripts can be found in the BUILD + directory of the MySQL Cluster source trees; it should not be + difficult to adapt them for other platforms. Generally, all that + is necessary is to compile MySQL Cluster with SCI Transporter + support is to configure the MySQL Cluster build using + . - To set up a server, use the following: + Once you have acquired the required Dolphin hardware and software, + you can obtain detailed information on how to adapt a MySQL + Cluster configured for normal TCP/IP communication to use SCI from + the Dolphin Express for MySQL Installation and + Reference Guide, available for download at + + (PDF file, 94 pages, 753 KB). This document provides instructions + for installing the SCI hardware and software, as well as + information concerning network topology and configuration. - -shell> cd /opt/DIS/bin/socket -shell> ./latency_bench -server - - - - To run a client, use latency_bench again, - except this time with the option: - - - -shell> cd /opt/DIS/bin/socket -shell> ./latency_bench -client server_hostname - - - - SCI socket configuration should now be complete and MySQL Cluster - ready to use both SCI Sockets and the SCI transporter (see - ). - - - - - Starting the Cluster - - - The next step in the process is to start MySQL Cluster. To - enable usage of SCI Sockets it is necessary to set the - environment variable LD_PRELOAD before - starting ndbd, mysqld, and - ndb_mgmd. This variable should point to the - kernel library for SCI Sockets. - - - - - - To start ndbd in a bash shell, do the - following: - - - -bash-shell> export LD_PRELOAD=/opt/DIS/lib/libkscisock.so -bash-shell> ndbd - - - - In a tcsh environment the same thing can be accomplished with: - - - -tcsh-shell> setenv LD_PRELOAD=/opt/DIS/lib/libkscisock.so -tcsh-shell> ndbd - - - - - MySQL Cluster can use only the kernel variant of SCI Sockets. - - -
Modified: trunk/refman-5.0/mysql-cluster-interconnects.xml =================================================================== --- trunk/refman-5.0/mysql-cluster-interconnects.xml 2008-11-28 03:28:32 UTC (rev 12687) +++ trunk/refman-5.0/mysql-cluster-interconnects.xml 2008-11-28 09:51:51 UTC (rev 12688) Changed blocks: 1, Lines Added: 48, Lines Deleted: 488; 19704 bytes @@ -96,515 +96,75 @@
- + Configuring MySQL Cluster to use SCI Sockets - + MySQL Cluster network transporters - + MySQL Cluster SCI (Scalable Coherent Interface) - + SCI (Scalable Coherent Interface) MySQL Cluster - + + + [js] Update following para to mention Windows when 6.4 becomes + available. + + - In this section, we show how to adapt a cluster configured for - normal TCP/IP communication to use SCI Sockets instead. This - documentation is based on SCI Sockets version 2.3.0 as of 01 - October 2004. + It is possible employing Scalable Coherent Interface (SCI) + technology to achieve a significant increase in connection speeds + and throughput between MySQL Cluster data and SQL nodes. To use + SCI, it is necessary to obtain and install Dolphin SCI network + cards and to use the drivers and other software supplied by + Dolphin. You can get information on obtaining these, from + Dolphin Interconnect + Solutions. SCI SuperSocket or SCI Transporter support is + available for 32-bit and 64-bit Linux, Solaris, and other + platforms. See the Dolphin documentation referenced later in this + section for more detailed information regarding platforms + supported for SCI. - - - - Prerequisites - - - Any machines with which you wish to use SCI Sockets must be - equipped with SCI cards. - - - - - - No special builds (other than the -max builds) - are needed for SCI Sockets because it uses normal TCP/IP socket - calls which are already available in MySQL Cluster. However, SCI - Sockets are currently supported only on the Linux 2.4 and 2.6 - kernels. For other operating systems, you can use SCI - Transporters, but this requires that the server be built using - . - - - - Prior to MySQL 5.0.44, there were issues with building MySQL - Cluster with SCI support (see Bug #25470), but these have been - resolved due to work contributed by Dolphin International. SCI - Sockets are now correctly supported for MySQL Cluster using the - -max builds, and versions of MySQL Cluster with - SCI Transporter support can be built using either of - compile-amd64-max-sci or - compile-pentium64-max-sci. Both of these build - scripts can be found in the BUILD directory - of the MySQL ¤t-series; source; it should not be difficult - to adapt them for other platforms. - - - - MySQL Cluster - SCI software requirements - - - - There are essentially four requirements for SCI Sockets: - - - - - - - Building the SCI Socket libraries. - - - - - - Installation of the SCI Socket kernel libraries. - - - - - - Installation of one or two configuration files. - - - - - - The SCI Socket kernel library must be enabled either for the - entire machine or for the shell where the MySQL Cluster - processes are started. - - - - - - - This process needs to be repeated for each machine in the cluster - where you plan to use SCI Sockets for inter-node communication. - - - - Two packages need to be retrieved to get SCI Sockets working: - - - - - - - The source code package containing the DIS support libraries - for the SCI Sockets libraries. - - - - - - The source code package for the SCI Socket libraries - themselves. - - - - - - - Currently, these are available only in source code format. The - latest versions of these packages at the time of this writing were - available as (respectively) - DIS_GPL_2_5_0_SEP_10_2004.tar.gz and - SCI_SOCKET_2_3_0_OKT_01_2004.tar.gz. You - should be able to find these (or possibly newer versions) at - . - - - - - Package Installation - - - MySQL Cluster - SCI software installation - - - - Once you have obtained the library packages, the next step is to - unpack them into appropriate directories, with the SCI Sockets - library unpacked into a directory below the DIS code. Next, you - need to build the libraries. This example shows the commands - used on Linux/x86 to perform this task: - - - - - -shell> tar xzf DIS_GPL_2_5_0_SEP_10_2004.tar.gz -shell> cd DIS_GPL_2_5_0_SEP_10_2004/src/ -shell> tar xzf ../../SCI_SOCKET_2_3_0_OKT_01_2004.tar.gz -shell> cd ../adm/bin/Linux_pkgs -shell> ./make_PSB_66_release - - - - It is possible to build these libraries for some 64-bit procesors. - To build the libraries for Opteron CPUs using the 64-bit - extensions, run make_PSB_66_X86_64_release - rather than make_PSB_66_release. If the build - is made on an Itanium machine, you should use - make_PSB_66_IA64_release. The X86-64 variant - should work for Intel EM64T architectures but this has not yet (to - our knowledge) been tested. - - - - Once the build process is complete, the compiled libraries will be - found in a zipped tar file with a name along the lines of - DIS-<operating-system>-time-date. - It is now time to install the package in the proper place. In this - example we will place the installation in - /opt/DIS. You most likely need to run the - following as the system root user. - - - -shell> cp DIS_Linux_2.4.20-8_181004.tar.gz /opt/ -shell> cd /opt -shell> tar xzf DIS_Linux_2.4.20-8_181004.tar.gz -shell> mv DIS_Linux_2.4.20-8_181004 DIS - - - - - Network Configuration - - - MySQL Cluster - network configuration (SCI) - - - - MySQL Cluster - SCI network configuration - - - - Now that all the libraries and binaries are in their proper - place, we need to ensure that the SCI cards have proper node IDs - within the SCI address space. - - - - - - It is also necessary to decide on the network structure before - proceeding. There are three types of network structures which can - be used in this context: - - - - - - - A simple one-dimensional ring - - - - - - One or more SCI switches with one ring per switch port - - - - - - A two- or three-dimensional torus. - - - - - - - Each of these topologies has its own method for providing node - IDs. We discuss each of them in brief. - - - - A simple ring uses node IDs which are non-zero multiples of 4: 4, - 8, 12,... - - - - The next possibility uses SCI switches. An SCI switch has 8 ports, - each of which can support a ring. It is necessary to make sure - that different rings use different node ID spaces. In a typical - configuration, the first port uses node IDs below 64 (4 − - 60), the next 64 node IDs (68 − 124) are assigned to the - next port, and so on, with node IDs 452 − 508 being assigned - to the eighth port. - - - - Two- and three-dimensional torus network structures take into - account where each node is located in each dimension, incrementing - by 4 for each node in the first dimension, by 64 in the second - dimension, and (where applicable) by 1024 in the third dimension. - See - Dolphin's - Web site for more thorough documentation. - - - - In our testing we have used switches, although most large cluster - installations use 2- or 3-dimensional torus structures. The - advantage provided by switches is that, with dual SCI cards and - dual switches, it is possible to build with relative ease a - redundant network where the average failover time on the SCI - network is on the order of 100 microseconds. This is supported by - the SCI transporter in MySQL Cluster and is also under development - for the SCI Socket implementation. - - - - Failover for the 2D/3D torus is also possible but requires sending - out new routing indexes to all nodes. However, this requires only - 100 milliseconds or so to complete and should be acceptable for - most high-availability cases. - - - - By placing cluster data nodes properly within the switched - architecture, it is possible to use 2 switches to build a - structure whereby 16 computers can be interconnected and no single - failure can hinder more than one of them. With 32 computers and 2 - switches it is possible to configure the cluster in such a manner - that no single failure can cause the loss of more than two nodes; - in this case, it is also possible to know which pair of nodes is - affected. Thus, by placing the two nodes in separate node groups, - it is possible to build a safe MySQL Cluster - installation. - - - - To set the node ID for an SCI card use the following command in - the /opt/DIS/sbin directory. In this example, - refers to the number of the SCI card (this - is always 1 if there is only 1 card in the machine); refers to adapter 0; and 68 is the - node ID: - - - -shell> ./sciconfig -c 1 -a 0 -n 68 - - - - If you have multiple SCI cards in the same machine, you can - determine which card has which slot by issuing the following - command (again we assume that the current working directory is - /opt/DIS/sbin): - - - -shell> ./sciconfig -c 1 -gsn - - - - This will give you the SCI card's serial number. Then repeat this - procedure with , and so on, for each card in - the machine. Once you have matched each card with a slot, you can - set node IDs for all cards. - - - - After the necessary libraries and binaries are installed, and the - SCI node IDs are set, the next step is to set up the mapping from - hostnames (or IP addresses) to SCI node IDs. This is done in the - SCI sockets configuration file, which should be saved as - /etc/sci/scisock.conf. In this file, each SCI - node ID is mapped through the proper SCI card to the hostname or - IP address that it is to communicate with. Here is a very simple - example of such a configuration file: - - - -#host #nodeId -alpha 8 -beta 12 -192.168.10.20 16 - - - - It is also possible to limit the configuration so that it applies - only to a subset of the available ports for these hosts. An - additional configuration file - /etc/sci/scisock_opt.conf can be used to - accomplish this, as shown here: - - - -#-key -type -values -EnablePortsByDefault yes -EnablePort tcp 2200 -DisablePort tcp 2201 -EnablePortRange tcp 2202 2219 -DisablePortRange tcp 2220 2231 - - - - - Driver Installation - - - MySQL Cluster - SCI drivers - - - - With the configuration files in place, the drivers can be - installed. - - - - - - First, the low-level drivers and then the SCI socket driver need - to be installed: - - - -shell> cd DIS/sbin/ -shell> ./drv-install add PSB66 -shell> ./scisocket-install add - - - - If desired, the installation can be checked by invoking a script - which verifies that all nodes in the SCI socket configuration - files are accessible: - - - -shell> cd /opt/DIS/sbin/ -shell> ./status.sh - - - - If you discover an error and need to change the SCI socket - configuration, it is necessary to use - ksocketconfig to accomplish this task: - - -shell> cd /opt/DIS/util -shell> ./ksocketconfig -f - - - For more information about ksocketconfig, - consult the documentation available from - . - - - - - Testing the setup - - - To ensure that SCI sockets are actually being used, you can - employ the latency_bench test program. Using - this utility's server component, clients can connect to the - server to test the latency of the connection. Determining - whether SCI is enabled should be fairly simple from observing - the latency. - - - - + - Before using latency_bench, it is necessary - to set the LD_PRELOAD environment variable as - shown later in this section. + Prior to MySQL 5.0.66, there were issues with building MySQL + Cluster with SCI support (see Bug #25470), but these have been + resolved due to work contributed by Dolphin. SCI Sockets are now + correctly supported for MySQL Cluster hosts running recent + versions of Linux using the -max builds, and + versions of MySQL Cluster with SCI Transporter support can be + built using either of compile-amd64-max-sci + or compile-pentium64-max-sci. Both of these + build scripts can be found in the BUILD + directory of the MySQL Cluster source trees; it should not be + difficult to adapt them for other platforms. Generally, all that + is necessary is to compile MySQL Cluster with SCI Transporter + support is to configure the MySQL Cluster build using + . - + - To set up a server, use the following: + Once you have acquired the required Dolphin hardware and software, + you can obtain detailed information on how to adapt a MySQL + Cluster configured for normal TCP/IP communication to use SCI from + the Dolphin Express for MySQL Installation and + Reference Guide, available for download at + + (PDF file, 94 pages, 753 KB). This document provides instructions + for installing the SCI hardware and software, as well as + information concerning network topology and configuration. - - -shell> cd /opt/DIS/bin/socket -shell> ./latency_bench -server - - - - To run a client, use latency_bench again, - except this time with the option: - - - -shell> cd /opt/DIS/bin/socket -shell> ./latency_bench -client server_hostname - - - - SCI socket configuration should now be complete and MySQL Cluster - ready to use both SCI Sockets and the SCI transporter (see - ). - - - - - Starting the cluster - - - The next step in the process is to start MySQL Cluster. To - enable usage of SCI Sockets it is necessary to set the - environment variable LD_PRELOAD before - starting ndbd, mysqld, and - ndb_mgmd. This variable should point to the - kernel library for SCI Sockets. - - - - - - To start ndbd in a bash shell, do the - following: - - - -bash-shell> export LD_PRELOAD=/opt/DIS/lib/libkscisock.so -bash-shell> ndbd - - - - In a tcsh environment the same thing can be accomplished with: - - - -tcsh-shell> setenv LD_PRELOAD=/opt/DIS/lib/libkscisock.so -tcsh-shell> ndbd - - - - - MySQL Cluster can use only the kernel variant of SCI Sockets. - - - +
Modified: trunk/refman-5.1/mysql-cluster-interconnects.xml =================================================================== --- trunk/refman-5.1/mysql-cluster-interconnects.xml 2008-11-28 03:28:32 UTC (rev 12687) +++ trunk/refman-5.1/mysql-cluster-interconnects.xml 2008-11-28 09:51:51 UTC (rev 12688) Changed blocks: 2, Lines Added: 41, Lines Deleted: 481; 19604 bytes @@ -92,7 +92,8 @@ the TCP/IP stack to one extent or another. We have experimented with both of these techniques using the SCI (Scalable Coherent Interface) technology developed by - Dolphin. + Dolphin Interconnect + Solutions.
@@ -114,498 +115,57 @@ MySQL Cluster - - In this section, we show how to adapt a cluster configured for - normal TCP/IP communication to use SCI Sockets instead. This - documentation is based on SCI Sockets version 2.3.0 as of 01 - October 2004. - + + [js] Update following para to mention Windows when 6.4 becomes + available. + - - - Prerequisites - - - Any machines with which you wish to use SCI Sockets must be - equipped with SCI cards. - - - - - No special builds (other than the -max builds) - are needed for SCI Sockets because it uses normal TCP/IP socket - calls which are already available in MySQL Cluster. However, SCI - Sockets are currently supported only on the Linux 2.4 and 2.6 - kernels. For other operating systems, you can use SCI - Transporters, but this requires that the server be built using - . + It is possible employing Scalable Coherent Interface (SCI) + technology to achieve a significant increase in connection speeds + and throughput between MySQL Cluster data and SQL nodes. To use + SCI, it is necessary to obtain and install Dolphin SCI network + cards and to use the drivers and other software supplied by + Dolphin. You can get information on obtaining these, from + Dolphin Interconnect + Solutions. SCI SuperSocket or SCI Transporter support is + available for 32-bit and 64-bit Linux, Solaris, and other + platforms. See the Dolphin documentation referenced later in this + section for more detailed information regarding platforms + supported for SCI. - - Prior to MySQL 5.1.20, there were issues with building MySQL - Cluster with SCI support (see Bug #25470), but these have been - resolved due to work contributed by Dolphin International. SCI - Sockets are now correctly supported for MySQL Cluster using the - -max builds, and versions of MySQL Cluster with - SCI Transporter support can be built using either of - compile-amd64-max-sci or - compile-pentium64-max-sci. Both of these build - scripts can be found in the BUILD directory - of the MySQL ¤t-series; source; it should not be difficult - to adapt them for other platforms. - - - - MySQL Cluster - SCI software requirements - - - - There are essentially four requirements for SCI Sockets: - - - - - - - Building the SCI Socket libraries. - - - - - - Installation of the SCI Socket kernel libraries. - - - - - - Installation of one or two configuration files. - - - - - - The SCI Socket kernel library must be enabled either for the - entire machine or for the shell where the MySQL Cluster - processes are started. - - - - - - - This process needs to be repeated for each machine in the cluster - where you plan to use SCI Sockets for inter-node communication. - - - - Two packages need to be retrieved to get SCI Sockets working: - - - - - - - The source code package containing the DIS support libraries - for the SCI Sockets libraries. - - - - - - The source code package for the SCI Socket libraries - themselves. - - - - - - - Currently, these are available only in source code format. The - latest versions of these packages at the time of this writing were - available as (respectively) - DIS_GPL_2_5_0_SEP_10_2004.tar.gz and - SCI_SOCKET_2_3_0_OKT_01_2004.tar.gz. You - should be able to find these (or possibly newer versions) at - . - - - - - Package Installation - - - MySQL Cluster - SCI software installation - - - - Once you have obtained the library packages, the next step is to - unpack them into appropriate directories, with the SCI Sockets - library unpacked into a directory below the DIS code. Next, you - need to build the libraries. This example shows the commands - used on Linux/x86 to perform this task: - - - - - -shell> tar xzf DIS_GPL_2_5_0_SEP_10_2004.tar.gz -shell> cd DIS_GPL_2_5_0_SEP_10_2004/src/ -shell> tar xzf ../../SCI_SOCKET_2_3_0_OKT_01_2004.tar.gz -shell> cd ../adm/bin/Linux_pkgs -shell> ./make_PSB_66_release - - - - It is possible to build these libraries for some 64-bit - processors. To build the libraries for Opteron CPUs using the - 64-bit extensions, run - make_PSB_66_X86_64_release rather than - make_PSB_66_release. If the build is made on an - Itanium machine, you should use - make_PSB_66_IA64_release. The X86-64 variant - should work for Intel EM64T architectures but this has not yet (to - our knowledge) been tested. - - - - Once the build process is complete, the compiled libraries will be - found in a zipped tar file with a name along the lines of - DIS-<operating-system>-time-date. - It is now time to install the package in the proper place. In this - example we will place the installation in - /opt/DIS. You most likely need to run the - following as the system root user. - - - -shell> cp DIS_Linux_2.4.20-8_181004.tar.gz /opt/ -shell> cd /opt -shell> tar xzf DIS_Linux_2.4.20-8_181004.tar.gz -shell> mv DIS_Linux_2.4.20-8_181004 DIS - - - - - Network configuration - - - MySQL Cluster - network configuration (SCI) - - - - MySQL Cluster - SCI network configuration - - - - Now that all the libraries and binaries are in their proper - place, we need to ensure that the SCI cards have proper node IDs - within the SCI address space. - - - - - - It is also necessary to decide on the network structure before - proceeding. There are three types of network structures which can - be used in this context: - - - - - - - A simple one-dimensional ring - - - - - - One or more SCI switches with one ring per switch port - - - - - - A two- or three-dimensional torus. - - - - - - - Each of these topologies has its own method for providing node - IDs. We discuss each of them in brief. - - - - A simple ring uses node IDs which are non-zero multiples of 4: 4, - 8, 12,... - - - - The next possibility uses SCI switches. An SCI switch has 8 ports, - each of which can support a ring. It is necessary to make sure - that different rings use different node ID spaces. In a typical - configuration, the first port uses node IDs below 64 (4 − - 60), the next 64 node IDs (68 − 124) are assigned to the - next port, and so on, with node IDs 452 − 508 being assigned - to the eighth port. - - - - Two- and three-dimensional torus network structures take into - account where each node is located in each dimension, incrementing - by 4 for each node in the first dimension, by 64 in the second - dimension, and (where applicable) by 1024 in the third dimension. - See - Dolphin's - Web site for more thorough documentation. - - - - In our testing we have used switches, although most large cluster - installations use 2- or 3-dimensional torus structures. The - advantage provided by switches is that, with dual SCI cards and - dual switches, it is possible to build with relative ease a - redundant network where the average failover time on the SCI - network is on the order of 100 microseconds. This is supported by - the SCI transporter in MySQL Cluster and is also under development - for the SCI Socket implementation. - - - - Failover for the 2D/3D torus is also possible but requires sending - out new routing indexes to all nodes. However, this requires only - 100 milliseconds or so to complete and should be acceptable for - most high-availability cases. - - - - By placing cluster data nodes properly within the switched - architecture, it is possible to use 2 switches to build a - structure whereby 16 computers can be interconnected and no single - failure can hinder more than one of them. With 32 computers and 2 - switches it is possible to configure the cluster in such a manner - that no single failure can cause the loss of more than two nodes; - in this case, it is also possible to know which pair of nodes is - affected. Thus, by placing the two nodes in separate node groups, - it is possible to build a safe MySQL Cluster - installation. - - - - To set the node ID for an SCI card use the following command in - the /opt/DIS/sbin directory. In this example, - refers to the number of the SCI card (this - is always 1 if there is only 1 card in the machine); refers to adapter 0; and 68 is the - node ID: - - - -shell> ./sciconfig -c 1 -a 0 -n 68 - - - - If you have multiple SCI cards in the same machine, you can - determine which card has which slot by issuing the following - command (again we assume that the current working directory is - /opt/DIS/sbin): - - - -shell> ./sciconfig -c 1 -gsn - - - - This will give you the SCI card's serial number. Then repeat this - procedure with , and so on, for each card in - the machine. Once you have matched each card with a slot, you can - set node IDs for all cards. - - - - After the necessary libraries and binaries are installed, and the - SCI node IDs are set, the next step is to set up the mapping from - hostnames (or IP addresses) to SCI node IDs. This is done in the - SCI sockets configuration file, which should be saved as - /etc/sci/scisock.conf. In this file, each SCI - node ID is mapped through the proper SCI card to the hostname or - IP address that it is to communicate with. Here is a very simple - example of such a configuration file: - - - -#host #nodeId -alpha 8 -beta 12 -192.168.10.20 16 - - - - It is also possible to limit the configuration so that it applies - only to a subset of the available ports for these hosts. An - additional configuration file - /etc/sci/scisock_opt.conf can be used to - accomplish this, as shown here: - - - -#-key -type -values -EnablePortsByDefault yes -EnablePort tcp 2200 -DisablePort tcp 2201 -EnablePortRange tcp 2202 2219 -DisablePortRange tcp 2220 2231 - - - - - Driver installation - - - MySQL Cluster - SCI drivers - - - - With the configuration files in place, the drivers can be - installed. - - - - - - First, the low-level drivers and then the SCI socket driver need - to be installed: - - - -shell> cd DIS/sbin/ -shell> ./drv-install add PSB66 -shell> ./scisocket-install add - - - - If desired, the installation can be checked by invoking a script - which verifies that all nodes in the SCI socket configuration - files are accessible: - - - -shell> cd /opt/DIS/sbin/ -shell> ./status.sh - - - - If you discover an error and need to change the SCI socket - configuration, it is necessary to use - ksocketconfig to accomplish this task: - - -shell> cd /opt/DIS/util -shell> ./ksocketconfig -f - - - For more information about ksocketconfig, - consult the documentation available from - . - - - - - Testing the setup - - - To ensure that SCI sockets are actually being used, you can - employ the latency_bench test program. Using - this utility's server component, clients can connect to the - server to test the latency of the connection. Determining - whether SCI is enabled should be fairly simple from observing - the latency. - - - - - Before using latency_bench, it is necessary - to set the LD_PRELOAD environment variable as - shown later in this section. + Prior to MySQL 5.1.20, there were issues with building MySQL + Cluster with SCI support (see Bug #25470), but these have been + resolved due to work contributed by Dolphin. SCI Sockets are now + correctly supported for MySQL Cluster hosts running recent + versions of Linux using the -max builds, and + versions of MySQL Cluster with SCI Transporter support can be + built using either of compile-amd64-max-sci + or compile-pentium64-max-sci. Both of these + build scripts can be found in the BUILD + directory of the MySQL Cluster source trees; it should not be + difficult to adapt them for other platforms. Generally, all that + is necessary is to compile MySQL Cluster with SCI Transporter + support is to configure the MySQL Cluster build using + . - To set up a server, use the following: + Once you have acquired the required Dolphin hardware and software, + you can obtain detailed information on how to adapt a MySQL + Cluster configured for normal TCP/IP communication to use SCI from + the Dolphin Express for MySQL Installation and + Reference Guide, available for download at + + (PDF file, 94 pages, 753 KB). This document provides instructions + for installing the SCI hardware and software, as well as + information concerning network topology and configuration. - -shell> cd /opt/DIS/bin/socket -shell> ./latency_bench -server - - - - To run a client, use latency_bench again, - except this time with the option: - - - -shell> cd /opt/DIS/bin/socket -shell> ./latency_bench -client server_hostname - - - - SCI socket configuration should now be complete and MySQL Cluster - ready to use both SCI Sockets and the SCI transporter (see - ). - - - - - Starting the cluster - - - The next step in the process is to start MySQL Cluster. To - enable usage of SCI Sockets it is necessary to set the - environment variable LD_PRELOAD before - starting ndbd, mysqld, and - ndb_mgmd. This variable should point to the - kernel library for SCI Sockets. - - - - - - To start ndbd in a bash shell, do the - following: - - - -bash-shell> export LD_PRELOAD=/opt/DIS/lib/libkscisock.so -bash-shell> ndbd - - - - In a tcsh environment the same thing can be accomplished with: - - - -tcsh-shell> setenv LD_PRELOAD=/opt/DIS/lib/libkscisock.so -tcsh-shell> ndbd - - - - - MySQL Cluster can use only the kernel variant of SCI Sockets. - - -