| List: | Commits | « Previous MessageNext Message » | |
| From: | David Li | Date: | January 28 2008 12:44pm |
| Subject: | bk commit into 5.1 tree (dli:1.2855) | ||
| View as plain text | |||
Below is the list of changes that have just been committed into a local 5.1 repository of dli. When dli does a push these changes will be propagated to the main repository and, within 24 hours after the push, to the public repository. For information on how to access the public repository see http://dev.mysql.com/doc/mysql/en/installing-source-tree.html ChangeSet@stripped, 2008-01-28 19:44:10+08:00, dli@stripped +35 -0 Merge dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-endian into dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-wl3615-merge MERGE: 1.2482.150.1 config/ac-macros/ha_ndbcluster.m4@stripped, 2008-01-28 18:11:28+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.28.1.1 storage/ndb/include/kernel/signaldata/ScanTab.hpp@stripped, 2008-01-28 18:11:28+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.19.1.1 storage/ndb/include/kernel/signaldata/TupCommit.hpp@stripped, 2008-01-28 19:44:01+08:00, dli@stripped +2 -3 Merge dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-endian into dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-wl3615-merge MERGE: 1.4.1.1 storage/ndb/include/ndb_version.h.in@stripped, 2008-01-28 19:44:01+08:00, dli@stripped +7 -7 Merge dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-endian into dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-wl3615-merge MERGE: 1.18.1.1 storage/ndb/src/common/transporter/Packer.cpp@stripped, 2008-01-28 19:44:01+08:00, dli@stripped +0 -1 Merge dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-endian into dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-wl3615-merge MERGE: 1.12.1.2 storage/ndb/src/common/util/Makefile.am@stripped, 2008-01-28 18:11:28+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.14.1.1 storage/ndb/src/kernel/blocks/backup/Backup.cpp@stripped, 2008-01-28 18:11:29+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.71.2.1 storage/ndb/src/kernel/blocks/cmvmi/Cmvmi.cpp@stripped, 2008-01-28 18:11:29+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.53.1.1 storage/ndb/src/kernel/blocks/dbacc/DbaccMain.cpp@stripped, 2008-01-28 18:11:29+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.88.1.1 storage/ndb/src/kernel/blocks/dbdict/Dbdict.cpp@stripped, 2008-01-28 19:44:01+08:00, dli@stripped +49 -58 Merge dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-endian into dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-wl3615-merge MERGE: 1.140.2.1 storage/ndb/src/kernel/blocks/dblqh/Dblqh.hpp@stripped, 2008-01-28 19:44:01+08:00, dli@stripped +8 -7 Merge dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-endian into dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-wl3615-merge MERGE: 1.78.1.1 storage/ndb/src/kernel/blocks/dblqh/DblqhMain.cpp@stripped, 2008-01-28 19:44:01+08:00, dli@stripped +3 -3 Merge dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-endian into dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-wl3615-merge MERGE: 1.180.2.1 storage/ndb/src/kernel/blocks/dbtc/Dbtc.hpp@stripped, 2008-01-28 19:44:02+08:00, dli@stripped +9 -8 Merge dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-endian into dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-wl3615-merge MERGE: 1.59.1.1 storage/ndb/src/kernel/blocks/dbtc/DbtcMain.cpp@stripped, 2008-01-28 19:44:02+08:00, dli@stripped +1 -1 Merge dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-endian into dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-wl3615-merge MERGE: 1.157.1.1 storage/ndb/src/kernel/blocks/dbtup/Dbtup.hpp@stripped, 2008-01-28 18:11:30+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.85.1.1 storage/ndb/src/kernel/blocks/dbtup/DbtupCommit.cpp@stripped, 2008-01-28 19:44:02+08:00, dli@stripped +6 -6 Merge dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-endian into dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-wl3615-merge MERGE: 1.37.1.1 storage/ndb/src/kernel/blocks/dbtup/DbtupExecQuery.cpp@stripped, 2008-01-28 18:11:31+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.88.2.1 storage/ndb/src/kernel/blocks/dbtup/DbtupGen.cpp@stripped, 2008-01-28 18:11:31+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.52.1.1 storage/ndb/src/kernel/blocks/dbtup/DbtupIndex.cpp@stripped, 2008-01-28 18:11:31+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.28.1.1 storage/ndb/src/kernel/blocks/dbtup/DbtupMeta.cpp@stripped, 2008-01-28 18:11:31+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.49.1.1 storage/ndb/src/kernel/blocks/dbtup/DbtupRoutines.cpp@stripped, 2008-01-28 18:11:31+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.42.1.1 storage/ndb/src/kernel/blocks/dbtup/DbtupScan.cpp@stripped, 2008-01-28 18:11:31+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.36.1.1 storage/ndb/src/kernel/blocks/dbtux/DbtuxGen.cpp@stripped, 2008-01-28 18:11:31+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.26.1.1 storage/ndb/src/kernel/blocks/dbutil/DbUtil.cpp@stripped, 2008-01-28 18:11:32+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.18.1.1 storage/ndb/src/kernel/blocks/ndbcntr/NdbcntrMain.cpp@stripped, 2008-01-28 18:11:32+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.66.1.1 storage/ndb/src/kernel/blocks/qmgr/QmgrMain.cpp@stripped, 2008-01-28 18:11:32+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.59.1.1 storage/ndb/src/kernel/blocks/restore.cpp@stripped, 2008-01-28 18:11:28+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.19.1.1 storage/ndb/src/kernel/blocks/suma/Suma.cpp@stripped, 2008-01-28 18:11:32+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.68.1.1 storage/ndb/src/kernel/vm/SimulatedBlock.cpp@stripped, 2008-01-28 18:11:32+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.42.2.1 storage/ndb/src/kernel/vm/SimulatedBlock.hpp@stripped, 2008-01-28 18:11:33+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.35.1.1 storage/ndb/src/mgmsrv/Services.cpp@stripped, 2008-01-28 18:11:33+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.89.1.1 storage/ndb/src/ndbapi/ClusterMgr.cpp@stripped, 2008-01-28 19:44:02+08:00, dli@stripped +1 -1 Merge dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-endian into dev3-190.dev.cn.tlan:/home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-wl3615-merge MERGE: 1.44.1.1 storage/ndb/src/ndbapi/NdbOperationExec.cpp@stripped, 2008-01-28 18:11:33+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.40.2.1 storage/ndb/src/ndbapi/NdbScanOperation.cpp@stripped, 2008-01-28 18:11:33+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.118.1.1 storage/ndb/src/ndbapi/TransporterFacade.hpp@stripped, 2008-01-28 18:11:33+08:00, dli@stripped +0 -0 Auto merged MERGE: 1.36.1.1 # This is a BitKeeper patch. What follows are the unified diffs for the # set of deltas contained in the patch. The rest of the patch, the part # that BitKeeper cares about, is below these diffs. # User: dli # Host: dev3-190.dev.cn.tlan # Root: /home/dli/mysql/mysql-5.1/mysql-5.1-telco-6.3-wl3615-merge/RESYNC --- 1.29/config/ac-macros/ha_ndbcluster.m4 2008-01-28 19:44:26 +08:00 +++ 1.30/config/ac-macros/ha_ndbcluster.m4 2008-01-28 19:44:26 +08:00 @@ -175,6 +175,46 @@ AC_DEFUN([NDBCLUSTER_WORKAROUNDS], [ esac ]) +AC_DEFUN([NDBCLUSTER_BYTEORDER], [ + + # This is for WL#3615(endian incompatability support) to check the + # architecture of the CPU so that can use assembler functions to + # improve the performance while swapping byte order. + + case $MACHINE_TYPE-$ac_cv_prog_gcc in + i?86-yes) + AC_DEFINE([NDB_ARCH_I386_GCC], [1], + [Define to 1 if processor is i386 and compiler is gcc]) + ;; + ia64-yes) + AC_DEFINE([NDB_ARCH_IA64_GCC], [1], + [Define to 1 if processor is ia64 and compiler is gcc]) + ;; + x86_64-yes) + AC_DEFINE([NDB_ARCH_X86_64_GCC], [1], + [Define to 1 if processor is x86_64 and compiler is gcc]) + ;; + sparc-yes) + AC_DEFINE([NDB_ARCH_SPARC_GCC], [1], + [Define to 1 if processor is sparc and compiler is gcc]) + ;; + sparc64-yes) + AC_DEFINE([NDB_ARCH_SPARC64_GCC], [1], + [Define to 1 if processor is sparc64 and compiler is gcc]) + ;; + ppc-yes) + AC_DEFINE([NDB_ARCH_PPC_GCC], [1], + [Define to 1 if processor is ppc and compiler is gcc]) + ;; + ppc64-yes) + AC_DEFINE([NDB_ARCH_PPC64_GCC], [1], + [Define to 1 if processor is ppc64 and compiler is gcc]) + ;; + *) + ;; + esac +]) + AC_DEFUN([MYSQL_SETUP_NDBCLUSTER], [ AC_MSG_RESULT([Using NDB Cluster]) @@ -189,6 +229,7 @@ AC_DEFUN([MYSQL_SETUP_NDBCLUSTER], [ MYSQL_CHECK_NDB_OPTIONS NDBCLUSTER_WORKAROUNDS + NDBCLUSTER_BYTEORDER MAKE_BINARY_DISTRIBUTION_OPTIONS="$MAKE_BINARY_DISTRIBUTION_OPTIONS --with-ndbcluster" --- 1.22/storage/ndb/src/kernel/blocks/restore.cpp 2008-01-28 19:44:26 +08:00 +++ 1.23/storage/ndb/src/kernel/blocks/restore.cpp 2008-01-28 19:44:26 +08:00 @@ -29,6 +29,7 @@ #include <Dblqh.hpp> #include <dbtup/Dbtup.hpp> #include <KeyDescriptor.hpp> +#include <ndb_byteorder.h> #define PAGES LCP_RESTORE_BUFFER @@ -1058,6 +1059,8 @@ Restore::parse_record(Signal* signal, Fi LqhKeyReq::setNoDiskFlag(tmp, disk ? 0 : 1); LqhKeyReq::setRowidFlag(tmp, 1); LqhKeyReq::setGCIFlag(tmp, gci); + // Set own byte order + LqhKeyReq::setApiByteOrder(tmp, MY_OWN_BYTE_ORDER); req->clientConnectPtr = file_ptr.i; req->hashValue = hashValue; req->requestInfo = tmp; --- 1.27/storage/ndb/include/ndb_version.h.in 2008-01-28 19:44:26 +08:00 +++ 1.28/storage/ndb/include/ndb_version.h.in 2008-01-28 19:44:26 +08:00 @@ -171,5 +171,12 @@ ndb_pnr(Uint32 version) version == NDB_VERSION_D || version >= NDBD_PNR; } +/** + * From which version do we support running on mixed endian hosts. + * FIXME: The version code should be changed before the endian + * patch being pushed into main tree. + */ +#define NDBD_ENDIAN_COMPATIBLE_VERSION MAKE_VERSION(6,3,8) + #endif --- 1.20/storage/ndb/include/kernel/signaldata/ScanTab.hpp 2008-01-28 19:44:26 +08:00 +++ 1.21/storage/ndb/include/kernel/signaldata/ScanTab.hpp 2008-01-28 19:44:26 +08:00 @@ -90,6 +90,7 @@ private: static Uint16 getScanBatch(const UintR & requestInfo); static Uint8 getDistributionKeyFlag(const UintR & requestInfo); static UintR getNoDiskFlag(const UintR & requestInfo); + static Uint32 getApiByteOrder(const UintR & requestInfo); /** * Set:ers for requestInfo @@ -106,6 +107,7 @@ private: static void setScanBatch(Uint32& requestInfo, Uint32 sz); static void setDistributionKeyFlag(Uint32& requestInfo, Uint32 flag); static void setNoDiskFlag(UintR & requestInfo, UintR val); + static void setApiByteOrder(UintR & requestInfo, Uint32 flag); }; /** @@ -126,10 +128,11 @@ private: b = Scan batch - 10 Bit 16-25 (max 1023) d = Distribution key flag - 1 Bit 26 n = No disk flag - 1 Bit 9 + e = byte order of API - 1 Bit 27 1111111111222222222233 01234567890123456789012345678901 - pppppppplnhcktzxbbbbbbbbbbd + pppppppplnhcktzxbbbbbbbbbbde */ #define PARALLEL_SHIFT (0) @@ -165,6 +168,9 @@ private: #define SCAN_NODISK_SHIFT (9) #define SCAN_NODISK_MASK (1) +#define API_BYTEORDER_SHIFT (27) +#define API_BYTEORDER_MASK (1) + inline Uint8 ScanTabReq::getParallelism(const UintR & requestInfo){ @@ -214,6 +220,12 @@ ScanTabReq::getScanBatch(const Uint32 & } inline +Uint32 +ScanTabReq::getApiByteOrder(const UintR & requestInfo){ + return (requestInfo >> API_BYTEORDER_SHIFT) & API_BYTEORDER_MASK; +} + +inline void ScanTabReq::clearRequestInfo(UintR & requestInfo){ requestInfo = 0; @@ -323,6 +335,14 @@ ScanTabReq::setNoDiskFlag(UintR & reques ASSERT_BOOL(flag, "TcKeyReq::setNoDiskFlag"); requestInfo= (requestInfo & ~(SCAN_NODISK_MASK << SCAN_NODISK_SHIFT)) | ((flag & SCAN_NODISK_MASK) << SCAN_NODISK_SHIFT); +} + +inline +void +ScanTabReq::setApiByteOrder(UintR & requestInfo, Uint32 flag){ + ASSERT_MAX(flag, API_BYTEORDER_MASK, "ScanTabReq::setApiByteOrder"); + requestInfo= (requestInfo & ~(API_BYTEORDER_MASK << API_BYTEORDER_SHIFT)) | + ((flag & API_BYTEORDER_MASK) << API_BYTEORDER_SHIFT); } /** --- 1.5/storage/ndb/include/kernel/signaldata/TupCommit.hpp 2008-01-28 19:44:27 +08:00 +++ 1.6/storage/ndb/include/kernel/signaldata/TupCommit.hpp 2008-01-28 19:44:27 +08:00 @@ -35,7 +35,7 @@ class TupCommitReq { friend bool printTUPCOMMITREQ(FILE * output, const Uint32 * theData, Uint32 len, Uint16 receiverBlockNo); public: - STATIC_CONST( SignalLength = 5 ); + STATIC_CONST( SignalLength = 6 ); private: @@ -47,6 +47,19 @@ private: Uint32 hashValue; Uint32 diskpage; Uint32 gci_lo; + /** + * After recieved TUP_COMMITREQ signal, it will send events from the SUMA + * block to MYSQLDs to handle MySQL Cluster replication by SUB_TABLE_DATA + * signal. Since there may be several subscribers of the SUB_TABLE_DATA + * event, and they could be mixed endian, we have to handle endian in the + * API when reading the event. + * For the first version of mixed endian support, we will only support + * events when all MYSQLDs(APIs) are on the same endian (though the endian + * of MYSQLD may differ from that of data nodes). + * The following variable member is used to transfer the byte order of API + * node from DBLQH block to DBTUP block. + */ + Uint32 apiByteOrder; }; #endif --- 1.15/storage/ndb/src/common/transporter/Packer.cpp 2008-01-28 19:44:27 +08:00 +++ 1.16/storage/ndb/src/common/transporter/Packer.cpp 2008-01-28 19:44:27 +08:00 @@ -14,6 +14,7 @@ Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */ #include <ndb_global.h> +#include <ndb_byteorder.h> #include "Packer.hpp" #include <TransporterRegistry.hpp> @@ -45,11 +46,23 @@ TransporterRegistry::unpack(Uint32 * rea Uint32 word3 = readPtr[2]; loop_count++; -#if 0 - if(Protocol6::getByteOrder(word1) != MY_OWN_BYTE_ORDER){ - //Do funky stuff + /** + * The bit 0, 7, 24 and 31 of the first word(word1) indicates byte order + * of the sending node. All the 4 bits are 0 indicates little-endian, all + * the 4 bits are 1 indicates big-endian. So it doesn't matter what + * endianess the receiving node is, by checking bit 31 on the receiving + * node will read a 0 in this bit if the sending node is little-endian + * and read 1 means that the sending node is big-endian. + * + * If the byte order of the sending node is different from the receiving + * node, should firstly swap byte order for the first 3 words to get the + * general information of the signal (e.g. the length of signal). + */ + if (Protocol6::getByteOrder(word1) != MY_OWN_BYTE_ORDER){ + ndbSwapOneWord32(&word1); + ndbSwapOneWord32(&word2); + ndbSwapOneWord32(&word3); } -#endif const Uint16 messageLen32 = Protocol6::getMessageLength(word1); const Uint32 messageLenBytes = ((Uint32)messageLen32) << 2; @@ -64,6 +77,20 @@ TransporterRegistry::unpack(Uint32 * rea break; }//if + /** + * Swap the byte order for each 32-bit word of the signal data if the + * endianess of the signal sender is different from the signal receiver. + * Since all protocols in NDB are centered around sending 32-bit(4 bytes) + * words, for the most signals, this simple conversion can make it work. + * For the character strings (and other non-32bit-word data) in the + * signal, their byte order should not be converted (but they also be + * converted here as a general method), they will be reverted back in + * the individual places in the code where the signal is executed. + */ + if (Protocol6::getByteOrder(word1) != MY_OWN_BYTE_ORDER){ + ndbSwapManyWords32(&readPtr[3], messageLen32 - 3); + } + if(Protocol6::getCheckSumIncluded(word1)){ const Uint32 tmpLen = messageLen32 - 1; const Uint32 checkSumSent = readPtr[tmpLen]; @@ -128,11 +155,12 @@ TransporterRegistry::unpack(Uint32 * rea Uint32 word3 = readPtr[2]; loop_count++; -#if 0 - if(Protocol6::getByteOrder(word1) != MY_OWN_BYTE_ORDER){ - //Do funky stuff - }//if -#endif + // Handle byte order conversion for the first 3 words + if (Protocol6::getByteOrder(word1) != MY_OWN_BYTE_ORDER){ + ndbSwapOneWord32(&word1); + ndbSwapOneWord32(&word2); + ndbSwapOneWord32(&word3); + } const Uint16 messageLen32 = Protocol6::getMessageLength(word1); const Uint32 messageLenBytes = ((Uint32)messageLen32) << 2; @@ -146,6 +174,10 @@ TransporterRegistry::unpack(Uint32 * rea break; }//if + if (Protocol6::getByteOrder(word1) != MY_OWN_BYTE_ORDER){ + ndbSwapManyWords32(&readPtr[3], messageLen32 - 3); + } + if(Protocol6::getCheckSumIncluded(word1)){ const Uint32 tmpLen = messageLen32 - 1; const Uint32 checkSumSent = readPtr[tmpLen]; @@ -226,11 +258,24 @@ TransporterRegistry::unpack(Uint32 * rea Uint32 word2 = readPtr[1]; Uint32 word3 = readPtr[2]; loop_count++; -#if 0 - if(Protocol6::getByteOrder(word1) != MY_OWN_BYTE_ORDER){ - //Do funky stuff + + /** + * The bit 0, 7, 24 and 31 of the first word(word1) indicates byte order + * of the sending node. All the 4 bits are 0 indicates little-endian, all + * the 4 bits are 1 indicates big-endian. So it doesn't matter what + * endianess the receiving node is, by checking bit 31 on the receiving + * node will read a 0 in this bit if the sending node is little-endian + * and read 1 means that the sending node is big-endian. + * + * If the byte order of the sending node is different from the receiving + * node, should firstly swap byte order for the first 3 words to get the + * general information of the signal (e.g. the length of signal). + */ + if (Protocol6::getByteOrder(word1) != MY_OWN_BYTE_ORDER){ + ndbSwapOneWord32(&word1); + ndbSwapOneWord32(&word2); + ndbSwapOneWord32(&word3); } -#endif const Uint16 messageLen32 = Protocol6::getMessageLength(word1); @@ -240,6 +285,20 @@ TransporterRegistry::unpack(Uint32 * rea return readPtr; }//if + /** + * Swap the byte order for each 32-bit word of the signal data if the + * endianess of the signal sender is different from the signal receiver. + * Since all protocols in NDB are centered around sending 32-bit(4 bytes) + * words, for the most signals, this simple conversion can make it work. + * For the character strings (and other non-32bit-word data) in the + * signal, their byte order should not be converted (but they also be + * converted here as a general method), they will be reverted back in + * the individual places in the code where the signal is executed. + */ + if (Protocol6::getByteOrder(word1) != MY_OWN_BYTE_ORDER){ + ndbSwapManyWords32(&readPtr[3], messageLen32 - 3); + } + if(Protocol6::getCheckSumIncluded(word1)){ const Uint32 tmpLen = messageLen32 - 1; const Uint32 checkSumSent = readPtr[tmpLen]; @@ -297,11 +356,13 @@ TransporterRegistry::unpack(Uint32 * rea Uint32 word2 = readPtr[1]; Uint32 word3 = readPtr[2]; loop_count++; -#if 0 - if(Protocol6::getByteOrder(word1) != MY_OWN_BYTE_ORDER){ - //Do funky stuff - }//if -#endif + + // Handle byte order conversion for the first 3 words + if (Protocol6::getByteOrder(word1) != MY_OWN_BYTE_ORDER){ + ndbSwapOneWord32(&word1); + ndbSwapOneWord32(&word2); + ndbSwapOneWord32(&word3); + } const Uint16 messageLen32 = Protocol6::getMessageLength(word1); if(messageLen32 == 0 || messageLen32 > MAX_MESSAGE_SIZE){ @@ -310,6 +371,10 @@ TransporterRegistry::unpack(Uint32 * rea return readPtr; }//if + if (Protocol6::getByteOrder(word1) != MY_OWN_BYTE_ORDER){ + ndbSwapManyWords32(&readPtr[3], messageLen32 - 3); + } + if(Protocol6::getCheckSumIncluded(word1)){ const Uint32 tmpLen = messageLen32 - 1; const Uint32 checkSumSent = readPtr[tmpLen]; @@ -380,6 +445,9 @@ Packer::Packer(bool signalId, bool check // Set the priority preComputedWord1 = 0; + /** + * Set the byte order of own node. + */ Protocol6::setByteOrder(preComputedWord1, MY_OWN_BYTE_ORDER); Protocol6::setSignalIdIncluded(preComputedWord1, signalIdUsed); Protocol6::setCheckSumIncluded(preComputedWord1, checksumUsed); --- 1.76/storage/ndb/src/kernel/blocks/backup/Backup.cpp 2008-01-28 19:44:27 +08:00 +++ 1.77/storage/ndb/src/kernel/blocks/backup/Backup.cpp 2008-01-28 19:44:27 +08:00 @@ -17,6 +17,7 @@ #include "Backup.hpp" #include <ndb_version.h> +#include <ndb_byteorder.h> #include <NdbTCP.h> #include <Bitmask.hpp> @@ -3737,6 +3738,7 @@ Backup::execBACKUP_FRAGMENT_REQ(Signal* ScanFragReq::setNoDiskFlag(req->requestInfo, 1); ScanFragReq::setLcpScanFlag(req->requestInfo, 1); } + ScanFragReq::setApiByteOrder(req->requestInfo, MY_OWN_BYTE_ORDER); req->transId1 = 0; req->transId2 = (BACKUP << 20) + (getOwnNodeId() << 8); req->clientOpPtr= filePtr.i; --- 1.59/storage/ndb/src/kernel/blocks/cmvmi/Cmvmi.cpp 2008-01-28 19:44:27 +08:00 +++ 1.60/storage/ndb/src/kernel/blocks/cmvmi/Cmvmi.cpp 2008-01-28 19:44:27 +08:00 @@ -41,6 +41,7 @@ #include <NdbSleep.h> #include <SafeCounter.hpp> +#include <ndb_byteorder.h> // Used here only to print event reports on stdout/console. EventLogger g_eventLogger; @@ -232,8 +233,49 @@ void Cmvmi::execEVENT_REP(Signal* signal if(ptr.p->logLevel.getLogLevel(eventCategory) < threshold){ continue; } + /** + * If the event type is InfoEvent or WarningEvent, it will send the char + * string event info to the MGM node. + * So if the endianness of MGM node and own node are different, should + * swap the char string event info here, so that no need to swap it on + * the MGM node. + * We should compare the endian of the sender node and own node here. + */ + const Uint32 senderRef = ptr.p->blockRef; + bool needConvertEndian = compareEndianWithOwn(senderRef); + + if (needConvertEndian) { + switch(eventType) { + case NDB_LE_InfoEvent: + case NDB_LE_WarningEvent: + { + Uint32 noOfWords = signal->header.theLength - 1; //subtract theData[0] + ndbSwapManyWords32(&(signal->theData[1]), noOfWords); + break; + } + default: + break; + } + } sendSignal(ptr.p->blockRef, GSN_EVENT_REP, signal, signal->length(), JBB); + + /** + * Turn it back again, so that it can be reused to send to next subscriber. + */ + if (needConvertEndian) { + switch(eventType) { + case NDB_LE_InfoEvent: + case NDB_LE_WarningEvent: + { + Uint32 noOfWords = signal->header.theLength - 1; //subtract theData[0] + ndbSwapManyWords32(&(signal->theData[1]), noOfWords); + break; + } + default: + break; + } + } } if(clogLevel.getLogLevel(eventCategory) < threshold){ --- 1.89/storage/ndb/src/kernel/blocks/dbacc/DbaccMain.cpp 2008-01-28 19:44:27 +08:00 +++ 1.90/storage/ndb/src/kernel/blocks/dbacc/DbaccMain.cpp 2008-01-28 19:44:27 +08:00 @@ -16,6 +16,7 @@ #define DBACC_C #include "Dbacc.hpp" #include <my_sys.h> +#include <ndb_byteorder.h> #include <AttributeHeader.hpp> #include <signaldata/AccFrag.hpp> @@ -3200,8 +3201,12 @@ Dbacc::readTablePk(Uint32 localkey1, Uin { Uint32 fragPageId = localkey1 >> MAX_TUPLES_BITS; Uint32 pageIndex = localkey1 & ((1 << MAX_TUPLES_BITS ) - 1); + /** + * Because the keyinfo in DBACC has local endian, use own + * byte order as argument to call accReadPk(). + */ ret = c_tup->accReadPk(tableId, fragId, fragPageId, pageIndex, - ckeys, true); + ckeys, true, MY_OWN_BYTE_ORDER); } else { --- 1.149/storage/ndb/src/kernel/blocks/dbdict/Dbdict.cpp 2008-01-28 19:44:27 +08:00 +++ 1.150/storage/ndb/src/kernel/blocks/dbdict/Dbdict.cpp 2008-01-28 19:44:27 +08:00 @@ -87,6 +87,7 @@ #include <SLList.hpp> #include <EventLogger.hpp> +#include <ndb_byteorder.h> extern EventLogger g_eventLogger; #define ZNOT_FOUND 626 @@ -343,12 +344,20 @@ void Dbdict::packTableIntoPages(Signal* const Uint32 type= signal->theData[2]; const Uint32 pageId= signal->theData[3]; + /** + * Compare the endian of the sender node and own node here + * to judge whether need convert endian. + */ + const Uint32 senderRef = c_retrieveRecord.blockRef; + bool needConvertEndian = compareEndianWithOwn(senderRef); + PageRecordPtr pagePtr; c_pageRecordArray.getPtr(pagePtr, pageId); memset(&pagePtr.p->word[0], 0, 4 * ZPAGE_HEADER_SIZE); LinearWriter w(&pagePtr.p->word[ZPAGE_HEADER_SIZE], - ZMAX_PAGES_OF_TABLE_DEFINITION * ZSIZE_OF_PAGES_IN_WORDS); + ZMAX_PAGES_OF_TABLE_DEFINITION * ZSIZE_OF_PAGES_IN_WORDS, + needConvertEndian); w.first(); switch((DictTabInfo::TableType)type) { case DictTabInfo::SystemTable: @@ -520,7 +529,7 @@ Dbdict::packTableIntoPages(SimplePropert ConstRope ng(c_rope_pool, tablePtr.p->ngData); ng.copy(ngData); w.add(DictTabInfo::FragmentDataLen, ng.size()); - w.add(DictTabInfo::FragmentData, ngData, ng.size()); + w.add16(DictTabInfo::FragmentData, ngData, ng.size()); ConstRope range(c_rope_pool, tablePtr.p->rangeData); range.copy(rangeData); @@ -2989,9 +2998,14 @@ Dbdict::restartCreateTab_readTableConf(S ParseDictTabInfoRecord parseRecord; parseRecord.requestType = DictTabInfo::GetTabInfoConf; parseRecord.errorCode = 0; - + + const Uint32 senderRef = signal->senderBlockRef(); + bool needConvertEndian = compareEndianWithOwn(senderRef); + Uint32 sz = c_readTableRecord.no_of_words; - SimplePropertiesLinearReader r(pageRecPtr.p->word+ZPAGE_HEADER_SIZE, sz); + SimplePropertiesLinearReader r(pageRecPtr.p->word+ZPAGE_HEADER_SIZE, + sz, + needConvertEndian); handleTabInfoInit(r, &parseRecord); if (parseRecord.errorCode != 0) { @@ -3115,6 +3129,10 @@ Dbdict::execGET_TABINFO_CONF(Signal* sig parseRecord.requestType = DictTabInfo::GetTabInfoConf; parseRecord.errorCode = 0; + /** + * Because we convert endian in the reply of GET_TABINFOREQ, + * no need to convert endian here. + */ SimplePropertiesSectionReader r(tabInfoPtr, getSectionSegmentPool()); handleTabInfoInit(r, &parseRecord); ndbrequire(parseRecord.errorCode == 0); @@ -3833,6 +3851,12 @@ Dbdict::execCREATE_TABLE_REQ(Signal* sig CreateTableReq* const req = (CreateTableReq*)signal->getDataPtr(); const Uint32 senderRef = req->senderRef; const Uint32 senderData = req->senderData; + + /** + * Compare the endian of the sender node and own node here + * to judge whether need convert endian. + */ + bool needConvertEndian = compareEndianWithOwn(senderRef); ParseDictTabInfoRecord parseRecord; do { @@ -3875,7 +3899,9 @@ Dbdict::execCREATE_TABLE_REQ(Signal* sig SegmentedSectionPtr ptr; handle.getSection(ptr, CreateTableReq::DICT_TAB_INFO); - SimplePropertiesSectionReader r(ptr, getSectionSegmentPool()); + SimplePropertiesSectionReader r(ptr, + getSectionSegmentPool(), + needConvertEndian); handleTabInfoInit(r, &parseRecord); releaseSections(handle); @@ -4026,6 +4052,12 @@ Dbdict::execALTER_TABLE_REQ(Signal* sign ParseDictTabInfoRecord aParseRecord; aParseRecord.errorCode = 0; + /** + * Compare the endian of the sender node and own node here + * to judge whether need convert endian. + */ + bool needConvertEndian = compareEndianWithOwn(senderRef); + // Get table definition TableRecordPtr tablePtr; c_tableRecordPool.getPtr(tablePtr, tableId, false); @@ -4272,6 +4304,12 @@ Dbdict::execALTER_TAB_REQ(Signal * signa AlterTabReq::RequestType requestType = (AlterTabReq::RequestType) req->requestType; + /** + * Compare the endian of the sender node and own node here + * to judge whether need convert endian. + */ + bool needConvertEndian = compareEndianWithOwn(senderRef); + SectionHandle handle(this, signal); SegmentedSectionPtr tabInfoPtr; handle.getSection(tabInfoPtr, AlterTabReq::DICT_TAB_INFO); @@ -4366,7 +4404,9 @@ Dbdict::execALTER_TAB_REQ(Signal * signa jam(); // Parse altered table defintion parseRecord.requestType = DictTabInfo::AlterTableFromAPI; - SimplePropertiesSectionReader r(tabInfoPtr, getSectionSegmentPool()); + SimplePropertiesSectionReader r(tabInfoPtr, + getSectionSegmentPool(), + needConvertEndian); handleTabInfoInit(r, &parseRecord, false); // Will not save info @@ -4460,6 +4500,41 @@ Dbdict::execALTER_TAB_REQ(Signal * signa case(AlterTabReq::AlterTableCommit): { jam(); + /** + * If the endian of sender and own node is different, + * should re-pack the table info into a new section + * with local endian. + */ + if (needConvertEndian) { + // Firstly, parse the table info into local endian. + ParseDictTabInfoRecord parseRecord; + + parseRecord.requestType = DictTabInfo::AlterTableFromAPI; + parseRecord.errorCode = 0; + + SimplePropertiesSectionReader r(tabInfoPtr, + getSectionSegmentPool(), + needConvertEndian); + + handleTabInfoInit(r, &parseRecord, false); + + if(parseRecord.errorCode != 0){ + jam(); + c_opCreateTable.release(alterTabPtr); + alterTabRef(signal, req, + (AlterTableRef::ErrorCode) parseRecord.errorCode, + &parseRecord); + return; + } + + // Then, re-pack the table info into a new section. + signal->setSection(tabInfoPtr, 0); + releaseSections(signal); + SimplePropertiesSectionWriter w(getSectionSegmentPool()); + packTableIntoPages(w, parseRecord.tablePtr); + w.getPtr(tabInfoPtr); + } + alterTabPtr.p->m_tabInfoPtrI = tabInfoPtr.i; handle.clear(); @@ -4563,6 +4638,10 @@ void Dbdict::execALTER_TAB_REF(Signal * Uint32 tableId = tablePtr.p->tableId; Uint32 tableVersion = tablePtr.p->tableVersion; Uint32 gci = tablePtr.p->gciTableCreated; + /** + * No need to handle endian here, it will be handled after + * signal ALTER_TAB_REQ is received. + */ SimplePropertiesSectionWriter w(getSectionSegmentPool()); packTableIntoPages(w, tablePtr); SegmentedSectionPtr spDataPtr; @@ -4716,6 +4795,7 @@ Dbdict::execALTER_TAB_CONF(Signal * sign Uint32 tableId = tablePtr.p->tableId; Uint32 tableVersion = tablePtr.p->tableVersion; Uint32 gci = tablePtr.p->gciTableCreated; + // On coordinator, the table info is local endian. SimplePropertiesSectionWriter w(getSectionSegmentPool()); packTableIntoPages(w, tablePtr); SegmentedSectionPtr spDataPtr; @@ -4745,6 +4825,7 @@ Dbdict::execALTER_TAB_CONF(Signal * sign // Send commit request to all alive nodes TableRecordPtr tablePtr; c_tableRecordPool.getPtr(tablePtr, tableId); + // On coordinator, the table info is local endian. SimplePropertiesSectionWriter w(getSectionSegmentPool()); packTableIntoPages(w, tablePtr); SegmentedSectionPtr spDataPtr; @@ -5476,6 +5557,12 @@ Dbdict::createTab_prepare(Signal* signal const Uint32 gci = req->gci; const Uint32 tableId = req->tableId; const Uint32 tableVersion = req->tableVersion; + /** + * Compare the endian of the sender node and own node here + * to judge whether need convert endian. + */ + const Uint32 senderRef = req->senderRef; + bool needConvertEndian = compareEndianWithOwn(senderRef); SectionHandle handle(this, signal); SegmentedSectionPtr tabInfoPtr; @@ -5507,7 +5594,9 @@ Dbdict::createTab_prepare(Signal* signal parseRecord.requestType = DictTabInfo::AddTableFromDict; parseRecord.errorCode = 0; - SimplePropertiesSectionReader r(tabInfoPtr, getSectionSegmentPool()); + SimplePropertiesSectionReader r(tabInfoPtr, + getSectionSegmentPool(), + needConvertEndian); handleTabInfoInit(r, &parseRecord); @@ -5520,7 +5609,6 @@ Dbdict::createTab_prepare(Signal* signal handle.getSection(fragPtr, CreateTabReq::FRAGMENTATION); handle.clear(); - createTabPtr.p->m_tabInfoPtrI = tabInfoPtr.i; createTabPtr.p->m_fragmentsPtrI = fragPtr.i; TableRecordPtr tabPtr; @@ -5529,6 +5617,22 @@ Dbdict::createTab_prepare(Signal* signal tabPtr.p->tableVersion = tableVersion; tabPtr.p->gciTableCreated = gci; + /** + * If the endian of the sender and own node is different, + * should re-pack the table info into a new section with + * local endian, so that it can write the table info into + * schema with local endian while writeTableFile(), and + * can create table correctly when system restart. + */ + if (needConvertEndian) { + signal->setSection(tabInfoPtr, 0); + releaseSections(signal); + SimplePropertiesSectionWriter w(getSectionSegmentPool()); + packTableIntoPages(w, tabPtr); + w.getPtr(tabInfoPtr); + } + createTabPtr.p->m_tabInfoPtrI = tabInfoPtr.i; + SchemaFile::TableEntry tabEntry; tabEntry.m_tableVersion = tableVersion; tabEntry.m_tableType = tabPtr.p->tableType; @@ -7660,6 +7764,12 @@ void Dbdict::execGET_TABINFOREQ(Signal* GetTabInfoReq * const req = (GetTabInfoReq *)&signal->theData[0]; SectionHandle handle(this, signal); + /** + * Compare the endian of the sender node and own node here + * to judge whether need convert endian. + */ + const Uint32 senderRef = req->senderRef; + bool needConvertEndian = compareEndianWithOwn(senderRef); /** * If I get a GET_TABINFO_REQ from myself @@ -7720,6 +7830,12 @@ void Dbdict::execGET_TABINFOREQ(Signal* SegmentedSectionPtr ssPtr; handle.getSection(ssPtr,GetTabInfoReq::TABLE_NAME); copy(tableName, ssPtr); + + // handle endian conversion for table name. + if (needConvertEndian) { + jam(); + ndbSwapManyWords32(tableName, (len + 3)/4); + } DictObject * old_ptr_p = get_object((char*)tableName, len); if(old_ptr_p) @@ -7886,6 +8002,11 @@ Dbdict::execLIST_TABLES_REQ(Signal* sign conf->senderData = senderData; conf->counter = 0; Uint32 pos = 0; + /** + * Compare the endian of the sender node and own node here + * to judge whether need convert endian. + */ + bool needConvertEndian = compareEndianWithOwn(senderRef); DLHashTable<DictObject>::Iterator iter; bool ok = c_obj_hash.first(iter); @@ -8031,6 +8152,13 @@ Dbdict::execLIST_TABLES_REQ(Signal* sign else *p++ = 0; } + // handle endian conversion for table name. + Uint32* tmpData = &conf->tableData[pos]; + if (needConvertEndian) { + jam(); + // need swap byte order for each 4-byte word of table name + ndbSwapOneWord32(tmpData); + } pos++; if (pos >= ListTablesConf::DataLength) { sendSignal(senderRef, GSN_LIST_TABLES_CONF, signal, @@ -8146,6 +8274,7 @@ Dbdict::execCREATE_INDX_REQ(Signal* sign // save attribute list SegmentedSectionPtr ssPtr; handle.getSection(ssPtr, CreateIndxReq::ATTRIBUTE_LIST_SECTION); + // Because type is Uint32, no need handle byte order here. SimplePropertiesSectionReader r0(ssPtr, getSectionSegmentPool()); r0.reset(); // undo implicit first() if (! r0.getWord(&opPtr.p->m_attrList.sz) || @@ -8157,9 +8286,18 @@ Dbdict::execCREATE_INDX_REQ(Signal* sign createIndex_sendReply(signal, opPtr, opPtr.p->m_isMaster); return; } + /** + * Compare the endian of the client node and own node here + * to judge whether need convert endian. + */ + const Uint32 clientRef = req->getUserRef(); + bool needConvertEndian = compareEndianWithOwn(clientRef); + // save name and index table properties handle.getSection(ssPtr, CreateIndxReq::INDEX_NAME_SECTION); - SimplePropertiesSectionReader r1(ssPtr, getSectionSegmentPool()); + SimplePropertiesSectionReader r1(ssPtr, + getSectionSegmentPool(), + needConvertEndian); c_tableDesc.init(); SimpleProperties::UnpackStatus status = SimpleProperties::unpack( r1, &c_tableDesc, @@ -8302,6 +8440,9 @@ Dbdict::createIndex_slavePrepare(Signal* void Dbdict::createIndex_toCreateTable(Signal* signal, OpCreateIndexPtr opPtr) { + /** + * Send CREATE_TABLE_REQ to own node, no need to handle byte order here. + */ union { char tableName[MAX_TAB_NAME_SIZE]; char attributeName[MAX_ATTR_NAME_SIZE]; @@ -9275,6 +9416,9 @@ Dbdict::prepareUtilTransaction(Callback Uint32 attrIds[], const char *attrNames[]) { + /** + * Send to own node, no need consider endian problem + */ jam(); EVENT_TRACE; @@ -9530,13 +9674,22 @@ Dbdict::createEvent_RT_USER_CREATE(Signa ndbout_c("mask = %s", buf); #endif + /** + * Compare the endian of the client node and own node here + * to judge whether need convert endian. + */ + const Uint32 clientRef = evntRecPtr.p->m_request.getUserRef(); + bool needConvertEndian = compareEndianWithOwn(clientRef); + // Interpret the long signal SegmentedSectionPtr ssPtr; // save name and event properties ndbrequire(handle.getSection(ssPtr, CreateEvntReq::EVENT_NAME_SECTION)); - SimplePropertiesSectionReader r0(ssPtr, getSectionSegmentPool()); + SimplePropertiesSectionReader r0(ssPtr, + getSectionSegmentPool(), + needConvertEndian); #ifdef EVENT_DEBUG r0.printAll(ndbout); #endif @@ -10057,11 +10210,20 @@ Dbdict::createEvent_RT_USER_GET(Signal* ndbout_c("DBDICT(Coordinator) got GSN_CREATE_EVNT_REQ::RT_USER_GET evntRecPtr.i = (%d), ref = %u", evntRecPtr.i, evntRecPtr.p->m_request.getUserRef()); #endif + /** + * Compare the endian of the client node and own node here + * to judge whether need convert endian. + */ + const Uint32 clientRef = evntRecPtr.p->m_request.getUserRef(); + bool needConvertEndian = compareEndianWithOwn(clientRef); + SegmentedSectionPtr ssPtr; handle.getSection(ssPtr, 0); - SimplePropertiesSectionReader r0(ssPtr, getSectionSegmentPool()); + SimplePropertiesSectionReader r0(ssPtr, + getSectionSegmentPool(), + needConvertEndian); #ifdef EVENT_DEBUG r0.printAll(ndbout); #endif @@ -10185,7 +10347,12 @@ void Dbdict::execCREATE_EVNT_CONF(Signal ptr[0].p = (Uint32 *)evntRecPtr.p->m_eventRec.TABLE_NAME; ptr[0].sz = (strlen(evntRecPtr.p->m_eventRec.TABLE_NAME)+4)/4; // to make sure we have a null - + /** + * If the user node and own node have different endian, + * should swap the byte order of table name. + * But because this method will be call many times, will + * swap in createEvent_sendReply() instead of this method. + */ createEvent_sendReply(signal, evntRecPtr, ptr, 1); return; @@ -10374,6 +10541,22 @@ void Dbdict::createEvent_sendReply(Signa if (ptr) { jam(); + + /** + * Compare the endian of the sender node and own node here + * to judge whether need convert endian. + */ + bool needConvertEndian = compareEndianWithOwn(senderRef); + + if (needConvertEndian) { + /** + * If the user node and own node have different endian, + * should swap the byte order of table name here, so that + * no need to handle it on user node. + */ + ndbSwapManyWords32(ptr[0].p, ptr[0].sz); + } + sendSignal(senderRef, gsn, signal, signalLength, JBB, ptr, noLSP); } else { jam(); @@ -10931,11 +11114,20 @@ Dbdict::execDROP_EVNT_REQ(Signal* signal OpDropEvent* evntRec = evntRecPtr.p; evntRec->init(req); + /** + * Compare the endian of the client node and own node here + * to judge whether need convert endian. + */ + const Uint32 clientRef = evntRecPtr.p->m_request.getUserRef(); + bool needConvertEndian = compareEndianWithOwn(clientRef); + SegmentedSectionPtr ssPtr; handle.getSection(ssPtr, 0); - SimplePropertiesSectionReader r0(ssPtr, getSectionSegmentPool()); + SimplePropertiesSectionReader r0(ssPtr, + getSectionSegmentPool(), + needConvertEndian); #ifdef EVENT_DEBUG r0.printAll(ndbout); #endif @@ -12596,6 +12788,13 @@ Dbdict::execCREATE_TRIG_REQ(Signal* sign &handle); return; } + /** + * Compare the endian of the client node and own node here + * to judge whether need convert endian. + */ + const Uint32 clientRef = req->getUserRef(); + bool needConvertEndian = compareEndianWithOwn(clientRef); + // seize operation record ndbrequire(signal->getLength() == CreateTrigReq::SignalLength + 1); const Uint32 opKey = req->getOpKey(); @@ -12620,7 +12819,9 @@ Dbdict::execCREATE_TRIG_REQ(Signal* sign // save name SegmentedSectionPtr ssPtr; handle.getSection(ssPtr, CreateTrigReq::TRIGGER_NAME_SECTION); - SimplePropertiesSectionReader ssReader(ssPtr, getSectionSegmentPool()); + SimplePropertiesSectionReader ssReader(ssPtr, + getSectionSegmentPool(), + needConvertEndian); if (ssReader.getKey() != CreateTrigReq::TriggerNameKey || ! ssReader.getString(opPtr.p->m_triggerName)) { jam(); @@ -13067,9 +13268,18 @@ Dbdict::execDROP_TRIG_REQ(Signal* signal OpDropTrigger opTmp; opPtr.p=&opTmp; + /** + * Compare the endian of the client node and own node here + * to judge whether need convert endian. + */ + const Uint32 clientRef = req->getUserRef(); + bool needConvertEndian = compareEndianWithOwn(clientRef); + SegmentedSectionPtr ssPtr; handle.getSection(ssPtr, DropTrigReq::TRIGGER_NAME_SECTION); - SimplePropertiesSectionReader ssReader(ssPtr, getSectionSegmentPool()); + SimplePropertiesSectionReader ssReader(ssPtr, + getSectionSegmentPool(), + needConvertEndian); if (ssReader.getKey() != DropTrigReq::TriggerNameKey || ! ssReader.getString(triggerName)) { jam(); @@ -16045,7 +16255,19 @@ Dbdict::create_fg_prepare_start(Signal* jam(); SegmentedSectionPtr objInfoPtr; getSection(objInfoPtr, ((OpCreateObj*)op)->m_obj_info_ptr_i); - SimplePropertiesSectionReader it(objInfoPtr, getSectionSegmentPool()); + /** + * Since we don't handle endian in the executing function of + * CREATE_FILEGROUP_REQ signal, otherwise we handle it after + * CREATE_OBJ_REQ signal is received, thus we should compare + * the endian of the client node and own node to judge whether + * need convert endian instead of comparing the sender node + * and own node. + */ + const Uint32 clientRef = op->m_clientRef; + bool needConvertEndian = compareEndianWithOwn(clientRef); + SimplePropertiesSectionReader it(objInfoPtr, + getSectionSegmentPool(), + needConvertEndian); Ptr<DictObject> obj_ptr; obj_ptr.setNull(); FilegroupPtr fg_ptr; fg_ptr.setNull(); @@ -16167,6 +16389,23 @@ Dbdict::create_fg_prepare_start(Signal* c_filegroup_hash.add(fg_ptr); op->m_obj_ptr_i = fg_ptr.i; + + /** + * If the endian of the client and own node is different, + * should re-pack the logfile group/tablespace info into + * a new section with local endian, so that it can write + * the logfile group/tablespace info with local endian + * into schema while writeTableFile(), and can create + * logfile group/tablespace correctly when system restart. + */ + if (needConvertEndian) { + signal->setSection(objInfoPtr, 0); + releaseSections(signal); + SimplePropertiesSectionWriter w(getSectionSegmentPool()); + packFilegroupIntoPages(w, fg_ptr, 0, 0); + w.getPtr(objInfoPtr); + } + ((OpCreateObj*)op)->m_obj_info_ptr_i = objInfoPtr.i; } while(0); error: @@ -16232,6 +16471,7 @@ Dbdict::create_fg_prepare_complete(Signa ndbrequire(false); } + // Send to own node, no need handle byte order. sendSignal(ref, GSN_CREATE_FILEGROUP_REQ, signal, len, JBB); } @@ -16293,11 +16533,24 @@ void Dbdict::create_file_prepare_start(Signal* signal, SchemaOp* op) { /** + * Since we don't handle endian in the executing function of + * CREATE_FILE_REQ signal, otherwise we handle it after + * CREATE_OBJ_REQ signal is received, thus we should compare + * the endian of the client node and own node to judge whether + * need convert endian instead of comparing the sender node + * and own node. + */ + const Uint32 clientRef = op->m_clientRef; + bool needConvertEndian = compareEndianWithOwn(clientRef); + + /** * Put data into table record */ SegmentedSectionPtr objInfoPtr; getSection(objInfoPtr, ((OpCreateObj*)op)->m_obj_info_ptr_i); - SimplePropertiesSectionReader it(objInfoPtr, getSectionSegmentPool()); + SimplePropertiesSectionReader it(objInfoPtr, + getSectionSegmentPool(), + needConvertEndian); Ptr<DictObject> obj_ptr; obj_ptr.setNull(); FilePtr filePtr; filePtr.setNull(); @@ -16442,6 +16695,22 @@ Dbdict::create_file_prepare_start(Signal c_file_hash.add(filePtr); op->m_obj_ptr_i = filePtr.i; + + /** + * If the endian of the client and own node is different, + * should re-pack the file info into a new section with + * local endian, so that it can write the file info with + * local endian into schema while writeTableFile(), and + * can create file correctly when system restart. + */ + if (needConvertEndian) { + signal->setSection(objInfoPtr, 0); + releaseSections(signal); + SimplePropertiesSectionWriter w(getSectionSegmentPool()); + packFileIntoPages(w, filePtr, 0); + w.getPtr(objInfoPtr); + } + ((OpCreateObj*)op)->m_obj_info_ptr_i = objInfoPtr.i; } while(0); if (op->m_errorCode) @@ -16528,6 +16797,7 @@ Dbdict::create_file_prepare_complete(Sig ndbrequire(false); } + // Send to own node, no need to handle byte order. char name[MAX_TAB_NAME_SIZE]; ConstRope tmp(c_rope_pool, f_ptr.p->m_path); tmp.copy(name); --- 1.87/storage/ndb/src/kernel/blocks/dblqh/Dblqh.hpp 2008-01-28 19:44:27 +08:00 +++ 1.88/storage/ndb/src/kernel/blocks/dblqh/Dblqh.hpp 2008-01-28 19:44:27 +08:00 @@ -2026,6 +2026,12 @@ public: Uint8 m_disk_table; Uint8 m_use_rowid; Uint8 m_dealloc; + /** + * The byte order of the API node. + * The KEYINFO and ATTRINFO in LQH block has the endian of the API node, + * this flag gives the byte order for KEYINFO and ATTRINFO. + */ + Uint8 m_api_byte_order; Uint32 m_log_part_ptr_i; Local_key m_row_id; @@ -2906,7 +2912,11 @@ private: public: bool is_same_trans(Uint32 opId, Uint32 trid1, Uint32 trid2); - void get_op_info(Uint32 opId, Uint32 *hash, Uint32* gci_hi, Uint32* gci_lo); + void get_op_info(Uint32 opId, + Uint32* hash, + Uint32* gci_hi, + Uint32* gci_lo, + Uint32* api_byte_order); void accminupdate(Signal*, Uint32 opPtrI, const Local_key*); /** @@ -3006,14 +3016,20 @@ Dblqh::is_same_trans(Uint32 opId, Uint32 inline void -Dblqh::get_op_info(Uint32 opId, Uint32 *hash, Uint32* gci_hi, Uint32* gci_lo) +Dblqh::get_op_info(Uint32 opId, + Uint32* hash, + Uint32* gci_hi, + Uint32* gci_lo, + Uint32* api_byte_order) { TcConnectionrecPtr regTcPtr; regTcPtr.i= opId; ptrCheckGuard(regTcPtr, ctcConnectrecFileSize, tcConnectionrec); + *hash = regTcPtr.p->hashValue; *gci_hi = regTcPtr.p->gci_hi; *gci_lo = regTcPtr.p->gci_lo; + *api_byte_order = regTcPtr.p->m_api_byte_order; } #include "../dbacc/Dbacc.hpp" --- 1.203/storage/ndb/src/kernel/blocks/dblqh/DblqhMain.cpp 2008-01-28 19:44:27 +08:00 +++ 1.204/storage/ndb/src/kernel/blocks/dblqh/DblqhMain.cpp 2008-01-28 19:44:27 +08:00 @@ -19,6 +19,7 @@ #include <md5_hash.hpp> #include <ndb_version.h> +#include <ndb_byteorder.h> #include <signaldata/TuxBound.hpp> #include <signaldata/AccScan.hpp> #include <signaldata/CopyActive.hpp> @@ -652,6 +653,7 @@ void Dblqh::execNDB_STTOR(Signal* signal // Dont setReturnedReadLenAIFlag // Dont setAPIVersion LqhKeyReq::setMarkerFlag(preComputedRequestInfoMask, 1); + LqhKeyReq::setApiByteOrder(preComputedRequestInfoMask, 1); //preComputedRequestInfoMask = 0x003d7fff; startphase1Lab(signal, /* dummy */ ~0, ownNodeId); @@ -3568,7 +3570,20 @@ void Dblqh::execLQHKEYREQ(Signal* signal regTcPtr->commitAckMarker = markerPtr.i; } + /** + * Set byte order of API node. + * If the sender node has endian patch, then no need to set + * API byte order flag for the reqinfo of regTcPtr, since it + * has been included in the requestInfo of lqhKeyReq. + * Otherwise, set the byte order of own node into the reqinfo + * of regTcPtr, so that it can be used later. + */ + if (unlikely(senderVersion < NDBD_ENDIAN_COMPATIBLE_VERSION)) + { + LqhKeyReq::setApiByteOrder(Treqinfo, MY_OWN_BYTE_ORDER); + } regTcPtr->reqinfo = Treqinfo; + regTcPtr->m_api_byte_order = LqhKeyReq::getApiByteOrder(Treqinfo); regTcPtr->lastReplicaNo = LqhKeyReq::getLastReplicaNo(Treqinfo); regTcPtr->dirtyOp = LqhKeyReq::getDirtyFlag(Treqinfo); regTcPtr->opExec = LqhKeyReq::getInterpretedFlag(Treqinfo); @@ -4071,6 +4086,25 @@ Dblqh::exec_acckeyreq(Signal* signal, Tc if (regTcPtr.p->primKeyLen > 4) { sendKeyinfoAcc(signal, 11); }//if + + /** + * Now handle the endian conversion for key data. + * If the endian of sender and receiver are different while + * sending LQHKEYREQ signal, the byte order of the key data + * will be swapped after LQHKEYREQ signal is received, so we must + * turn them back again for CHAR (and similar) types. + */ + const Uint32 apiByteOrder = regTcPtr.p->m_api_byte_order; + bool needConvertEndian = (apiByteOrder != MY_OWN_BYTE_ORDER) ? true : false; + if (needConvertEndian) + { + jam(); + Uint32 tableId = regTcPtr.p->tableref; + Uint32* keyData = &(signal->theData[7]); + // XXX: not handle the returned value of convert_endian_key() here. + convert_endian_key(tableId, keyData); + } + EXECUTE_DIRECT(refToBlock(regTcPtr.p->tcAccBlockref), GSN_ACCKEYREQ, signal, 7 + regTcPtr.p->primKeyLen); if (signal->theData[0] < RNIL) { @@ -4085,7 +4119,7 @@ Dblqh::exec_acckeyreq(Signal* signal, Tc execACCKEYREF(signal); }//if return; -}//Dblqh::prepareContinueAfterBlockedLab() +}//Dblqh::exec_acckeyreq() void Dblqh::handle_nr_copy(Signal* signal, Ptr<TcConnectionrec> regTcPtr) @@ -4113,8 +4147,10 @@ Dblqh::handle_nr_copy(Signal* signal, Pt regTcPtr.p->m_nr_delete.m_cnt = 1; // Wait for real op aswell Uint32* dst = signal->theData+24; bool uncommitted; + // Get the byte order of API node + const Uint32 apiByteOrder = regTcPtr.p->m_api_byte_order; const int len = c_tup->nr_read_pk(fragPtr, ®TcPtr.p->m_row_id, dst, - uncommitted); + uncommitted, apiByteOrder); const bool match = (len>0) ? compare_key(regTcPtr.p, dst, len) == 0 : false; if (TRACENR_FLAG) @@ -4313,6 +4349,23 @@ Dblqh::nr_copy_delete_row(Signal* signal { jam(); keylen = 1; + /** + * If the endian of sender and receiver are different while + * sending LQHKEYREQ signal, the byte order of the key data + * will be swapped after LQHKEYREQ signal is received, so we must + * turn them back again for CHAR (and similar) types before + * calculating hash. + */ + const Uint32 apiByteOrder = regTcPtr.p->m_api_byte_order; + bool needConvertEndian = (apiByteOrder != MY_OWN_BYTE_ORDER) ? true : false; + if (needConvertEndian) + { + jam(); + Uint32* keyData = signal->theData+24; + // XXX: not handle the returned value of convert_endian_key() here. + convert_endian_key(tableId, keyData); + } + if (g_key_descriptor_pool.getPtr(tableId)->hasCharAttr) { signal->theData[3] = calculateHash(tableId, signal->theData+24); @@ -4336,6 +4389,22 @@ Dblqh::nr_copy_delete_row(Signal* signal signal->theData[10] = regTcPtr.p->tupkeyData[3]; if (keylen > 4) sendKeyinfoAcc(signal, 11); + /** + * Now handle the endian conversion for key data. + * If the endian of sender and receiver are different while + * sending LQHKEYREQ signal, the byte order of the key data + * will be swapped after LQHKEYREQ signal is received, so we must + * turn them back again for CHAR (and similar) types. + */ + const Uint32 apiByteOrder = regTcPtr.p->m_api_byte_order; + bool needConvertEndian = (apiByteOrder != MY_OWN_BYTE_ORDER) ? true : false; + if (needConvertEndian) + { + jam(); + Uint32* keyData = &(signal->theData[7]); + // XXX: not handle the returned value of convert_endian_key() here. + convert_endian_key(tableId, keyData); + } } const Uint32 ref = refToBlock(regTcPtr.p->tcAccBlockref); EXECUTE_DIRECT(ref, GSN_ACCKEYREQ, signal, 7 + keylen); @@ -4527,6 +4596,21 @@ Dblqh::readPrimaryKeys(Uint32 opPtrI, Ui } while(pos < keyLen); } + /** + * This function is called from ACC to get the key. Since all the key + * data in ACC has local endian, it needs to convert the key data to + * local endian, if needed (ie. if the endian bit from LQHKEYREQ + * differs from own endian). + */ + const Uint32 apiByteOrder = regTcPtr.p->m_api_byte_order; + bool needConvertEndian = (apiByteOrder != MY_OWN_BYTE_ORDER) ? true : false; + if (needConvertEndian) + { + jam(); + // XXX: not handle the returned value of convert_endian_key() here. + convert_endian_key(tableId, tmp); + } + if (xfrm) { jam(); @@ -4715,6 +4799,9 @@ Dblqh::acckeyconf_tupkeyreq(Signal* sign Ttupreq = Ttupreq + (regTcPtr->opExec << 10); Ttupreq = Ttupreq + (regTcPtr->apiVersionNo << 11); Ttupreq = Ttupreq + (regTcPtr->m_use_rowid << 11); + // Set the byte order flag for API node. + Uint32 apiByteOrder = regTcPtr->m_api_byte_order; + TupKeyReq::setApiByteOrder(Ttupreq, apiByteOrder); /* --------------------------------------------------------------------- * Clear interpreted mode bit since we do not want the next replica to @@ -4771,7 +4858,7 @@ Dblqh::acckeyconf_tupkeyreq(Signal* sign regTcPtr->m_row_id.m_page_idx = page_idx; EXECUTE_DIRECT(tup, GSN_TUPKEYREQ, signal, TupKeyReq::SignalLength); -}//Dblqh::execACCKEYCONF() +}//Dblqh::acckeyconf_tupkeyreq() void Dblqh::acckeyconf_load_diskpage(Signal* signal, TcConnectionrecPtr regTcPtr, @@ -5599,6 +5686,22 @@ void Dblqh::writeLogHeader(Signal* signa Uint32 aiLen = tcConnectptr.p->currTupAiLen; Local_key rowid = tcConnectptr.p->m_row_id; Uint32 totLogLen = ZLOG_HEAD_SIZE + aiLen + keyLen; + + /** + * If the byte order of the API node differs from own node, the keyinfo + * and attrinfo will have the same byte order as API node in DBLQH block. + * We write the byte order of the API node into log header, and write + * the original keyinfo and attrinfo (with the same byte order as API) + * into log. Then on system restarting, we can get the byte order of + * the API node from log header and transfer it to the TUP block by + * LQHKEYREQ signal, so that we can handle the endian of the redo log + * correctly. + * As tcConnectptr.p->operation is 8 bits, we can set the byte order + * flag of API node into bit 8 of operation word (dataPtr[3]) in the + * log header. + */ + Uint32 apiByteOrder = tcConnectptr.p->m_api_byte_order; + operation |= (apiByteOrder << 8); if ((logPos + ZLOG_HEAD_SIZE) < ZPAGE_SIZE) { Uint32* dataPtr = &logPagePtr.p->logPageWord[logPos]; @@ -6529,6 +6632,7 @@ void Dblqh::commitContinueAfterBlockedLa tupCommitReq->hashValue = regTcPtr.p->hashValue; tupCommitReq->diskpage = RNIL; tupCommitReq->gci_lo = regTcPtr.p->gci_lo; + tupCommitReq->apiByteOrder = regTcPtr.p->m_api_byte_order; EXECUTE_DIRECT(tup, GSN_TUP_COMMITREQ, signal, TupCommitReq::SignalLength); @@ -8809,6 +8913,11 @@ void Dblqh::accScanConfScanLab(Signal* s req->errorCode = RNIL; req->tuxScanPtrI = scanptr.p->scanAccPtr; Uint32 len = req->boundAiLength = copy_bounds(req->data, tcConnectptr.p); + /* + * Set the byte order bit flag for API node. + */ + TuxBoundInfo::setApiByteOrder(req->boundInfoFlags, + tcConnectptr.p->m_api_byte_order); EXECUTE_DIRECT(DBTUX, GSN_TUX_BOUND_INFO, signal, TuxBoundInfo::SignalLength + len); @@ -9340,6 +9449,10 @@ Dblqh::next_scanconf_tupkeyreq(Signal* s tupKeyReq->attrBufLen = 0; tupKeyReq->opRef = scanPtr.p->scanApiOpPtr; tupKeyReq->applRef = scanPtr.p->scanApiBlockref; + /** + * Set the byte order flag for API node. + */ + TupKeyReq::setApiByteOrder(tupKeyReq->request, regTcPtr->m_api_byte_order); tupKeyReq->storedProcedure = scanPtr.p->scanStoredProcId; tupKeyReq->transId1 = regTcPtr->transid[0]; tupKeyReq->transId2 = regTcPtr->transid[1]; @@ -9390,7 +9503,9 @@ Dblqh::readPrimaryKeys(ScanRecord *scanP tableId = tFragPtr.p->tabRef; } - int ret = c_tup->accReadPk(tableId, fragId, fragPageId, pageIndex, dst, false); + Uint32 apiByteOrder = tcConP->m_api_byte_order; + int ret = c_tup->accReadPk(tableId, fragId, fragPageId, + pageIndex, dst, false, apiByteOrder); jamEntry(); if(0) ndbout_c("readPrimaryKeys(table: %d fragment: %d [ %d %d ] -> %d", @@ -9896,6 +10011,25 @@ void Dblqh::initScanTc(const ScanFragReq ptrCheckGuard(tTablePtr, ctabrecFileSize, tablerec); tcConnectptr.p->m_disk_table = tTablePtr.p->m_disk_table && (!req || !ScanFragReq::getNoDiskFlag(req->requestInfo)); + /** + * Get the byte order of API node from SCAN_FRAGREQ signal, + * and set it into tcConnectptr. + * In the case of executing COPY_FRAGREQ, req is NULL, we + * should set the API byte order to that of own node, since + * it reads data on own node and send to the restarting node. + */ + Uint32 apiByteOrder = MY_OWN_BYTE_ORDER; + if (req) { + BlockReference apiRef = req->resultRef; + Uint32 version = getNodeInfo(refToNode(apiRef)).m_version; + if (likely(version >= NDBD_ENDIAN_COMPATIBLE_VERSION)) { + apiByteOrder = ScanFragReq::getApiByteOrder(req->requestInfo); + } + } + Uint32 reqinfo = 0; + LqhKeyReq::setApiByteOrder(reqinfo, apiByteOrder); + tcConnectptr.p->reqinfo = reqinfo; + tcConnectptr.p->m_api_byte_order = apiByteOrder; tabptr.p->usageCount++; }//Dblqh::initScanTc() @@ -10745,6 +10879,27 @@ void Dblqh::copyTupkeyConfLab(Signal* si tcConP->gci_hi = tmp[len]; tcConP->gci_lo = 0; + // Move into databuffer to make packLqhkeyreqLab happy + memcpy(tcConP->tupkeyData, tmp, 4*4); + if(len > 4) + keyinfoLab(tmp+4, tmp + len); + LqhKeyReq::setKeyLen(tcConP->reqinfo, len); + + /** + * If the endian of API node and own node is different, the byte order + * of the CHAR (and similar) types key data will be swapped after + * readPrimaryKeys() is called, so we must turn them back before + * calculating hash. + */ + const Uint32 apiByteOrder = tcConnectptr.p->m_api_byte_order; + bool needConvertEndian = (apiByteOrder != MY_OWN_BYTE_ORDER) ? true : false; + if (needConvertEndian) + { + jam(); + // XXX: not handle the returned value of convert_endian_key() here. + convert_endian_key(tableId, tmp); + } + // Calculate hash (no need to linearies key) if (g_key_descriptor_pool.getPtr(tableId)->hasCharAttr) { @@ -10755,12 +10910,6 @@ void Dblqh::copyTupkeyConfLab(Signal* si tcConnectptr.p->hashValue = md5_hash((Uint64*)tmp, len); } - // Move into databuffer to make packLqhkeyreqLab happy - memcpy(tcConP->tupkeyData, tmp, 4*4); - if(len > 4) - keyinfoLab(tmp+4, tmp + len); - LqhKeyReq::setKeyLen(tcConP->reqinfo, len); - /*---------------------------------------------------------------------------*/ // To avoid using up to many operation records in ACC we will increase the // constant to ensure that we never send more than 40 records at a time. @@ -11372,6 +11521,11 @@ void Dblqh::initCopyTc(Signal* signal, O LqhKeyReq::setOperation(reqinfo, op); LqhKeyReq::setGCIFlag(reqinfo, 1); LqhKeyReq::setNrCopyFlag(reqinfo, 1); + /** + * the API byte order has been set in tcConnectPtr while executing + * SCAN_FRAGREQ signal. + */ + LqhKeyReq::setApiByteOrder(reqinfo, tcConnectptr.p->m_api_byte_order); /* AILen in LQHKEYREQ IS ZERO */ tcConnectptr.p->reqinfo = reqinfo; /* ------------------------------------------------------------------------ */ @@ -17568,6 +17722,10 @@ void Dblqh::initReqinfoExecSr(Signal* si /* SET OPERATION TYPE AND LOCK MODE (NEVER READ OPERATION OR SCAN IN LOG) */ /* ------------------------------------------------------------------------- */ LqhKeyReq::setOperation(Treqinfo, regTcPtr->operation); +/* ------------------------------------------------------------------------- */ +/* Set byte order of API node */ +/* ------------------------------------------------------------------------- */ + LqhKeyReq::setApiByteOrder(Treqinfo, regTcPtr->m_api_byte_order); regTcPtr->reqinfo = Treqinfo; /* ------------------------------------------------------------------------ */ /* NO OF BACKUP IS SET TO ONE AND NUMBER OF STANDBY NODES IS SET TO ZERO. */ @@ -18069,6 +18227,9 @@ void Dblqh::readLogHeader(Signal* signal jam(); tcConnectptr.p->hashValue = logPagePtr.p->logPageWord[logPos + 2]; tcConnectptr.p->operation = logPagePtr.p->logPageWord[logPos + 3]; + // get the byte order of API node + Uint32 tmpData = logPagePtr.p->logPageWord[logPos + 3]; + tcConnectptr.p->m_api_byte_order = (tmpData >> 8) & 1; tcConnectptr.p->totSendlenAi = logPagePtr.p->logPageWord[logPos + 4]; tcConnectptr.p->primKeyLen = logPagePtr.p->logPageWord[logPos + 5]; tcConnectptr.p->m_row_id.m_page_no = logPagePtr.p->logPageWord[logPos + 6]; @@ -18079,7 +18240,10 @@ void Dblqh::readLogHeader(Signal* signal readLogwordExec(signal); /* IGNORE PREPARE LOG RECORD TYPE */ readLogwordExec(signal); /* IGNORE LOG RECORD SIZE */ tcConnectptr.p->hashValue = readLogwordExec(signal); - tcConnectptr.p->operation = readLogwordExec(signal); + Uint32 tmpData = readLogwordExec(signal); + tcConnectptr.p->operation = (Uint8)tmpData; + // get the byte order of API node + tcConnectptr.p->m_api_byte_order = (tmpData >> 8) & 1; tcConnectptr.p->totSendlenAi = readLogwordExec(signal); tcConnectptr.p->primKeyLen = readLogwordExec(signal); tcConnectptr.p->m_row_id.m_page_no = readLogwordExec(signal); @@ -19494,6 +19658,12 @@ Dblqh::execDUMP_STATE_ORD(Signal* signal << " applOprec = " << hex << tcRec.p->applOprec << " abortState = " << tcRec.p->abortState << endl; + /** + * NOTE: + * If the byte order of the API node and own node is different, + * the tupKeyData[] will has the same endian as API node, other + * than the local endian. + */ ndbout << " transid0 = " << hex << tcRec.p->transid[0] << " transid1 = " << hex << tcRec.p->transid[1] << " tupkeyData0 = " << tcRec.p->tupkeyData[0] @@ -19715,22 +19885,38 @@ Dblqh::execDUMP_STATE_ORD(Signal* signal Uint32 keyLen = tcRec.p->primKeyLen; BaseString key; - for(i = 0; i<keyLen && i < 4; i++) + memcpy(&(signal->theData[7]), + tcRec.p->tupkeyData, + sizeof(tcRec.p->tupkeyData)); + if (keyLen > 4) { jam(); - key.appfmt("0x%x ", tcRec.p->tupkeyData[i]); + tcConnectptr = tcRec; + sendKeyinfoAcc(signal, 11); } - - if (keyLen > 4) + + /** + * Now handle the endian conversion for key data. + * If the endian of sender and receiver are different while + * sending LQHKEYREQ signal, the byte order of the key data + * will be swapped after LQHKEYREQ signal is received, so we must + * turn them back again for CHAR (and similar) types. + */ + const Uint32 apiByteOrder = tcRec.p->m_api_byte_order; + bool needConvertEndian = (apiByteOrder != MY_OWN_BYTE_ORDER) ? true : false; + if (needConvertEndian) + { + jam(); + Uint32 tableId = tcRec.p->tableref; + Uint32* keyData = &(signal->theData[7]); + // XXX: not handle the returned value of convert_endian_key() here. + convert_endian_key(tableId, keyData); + } + + for(i = 7; i < (keyLen + 7); i++) { jam(); - tcConnectptr = tcRec; - sendKeyinfoAcc(signal, 4); - for (i = 4; i<keyLen; i++) - { - jam(); - key.appfmt("0x%x ", signal->theData[i]); - } + key.appfmt("0x%x ", signal->theData[i]); } char buf[100]; --- 1.65/storage/ndb/src/kernel/blocks/dbtc/Dbtc.hpp 2008-01-28 19:44:27 +08:00 +++ 1.66/storage/ndb/src/kernel/blocks/dbtc/Dbtc.hpp 2008-01-28 19:44:27 +08:00 @@ -697,6 +697,15 @@ public: Uint8 apiFailState; // Changed R Uint8 timeOutCounter; Uint8 singleUserMode; + + /** + * The byte order of API node. + * The KEYINFO and ATTRINFO in TC block has the endian of the API node, + * this flag is used to store the endian of the API node, so that can + * calculate hash correctly and send KEYINFO and ATTRINFO with API + * endian to LQH block. + */ + Uint8 apiByteOrder; Uint16 returncode; Uint16 takeOverInd; --- 1.181/storage/ndb/src/kernel/blocks/dbtc/DbtcMain.cpp 2008-01-28 19:44:27 +08:00 +++ 1.182/storage/ndb/src/kernel/blocks/dbtc/DbtcMain.cpp 2008-01-28 19:44:27 +08:00 @@ -21,6 +21,7 @@ #include <ndb_limits.h> #include <my_sys.h> #include <ndb_rand.h> +#include <ndb_byteorder.h> #include <signaldata/DiGetNodes.hpp> #include <signaldata/EventReport.hpp> @@ -2324,6 +2325,28 @@ void Dbtc::hash(Signal* signal) }//while }//if + /** + * Now handle the endian conversion for key data. + * If the endian of sender and receiver are different while + * sending TCKEYREQ/KEYINFO signal, the byte order of the key data + * will be swapped after TCKEYREQ signal is received, so we must + * turn them back again for CHAR (and similar) types. + */ + ApiConnectRecord * const regApiPtr = apiConnectptr.p; + const Uint32 apiByteOrder = regApiPtr->apiByteOrder; + bool needConvertEndian = (apiByteOrder != MY_OWN_BYTE_ORDER) ? true : false; + if (needConvertEndian) + { + jam(); + Uint32 tabPtrI = regCachePtr->tableref; + bool ok = convert_endian_key(tabPtrI, Tdata32); + if (unlikely(!ok)) + { + terrorCode = ZINVALID_KEY; + return; + } + } + UintR keylen = (UintR)regCachePtr->keylen; Uint32 distKey = regCachePtr->distributionKeyIndicator; @@ -2588,6 +2611,16 @@ void Dbtc::execTCKEYREQ(Signal* signal) bool isIndexOpReturn = regApiPtr->indexOpReturn; regApiPtr->isIndexOp = false; // Reset marker regApiPtr->m_exec_flag |= TexecFlag; + /** + * Get the byte order of API and set it into regApiPtr. + */ + Uint32 apiByteOrder = MY_OWN_BYTE_ORDER; + BlockReference apiRef = regApiPtr->ndbapiBlockref; + Uint32 version = getNodeInfo(refToNode(apiRef)).m_version; + if (likely(version >= NDBD_ENDIAN_COMPATIBLE_VERSION)) { + apiByteOrder = TcKeyReq::getApiByteOrder(Treqinfo); + } + regApiPtr->apiByteOrder = apiByteOrder; TableRecordPtr localTabptr; localTabptr.i = TtabIndex; localTabptr.p = &tableRecord[TtabIndex]; @@ -3290,6 +3323,8 @@ void Dbtc::sendlqhkeyreq(Signal* signal, LqhKeyReq::setSimpleFlag(Tdata10, sig0); LqhKeyReq::setOperation(Tdata10, sig1); LqhKeyReq::setNoDiskFlag(Tdata10, regCachePtr->m_no_disk_flag); + // Set byte order of API + LqhKeyReq::setApiByteOrder(Tdata10, regApiPtr->apiByteOrder); /* ----------------------------------------------------------------------- * Sequential Number of first LQH = 0, bit 22-23 @@ -9161,6 +9196,16 @@ void Dbtc::initScanrec(ScanRecordPtr sca ScanFragReq::setTupScanFlag(tmp, ScanTabReq::getTupScanFlag(ri)); ScanFragReq::setAttrLen(tmp, scanTabReq->attrLenKeyLen & 0xFFFF); ScanFragReq::setNoDiskFlag(tmp, ScanTabReq::getNoDiskFlag(ri)); + /* + * Set the byte order flag of API node + */ + BlockReference apiRef = apiConnectptr.p->ndbapiBlockref; + Uint32 version = getNodeInfo(refToNode(apiRef)).m_version; + if (unlikely(version < NDBD_ENDIAN_COMPATIBLE_VERSION)) { + ScanFragReq::setApiByteOrder(tmp, MY_OWN_BYTE_ORDER); + } else { + ScanFragReq::setApiByteOrder(tmp, ScanTabReq::getApiByteOrder(ri)); + } scanptr.p->scanRequestInfo = tmp; scanptr.p->scanStoredProcId = scanTabReq->storedProcId; @@ -13485,6 +13530,16 @@ void Dbtc::insertIntoIndexTable(Signal* // No ATTRINFO in TCKEYREQ TcKeyReq::setAIInTcKeyReq(tcKeyRequestInfo, 0); } + /** + * Set the byte order of API node. + * Since the trigger attribute info read in TUP block has the byte order + * of the API node, we should also set the byte order of API node into + * the request info of TCKEYREQ signal, and it indicates the byte order + * of the tirgger attribute info, so that we can handle the byte order + * of it correctly in TC block. + */ + Uint32 apiByteOrder = regApiPtr->apiByteOrder; + TcKeyReq::setApiByteOrder(tcKeyRequestInfo, apiByteOrder); tcKeyReq->requestInfo = tcKeyRequestInfo; /** @@ -13845,6 +13900,16 @@ void Dbtc::deleteFromIndexTable(Signal* } tcKeyLength += dataPos; + /** + * Set the byte order of API. + * Since the trigger attribute info read in TUP block has the byte order + * of the API node, we should also set the byte order of API node into + * the request info of TCKEYREQ signal, and it indicates the byte order + * of the tirgger attribute info, so that we can handle the byte order + * of it correctly in TC block. + */ + Uint32 apiByteOrder = regApiPtr->apiByteOrder; + TcKeyReq::setApiByteOrder(tcKeyRequestInfo, apiByteOrder); tcKeyReq->requestInfo = tcKeyRequestInfo; /** --- 1.96/storage/ndb/src/kernel/blocks/dbtup/Dbtup.hpp 2008-01-28 19:44:27 +08:00 +++ 1.97/storage/ndb/src/kernel/blocks/dbtup/Dbtup.hpp 2008-01-28 19:44:27 +08:00 @@ -352,6 +352,10 @@ typedef bool (Dbtup::* ReadFunction)(Uin typedef bool (Dbtup::* UpdateFunction)(Uint32*, KeyReqStruct*, Uint32); +typedef void (Dbtup::* ReadEndianFunction)(Uint8*, + KeyReqStruct*); +typedef void (Dbtup::* UpdateEndianFunction)(Uint32*, + KeyReqStruct*); private: typedef Tup_fixsize_page Fix_page; @@ -442,7 +446,7 @@ typedef Ptr<Fragoperrec> FragoperrecPtr; Uint32 noOfDynVar; Uint32 noOfDynFix; Uint32 noOfDynamic; - Uint32 tabDesOffset[7]; + Uint32 tabDesOffset[9]; Uint32 desAllocSize; Uint32 tableDescriptor; Uint32 dynTabDesOffset[3]; @@ -996,6 +1000,8 @@ ArrayPool<TupTriggerData> c_triggerPool; ReadFunction* readFunctionArray; UpdateFunction* updateFunctionArray; + ReadEndianFunction* readEndianFunctionArray; + UpdateEndianFunction* updateEndianFunctionArray; CHARSET_INFO** charsetArray; Uint32 readKeyArray; @@ -1522,11 +1528,27 @@ struct KeyReqStruct { Uint32 in_buf_index; Uint32 in_buf_len; Uint32 attr_descriptor; + /* the out buffer index for endian conversion */ + Uint32 endian_out_buf_index; + /* the endian swapped position in the out buffer */ + Uint32 endian_out_buf_pos; + /** + * The following flag is temporary, when all read attributes functions + * are implemented as read_packed, it should be removed. + */ + bool is_packed; bool xfrm_flag; /* Flag: is tuple in expanded or in shrunken/stored format? */ bool is_expanded; + /** + * Flag for API node byte order. + * 1: big-endian + * 0: little-endian + */ + bool api_byte_order; + struct Var_data { /* These are the pointers and offsets to the variable-sized part of the row @@ -1658,14 +1680,17 @@ public: * for md5 summing and when returning keyinfo. Returns number of * words or negative (-terrorCode) on error. */ - int tuxReadPk(Uint32 fragPtrI, Uint32 pageId, Uint32 pageOffset, Uint32* dataOut, bool xfrmFlag); + int tuxReadPk(Uint32 fragPtrI, Uint32 pageId, Uint32 pageOffset, + Uint32* dataOut, bool xfrmFlag, Uint32 apiByteOrder); /* * ACC reads primary key without headers into an array of words. At * this point in ACC deconstruction, ACC still uses logical references * to fragment and tuple. */ - int accReadPk(Uint32 tableId, Uint32 fragId, Uint32 fragPageId, Uint32 pageIndex, Uint32* dataOut, bool xfrmFlag); + int accReadPk(Uint32 tableId, Uint32 fragId, Uint32 fragPageId, + Uint32 pageIndex, Uint32* dataOut, bool xfrmFlag, + Uint32 apiByteOrder); /* * TUX checks if tuple is visible to scan. @@ -1682,7 +1707,8 @@ public: void complete_restore_lcp(Signal*, Uint32 ref, Uint32 data, Uint32 tableId, Uint32 fragmentId); - int nr_read_pk(Uint32 fragPtr, const Local_key*, Uint32* dataOut, bool©); + int nr_read_pk(Uint32 fragPtr, const Local_key*, Uint32* dataOut, + bool& copy, Uint32 apiByteOrder); int nr_update_gci(Uint32 fragPtr, const Local_key*, Uint32 gci); int nr_delete(Signal*, Uint32, Uint32 fragPtr, const Local_key*, Uint32 gci); @@ -2052,12 +2078,27 @@ private: //------------------------------------------------------------------ //------------------------------------------------------------------ + int readEndianAttributes(KeyReqStruct* req_struct, + const Uint32* inBuffer, + Uint32 inBufLen, + Uint32* outBuffer, + Uint32 TmaxRead, + bool xfrmFlag); + +//------------------------------------------------------------------ +//------------------------------------------------------------------ int updateAttributes(KeyReqStruct *req_struct, Uint32* inBuffer, Uint32 inBufLen); //------------------------------------------------------------------ //------------------------------------------------------------------ + int updateEndianAttributes(KeyReqStruct *req_struct, + Uint32* inBuffer, + Uint32 inBufLen); + +//------------------------------------------------------------------ +//------------------------------------------------------------------ bool readFixedSizeTHOneWordNotNULL(Uint8* outBuffer, KeyReqStruct *req_struct, AttributeHeader* ahOut, @@ -2410,7 +2451,17 @@ private: bool nullFlagCheck(KeyReqStruct *req_struct, Uint32 attrDes2); bool disk_nullFlagCheck(KeyReqStruct *req_struct, Uint32 attrDes2); Uint32 read_pseudo(const Uint32 *, Uint32, KeyReqStruct*, Uint32*); - Uint32 read_packed(const Uint32 *, Uint32, KeyReqStruct*, Uint32*); + /* read pseudo attributes when API and DB nodes have different endian */ + Uint32 read_pseudo_endian(const Uint32 *, Uint32, KeyReqStruct*, Uint32*); + Uint32 read_packed(const Uint32* inBuf, + Uint32 inPos, + KeyReqStruct *req_struct, + Uint32* outBuffer); + /* read packed attributes when API and DB nodes have different endian */ + Uint32 read_packed_endian(const Uint32* inBuf, + Uint32 inPos, + KeyReqStruct *req_struct, + Uint32* outBuffer); /* Fast bit counting (16 instructions on x86_64, gcc -O3). */ static inline uint32_t count_bits(uint32_t x) @@ -2427,6 +2478,37 @@ private: void setUpQueryRoutines(Tablerec* regTabPtr); // ***************************************************************** +// Endian handling methods. +// ***************************************************************** + void setUpEndianRoutines(Tablerec *regTabPtr); + + void read_endian_conversion_abit(Uint8* outBuffer, + KeyReqStruct *req_struct); + void read_endian_conversion_an8bit(Uint8* outBuffer, + KeyReqStruct *req_struct); + void read_endian_conversion_blob(Uint8* outBuffer, + KeyReqStruct *req_struct); + void read_endian_conversion_a16bit(Uint8* outBuffer, + KeyReqStruct *req_struct); + void read_endian_conversion_a32bit(Uint8* outBuffer, + KeyReqStruct *req_struct); + void read_endian_conversion_a64bit(Uint8* outBuffer, + KeyReqStruct *req_struct); + void read_endian_conversion_a128bit(Uint8* outBuffer, + KeyReqStruct *req_struct); + void update_endian_conversion_an8bit(Uint32* inBuffer, + KeyReqStruct *req_struct); + void update_endian_conversion_var(Uint32* inBuffer, + KeyReqStruct *req_struct); + void update_endian_conversion_blob(Uint32* inBuffer, + KeyReqStruct *req_struct); + void update_endian_conversion_a16bit(Uint32* inBuffer, + KeyReqStruct *req_struct); + void update_endian_conversion_a64bit(Uint32* inBuffer, + KeyReqStruct *req_struct); + void update_endian_conversion_a128bit(Uint32* inBuffer, + KeyReqStruct *req_struct); +// ***************************************************************** // Service methods. // ***************************************************************** TransState get_trans_state(Operationrec * const); @@ -2845,6 +2927,7 @@ private: // Private methods Uint32 sizeOfReadFunction(); + Uint32 sizeOfReadEndianFunction(); void removeTdArea(Uint32 tabDesRef, Uint32 list); void insertTdArea(Uint32 tabDesRef, Uint32 list); void itdaMergeTabDescr(Uint32& retRef, Uint32& retNo, bool normal); --- 1.46/storage/ndb/src/kernel/blocks/dbtup/DbtupCommit.cpp 2008-01-28 19:44:27 +08:00 +++ 1.47/storage/ndb/src/kernel/blocks/dbtup/DbtupCommit.cpp 2008-01-28 19:44:27 +08:00 @@ -415,12 +415,17 @@ Dbtup::disk_page_commit_callback(Signal* { Uint32 hash_value; Uint32 gci_hi, gci_lo; + Uint32 api_byte_order; OperationrecPtr regOperPtr; jamEntry(); c_operation_pool.getPtr(regOperPtr, opPtrI); - c_lqh->get_op_info(regOperPtr.p->userpointer, &hash_value, &gci_hi, &gci_lo); + c_lqh->get_op_info(regOperPtr.p->userpointer, + &hash_value, + &gci_hi, + &gci_lo, + &api_byte_order); TupCommitReq * const tupCommitReq= (TupCommitReq *)signal->getDataPtr(); @@ -429,6 +434,7 @@ Dbtup::disk_page_commit_callback(Signal* tupCommitReq->gci_hi= gci_hi; tupCommitReq->gci_lo= gci_lo; tupCommitReq->diskpage = page_id; + tupCommitReq->apiByteOrder = api_byte_order; regOperPtr.p->op_struct.m_load_diskpage_on_commit= 0; regOperPtr.p->m_commit_disk_callback_page= page_id; @@ -456,12 +462,17 @@ Dbtup::disk_page_log_buffer_callback(Sig { Uint32 hash_value; Uint32 gci_hi, gci_lo; + Uint32 api_byte_order; OperationrecPtr regOperPtr; jamEntry(); c_operation_pool.getPtr(regOperPtr, opPtrI); - c_lqh->get_op_info(regOperPtr.p->userpointer, &hash_value, &gci_hi, &gci_lo); + c_lqh->get_op_info(regOperPtr.p->userpointer, + &hash_value, + &gci_hi, + &gci_lo, + &api_byte_order); Uint32 page= regOperPtr.p->m_commit_disk_callback_page; TupCommitReq * const tupCommitReq= (TupCommitReq *)signal->getDataPtr(); @@ -471,6 +482,7 @@ Dbtup::disk_page_log_buffer_callback(Sig tupCommitReq->gci_hi= gci_hi; tupCommitReq->gci_lo= gci_lo; tupCommitReq->diskpage = page; + tupCommitReq->apiByteOrder = api_byte_order; ndbassert(regOperPtr.p->op_struct.m_load_diskpage_on_commit == 0); regOperPtr.p->op_struct.m_wait_log_buffer= 0; @@ -563,6 +575,12 @@ void Dbtup::execTUP_COMMITREQ(Signal* si req_struct.gci_hi = gci_hi; req_struct.gci_lo = gci_lo; regOperPtr.p->m_commit_disk_callback_page = tupCommitReq->diskpage; + /** + * Set the byte order of API node. + * Since this signal is only executed on local host, there is no + * need to consider online upgrade. + */ + req_struct.api_byte_order = tupCommitReq->apiByteOrder; #ifdef VM_TRACE if (tupCommitReq->diskpage == RNIL) --- 1.97/storage/ndb/src/kernel/blocks/dbtup/DbtupExecQuery.cpp 2008-01-28 19:44:27 +08:00 +++ 1.98/storage/ndb/src/kernel/blocks/dbtup/DbtupExecQuery.cpp 2008-01-28 19:44:27 +08:00 @@ -27,6 +27,7 @@ #include <signaldata/TupKey.hpp> #include <signaldata/AttrInfo.hpp> #include <NdbSqlUtil.hpp> +#include <ndb_byteorder.h> /* For debugging */ static void @@ -632,6 +633,10 @@ void Dbtup::execTUPKEYREQ(Signal* signal req_struct.no_changed_attrs= 0; req_struct.last_row= false; req_struct.changeMask.clear(); + /** + * Set the byte order flag for API node. + */ + req_struct.api_byte_order = TupKeyReq::getApiByteOrder(TrequestInfo); if (unlikely(get_trans_state(regOperPtr) != TRANS_IDLE)) { @@ -2097,6 +2102,14 @@ int Dbtup::interpreterNextLab(Signal* si // as long as it fits in the 64 bits of the register. /* ---------------------------------------------------------------- */ { + /* ---------------------------------------------------------------- */ + // Read an attribute from the tuple into a register on local node, + // so no need to consider endian. + // Set req_struct->api_byte_order to the byte order of own node, so + // that it doesn't handle endian while reading attributes. + /* ---------------------------------------------------------------- */ + const Uint32 apiByteOrder = req_struct->api_byte_order; + req_struct->api_byte_order = MY_OWN_BYTE_ORDER; Uint32 theAttrinfo= theInstruction; int TnoDataRW= readAttributes(req_struct, &theAttrinfo, @@ -2104,6 +2117,8 @@ int Dbtup::interpreterNextLab(Signal* si &TregMemBuffer[theRegister], (Uint32)3, false); + // restore API byte order in case use it later. + req_struct->api_byte_order = apiByteOrder; if (TnoDataRW == 2) { /* ------------------------------------------------------------- */ // Two words read means that we get the instruction plus one 32 @@ -2181,9 +2196,22 @@ int Dbtup::interpreterNextLab(Signal* si TdataForUpdate[0]= ah.m_value; Tlen= 1; } + + /* ------------------------------------------------------------*/ + // Write an attribute from a register into the tuple on local + // node, so no need to consider endian. + // Set req_struct->api_byte_order to the byte order of own node, + // so that it doesn't handle endian in updateAttributes function. + /* ------------------------------------------------------------*/ + const Uint32 apiByteOrder = req_struct->api_byte_order; + req_struct->api_byte_order = MY_OWN_BYTE_ORDER; + int TnoDataRW= updateAttributes(req_struct, &TdataForUpdate[0], Tlen); + // restore API byte order in req_struct in case use it later. + req_struct->api_byte_order = apiByteOrder; + if (TnoDataRW != -1) { /* --------------------------------------------------------- */ // Write the written data also into the log buffer so that it @@ -2227,12 +2255,32 @@ int Dbtup::interpreterNextLab(Signal* si case Interpreter::LOAD_CONST64: jam(); - TregMemBuffer[theRegister]= 0x60; /* 64 BIT UNSIGNED CONSTANT */ - TregMemBuffer[theRegister + 2 ]= * (TcurrentProgram + - TprogramCounter++); - TregMemBuffer[theRegister + 3 ]= * (TcurrentProgram + - TprogramCounter++); - break; + { + /* --------------------------------------------------------- */ + // Compare the endian of client node and own node here. + // If they are different, the byte order of 64-bit data will + // be swapped for each 32-bit data while receiving signal. + // Thus we must convert it back to the real 64-bit data with + // local endian. + /* --------------------------------------------------------- */ + const Uint32 apiByteOrder = req_struct->api_byte_order; + bool needConvertEndian = (apiByteOrder != MY_OWN_BYTE_ORDER) ? true : false; + + if (needConvertEndian) { + // allow unaligned + Uint64 tmp; + memcpy(&tmp, TcurrentProgram + TprogramCounter, sizeof(Uint64)); + ndbSwapOneWord64(&tmp); + memcpy(TcurrentProgram + TprogramCounter, &tmp, sizeof(Uint64)); + } + + TregMemBuffer[theRegister]= 0x60; /* 64 BIT UNSIGNED CONSTANT */ + TregMemBuffer[theRegister + 2 ]= * (TcurrentProgram + + TprogramCounter++); + TregMemBuffer[theRegister + 3 ]= * (TcurrentProgram + + TprogramCounter++); + break; + } case Interpreter::ADD_REG_REG: jam(); @@ -2438,16 +2486,30 @@ int Dbtup::interpreterNextLab(Signal* si case Interpreter::BRANCH_ATTR_OP_ARG:{ jam(); + // Compare the endian of client node and own node. + const Uint32 apiByteOrder = req_struct->api_byte_order; + bool needConvertEndian = (apiByteOrder != MY_OWN_BYTE_ORDER) ? true : false; + Uint32 cond = Interpreter::getBinaryCondition(theInstruction); Uint32 ins2 = TcurrentProgram[TprogramCounter]; Uint32 attrId = Interpreter::getBranchCol_AttrId(ins2) << 16; Uint32 argLen = Interpreter::getBranchCol_Len(ins2); if(tmpHabitant != attrId){ + /* ---------------------------------------------------------------- */ + // Read attribute on local node to compare, so no need to consider + // endian. + // Set req_struct->api_byte_order to the byte order of own node, so + // that it doesn't handle endian in readAttributes function. + /* ---------------------------------------------------------------- */ + const Uint32 apiByteOrder = req_struct->api_byte_order; + req_struct->api_byte_order = MY_OWN_BYTE_ORDER; Int32 TnoDataR = readAttributes(req_struct, &attrId, 1, tmpArea, tmpAreaSz, false); + // restore API byte order of req_struct in case use it later. + req_struct->api_byte_order = apiByteOrder; if (TnoDataR == -1) { jam(); @@ -2493,6 +2555,21 @@ int Dbtup::interpreterNextLab(Signal* si { return TUPKEY_abort(signal, 40); } + /* ---------------------------------------------------------------- */ + // If need convert endian, should swap the byte order of program data + // in order to compare with attribute read. + // For not LIKE and NOT_LIKE operations, call convert_endian_attr() + // function to swap the byte order of program data. + /* ---------------------------------------------------------------- */ + if (needConvertEndian) { + Uint32 srcPos = 0; + bool ok = convert_endian_attr(TattrDesc1, (CHARSET_INFO*)cs, (Uint32*)s2, srcPos); + if (unlikely(!ok)) { + jam(); + // report ATTRINFO error + return TUPKEY_abort(signal, 39); + }//if + }//if res1 = (*sqlType.m_cmp)(cs, s1, attrLen, s2, argLen, true); } } else { @@ -2505,6 +2582,16 @@ int Dbtup::interpreterNextLab(Signal* si { return TUPKEY_abort(signal, 40); } + /* ---------------------------------------------------------------- */ + // If need convert endian, should swap the byte order of program data + // in order to compare with attribute read. + // For LIKE and NOT_LIKE operations, treat program data as char string, + // and swap the byte order of it. + /* ---------------------------------------------------------------- */ + if (needConvertEndian) { + Uint32 swapWords = (argLen + 3) >> 2; + ndbSwapManyWords32((Uint32*)s2, swapWords); + }//if res1 = (*sqlType.m_like)(cs, s1, attrLen, s2, argLen); } } @@ -2559,10 +2646,20 @@ int Dbtup::interpreterNextLab(Signal* si Uint32 attrId= Interpreter::getBranchCol_AttrId(ins2) << 16; if (tmpHabitant != attrId){ + /* ---------------------------------------------------------------- */ + // Read attribute on local node to compare, so no need to consider + // endian. + // Set req_struct->api_byte_order to the byte order of own node, so + // that it doesn't handle endian in readAttributes function. + /* ---------------------------------------------------------------- */ + const Uint32 apiByteOrder = req_struct->api_byte_order; + req_struct->api_byte_order = MY_OWN_BYTE_ORDER; Int32 TnoDataR= readAttributes(req_struct, &attrId, 1, tmpArea, tmpAreaSz, false); + // restore API byte order of req_struct in case use it later. + req_struct->api_byte_order = apiByteOrder; if (TnoDataR == -1) { jam(); @@ -2587,11 +2684,21 @@ int Dbtup::interpreterNextLab(Signal* si Uint32 attrId= Interpreter::getBranchCol_AttrId(ins2) << 16; if (tmpHabitant != attrId){ + /* ---------------------------------------------------------------- */ + // Read attribute on local node to compare, so no need to consider + // endian. + // Set req_struct->api_byte_order to the byte order of own node, so + // that it doesn't handle endian in readAttributes function. + /* ---------------------------------------------------------------- */ + const Uint32 apiByteOrder = req_struct->api_byte_order; + req_struct->api_byte_order = MY_OWN_BYTE_ORDER; Int32 TnoDataR= readAttributes(req_struct, &attrId, 1, tmpArea, tmpAreaSz, false); - + // restore API byte order of req_struct in case use it later. + req_struct->api_byte_order = apiByteOrder; + if (TnoDataR == -1) { jam(); tupkeyErrorLab(signal); @@ -3537,7 +3644,10 @@ Dbtup::nr_update_gci(Uint32 fragPtrI, co int Dbtup::nr_read_pk(Uint32 fragPtrI, - const Local_key* key, Uint32* dst, bool& copy) + const Local_key* key, + Uint32* dst, + bool& copy, + Uint32 apiByteOrder) { FragrecordPtr fragPtr; @@ -3562,6 +3672,8 @@ Dbtup::nr_read_pk(Uint32 fragPtrI, req_struct.m_page_ptr = pagePtr; req_struct.m_tuple_ptr = (Tuple_header*)ptr; Uint32 bits = req_struct.m_tuple_ptr->m_header_bits; + // Set the byte order of API node + req_struct.api_byte_order = apiByteOrder; int ret = 0; copy = false; --- 1.53/storage/ndb/src/kernel/blocks/dbtup/DbtupGen.cpp 2008-01-28 19:44:27 +08:00 +++ 1.54/storage/ndb/src/kernel/blocks/dbtup/DbtupGen.cpp 2008-01-28 19:44:27 +08:00 @@ -659,6 +659,8 @@ Dbtup::initTab(Tablerec* const regTabPtr }//for regTabPtr->readFunctionArray = NULL; regTabPtr->updateFunctionArray = NULL; + regTabPtr->readEndianFunctionArray = NULL; + regTabPtr->updateEndianFunctionArray = NULL; regTabPtr->charsetArray = NULL; regTabPtr->tabDescriptor = RNIL; --- 1.30/storage/ndb/src/kernel/blocks/dbtup/DbtupIndex.cpp 2008-01-28 19:44:27 +08:00 +++ 1.31/storage/ndb/src/kernel/blocks/dbtup/DbtupIndex.cpp 2008-01-28 19:44:27 +08:00 @@ -18,6 +18,7 @@ #include "Dbtup.hpp" #include <RefConvert.hpp> #include <ndb_limits.h> +#include <ndb_byteorder.h> #include <pc.hpp> #include <AttributeDescriptor.hpp> #include "AttributeOffset.hpp" @@ -141,6 +142,11 @@ Dbtup::tuxReadAttrs(Uint32 fragPtrI, Operationrec tmpOp; KeyReqStruct req_struct; + /** + * Set the byte order flag for API node in req_struct to byte order + * of own node, so that it doesn't handle endian while reading attributes. + */ + req_struct.api_byte_order = MY_OWN_BYTE_ORDER; tmpOp.m_tuple_location.m_page_no= pageId; tmpOp.m_tuple_location.m_page_idx= pageIndex; @@ -198,7 +204,8 @@ Dbtup::tuxReadAttrs(Uint32 fragPtrI, return ret; } int -Dbtup::tuxReadPk(Uint32 fragPtrI, Uint32 pageId, Uint32 pageIndex, Uint32* dataOut, bool xfrmFlag) +Dbtup::tuxReadPk(Uint32 fragPtrI, Uint32 pageId, Uint32 pageIndex, + Uint32* dataOut, bool xfrmFlag, Uint32 apiByteOrder) { jamEntry(); // use own variables instead of globals @@ -214,6 +221,10 @@ Dbtup::tuxReadPk(Uint32 fragPtrI, Uint32 tmpOp.m_tuple_location.m_page_idx= pageIndex; KeyReqStruct req_struct; + /** + * Set the byte order flag for API node in req_struct. + */ + req_struct.api_byte_order = apiByteOrder; PagePtr page_ptr; Uint32* ptr= get_ptr(&page_ptr, &tmpOp.m_tuple_location, tablePtr.p); @@ -301,7 +312,9 @@ Dbtup::tuxReadPk(Uint32 fragPtrI, Uint32 } int -Dbtup::accReadPk(Uint32 tableId, Uint32 fragId, Uint32 fragPageId, Uint32 pageIndex, Uint32* dataOut, bool xfrmFlag) +Dbtup::accReadPk(Uint32 tableId, Uint32 fragId, Uint32 fragPageId, + Uint32 pageIndex, Uint32* dataOut, bool xfrmFlag, + Uint32 apiByteOrder) { jamEntry(); // get table @@ -315,7 +328,8 @@ Dbtup::accReadPk(Uint32 tableId, Uint32 Uint32 pageId = getRealpid(fragPtr.p, fragPageId); // use TUX routine - optimize later - int ret = tuxReadPk(fragPtr.i, pageId, pageIndex, dataOut, xfrmFlag); + int ret = tuxReadPk(fragPtr.i, pageId, pageIndex, + dataOut, xfrmFlag, apiByteOrder); return ret; } @@ -340,6 +354,8 @@ Dbtup::tuxQueryTh(Uint32 fragPtrI, // use temp op rec Operationrec tempOp; KeyReqStruct req_struct; + // initialize API byte order to byte order of own node in case use it later + req_struct.api_byte_order = MY_OWN_BYTE_ORDER; tempOp.m_tuple_location.m_page_no= getRealpid(fragPtr.p, fragPageId); tempOp.m_tuple_location.m_page_idx= pageIndex; tempOp.savepointId= savePointId; --- 1.57/storage/ndb/src/kernel/blocks/dbtup/DbtupMeta.cpp 2008-01-28 19:44:27 +08:00 +++ 1.58/storage/ndb/src/kernel/blocks/dbtup/DbtupMeta.cpp 2008-01-28 19:44:27 +08:00 @@ -715,10 +715,10 @@ Dbtup::handleAlterTabPrepare(Signal *sig */ Uint32* desc= &tableDescriptor[tableDescriptorRef].tabDescr; CHARSET_INFO** CharsetArray= - (CHARSET_INFO**)(desc + regAlterTabOpPtr.p->tabDesOffset[2]); + (CHARSET_INFO**)(desc + regAlterTabOpPtr.p->tabDesOffset[4]); memcpy(CharsetArray, regTabPtr->charsetArray, sizeof(*CharsetArray)*regTabPtr->noOfCharsets); - Uint32 *attrDesPtr= desc + regAlterTabOpPtr.p->tabDesOffset[4]; + Uint32 *attrDesPtr= desc + regAlterTabOpPtr.p->tabDesOffset[6]; memcpy(attrDesPtr, &tableDescriptor[regTabPtr->tabDescriptor].tabDescr, (ZAD_SIZE<<2)*oldNoOfAttr); @@ -1166,6 +1166,7 @@ Dbtup::computeTableMetaData(Tablerec *re regTabPtr->total_rec_size= total_rec_size; setUpQueryRoutines(regTabPtr); + setUpEndianRoutines(regTabPtr); setUpKeyArray(regTabPtr); } @@ -1233,9 +1234,11 @@ Dbtup::undo_createtable_logsync_callback * * 0 readFunctionArray ( one for each attribute ) * 1 updateFunctionArray ( ditto ) - * 2 charsetArray ( pointers to distinct CHARSET_INFO ) - * 3 readKeyArray ( attribute ids of keys ) - * 5 tabDescriptor ( attribute descriptors, each ZAD_SIZE ) + * 2 readEndianFunctionArray ( one for each attribute ) + * 3 updateEndianFunctionArray ( one for each attribute ) + * 4 charsetArray ( pointers to distinct CHARSET_INFO ) + * 5 readKeyArray ( attribute ids of keys ) + * 6 tabDescriptor ( attribute descriptors, each ZAD_SIZE ) */ void Dbtup::setUpDescriptorReferences(Uint32 descriptorReference, Tablerec* const regTabPtr, @@ -1244,10 +1247,12 @@ void Dbtup::setUpDescriptorReferences(Ui Uint32* desc= &tableDescriptor[descriptorReference].tabDescr; regTabPtr->readFunctionArray= (ReadFunction*)(desc + offset[0]); regTabPtr->updateFunctionArray= (UpdateFunction*)(desc + offset[1]); - regTabPtr->charsetArray= (CHARSET_INFO**)(desc + offset[2]); - regTabPtr->readKeyArray= descriptorReference + offset[3]; - regTabPtr->tabDescriptor= descriptorReference + offset[4]; - regTabPtr->m_real_order_descriptor = descriptorReference + offset[5]; + regTabPtr->readEndianFunctionArray= (ReadEndianFunction*)(desc + offset[2]); + regTabPtr->updateEndianFunctionArray= (UpdateEndianFunction*)(desc + offset[3]); + regTabPtr->charsetArray= (CHARSET_INFO**)(desc + offset[4]); + regTabPtr->readKeyArray= descriptorReference + offset[5]; + regTabPtr->tabDescriptor= descriptorReference + offset[6]; + regTabPtr->m_real_order_descriptor = descriptorReference + offset[7]; } void Dbtup::setupDynDescriptorReferences(Uint32 dynDescr, @@ -1269,6 +1274,15 @@ Dbtup::sizeOfReadFunction() return (Uint32)(end - start); } +Uint32 +Dbtup::sizeOfReadEndianFunction() +{ + ReadEndianFunction* tmp= (ReadEndianFunction*)&tableDescriptor[0]; + TableDescriptor* start= &tableDescriptor[0]; + TableDescriptor * end= (TableDescriptor*)(tmp + 1); + return (Uint32)(end - start); +} + void Dbtup::setUpKeyArray(Tablerec* const regTabPtr) { ndbrequire((regTabPtr->readKeyArray + regTabPtr->noOfKeyAttr) < @@ -1487,10 +1501,12 @@ void Dbtup::releaseTabDescr(Tablerec* co regTabPtr->readKeyArray= RNIL; regTabPtr->readFunctionArray= NULL; regTabPtr->updateFunctionArray= NULL; + regTabPtr->readEndianFunctionArray= NULL; + regTabPtr->updateEndianFunctionArray= NULL; regTabPtr->charsetArray= NULL; // move to start of descriptor - descriptor -= offset[3]; + descriptor -= offset[5]; Uint32 retNo= getTabDescrWord(descriptor + ZTD_DATASIZE); ndbrequire(getTabDescrWord(descriptor + ZTD_HEADER) == ZTD_TYPE_NORMAL); ndbrequire(retNo == getTabDescrWord((descriptor + retNo) - ZTD_TR_SIZE)); --- 1.49/storage/ndb/src/kernel/blocks/dbtup/DbtupRoutines.cpp 2008-01-28 19:44:27 +08:00 +++ 1.50/storage/ndb/src/kernel/blocks/dbtup/DbtupRoutines.cpp 2008-01-28 19:44:27 +08:00 @@ -23,6 +23,7 @@ #include <AttributeDescriptor.hpp> #include "AttributeOffset.hpp" #include <AttributeHeader.hpp> +#include <ndb_byteorder.h> void Dbtup::setUpQueryRoutines(Tablerec *regTabPtr) @@ -280,6 +281,121 @@ Dbtup::setUpQueryRoutines(Tablerec *regT } } +void +Dbtup::setUpEndianRoutines(Tablerec *regTabPtr) +{ + Uint32 startDescriptor = regTabPtr->tabDescriptor; + ndbrequire((startDescriptor + (regTabPtr->m_no_of_attributes << ZAD_LOG_SIZE)) + <= cnoOfTabDescrRec); + for (Uint32 i = 0; i < regTabPtr->m_no_of_attributes; i++) { + Uint32 attrDescrStart = startDescriptor + (i << ZAD_LOG_SIZE); + Uint32 attrDescr = tableDescriptor[attrDescrStart].tabDescr; + + Uint32 type = AttributeDescriptor::getType(attrDescr); + Uint32 size = AttributeDescriptor::getSize(attrDescr); + + switch (size) { + case DictTabInfo::aBit: + /** + * Bit + * Bit type attribute need not consider endian, but should handle endian + * conversion for the left unswapped part of the previous attribute + * in read_packed. + */ + jam(); + regTabPtr->readEndianFunctionArray[i] = &Dbtup::read_endian_conversion_abit; + regTabPtr->updateEndianFunctionArray[i] = NULL; + break; + case DictTabInfo::an8Bit: + /** + * Tinyint, Tinyunsigned, Mediumint, Mediumunsigned, + * Olddecimal, Char, Varchar, Binary, Varbinary, + * Datetime, Date, Blob, Text, Longvarchar, + * Longvarbinary, Time, Year, Timestamp, Olddecimalunsigned, + * Decimal, Decimalunsigned. + */ + jam(); + switch (type) { + case NdbSqlUtil::Type::Varchar: + case NdbSqlUtil::Type::Varbinary: + case NdbSqlUtil::Type::Longvarchar: + case NdbSqlUtil::Type::Longvarbinary: + { + jam(); + regTabPtr->readEndianFunctionArray[i] = &Dbtup::read_endian_conversion_an8bit; + /** + * For variable disk attribute, should handle as fixed size attribute while updating. + */ + Uint32 disk = AttributeDescriptor::getDiskBased(attrDescr); + Uint32 array = AttributeDescriptor::getArrayType(attrDescr); + if (disk && array == NDB_ARRAYTYPE_FIXED) { + regTabPtr->updateEndianFunctionArray[i] = &Dbtup::update_endian_conversion_an8bit; + } else { + regTabPtr->updateEndianFunctionArray[i] = &Dbtup::update_endian_conversion_var; + } + break; + } + case NdbSqlUtil::Type::Blob: + case NdbSqlUtil::Type::Text: + regTabPtr->readEndianFunctionArray[i] = &Dbtup::read_endian_conversion_blob; + regTabPtr->updateEndianFunctionArray[i] = &Dbtup::update_endian_conversion_blob; + break; + case NdbSqlUtil::Type::Datetime: + regTabPtr->readEndianFunctionArray[i] = &Dbtup::read_endian_conversion_a64bit; + regTabPtr->updateEndianFunctionArray[i] = &Dbtup::update_endian_conversion_a64bit; + break; + case NdbSqlUtil::Type::Timestamp: + /** + * Timestamp type attribute need not consider endian, but should handle endian + * conversion for the left unswapped part of the previous attribute + * in read_packed. + */ + regTabPtr->readEndianFunctionArray[i] = &Dbtup::read_endian_conversion_a32bit; + regTabPtr->updateEndianFunctionArray[i] = NULL; + break; + default: + regTabPtr->readEndianFunctionArray[i] = &Dbtup::read_endian_conversion_an8bit; + regTabPtr->updateEndianFunctionArray[i] = &Dbtup::update_endian_conversion_an8bit; + } //switch (type) + break; + case DictTabInfo::a16Bit: + /** + * Smallint, Smallunsigned + */ + jam(); + regTabPtr->readEndianFunctionArray[i] = &Dbtup::read_endian_conversion_a16bit; + regTabPtr->updateEndianFunctionArray[i] = &Dbtup::update_endian_conversion_a16bit; + break; + case DictTabInfo::a32Bit: + /** + * Int, Unsigned, Float + * a32Bit size attribute need not consider endian, but should handle endian + * conversion for the left unswapped part of the previous attribute + * in read_packed. + */ + jam(); + regTabPtr->readEndianFunctionArray[i] = &Dbtup::read_endian_conversion_a32bit; + regTabPtr->updateEndianFunctionArray[i] = NULL; + break; + case DictTabInfo::a64Bit: + /** + * Bigint, Bigunsigned, Double + */ + jam(); + regTabPtr->readEndianFunctionArray[i] = &Dbtup::read_endian_conversion_a64bit; + regTabPtr->updateEndianFunctionArray[i] = &Dbtup::update_endian_conversion_a64bit; + break; + case DictTabInfo::a128Bit: + jam(); + regTabPtr->readEndianFunctionArray[i] = &Dbtup::read_endian_conversion_a128bit; + regTabPtr->updateEndianFunctionArray[i] = &Dbtup::update_endian_conversion_a128bit; + break; + default: + jam(); + }//switch(size) + } +} + /* Dump a byte buffer, for debugging. */ static void dump_buf_hex(unsigned char *p, Uint32 bytes) { @@ -314,6 +430,624 @@ pad32(Uint32 bytepos, Uint32 bitsused) } /* ---------------------------------------------------------------- */ +/* This routine is used to swap the byte order for the a bit */ +/* attribute size attribute while reading. */ +/* */ +/* This type attribute need not consider endian, but should */ +/* handle endian conversion for the left unswapped part of */ +/* the previous attribute while reading packed. */ +/* ---------------------------------------------------------------- */ +void +Dbtup::read_endian_conversion_abit(Uint8* outBuffer, + KeyReqStruct *req_struct) +{ + jam(); + if (req_struct->is_packed) { + Uint32 preOutIndex = req_struct->endian_out_buf_index; + Uint32 outIndex = req_struct->out_buf_index; + Uint32 swapPos = req_struct->endian_out_buf_pos; + ndbrequire(preOutIndex <= outIndex); + ndbrequire(swapPos <= outIndex); + + /** + * If there is unswapped part of the previous attribute, + * swap the left unswapped part of the previous attribute. + */ + if (swapPos != preOutIndex) { + Uint32* attrBuffer = (Uint32*)(outBuffer + swapPos); + // the left unswapped part can only be 4 bytes. + ndbSwapOneWord32(attrBuffer); + } + + // set new swap position + req_struct->endian_out_buf_pos = (outIndex + 3) & ~(Uint32)3; + }//if +}//Dbtup::read_endian_conversion_abit() + +/* ---------------------------------------------------------------- */ +/* This routine is used to swap the byte order for the 8 bit */ +/* attribute size attribute while reading. */ +/* */ +/* e.g. */ +/* create table t1 (a char (3) not null primary key, b char(3) */ +/* not null) engine = ndbcluster; */ +/* insert into t1 ('aaa', 'bbb'); */ +/* select * from t1 where a = 'aaa'; */ +/* */ +/* The out buffer format before swap should be: */ +/* [ READ_PACKED 4 ] [ 3 ] [ 'aaa' 'b' ] [ 'bb' p p ] */ +/* */ +/* The out buffer format after swapped should be: */ +/* [ READ_PACKED 4 ] [ 3 ] [ 'b' 'aaa' ] [ p p 'bb' ] */ +/* ---------------------------------------------------------------- */ +void +Dbtup::read_endian_conversion_an8bit(Uint8* outBuffer, + KeyReqStruct *req_struct) +{ + jam(); + Uint32 preOutIndex = req_struct->endian_out_buf_index; + Uint32 outIndex = req_struct->out_buf_index; + Uint32 swapPos = req_struct->endian_out_buf_pos; + ndbrequire(preOutIndex <= outIndex); + ndbrequire(swapPos <= outIndex); + Uint32 outBits = req_struct->out_buf_bits; + Uint32* attrBuffer; + Uint32 attrBytes = 0; + Uint32 swapWords = 0; + Uint32 swapBytes = 0; + + if (! req_struct->is_packed) { + attrBytes = outIndex - preOutIndex; + if (attrBytes > 0) { + // the attribute is not NULL + swapWords = (attrBytes + 3) >> 2; + attrBuffer = (Uint32*)(outBuffer + preOutIndex); + + ndbSwapManyWords32(attrBuffer, swapWords); + } + } else { + attrBytes = outIndex - swapPos; + + if (attrBytes > 0) { + /** + * If out_buf_bits is not 0, should swap the whole attribute, + * else only swap the beginning 4-byte aligned part of the + * attribute. + */ + if (outBits) { + swapWords = (attrBytes + 3) >> 2; + } else { + swapWords = attrBytes >> 2; + } //if (outBits) + if (swapWords > 0) { + attrBuffer = (Uint32*)(outBuffer + swapPos); + ndbSwapManyWords32(attrBuffer, swapWords); + } + } + + // set new swap position + swapBytes = swapWords << 2; + if (outBits || (attrBytes == swapBytes)) { + swapPos = outIndex + 4 * ((outBits + 31) >> 5); + } else { + swapPos += swapBytes; + } + req_struct->endian_out_buf_pos = swapPos; + }//if +}//Dbtup::read_endian_conversion_an8bit() + +/* ---------------------------------------------------------------- */ +/* This routine is used to swap the byte order for the blob/text */ +/* type attribute while reading. */ +/* ---------------------------------------------------------------- */ +void +Dbtup::read_endian_conversion_blob(Uint8* outBuffer, + KeyReqStruct *req_struct) +{ + jam(); + Uint32 preOutIndex = req_struct->endian_out_buf_index; + Uint32 outIndex = req_struct->out_buf_index; + Uint32 swapPos = req_struct->endian_out_buf_pos; + ndbrequire(preOutIndex <= outIndex); + ndbrequire(swapPos <= outIndex); + Uint32 outBits = req_struct->out_buf_bits; + Uint32* attrBuffer; + Uint32 attrBytes = 0; + Uint32 swapBytes = 0; + Uint32 swapWords = 0; + Uint32 attrDescriptor = req_struct->attr_descriptor; + Uint32 array = AttributeDescriptor::getArrayType(attrDescriptor); + + if (! req_struct->is_packed) { + attrBytes = outIndex - preOutIndex; + if (attrBytes > 0) { + // the attribute is not NULL + swapWords = (attrBytes + 3) >> 2; + attrBuffer = (Uint32*)(outBuffer + preOutIndex); + + if (array == NDB_ARRAYTYPE_FIXED) { + /** + * Blob V1: + * Blob head V1 is 8 bytes: + * 8 bytes blob length - native endian + * Thus we should swap the byte-order of blob head as Uint64. + * The inline bytes of blob is stored as character string, + * should swap the byte-order for each byte of it. + */ + // allow unaligned + Uint64 tmp; + memcpy(&tmp, attrBuffer, sizeof(Uint64)); + ndbSwapOneWord64(&tmp); + memcpy(attrBuffer, &tmp, sizeof(Uint64)); + ndbSwapManyWords32(attrBuffer + 2, swapWords - 2); + } else if (array == NDB_ARRAYTYPE_MEDIUM_VAR) { + /** + * Blob V2: + * Blob head V2 is 16 bytes: + * 2 bytes head+inline length - little-endian + * 2 bytes reserved (zero) + * 4 bytes NDB$PKID - little-endian + * 8 bytes blob length - little-endian + * Thus we should swap the byte-order for each byte of blob head. + * The inline bytes of blob is stored as character string, + * should also swap the byte-order for each byte of it. + */ + ndbSwapManyWords32(attrBuffer, swapWords); + }//if + } + } else { + attrBytes = outIndex - swapPos; + + if (attrBytes > 0) { + /** + * If out_buf_bits is not 0, should swap the whole attribute, + * else only swap the beginning 4-byte aligned part of the + * attribute. + */ + if (outBits) { + swapWords = (attrBytes + 3) >> 2; + } else { + swapWords = attrBytes >> 2; + } //if (outBits) + if (swapWords > 0) { + if (array == NDB_ARRAYTYPE_FIXED) { + /** + * Blob V1: + * Blob head V1 is 8 bytes: + * 8 bytes blob length - native endian + * Thus we should swap the byte-order of blob head as Uint64. + * The inline bytes of blob is stored as character string, + * should swap the byte-order for each byte of it. + * Firstly swap the byte order of 64-bit part of the attribute, + * then treat it as 8-bit data and swap it. + */ + attrBuffer = (Uint32*)(outBuffer + preOutIndex); + // allow unaligned + Uint64 tmp; + memcpy(&tmp, attrBuffer, sizeof(Uint64)); + ndbSwapOneWord64_2(&tmp); + memcpy(attrBuffer, &tmp, sizeof(Uint64)); + attrBuffer = (Uint32*)(outBuffer + swapPos); + ndbSwapManyWords32(attrBuffer, swapWords); + } else if (array == NDB_ARRAYTYPE_MEDIUM_VAR) { + /** + * Blob V2: + * Blob head V2 is 16 bytes: + * 2 bytes head+inline length - little-endian + * 2 bytes reserved (zero) + * 4 bytes NDB$PKID - little-endian + * 8 bytes blob length - little-endian + * Thus we should swap the byte-order for each byte of blob head. + * The inline bytes of blob is stored as character string, + * should also swap the byte-order for each byte of it. + */ + attrBuffer = (Uint32*)(outBuffer + swapPos); + ndbSwapManyWords32(attrBuffer, swapWords); + }//if + } + } + + // set new swap position + swapBytes = swapWords << 2; + if (outBits || (attrBytes == swapBytes)) { + swapPos = outIndex + 4 * ((outBits + 31) >> 5); + } else { + swapPos += swapBytes; + } + req_struct->endian_out_buf_pos = swapPos; + }//if +}//Dbtup::read_endian_conversion_blob() + +/* ---------------------------------------------------------------- */ +/* This routine is used to swap the byte order for the 16 bit */ +/* attribute size attribute while reading. */ +/* */ +/* e.g. */ +/* create table t1 (a small int unsigned not null primary key, */ +/* b small int unsigned not null) */ +/* engine = ndbcluster; */ +/* insert into t1 (0x0a0b, 0x0c0d); */ +/* select * from t1 where a = 0x0a0b; */ +/* */ +/* The out buffer format before swap should be: */ +/* [ READ_PACKED 4 ] [ 3 ] [ '0a0b' '0c0d' ] */ +/* */ +/* The out buffer format after swapped should be: */ +/* [ READ_PACKED 4 ] [ 3 ] [ '0c0d' '0a0b' ] */ +/* ---------------------------------------------------------------- */ +void +Dbtup::read_endian_conversion_a16bit(Uint8* outBuffer, + KeyReqStruct *req_struct) +{ + jam(); + Uint32 preOutIndex = req_struct->endian_out_buf_index; + Uint32 outIndex = req_struct->out_buf_index; + Uint32 swapPos = req_struct->endian_out_buf_pos; + ndbrequire(preOutIndex <= outIndex); + ndbrequire(swapPos <= outIndex); + Uint32 outBits = req_struct->out_buf_bits; + Uint32* attrBuffer; + Uint32 attrBytes = 0; + Uint32 swapBytes = 0; + Uint32 swapWords = 0; + + if (! req_struct->is_packed) { + attrBytes = outIndex - preOutIndex; + if (attrBytes > 0) { + // the attribute is not NULL + swapWords = (attrBytes + 3) >> 2; + attrBuffer = (Uint32*)(outBuffer + preOutIndex); + + ndbSwapManyWords16(attrBuffer, swapWords); + } + } else { + /** + * Firstly swap the byte order of 16-bit attribute, + * then treat it as 8-bit data and swap it. + */ + attrBytes = outIndex - preOutIndex; + if (attrBytes > 0) { + // allow unaligned + Uint16 tmp; + memcpy(&tmp, outBuffer + preOutIndex, sizeof(Uint16)); + ndbSwapOneWord16(&tmp); + memcpy(outBuffer + preOutIndex, &tmp, sizeof(Uint16)); + } + + attrBytes = outIndex - swapPos; + if (attrBytes > 0) { + /** + * If out_buf_bits is not 0, should swap the whole attribute, + * else only swap the beginning 4-byte aligned part of the + * attribute. + */ + if (outBits) { + swapWords = (attrBytes + 3) >> 2; + } else { + swapWords = attrBytes >> 2; + } //if (outBits) + if (swapWords != 0) { + attrBuffer = (Uint32*)(outBuffer + swapPos); + ndbSwapManyWords32(attrBuffer, swapWords); + } + } + + // set new swap position + swapBytes = swapWords << 2; + if (outBits || (attrBytes == swapBytes)) { + swapPos = outIndex + 4 * ((outBits + 31) >> 5); + } else { + swapPos += swapBytes; + } + req_struct->endian_out_buf_pos = swapPos; + }//if +}//Dbtup::read_endian_conversion_a16bit() + +/* ---------------------------------------------------------------- */ +/* This routine is used to swap the byte order for the 32 bit */ +/* attribute size attribute while reading. */ +/* */ +/* This type attribute need not consider endian, but should */ +/* handle endian conversion for the left unswapped part of */ +/* the previous attribute while reading packed. */ +/* ---------------------------------------------------------------- */ +void +Dbtup::read_endian_conversion_a32bit(Uint8* outBuffer, + KeyReqStruct *req_struct) +{ + jam(); + if (req_struct->is_packed) { + Uint32 preOutIndex = req_struct->endian_out_buf_index; + Uint32 outIndex = req_struct->out_buf_index; + Uint32 swapPos = req_struct->endian_out_buf_pos; + ndbrequire(preOutIndex <= outIndex); + ndbrequire(swapPos <= outIndex); + Uint32 outBits = req_struct->out_buf_bits; + Uint32* attrBuffer; + + /** + * If there is unswapped part of the previous attribute, + * swap the left unswapped part of the previous attribute. + */ + if (swapPos != preOutIndex) { + attrBuffer = (Uint32*)(outBuffer + swapPos); + // the left unswapped part can only be 4 bytes. + ndbSwapOneWord32(attrBuffer); + } + + // set new swap position + req_struct->endian_out_buf_pos = pad32(outIndex, outBits); + }//if +}//Dbtup::read_endian_conversion_a32bit() + +/* ---------------------------------------------------------------- */ +/* This routine is used to swap the byte order for the 64 bit */ +/* attribute size attribute while reading. */ +/* ---------------------------------------------------------------- */ +void +Dbtup::read_endian_conversion_a64bit(Uint8* outBuffer, + KeyReqStruct *req_struct) +{ + jam(); + Uint32 preOutIndex = req_struct->endian_out_buf_index; + Uint32 outIndex = req_struct->out_buf_index; + Uint32 swapPos = req_struct->endian_out_buf_pos; + ndbrequire(preOutIndex <= outIndex); + ndbrequire(swapPos <= outIndex); + Uint32 outBits = req_struct->out_buf_bits; + Uint32* attrBuffer; + Uint32 attrBytes = 0; + + if (req_struct->is_packed) { + /** + * If there is unswapped part of the previous attribute, + * swap the left unswapped part of the previous attribute. + */ + if (swapPos != preOutIndex) { + attrBuffer = (Uint32*)(outBuffer + swapPos); + // the left unswapped part can only be 4 bytes. + ndbSwapOneWord32(attrBuffer); + } + }//if + + attrBytes = outIndex - preOutIndex; + if (attrBytes > 0) { + // the attribute is not NULL + attrBuffer = (Uint32*)(outBuffer + preOutIndex); + // allow unaligned + Uint64 tmp; + memcpy(&tmp, attrBuffer, sizeof(Uint64)); + ndbSwapOneWord64(&tmp); + memcpy(attrBuffer, &tmp, sizeof(Uint64)); + } + + // set new swap position + if (req_struct->is_packed) { + req_struct->endian_out_buf_pos = pad32(outIndex, outBits); + } +}//Dbtup::read_endian_conversion_a64bit() + +/* ---------------------------------------------------------------- */ +/* This routine is used to swap the byte order for the 128 bit */ +/* attribute size attribute while reading. */ +/* XXX: This function is not tested yet, don't have a good method */ +/* to test 128bit attribute. */ +/* ---------------------------------------------------------------- */ +void +Dbtup::read_endian_conversion_a128bit(Uint8* outBuffer, + KeyReqStruct *req_struct) +{ + jam(); + Uint32 preOutIndex = req_struct->endian_out_buf_index; + Uint32 outIndex = req_struct->out_buf_index; + Uint32 swapPos = req_struct->endian_out_buf_pos; + ndbrequire(preOutIndex <= outIndex); + ndbrequire(swapPos <= outIndex); + Uint32 outBits = req_struct->out_buf_bits; + Uint32* attrBuffer; + Uint32 attrBytes = 0; + + if (req_struct->is_packed) { + /** + * If there is unswapped part of the previous attribute, + * swap the left unswapped part of the previous attribute. + */ + if (swapPos != preOutIndex) { + attrBuffer = (Uint32*)(outBuffer + swapPos); + // the left unswapped part can only be 4 bytes. + ndbSwapOneWord32(attrBuffer); + } + } + + attrBytes = outIndex - preOutIndex; + if (attrBytes > 0) { + // the attribute is not NULL + attrBuffer = (Uint32*)(outBuffer + preOutIndex); + ndbSwapOneWord128(attrBuffer); + } + + // set new swap position + if (req_struct->is_packed) { + req_struct->endian_out_buf_pos = pad32(outIndex, outBits); + } +}//Dbtup::read_endian_conversion_a128bit() + +/* ---------------------------------------------------------------- */ +/* This routine is used to swap the byte order for the 8 bit */ +/* attribute size attribute while updating. */ +/* ---------------------------------------------------------------- */ +void +Dbtup::update_endian_conversion_an8bit(Uint32* inBuffer, + KeyReqStruct *req_struct) +{ + Uint32 attrBufIndex = req_struct->in_buf_index; + AttributeHeader ahIn(inBuffer[attrBufIndex]); + Uint32 nullIndicator= ahIn.isNULL(); + + if (!nullIndicator) { + jam(); + + Uint32 attrDescriptor = req_struct->attr_descriptor; + Uint32 attrWords = AttributeDescriptor::getSizeInWords(attrDescriptor); + Uint32* attrBuffer = inBuffer + attrBufIndex + 1; + + ndbSwapManyWords32(attrBuffer, attrWords); + } +}//Dbtup::update_endian_conversion_an8bit() + +/* ---------------------------------------------------------------- */ +/* This routine is used to swap the byte order for the variable */ +/* size attribute while updating. */ +/* ---------------------------------------------------------------- */ +void +Dbtup::update_endian_conversion_var(Uint32* inBuffer, + KeyReqStruct *req_struct) +{ + Uint32 attrBufIndex = req_struct->in_buf_index; + AttributeHeader ahIn(inBuffer[attrBufIndex]); + Uint32 nullIndicator= ahIn.isNULL(); + + if (!nullIndicator) { + jam(); + + Uint32 attrBytes = ahIn.getByteSize(); + Uint32 attrWords = (attrBytes + 3) >> 2; + Uint32* attrBuffer = inBuffer + attrBufIndex + 1; + + ndbSwapManyWords32(attrBuffer, attrWords); + } +}//Dbtup::update_endian_conversion_var() + +/* ---------------------------------------------------------------- */ +/* This routine is used to swap the byte order for the blob/text */ +/* type attribute while updating. */ +/* ---------------------------------------------------------------- */ +void +Dbtup::update_endian_conversion_blob(Uint32* inBuffer, + KeyReqStruct *req_struct) +{ + Uint32 attrBufIndex = req_struct->in_buf_index; + AttributeHeader ahIn(inBuffer[attrBufIndex]); + Uint32 nullIndicator= ahIn.isNULL(); + + if (!nullIndicator) { + jam(); + + Uint32 attrDescriptor = req_struct->attr_descriptor; + Uint32 array = AttributeDescriptor::getArrayType(attrDescriptor); + Uint32 attrWords; + Uint32 attrBytes; + Uint32* attrBuffer = inBuffer + attrBufIndex + 1; + + if (array == NDB_ARRAYTYPE_FIXED) { + /** + * Blob V1: + * Blob head V1 is 8 bytes: + * 8 bytes blob length - native endian + * Thus we should swap the byte-order of blob head as Uint64. + * The inline bytes of blob is stored as character string, + * should swap the byte-order for each byte of it. + */ + attrWords = AttributeDescriptor::getSizeInWords(attrDescriptor); + // allow unaligned + Uint64 tmp; + memcpy(&tmp, attrBuffer, sizeof(Uint64)); + ndbSwapOneWord64(&tmp); + memcpy(attrBuffer, &tmp, sizeof(Uint64)); + ndbSwapManyWords32(attrBuffer + 2, attrWords - 2); + } else if (array == NDB_ARRAYTYPE_MEDIUM_VAR) { + /** + * Blob V2: + * Blob head V2 is 16 bytes: + * 2 bytes head+inline length - little-endian + * 2 bytes reserved (zero) + * 4 bytes NDB$PKID - little-endian + * 8 bytes blob length - little-endian + * We treat it as a varchar. + */ + attrBytes = ahIn.getByteSize(); + attrWords = (attrBytes + 3) >> 2; + ndbSwapManyWords32(attrBuffer, attrWords); + } + } +}//Dbtup::update_endian_conversion_blob() + +/* ---------------------------------------------------------------- */ +/* This routine is used to swap the byte order for the 16 bit */ +/* attribute size attribute while updating. */ +/* ---------------------------------------------------------------- */ +void +Dbtup::update_endian_conversion_a16bit(Uint32* inBuffer, + KeyReqStruct *req_struct) +{ + Uint32 attrBufIndex = req_struct->in_buf_index; + AttributeHeader ahIn(inBuffer[attrBufIndex]); + Uint32 nullIndicator= ahIn.isNULL(); + + if (!nullIndicator) { + jam(); + + Uint32 attrDescriptor = req_struct->attr_descriptor; + Uint32 attrWords = AttributeDescriptor::getSizeInWords(attrDescriptor); + Uint32* attrBuffer = inBuffer + attrBufIndex + 1; + + ndbSwapManyWords16(attrBuffer, attrWords); + } +}//Dbtup::update_endian_conversion_a16bit() + +/* ---------------------------------------------------------------- */ +/* This routine is used to swap the byte order for the 64 bit */ +/* attribute size attribute while updating. */ +/* ---------------------------------------------------------------- */ +void +Dbtup::update_endian_conversion_a64bit(Uint32* inBuffer, + KeyReqStruct *req_struct) +{ + Uint32 attrBufIndex = req_struct->in_buf_index; + AttributeHeader ahIn(inBuffer[attrBufIndex]); + Uint32 nullIndicator= ahIn.isNULL(); + + if (!nullIndicator) { + jam(); + + Uint32 attrDescriptor = req_struct->attr_descriptor; + Uint32 attrWords = AttributeDescriptor::getSizeInWords(attrDescriptor); + Uint32 attrWords64 = (attrWords + 1) >> 1; + Uint32* attrBuffer = inBuffer + attrBufIndex + 1; + // allow unaligned + Uint64 tmp; + memcpy(&tmp, attrBuffer, attrWords64*sizeof(Uint64)); + ndbSwapManyWords64(&tmp, attrWords64); + memcpy(attrBuffer, &tmp, attrWords64*sizeof(Uint64)); + } +}//Dbtup::update_endian_conversion_a64bit() + +/* ---------------------------------------------------------------- */ +/* This routine is used to swap the byte order for the 128 bit */ +/* attribute size attribute while updating. */ +/* XXX: This function is not tested yet, don't have a good method */ +/* to test 128bit attribute. */ +/* ---------------------------------------------------------------- */ +void +Dbtup::update_endian_conversion_a128bit(Uint32* inBuffer, + KeyReqStruct *req_struct) +{ + Uint32 attrBufIndex = req_struct->in_buf_index; + AttributeHeader ahIn(inBuffer[attrBufIndex]); + Uint32 nullIndicator= ahIn.isNULL(); + + if (!nullIndicator) { + jam(); + + Uint32 attrDescriptor = req_struct->attr_descriptor; + Uint32 attrWords = AttributeDescriptor::getSizeInWords(attrDescriptor); + Uint32* attrBuffer = inBuffer + attrBufIndex + 1; + + ndbSwapManyWords128(attrBuffer, attrWords); + } +}//Dbtup::update_endian_conversion_a128bit() + +/* ---------------------------------------------------------------- */ /* THIS ROUTINE IS USED TO READ A NUMBER OF ATTRIBUTES IN THE */ /* DATABASE AND PLACE THE RESULT IN ATTRINFO RECORDS. */ // @@ -344,6 +1078,23 @@ int Dbtup::readAttributes(KeyReqStruct * Uint32 maxRead, bool xfrm_flag) { + /** + * Compare the enidan here, so that there is no influence to the + * performance while client and TUP nodes have the same endian. + */ + const Uint32 apiByteOrder = req_struct->api_byte_order; + bool needConvertEndian = (apiByteOrder != MY_OWN_BYTE_ORDER) ? true : false; + if (needConvertEndian) { + int ret; + ret = readEndianAttributes(req_struct, + inBuffer, + inBufLen, + outBuf, + maxRead, + xfrm_flag); + return ret; + } + Uint32 attributeId, descr_index, tmpAttrBufIndex, tmpAttrBufBits, inBufIndex; Uint32 attributeOffset; TableDescriptor* attr_descr; @@ -403,6 +1154,91 @@ int Dbtup::readAttributes(KeyReqStruct * return pad32(req_struct->out_buf_index, req_struct->out_buf_bits) >> 2; } +/* ---------------------------------------------------------------- */ +/* This routine is used to read attributes in the case that */ +/* API nodes and DB nodes have different endian. */ +/* ---------------------------------------------------------------- */ +int Dbtup::readEndianAttributes(KeyReqStruct *req_struct, + const Uint32* inBuffer, + Uint32 inBufLen, + Uint32* outBuf, + Uint32 maxRead, + bool xfrm_flag) +{ + Uint32 attributeId, descr_index, tmpAttrBufIndex, tmpAttrBufBits, inBufIndex; + Uint32 attributeOffset; + TableDescriptor* attr_descr; + AttributeHeader* ahOut; + + Tablerec* const regTabPtr= tabptr.p; + Uint32 numAttributes= regTabPtr->m_no_of_attributes; + + inBufIndex= 0; + req_struct->out_buf_index= 0; + req_struct->out_buf_bits = 0; + req_struct->endian_out_buf_pos = 0; + req_struct->max_read= 4*maxRead; + req_struct->xfrm_flag= xfrm_flag; + Uint8*outBuffer = (Uint8*)outBuf; + while (inBufIndex < inBufLen) { + tmpAttrBufIndex= req_struct->out_buf_index; + tmpAttrBufBits = req_struct->out_buf_bits; + AttributeHeader ahIn(inBuffer[inBufIndex]); + inBufIndex++; + attributeId= ahIn.getAttributeId(); + descr_index= attributeId << ZAD_LOG_SIZE; + jam(); + + tmpAttrBufIndex = pad32(tmpAttrBufIndex, tmpAttrBufBits); + AttributeHeader::init(&outBuffer[tmpAttrBufIndex], attributeId, 0); + ahOut= (AttributeHeader*)&outBuffer[tmpAttrBufIndex]; + req_struct->out_buf_index= tmpAttrBufIndex + 4; + req_struct->out_buf_bits = 0; + attr_descr= req_struct->attr_descr; + if (attributeId < numAttributes) { + attributeOffset= attr_descr[descr_index + 1].tabDescr; + ReadFunction f= regTabPtr->readFunctionArray[attributeId]; + req_struct->attr_descriptor= attr_descr[descr_index].tabDescr; + req_struct->endian_out_buf_index = req_struct->out_buf_index; + // the out buffer format is not packed + req_struct->is_packed = false; + ReadEndianFunction f_read_endian= regTabPtr->readEndianFunctionArray[attributeId]; + if ((this->*f)(outBuffer, + req_struct, + ahOut, + attributeOffset)) { + /** + * If the endian of API node and own node are different, + * the byte-order of each 4-byte word of the attributes read will be + * swapped while sending to the API node. + * Thus we should swap the byte-order for the attributes read in + * advance, so that it is no need to swap it on the API node again. + * We handle the outBuffer after reading function. + */ + if (f_read_endian) { + (this->*f_read_endian)(outBuffer, req_struct); + } + continue; + } else { + return -1; + } + } + else if(attributeId & AttributeHeader::PSEUDO) + { + Uint32 sz = read_pseudo_endian(inBuffer, inBufIndex, + req_struct, + (Uint32*)outBuffer); + inBufIndex += sz; + } + else + { + terrorCode = ZATTRIBUTE_ID_ERROR; + return -1; + }//if + }//while + return pad32(req_struct->out_buf_index, req_struct->out_buf_bits) >> 2; +} + bool Dbtup::readFixedSizeTHOneWordNotNULL(Uint8* outBuffer, KeyReqStruct *req_struct, @@ -1480,6 +2316,20 @@ int Dbtup::updateAttributes(KeyReqStruct Uint32* inBuffer, Uint32 inBufLen) { + /** + * Compare the enidan here, so that there is no influence to the + * performance while client and TUP nodes have the same endian. + */ + const Uint32 apiByteOrder = req_struct->api_byte_order; + bool needConvertEndian = (apiByteOrder != MY_OWN_BYTE_ORDER) ? true : false; + if (needConvertEndian) { + int ret; + ret = updateEndianAttributes(req_struct, + inBuffer, + inBufLen); + return ret; + } + Tablerec* const regTabPtr= tabptr.p; Operationrec* const regOperPtr= operPtr.p; Uint32 numAttributes= regTabPtr->m_no_of_attributes; @@ -1571,6 +2421,108 @@ int Dbtup::updateAttributes(KeyReqStruct return 0; } +/* ---------------------------------------------------------------- */ +/* This routine is used to update attributes in the case that */ +/* API nodes and DB nodes have different endian. */ +/* ---------------------------------------------------------------- */ +int Dbtup::updateEndianAttributes(KeyReqStruct *req_struct, + Uint32* inBuffer, + Uint32 inBufLen) +{ + Tablerec* const regTabPtr= tabptr.p; + Operationrec* const regOperPtr= operPtr.p; + Uint32 numAttributes= regTabPtr->m_no_of_attributes; + TableDescriptor *attr_descr= req_struct->attr_descr; + + Uint32 inBufIndex= 0; + req_struct->in_buf_index= 0; + req_struct->in_buf_len= inBufLen; + + while (inBufIndex < inBufLen) { + AttributeHeader ahIn(inBuffer[inBufIndex]); + Uint32 attributeId= ahIn.getAttributeId(); + Uint32 attrDescriptorIndex= attributeId << ZAD_LOG_SIZE; + if (likely(attributeId < numAttributes)) { + Uint32 attrDescriptor= attr_descr[attrDescriptorIndex].tabDescr; + Uint32 attributeOffset= attr_descr[attrDescriptorIndex + 1].tabDescr; + req_struct->attr_descriptor= attrDescriptor; + /** + * If the endian of API node and own node are different, + * the byte-order of each 4-byte word of the attributes will be + * swapped while receiving from the API node. + * Thus we should swap back the byte-order for the attributes. + * We handle the inBuffer before updating function. + */ + UpdateEndianFunction f_update_endian= regTabPtr->updateEndianFunctionArray[attributeId]; + if (f_update_endian) { + (this->*f_update_endian)(inBuffer, req_struct); + } + if ((AttributeDescriptor::getPrimaryKey(attrDescriptor)) && + (regOperPtr->op_struct.op_type != ZINSERT)) { + if (checkUpdateOfPrimaryKey(req_struct, + &inBuffer[inBufIndex], + regTabPtr)) { + jam(); + terrorCode= ZTRY_UPDATE_PRIMARY_KEY; + return -1; + } + } + UpdateFunction f= regTabPtr->updateFunctionArray[attributeId]; + jam(); + req_struct->changeMask.set(attributeId); + if (attributeId >= 64) { + if (req_struct->max_attr_id_updated < attributeId) { + Uint32 no_changed_attrs= req_struct->no_changed_attrs; + req_struct->max_attr_id_updated= attributeId; + req_struct->no_changed_attrs= no_changed_attrs + 1; + } + } + if ((this->*f)(inBuffer, + req_struct, + attributeOffset)) { + inBufIndex= req_struct->in_buf_index; + continue; + } else { + jam(); + return -1; + } + } + else if(attributeId == AttributeHeader::DISK_REF) + { + /** + * No need consider endian here, the data is Uint32. + */ + jam(); + Uint32 sz= ahIn.getDataSize(); + ndbrequire(sz == 2); + req_struct->m_tuple_ptr->m_header_bits |= Tuple_header::DISK_PART; + memcpy(req_struct->m_tuple_ptr->get_disk_ref_ptr(regTabPtr), + inBuffer+inBufIndex+1, sz << 2); + inBufIndex += 1 + sz; + req_struct->in_buf_index = inBufIndex; + } + else if(attributeId == AttributeHeader::ANY_VALUE) + { + /** + * No need consider endian here. + */ + jam(); + Uint32 sz= ahIn.getDataSize(); + ndbrequire(sz == 1); + regOperPtr->m_any_value = * (inBuffer + inBufIndex + 1); + inBufIndex += 1 + sz; + req_struct->in_buf_index = inBufIndex; + } + else + { + jam(); + terrorCode= ZATTRIBUTE_ID_ERROR; + return -1; + } + } + return 0; +} + bool Dbtup::checkUpdateOfPrimaryKey(KeyReqStruct* req_struct, Uint32* updateBuffer, @@ -2300,6 +3252,147 @@ Dbtup::read_pseudo(const Uint32 * inBuff return 0; } +/* ---------------------------------------------------------------- */ +/* This routine is used to read pseudo attributes in the case */ +/* that API and DB nodes have different endian. */ +/* ---------------------------------------------------------------- */ +Uint32 +Dbtup::read_pseudo_endian(const Uint32 * inBuffer, Uint32 inPos, + KeyReqStruct *req_struct, + Uint32* outBuf) +{ + ndbassert(inPos); + ndbassert(req_struct->out_buf_index); + ndbassert(req_struct->out_buf_bits == 0); + ndbassert((req_struct->out_buf_index & 3) == 0); + + Uint32 attrId = (* (inBuffer + inPos - 1)) >> 16; + Uint32 outPos = req_struct->out_buf_index; + Uint32* outBuffer = outBuf + ((outPos - 1) >> 2); + + Uint32 sz; + Uint32 tmp[sizeof(SignalHeader)+25]; + Signal * signal = (Signal*)&tmp; + switch(attrId){ + case AttributeHeader::READ_PACKED: + case AttributeHeader::READ_ALL: + return read_packed_endian(inBuffer, inPos, req_struct, outBuf); + case AttributeHeader::FRAGMENT: + outBuffer[1] = fragptr.p->fragmentId; + sz = 1; + break; + case AttributeHeader::FRAGMENT_FIXED_MEMORY: + { + Uint64 tmp= fragptr.p->noOfPages; + tmp*= 32768; + ndbSwapOneWord64(&tmp); + memcpy(outBuffer + 1, &tmp, 8); + sz = 2; + break; + } + case AttributeHeader::FRAGMENT_VARSIZED_MEMORY: + { + Uint64 tmp= fragptr.p->noOfVarPages; + tmp*= 32768; + ndbSwapOneWord64(&tmp); + memcpy(outBuffer + 1, &tmp, 8); + sz = 2; + break; + } + case AttributeHeader::ROW_SIZE: + outBuffer[1] = tabptr.p->m_offsets[MM].m_fix_header_size << 2; + sz = 1; + break; + case AttributeHeader::ROW_COUNT: + case AttributeHeader::COMMIT_COUNT: + signal->theData[0] = operPtr.p->userpointer; + signal->theData[1] = attrId; + + EXECUTE_DIRECT(DBLQH, GSN_READ_PSEUDO_REQ, signal, 2); + outBuffer[1] = signal->theData[0]; + outBuffer[2] = signal->theData[1]; + sz = 2; + // allow unaligned + Uint64 tmp; + memcpy(&tmp, outBuffer + 1, 8); + ndbSwapOneWord64(&tmp); + memcpy(outBuffer + 1, &tmp, 8); + break; + case AttributeHeader::RANGE_NO: + signal->theData[0] = operPtr.p->userpointer; + signal->theData[1] = attrId; + + EXECUTE_DIRECT(DBLQH, GSN_READ_PSEUDO_REQ, signal, 2); + outBuffer[1] = signal->theData[0]; + sz = 1; + break; + case AttributeHeader::DISK_REF: + { + Uint32 *ref= req_struct->m_tuple_ptr->get_disk_ref_ptr(tabptr.p); + outBuffer[1] = ref[0]; + outBuffer[2] = ref[1]; + sz = 2; + break; + } + case AttributeHeader::RECORDS_IN_RANGE: + signal->theData[0] = operPtr.p->userpointer; + signal->theData[1] = attrId; + + EXECUTE_DIRECT(DBLQH, GSN_READ_PSEUDO_REQ, signal, 2); + outBuffer[1] = signal->theData[0]; + outBuffer[2] = signal->theData[1]; + outBuffer[3] = signal->theData[2]; + outBuffer[4] = signal->theData[3]; + sz = 4; + break; + case AttributeHeader::ROWID: + outBuffer[1] = req_struct->frag_page_id; + outBuffer[2] = operPtr.p->m_tuple_location.m_page_idx; + sz = 2; + break; + case AttributeHeader::ROW_GCI: + sz = 0; + if (tabptr.p->m_bits & Tablerec::TR_RowGCI) + { + Uint64 tmp = * req_struct->m_tuple_ptr->get_mm_gci(tabptr.p); + ndbSwapOneWord64(&tmp); + memcpy(outBuffer + 1, &tmp, sizeof(tmp)); + sz = 2; + } + break; + case AttributeHeader::ANY_VALUE: + { + /** + * Read ANY_VALUE does not actually read anything + * but...sets operPtr.p->m_any_value and + * and puts it into clogMemBuffer so that it's also sent + * to backup replica(s) + * + * This nifty features is used for delete+read with circ. replication + */ + jam(); + Uint32 RlogSize = req_struct->log_size; + operPtr.p->m_any_value = inBuffer[inPos]; + * (clogMemBuffer + RlogSize) = inBuffer[inPos - 1]; + * (clogMemBuffer + RlogSize + 1) = inBuffer[inPos]; + req_struct->out_buf_index = outPos - 4; + req_struct->log_size = RlogSize + 2; + return 1; + } + case AttributeHeader::COPY_ROWID: + sz = 2; + outBuffer[1] = operPtr.p->m_copy_tuple_location.m_page_no; + outBuffer[2] = operPtr.p->m_copy_tuple_location.m_page_idx; + break; + default: + return 0; + } + + AttributeHeader::init(outBuffer, attrId, sz << 2); + req_struct->out_buf_index = outPos + 4*sz; + return 0; +} + Uint32 Dbtup::read_packed(const Uint32* inBuf, Uint32 inPos, KeyReqStruct *req_struct, @@ -2426,6 +3519,165 @@ Dbtup::read_packed(const Uint32* inBuf, } error: + ndbrequire(false); + return 0; +} + +/* ---------------------------------------------------------------- */ +/* This routine is used to read packed attributes in the case */ +/* that API and DB nodes have different endian. */ +/* ---------------------------------------------------------------- */ +Uint32 +Dbtup::read_packed_endian(const Uint32* inBuf, Uint32 inPos, + KeyReqStruct *req_struct, + Uint32* outBuffer) +{ + ndbassert(req_struct->out_buf_index >= 4); + ndbassert((req_struct->out_buf_index & 3) == 0); + ndbassert(req_struct->out_buf_bits == 0); + + Tablerec* const regTabPtr = tabptr.p; + Uint32 outPos = req_struct->out_buf_index; + Uint32 outBits = req_struct->out_buf_bits; + Uint32 maxRead = req_struct->max_read; + + Uint32 cnt; + Uint32 numAttributes = regTabPtr->m_no_of_attributes; + Uint32 attrDescriptorStart = regTabPtr->tabDescriptor; + Uint32 attrId = (* (inBuf + inPos - 1)) >> 16; + Uint32 bmlen32 = ((* (inBuf + inPos - 1)) & 0xFFFF); + + Bitmask<MAXNROFATTRIBUTESINWORDS> mask; + if (attrId == AttributeHeader::READ_ALL) + { + cnt = bmlen32; + for (Uint32 i = 0; i<cnt; i++) + mask.set(i); + bmlen32 = 0; + } + else + { + bmlen32 /= 4; + cnt = 32*bmlen32 <= numAttributes ? 32*bmlen32 : numAttributes; + mask.assign(bmlen32, inBuf + inPos); + } + + // Compute result bitmap len + Bitmask<MAXNROFATTRIBUTESINWORDS> nullable = mask; + nullable.bitANDC(regTabPtr->notNullAttributeMask); + Uint32 nullcnt = nullable.count(); + Uint32 masksz = (cnt + nullcnt + 31) >> 5; + + Uint32* dst = (Uint32*)(outBuffer + ((outPos - 4) >> 2)); + Uint32* dstmask = dst + 1; + AttributeHeader::init(dst, AttributeHeader::READ_PACKED, 4*masksz); + bzero(dstmask, 4*masksz); + + AttributeHeader ahOut; + Uint8* outBuf = (Uint8*)outBuffer; + outPos += 4*masksz; + req_struct->endian_out_buf_pos = outPos; + req_struct->is_packed = true; + if (likely(outPos <= maxRead)) + { + jam(); + for (Uint32 attrId = 0, maskpos = 0; attrId<cnt; attrId++, maskpos++) + { + jam(); + if (mask.get(attrId)) + { + jam(); + Uint32 attrDescrIdx = attrDescriptorStart + (attrId << ZAD_LOG_SIZE); + Uint32 attrDesc1 = tableDescriptor[attrDescrIdx].tabDescr; + Uint32 attrDesc2 = tableDescriptor[attrDescrIdx + 1].tabDescr; + ReadFunction f = regTabPtr->readFunctionArray[attrId]; + + if (outBits) + { + ndbassert((outPos & 3) == 0); + } + + Uint32 save[2] = { outPos, outBits }; + switch(AttributeDescriptor::getSize(attrDesc1)){ + case DictTabInfo::aBit: // bit + outPos = (outPos + 3) & ~(Uint32)3; + break; + case DictTabInfo::an8Bit: // char + case DictTabInfo::a16Bit: // uint16 + outPos = outPos + 4 * ((outBits + 31) >> 5); + outBits = 0; + break; + case DictTabInfo::a32Bit: // uint32 + case DictTabInfo::a64Bit: // uint64 + case DictTabInfo::a128Bit: + outPos = ((outPos + 3) & ~(Uint32)3) + 4 * ((outBits + 31) >> 5); + outBits = 0; + break; +#ifdef VM_TRACE + default: + ndbrequire(false); +#endif + } + + req_struct->out_buf_index = outPos; + req_struct->out_buf_bits = outBits; + req_struct->attr_descriptor = attrDesc1; + req_struct->endian_out_buf_index = req_struct->out_buf_index; + ReadEndianFunction f_read_endian = regTabPtr->readEndianFunctionArray[attrId]; + if ((this->*f)(outBuf, req_struct, &ahOut, attrDesc2)) + { + jam(); + BitmaskImpl::set(masksz, dstmask, maskpos); + + outPos = req_struct->out_buf_index; + outBits = req_struct->out_buf_bits; + + if (nullable.get(attrId)) + { + jam(); + maskpos++; + if (ahOut.isNULL()) + { + jam(); + BitmaskImpl::set(masksz, dstmask, maskpos); + outPos = save[0]; + outBits = save[1]; + } + } + /** + * If the endian of API node and own node are different, + * the byte-order of each 4-byte word of the attributes read will be + * swapped while sending to the API node. + * Thus we should swap the byte-order for the attributes read in + * advance, so that it is no need to swap it on the API node again. + * We handle the outBuf after reading function. + */ + if (f_read_endian) + { + (this->*f_read_endian)(outBuf, req_struct); + } + continue; + } else { + goto error; + }//if + } + } + + req_struct->out_buf_index = pad32(outPos, outBits); + req_struct->out_buf_bits = 0; + + // Swap the left unswapped buffer, it can only be 4 bytes. + if (req_struct->endian_out_buf_pos != req_struct->out_buf_index) { + ndbrequire(req_struct->endian_out_buf_pos + 4 <= req_struct->out_buf_index); + Uint32* leftBuf = (Uint32*)(outBuf + req_struct->endian_out_buf_pos); + ndbSwapOneWord32(leftBuf); + req_struct->endian_out_buf_pos = req_struct->out_buf_index; + } + + return bmlen32; + } + +error: ndbrequire(false); return 0; } --- 1.27/storage/ndb/src/kernel/blocks/dbtux/DbtuxGen.cpp 2008-01-28 19:44:27 +08:00 +++ 1.28/storage/ndb/src/kernel/blocks/dbtux/DbtuxGen.cpp 2008-01-28 19:44:27 +08:00 @@ -15,6 +15,7 @@ #define DBTUX_GEN_CPP #include "Dbtux.hpp" +#include <ndb_byteorder.h> #include <signaldata/NodeStateSignalData.hpp> @@ -298,7 +299,16 @@ Dbtux::readTablePk(const Frag& frag, Tre { const Uint32 tableFragPtrI = frag.m_tupTableFragPtrI; const TupLoc tupLoc = ent.m_tupLoc; - int ret = c_tup->tuxReadPk(tableFragPtrI, tupLoc.getPageId(), tupLoc.getPageOffset(), pkData, true); + /** + * Use own node byte order as the argument to call tuxReadPk() + * to get tuple key. + */ + int ret = c_tup->tuxReadPk(tableFragPtrI, + tupLoc.getPageId(), + tupLoc.getPageOffset(), + pkData, + true, + MY_OWN_BYTE_ORDER); jamEntry(); // TODO handle error ndbrequire(ret > 0); --- 1.23/storage/ndb/src/kernel/blocks/dbutil/DbUtil.cpp 2008-01-28 19:44:27 +08:00 +++ 1.24/storage/ndb/src/kernel/blocks/dbutil/DbUtil.cpp 2008-01-28 19:44:27 +08:00 @@ -18,6 +18,7 @@ #include "DbUtil.hpp" #include <ndb_version.h> +#include <ndb_byteorder.h> #include <signaldata/WaitGCP.hpp> #include <signaldata/KeyInfo.hpp> @@ -1921,6 +1922,10 @@ DbUtil::runTransaction(Signal* signal, T /* First operation */ Uint32 start = 0; TcKeyReq::setStartFlag(start, 1); + /** + * Set the byte order of API(my own). + */ + TcKeyReq::setApiByteOrder(start, MY_OWN_BYTE_ORDER); runOperation(signal, transPtr, opPtr, start); transPtr.p->sent ++; --- 1.72/storage/ndb/src/kernel/blocks/ndbcntr/NdbcntrMain.cpp 2008-01-28 19:44:27 +08:00 +++ 1.73/storage/ndb/src/kernel/blocks/ndbcntr/NdbcntrMain.cpp 2008-01-28 19:44:27 +08:00 @@ -18,6 +18,7 @@ #include <ndb_limits.h> #include <ndb_version.h> +#include <ndb_byteorder.h> #include <SimpleProperties.hpp> #include <signaldata/DictTabInfo.hpp> #include <signaldata/CreateTable.hpp> @@ -1925,6 +1926,7 @@ void Ndbcntr::crSystab7Lab(Signal* signa tcKeyReq->setKeyLength (reqInfo_Start, 1); tcKeyReq->setAIInTcKeyReq (reqInfo_Start, 5); tcKeyReq->setAbortOption (reqInfo_Start, TcKeyReq::AbortOnError); + tcKeyReq->setApiByteOrder (reqInfo_Start, MY_OWN_BYTE_ORDER); /* KEY LENGTH = 1, ATTRINFO LENGTH IN TCKEYREQ = 5 */ cresponses = 0; --- 1.64/storage/ndb/src/kernel/blocks/qmgr/QmgrMain.cpp 2008-01-28 19:44:27 +08:00 +++ 1.65/storage/ndb/src/kernel/blocks/qmgr/QmgrMain.cpp 2008-01-28 19:44:27 +08:00 @@ -36,6 +36,7 @@ #include <signaldata/Upgrade.hpp> #include <ndb_version.h> +#include <ndb_byteorder.h> #ifdef DEBUG_ARBIT #include <NdbOut.hpp> @@ -1069,6 +1070,7 @@ Qmgr::sendCmNodeInfoReq(Signal* signal, req->dynamicId = self->ndynamicId; req->version = getNodeInfo(getOwnNodeId()).m_version; req->mysql_version = getNodeInfo(getOwnNodeId()).m_mysql_version; + req->setByteOrder(MY_OWN_BYTE_ORDER); const Uint32 ref = calcQmgrBlockRef(nodeId); sendSignal(ref,GSN_CM_NODEINFOREQ, signal, CmNodeInfoReq::SignalLength, JBB); DEBUG_START(GSN_CM_NODEINFOREQ, nodeId, ""); @@ -1647,6 +1649,16 @@ void Qmgr::execCM_NODEINFOCONF(Signal* s recompute_version_info(NodeInfo::DB, version); + /** + * Store the byte order of the sender node + */ + Uint32 byteOrder = MY_OWN_BYTE_ORDER; + if (version >= NDBD_ENDIAN_COMPATIBLE_VERSION) { + jam(); + byteOrder = conf->getByteOrder(); + } + setNodeInfo(replyNodePtr.i).setByteOrder(byteOrder); + if(!c_start.m_nodes.done()){ jam(); return; @@ -1703,6 +1715,17 @@ void Qmgr::execCM_NODEINFOREQ(Signal* si mysql_version = 0; setNodeInfo(addNodePtr.i).m_mysql_version = mysql_version; + + /** + * Store the byte order of the sender node + */ + Uint32 byteOrder = MY_OWN_BYTE_ORDER; + if (req->version >= NDBD_ENDIAN_COMPATIBLE_VERSION) { + jam(); + byteOrder = req->getByteOrder(); + } + setNodeInfo(addNodePtr.i).setByteOrder(byteOrder); + c_maxDynamicId = req->dynamicId; cmAddPrepare(signal, addNodePtr, nodePtr.p); @@ -1759,6 +1782,7 @@ Qmgr::cmAddPrepare(Signal* signal, NodeR conf->dynamicId = self->ndynamicId; conf->version = getNodeInfo(getOwnNodeId()).m_version; conf->mysql_version = getNodeInfo(getOwnNodeId()).m_mysql_version; + conf->setByteOrder(MY_OWN_BYTE_ORDER); sendSignal(nodePtr.p->blockRef, GSN_CM_NODEINFOCONF, signal, CmNodeInfoConf::SignalLength, JBB); DEBUG_START(GSN_CM_NODEINFOCONF, refToNode(nodePtr.p->blockRef), ""); @@ -2277,6 +2301,10 @@ void Qmgr::initData(Signal* signal) } setNodeInfo(getOwnNodeId()).m_mysql_version = NDB_MYSQL_VERSION_D; + /** + * Set the byte order of own node. + */ + setNodeInfo(getOwnNodeId()).setByteOrder(MY_OWN_BYTE_ORDER); }//Qmgr::initData() @@ -2871,6 +2899,15 @@ void Qmgr::execAPI_REGREQ(Signal* signal ApiRegReq* req = (ApiRegReq*)signal->getDataPtr(); const Uint32 version = req->version; const BlockReference ref = req->ref; + + /** + * Get the byte order of the API node + */ + Uint32 byteOrder = MY_OWN_BYTE_ORDER; + if (version >= NDBD_ENDIAN_COMPATIBLE_VERSION) { + jam(); + byteOrder = req->getByteOrder(); + } Uint32 mysql_version = req->mysql_version; if (version < NDBD_SPLIT_VERSION) @@ -2931,6 +2968,8 @@ void Qmgr::execAPI_REGREQ(Signal* signal setNodeInfo(apiNodePtr.i).m_version = version; setNodeInfo(apiNodePtr.i).m_mysql_version = mysql_version; + // store the byte order of API node + setNodeInfo(apiNodePtr.i).setByteOrder(byteOrder); setNodeInfo(apiNodePtr.i).m_heartbeat_cnt= 0; ApiRegConf * const apiRegConf = (ApiRegConf *)&signal->theData[0]; @@ -2938,6 +2977,8 @@ void Qmgr::execAPI_REGREQ(Signal* signal apiRegConf->apiHeartbeatFrequency = (chbApiDelay / 10); apiRegConf->version = NDB_VERSION; apiRegConf->mysql_version = NDB_MYSQL_VERSION_D; + // set the byte order of own + apiRegConf->setByteOrder(MY_OWN_BYTE_ORDER); NodeState state= apiRegConf->nodeState = getNodeState(); { NodeRecPtr nodePtr; @@ -3030,6 +3071,9 @@ Qmgr::execAPI_VERSION_REQ(Signal * signa Uint32 senderRef = req->senderRef; Uint32 nodeId = req->nodeId; + // judge whether need endian conversion + bool needConvertEndian = compareEndianWithOwn(senderRef); + ApiVersionConf * conf = (ApiVersionConf *)req; if(getNodeInfo(nodeId).m_connected) { @@ -3037,6 +3081,17 @@ Qmgr::execAPI_VERSION_REQ(Signal * signa conf->mysql_version = getNodeInfo(nodeId).m_mysql_version; struct in_addr in= globalTransporterRegistry.get_connect_address(nodeId); conf->inet_addr= in.s_addr; + /** + * If the endian of the sender and receiver are different, the byte + * order of conf->inet_addr will be swapped, but conf->inet_addr + * already is network byte order before sending, so it is needed to + * turn back its byte order again. + * Swap the byte order before sending instead of turning it back on + * API node. + */ + if (needConvertEndian) { + ndbSwapManyWords32(&(conf->inet_addr), sizeof(conf->inet_addr)>>2); + } } else { --- 1.83/storage/ndb/src/kernel/blocks/suma/Suma.cpp 2008-01-28 19:44:27 +08:00 +++ 1.84/storage/ndb/src/kernel/blocks/suma/Suma.cpp 2008-01-28 19:44:27 +08:00 @@ -17,6 +17,7 @@ #include "Suma.hpp" #include <ndb_version.h> +#include <ndb_byteorder.h> #include <NdbTCP.h> #include <Bitmask.hpp> @@ -2145,6 +2146,7 @@ Suma::SyncRecord::nextScan(Signal* signa ScanFragReq::setHoldLockFlag(req->requestInfo, 1); ScanFragReq::setKeyinfoFlag(req->requestInfo, 0); ScanFragReq::setAttrLen(req->requestInfo, attrLen); + ScanFragReq::setApiByteOrder(req->requestInfo, MY_OWN_BYTE_ORDER); req->fragmentNoKeyLen = fd.m_fragDesc.m_fragmentNo; req->schemaVersion = tabPtr.p->m_schemaVersion; req->transId1 = 0; --- 1.45/storage/ndb/src/kernel/vm/SimulatedBlock.cpp 2008-01-28 19:44:27 +08:00 +++ 1.46/storage/ndb/src/kernel/vm/SimulatedBlock.cpp 2008-01-28 19:44:27 +08:00 @@ -40,6 +40,7 @@ #include <NdbSqlUtil.hpp> #include <EventLogger.hpp> +#include <ndb_byteorder.h> extern EventLogger g_eventLogger; @@ -2356,6 +2357,230 @@ SimulatedBlock::create_distr_key(Uint32 } } return dstPos; +} + +/* + * Compare the byte order of sender or receiver node with own node. + * If they are different, return TRUE. + * Else, return FALSE. + */ +bool +SimulatedBlock::compareEndianWithOwn(Uint32 otherBlockRef) +{ + const Uint32 otherNodeId = refToNode(otherBlockRef); + const Uint32 ownNodeId = getOwnNodeId(); + if (otherNodeId != 0 && otherNodeId != ownNodeId) { + /** + * execute remotely + */ + Uint32 otherByteOrder = getNodeInfo(otherNodeId).getByteOrder(); + + if (otherByteOrder != MY_OWN_BYTE_ORDER) { + return true; + } + } + + return false; +} + +bool +SimulatedBlock::convert_endian_key(Uint32 tab, Uint32* src) const +{ + const KeyDescriptor * desc = g_key_descriptor_pool.getPtr(tab); + const Uint32 noOfKeyAttr = desc->noOfKeyAttr; + + Uint32 i = 0; + Uint32 srcPos = 0; + while (i < noOfKeyAttr) + { + const KeyDescriptor::KeyAttr& keyAttr = desc->keyAttr[i]; + bool ok = + convert_endian_attr(keyAttr.attributeDescriptor, keyAttr.charsetInfo, + src, srcPos); + if (unlikely(!ok)) + return false; + + i++; + } + + return true; +} + +bool +SimulatedBlock::convert_endian_attr(Uint32 attrDesc, CHARSET_INFO* cs, + Uint32* src, Uint32& srcPos) const +{ + Uint32 attrBytes = AttributeDescriptor::getSizeInBytes(attrDesc); + Uint32 attrNoOfWords = AttributeDescriptor::getSizeInWords(attrDesc); + ndbassert(attrNoOfWords > 0); + Uint32 typeId = AttributeDescriptor::getType(attrDesc); + Uint32* attrBuf = (Uint32*)((char*)src + srcPos); + bool ok = false; + + switch (typeId) { + case NdbSqlUtil::Type::Tinyint: + case NdbSqlUtil::Type::Tinyunsigned: + /** + * 8 bits + * data is stored in the lowest 1 byte of a 4-byte word. + * should swap byte order for each byte of the word. + */ + ndbSwapManyWords32(attrBuf, attrNoOfWords); + break; + case NdbSqlUtil::Type::Smallint: + case NdbSqlUtil::Type::Smallunsigned: + /** + * 16 bits + * data is stored as Uint16. + * should swap byte order as Uint16. + */ + ndbSwapManyWords16(attrBuf, attrNoOfWords); + break; + case NdbSqlUtil::Type::Mediumint: + case NdbSqlUtil::Type::Mediumunsigned: + /** + * 24 bits + * data is stored in the lowest 3 bytes of a 4-byte word. + * should swap byte order for each byte of the word. + */ + ndbSwapManyWords32(attrBuf, attrNoOfWords); + break; + case NdbSqlUtil::Type::Bigint: + case NdbSqlUtil::Type::Bigunsigned: + case NdbSqlUtil::Type::Double: + case NdbSqlUtil::Type::Datetime: + { + /** + * 64 bits + * data is stored as Uint64. + * should swap byte order as Uint64. + */ + // allow unaligned + Uint64 tmp; + memcpy(&tmp, attrBuf, sizeof(Uint64)); + ndbSwapOneWord64(&tmp); + memcpy(attrBuf, &tmp, sizeof(Uint64)); + break; + } + case NdbSqlUtil::Type::Olddecimal: + case NdbSqlUtil::Type::Olddecimalunsigned: + case NdbSqlUtil::Type::Decimal : + case NdbSqlUtil::Type::Decimalunsigned : + case NdbSqlUtil::Type::Char: + case NdbSqlUtil::Type::Binary: + /** + * data is stored in the byte order of character string. + * should swap byte order for each byte. + */ + ndbSwapManyWords32(attrBuf, attrNoOfWords); + break; + case NdbSqlUtil::Type::Varchar: + case NdbSqlUtil::Type::Varbinary: + case NdbSqlUtil::Type::Longvarchar: + case NdbSqlUtil::Type::Longvarbinary: + ok = convert_endian_var_attr(attrDesc, cs, src, srcPos, attrBytes); + if (unlikely(!ok)) + return false; + // reset attrNoOfWords + attrNoOfWords = (attrBytes + 3) >> 2; + break; + case NdbSqlUtil::Type::Date: + case NdbSqlUtil::Type::Time: + /** + * 24 bits + * data is stored in the lowest 3 bytes of a 4-byte word. + * should swap byte order for each byte of the word. + */ + ndbSwapManyWords32(attrBuf, attrNoOfWords); + break; + case NdbSqlUtil::Type::Year: + /** + * 8 bits + * data is stored in the lowest 1 byte of a 4-byte word. + * should swap byte order for each byte of the word. + */ + ndbSwapManyWords32(attrBuf, attrNoOfWords); + break; + default: + /** + * the types that no need to handle endian: + * Int, Unsigned, Float, Timestamp, Bit. + * Blob, Text cannot be primary key. + */ + break; + } + + // use attrNoOfWords to calculate the real attrBytes + attrBytes = attrNoOfWords << 2; + srcPos += attrBytes; + + return true; +} + +bool +SimulatedBlock::convert_endian_var_attr(Uint32 attrDesc, CHARSET_INFO* cs, + Uint32* src, Uint32& srcPos, Uint32& attrBytes) const +{ + Uint32 typeId = AttributeDescriptor::getType(attrDesc); + Uint32 array = AttributeDescriptor::getArrayType(attrDesc); + Uint32* attrBuf = (Uint32*)((char*)src + srcPos); + Uint32 noOfWords; + + /** + * The first 1 word(4 bytes) stores the length of the variable attribute, + * so swap the byte order of the first word firstly to get the length + * of the variable attribute. + */ + Uint32 srcTmp; + memcpy(&srcTmp, attrBuf, 4); + ndbSwapOneWord32(&srcTmp); + + const uchar* srcPtr = (const uchar*)&srcTmp; + Uint32 varLen, varWords; + + if (cs == NULL) + { + jam(); + Uint32 len; + LINT_INIT(len); + switch(array){ + case NDB_ARRAYTYPE_SHORT_VAR: + len = 1 + srcPtr[0]; + break; + case NDB_ARRAYTYPE_MEDIUM_VAR: + len = 2 + srcPtr[0] + (srcPtr[1] << 8); + break; + default: + len = attrBytes; + break; + } + varLen = len; + varWords = (varLen + 3) >> 2; + } + else + { + jam(); + Uint32 lb, len; + bool ok = NdbSqlUtil::get_var_length(typeId, srcPtr, attrBytes, lb, len); + if (unlikely(!ok)) + { + ndbout_c("get_var_length failed in SimulatedBlock::convert_endian_var_attr"); + return false; + } + varLen = lb + len; + varWords = (varLen + 3) >> 2; + } + + attrBytes = varLen; + noOfWords = varWords; + + // swap the var attribute + ndbSwapManyWords32(attrBuf, noOfWords); + + // the offset to the next key should be (noOfWords << 2) + attrBytes = noOfWords << 2; + + return true; } CArray<KeyDescriptor> g_key_descriptor_pool; --- 1.41/storage/ndb/src/kernel/vm/SimulatedBlock.hpp 2008-01-28 19:44:27 +08:00 +++ 1.42/storage/ndb/src/kernel/vm/SimulatedBlock.hpp 2008-01-28 19:44:27 +08:00 @@ -462,6 +462,17 @@ protected: Uint32 *data, const Uint32 keyPaLen[MAX_ATTRIBUTES_IN_INDEX])const; + bool compareEndianWithOwn(Uint32 otherBlockRef); + + /** + * convert endian for keyinfo + */ + bool convert_endian_key(Uint32 tab, Uint32* src) const; + bool convert_endian_attr(Uint32 attrDesc, CHARSET_INFO* cs, + Uint32* src, Uint32& srcPos) const; + bool convert_endian_var_attr(Uint32 attrDesc, CHARSET_INFO* cs, + Uint32* src, Uint32& srcPos, Uint32& attrBytes) const; + private: NewVARIABLE* NewVarRef; /* New Base Address Table for block */ Uint16 theBATSize; /* # entries in BAT */ @@ -771,11 +782,16 @@ SimulatedBlock::EXECUTE_DIRECT(Uint32 bl Signal* signal, Uint32 len){ signal->setLength(len); + /** + * For: WL#3615, Endian Task. + * Set the sender blockref here, so that can judge the endian + * of the sender node while executing signal. + */ + signal->header.theSendersBlockRef = reference(); #ifdef VM_TRACE if(globalData.testOn){ signal->header.theVerId_signalNumber = gsn; signal->header.theReceiversBlockNumber = block; - signal->header.theSendersBlockRef = reference(); globalSignalLoggers.executeDirect(signal->header, 0, // in &signal->theData[0], @@ -806,7 +822,6 @@ SimulatedBlock::EXECUTE_DIRECT(Uint32 bl if(globalData.testOn){ signal->header.theVerId_signalNumber = gsn; signal->header.theReceiversBlockNumber = block; - signal->header.theSendersBlockRef = reference(); globalSignalLoggers.executeDirect(signal->header, 1, // out &signal->theData[0], --- 1.91/storage/ndb/src/mgmsrv/Services.cpp 2008-01-28 19:44:28 +08:00 +++ 1.92/storage/ndb/src/mgmsrv/Services.cpp 2008-01-28 19:44:28 +08:00 @@ -423,6 +423,62 @@ backward(const char * base, const Proper return ret; } +/** + * Check the endian and version compatible of the connecting node. + * If the connecting node is of a different endian than the management + * server or any of all connected data nodes _and_ of an older version + * than NDBD_ENDIAN_COMPATIBLE_VERSION(the version from which it supports + * mixed endian), it should not be allowed to connect. + * But for all-same endian systems, even old version node also is allowed + * to connect. + */ +bool +MgmApiSession::check_endian_compatible(const char* otherEndian, + Uint32 otherVersion) +{ + // compare the endian of connecting node with management node + union { long l; char c[sizeof(long)]; } endian_check; + endian_check.l = 1; + if ((strcmp(otherEndian, + (endian_check.c[sizeof(long)-1])?"big":"little") != 0) && + (otherVersion < NDBD_ENDIAN_COMPATIBLE_VERSION)) { + m_output->println("get nodeid reply"); + m_output->println("result: Node does not have the same endianness as " + "management server, the node should be upgraded to " + "a version newer than ndb-%u.%u.%u", + getMajor(NDBD_ENDIAN_COMPATIBLE_VERSION), + getMinor(NDBD_ENDIAN_COMPATIBLE_VERSION), + getBuild(NDBD_ENDIAN_COMPATIBLE_VERSION)); + m_output->println(""); + return false; + } + + // compare the endian of connecting node with all connected data nodes + NodeId nodeId = 0; + Uint32 endian = 0; + while (m_mgmsrv.getNextNodeId(&nodeId, NDB_MGM_NODE_TYPE_NDB)) { + const ClusterMgr::Node &node = + m_mgmsrv.theFacade->theClusterMgr->getNodeInfo(nodeId); + if (node.connected) { + endian = node.m_info.getByteOrder(); + if ((strcmp(otherEndian, (endian == 1)?"big":"little") != 0) && + (otherVersion < NDBD_ENDIAN_COMPATIBLE_VERSION)) { + m_output->println("get nodeid reply"); + m_output->println("result: Node does not have the same endianness as " + "DB node, the node should be upgraded to a version " + "newer than ndb-%u.%u.%u", + getMajor(NDBD_ENDIAN_COMPATIBLE_VERSION), + getMinor(NDBD_ENDIAN_COMPATIBLE_VERSION), + getBuild(NDBD_ENDIAN_COMPATIBLE_VERSION)); + m_output->println(""); + return false; + } + } + } + + return true; +} + void MgmApiSession::get_nodeid(Parser_t::Context &, const class Properties &args) @@ -438,7 +494,6 @@ MgmApiSession::get_nodeid(Parser_t::Cont const char * name= NULL; Uint32 log_event= 1; bool log_event_version; - union { long l; char c[sizeof(long)]; } endian_check; args.get("version", &version); args.get("nodetype", &nodetype); @@ -453,16 +508,16 @@ MgmApiSession::get_nodeid(Parser_t::Cont /* for backwards compatability keep track if client uses new protocol */ log_event_version= args.get("log_event", &log_event); - endian_check.l = 1; - if(endian - && strcmp(endian,(endian_check.c[sizeof(long)-1])?"big":"little")!=0) { - m_output->println(cmd); - m_output->println("result: Node does not have the same endianness as the management server."); - m_output->println(""); - return; + bool compatible; + + // check endian compatible + if (endian) { + compatible = check_endian_compatible(endian, version); + if (!compatible) { + return; + } } - bool compatible; switch (nodetype) { case NODE_TYPE_MGM: case NODE_TYPE_API: --- 1.46/storage/ndb/src/ndbapi/ClusterMgr.cpp 2008-01-28 19:44:28 +08:00 +++ 1.47/storage/ndb/src/ndbapi/ClusterMgr.cpp 2008-01-28 19:44:28 +08:00 @@ -17,6 +17,7 @@ #include <my_pthread.h> #include <ndb_limits.h> #include <util/version.h> +#include <ndb_byteorder.h> #include "TransporterFacade.hpp" #include "ClusterMgr.hpp" @@ -193,6 +194,10 @@ ClusterMgr::forceHB() req->ref = numberToRef(API_CLUSTERMGR, theFacade.ownId()); req->version = NDB_VERSION; req->mysql_version = NDB_MYSQL_VERSION_D; + /** + * Set the byte order of own node + */ + req->setByteOrder(MY_OWN_BYTE_ORDER); int nodeId= 0; for(int i=0; @@ -229,6 +234,13 @@ ClusterMgr::threadMain( ){ req->ref = numberToRef(API_CLUSTERMGR, theFacade.ownId()); req->version = NDB_VERSION; req->mysql_version = NDB_MYSQL_VERSION_D; + /** + * Set the byte order of own node + */ + const NodeId ownNodeId = theFacade.ownId(); + Node & ownNode = theNodes[ownNodeId]; + ownNode.m_info.setByteOrder(MY_OWN_BYTE_ORDER); + req->setByteOrder(MY_OWN_BYTE_ORDER); Uint32 timeSlept = 100; Uint64 now = NdbTick_CurrentMillisecond(); @@ -396,6 +408,15 @@ ClusterMgr::execAPI_REGCONF(const Uint32 node.compatible = ndbCompatible_api_ndb(NDB_VERSION, node.m_info.m_version); } + + /** + * Store the byte order of remote node + */ + Uint32 byteOrder = MY_OWN_BYTE_ORDER; + if (node.m_info.m_version >= NDBD_ENDIAN_COMPATIBLE_VERSION) { + byteOrder = apiRegConf->getByteOrder(); + } + node.m_info.setByteOrder(byteOrder); node.m_api_reg_conf = true; --- 1.43/storage/ndb/src/ndbapi/NdbOperationExec.cpp 2008-01-28 19:44:28 +08:00 +++ 1.44/storage/ndb/src/ndbapi/NdbOperationExec.cpp 2008-01-28 19:44:28 +08:00 @@ -32,6 +32,7 @@ #include <signaldata/ScanTab.hpp> #include <ndb_version.h> +#include <ndb_byteorder.h> #include "API.hpp" #include <NdbOut.hpp> @@ -204,6 +205,8 @@ NdbOperation::prepareSend(Uint32 aTC_Con tcKeyReq->setStartFlag(tReqInfo, tStartIndicator); tcKeyReq->setInterpretedFlag(tReqInfo, tInterpretIndicator); tcKeyReq->setNoDiskFlag(tReqInfo, tNoDisk); + /* Set the byte order of API node(own node). */ + tcKeyReq->setApiByteOrder(tReqInfo, MY_OWN_BYTE_ORDER); OperationType tOperationType = theOperationType; Uint32 tTupKeyLen = theTupKeyLen; @@ -956,6 +959,8 @@ NdbOperation::fillTcKeyReqHdr(TcKeyReq * TcKeyReq::setAbortOption(reqInfo, m_abortOption); TcKeyReq::setDistributionKeyFlag(reqInfo, theDistrKeyIndicator_); TcKeyReq::setScanIndFlag(reqInfo, theScanInfo & 1); + /* Set the byte order of API node(own node). */ + TcKeyReq::setApiByteOrder(reqInfo, MY_OWN_BYTE_ORDER); /* We will setAIInTcKeyReq() and setKeyLength() later. */ tcKeyReq->requestInfo= reqInfo; --- 1.123/storage/ndb/src/ndbapi/NdbScanOperation.cpp 2008-01-28 19:44:28 +08:00 +++ 1.124/storage/ndb/src/ndbapi/NdbScanOperation.cpp 2008-01-28 19:44:28 +08:00 @@ -14,6 +14,7 @@ Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */ #include <ndb_global.h> +#include <ndb_byteorder.h> #include <Ndb.hpp> #include <NdbScanOperation.hpp> #include <NdbIndexScanOperation.hpp> @@ -204,6 +205,7 @@ NdbScanOperation::readTuples(NdbScanOper ScanTabReq::setScanBatch(reqInfo, 0); ScanTabReq::setRangeScanFlag(reqInfo, rangeScan); ScanTabReq::setTupScanFlag(reqInfo, tupScan); + ScanTabReq::setApiByteOrder(reqInfo, MY_OWN_BYTE_ORDER); req->requestInfo = reqInfo; m_keyInfo = (scan_flags & SF_KeyInfo) ? 1 : 0; --- 1.37/storage/ndb/src/ndbapi/TransporterFacade.hpp 2008-01-28 19:44:28 +08:00 +++ 1.38/storage/ndb/src/ndbapi/TransporterFacade.hpp 2008-01-28 19:44:28 +08:00 @@ -169,6 +169,7 @@ private: friend class ClusterMgr; friend class ArbitMgr; friend class MgmtSrvr; + friend class MgmApiSession; friend class SignalSender; friend class GrepPS; friend class ExtSender; ///< @todo Hack to be able to sendSignalUnCond --- 1.44/storage/ndb/src/kernel/blocks/dbtup/DbtupScan.cpp 2008-01-28 19:44:28 +08:00 +++ 1.45/storage/ndb/src/kernel/blocks/dbtup/DbtupScan.cpp 2008-01-28 19:44:28 +08:00 @@ -20,6 +20,7 @@ #include <signaldata/NextScan.hpp> #include <signaldata/AccLock.hpp> #include <md5_hash.hpp> +#include <ndb_byteorder.h> #undef jam #undef jamEntry @@ -309,8 +310,12 @@ Dbtup::scanReply(Signal* signal, ScanOpP // read tuple key - use TUX routine const ScanPos& pos = scan.m_scanPos; const Local_key& key_mm = pos.m_key_mm; + /** + * Since the key info in DBACC is local endian, thus use own + * node byte order as the argument to call tuxReadPk(). + */ int ret = tuxReadPk(fragPtr.i, pos.m_realpid_mm, key_mm.m_page_idx, - pkData, true); + pkData, true, MY_OWN_BYTE_ORDER); ndbrequire(ret > 0); pkSize = ret; dbg((DBTUP, "PK size=%d data=%08x", pkSize, pkData[0])); --- 1.16/storage/ndb/src/common/util/Makefile.am 2008-01-28 19:44:28 +08:00 +++ 1.17/storage/ndb/src/common/util/Makefile.am 2008-01-28 19:44:28 +08:00 @@ -21,7 +21,7 @@ libgeneral_la_SOURCES = \ SocketServer.cpp SocketClient.cpp SocketAuthenticator.cpp\ OutputStream.cpp NdbOut.cpp BaseString.cpp \ NdbSqlUtil.cpp new.cpp \ - uucode.c random.c version.c \ + uucode.c random.c version.c ndb_byteorder.c \ strdup.c \ ConfigValues.cpp ndb_init.c basestring_vsnprintf.c \ Bitmask.cpp \
| Thread | ||
|---|---|---|
| • bk commit into 5.1 tree (dli:1.2855) | David Li | 28 Jan |
© 1995-2008 MySQL AB, 2008-2009 Sun Microsystems, Inc.
Page generated in 2.955 sec. using MySQL 5.1.14-beta-log