From: Stewart Smith
Date: July 19 2012 4:59am
Subject: [PATCH] Sid_map::Sid_map calls DBUG which may have unitialized THR_KEY_mysys and thus crash
List-Archive: http://lists.mysql.com/internals/38552
Message-Id: <878veg5ote.fsf@flamingspork.com>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
	micalg=pgp-sha1; protocol="application/pgp-signature"

--=-=-=
Content-Transfer-Encoding: quoted-printable

IF you are really lucky (or unlucky), the order in which things are
initialized on server startup goes a bit diferent than you'd expect. I
managed to get this to reproduce by always LD_PRELOADING
libeatmydata. This may also occur if LD_PRELOADING other libraries, but
I haven't tested.

Namely, we get the Sid_map::Sid_map constructor firing before
my_thread_global_init() is run. What happens here is that
Sid_map::Sid_map calls a DBUG function, which will try to get a thread
specific variable for THR_KEY_mysys, but the thread key hasn't yet been
initialized, so instead we get a wonderful pile of garbage.

The end result, a crash:

#0  0x0000000000b8e825 in _db_enter_ (
    _func_=3D<error reading variable: Cannot access memory at address 0xc>,=
=20
    _file_=3D0xcbe490 "dno, const rpl_sid *)", _line_=3D27,=20
    _stack_frame_=3D0x7fff2d95d1f0)
    at dbug/dbug.c:1112
#1  0x000000000084741b in Sid_map::Sid_map (this=3D0x140e2e0,=20
    _sid_lock=3D<optimized out>)
    at sql/rpl_gtid_sid_map.cc:27
#2  0x0000000000847252 in __static_initialization_and_destruction_0 (
    __priority=3D65535, __initialize_p=3D1)
    at sql/rpl_gtid_misc.cc:34
#3  _GLOBAL__sub_I_rpl_gtid_misc.cc(void) ()
    at sql/rpl_gtid_misc.cc:174
#4  0x0000000000bf481d in __do_global_ctors_aux ()


Running under Valgrind gives you instead:

=3D=3D3390=3D=3D Invalid read of size 1
=3D=3D3390=3D=3D    at 0x636D1D2: internal_fnmatch (fnmatch_loop.c:69)
=3D=3D3390=3D=3D    by 0x636E9EC: fnmatch@@GLIBC_2.2.5 (fnmatch.c:460)
=3D=3D3390=3D=3D    by 0xB8D804: InList (dbug.c:1501)

followed by:

=3D=3D3390=3D=3D  General Protection Fault
=3D=3D3390=3D=3D    at 0x636D1D2: internal_fnmatch (fnmatch_loop.c:69)
=3D=3D3390=3D=3D    by 0x636E9EC: fnmatch@@GLIBC_2.2.5 (fnmatch.c:460)
=3D=3D3390=3D=3D    by 0xB8D804: InList (dbug.c:1501)
=3D=3D3390=3D=3D    by 0xB8D871: DoTrace (dbug.c:1670)
=3D=3D3390=3D=3D    by 0xB8E84C: _db_enter_ (dbug.c:1115)
=3D=3D3390=3D=3D    by 0x84741A: Sid_map::Sid_map(Checkable_rwlock*) (rpl_g=
tid_sid_map.cc:27)
=3D=3D3390=3D=3D    by 0x847251: _GLOBAL__sub_I_rpl_gtid_misc.cc (rpl_gtid_=
misc.cc:34)
=3D=3D3390=3D=3D    by 0xBF47CC: __libc_csu_init (in /home/stewart/percona-=
server/5.6/Percona-Server/build/sql/mysqld-debug)
=3D=3D3390=3D=3D    by 0x62C96FF: (below main) (libc-start.c:185)


Clearly, this isn't ideal. basically, it prevents even mtr from starting
the debug server to find what version it is.

The following patch solves the problem for me, basicall, if the thread
specific value hasn't been inited, let's do the right thing and return
NULL. Everything seems to work okay and I believe this to be correct.

Another option could be to not have the global_sid_map in
rpl_gtid_misc.cc be inited as part of setup, or have it not use
DBUG... but this is possibly undesirable and could hide the problem in
the future.

I've also filed this as a bug http://bugs.mysql.com/bug.php?id=3D65946


=2D-- a/mysys/my_thr_init.c	2011-08-10 06:26:45 +0000
+++ b/mysys/my_thr_init.c	2012-07-19 04:52:08 +0000
@@ -458,8 +458,12 @@ const char *my_thread_name(void)
=20
 extern void **my_thread_var_dbug()
 {
=2D  struct st_my_thread_var *tmp=3D
=2D    my_pthread_getspecific(struct st_my_thread_var*,THR_KEY_mysys);
+  struct st_my_thread_var *tmp;
+
+  if (!my_thread_global_init_done)
+    return NULL;
+
+  tmp=3D my_pthread_getspecific(struct st_my_thread_var*,THR_KEY_mysys);
   return tmp && tmp->init ? &tmp->dbug : 0;
 }
 #endif /* DBUG_OFF */



=2D-=20
Stewart Smith

--=-=-=
Content-Type: application/pgp-signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQIcBAEBAgAGBQJQB5QdAAoJEP08notdpDyvl00P/RRtCxMmioNBGRKT48GTKz+T
v9sqb5Ztbo3dFkx8YHchdAZn1ADnx5pbF090ZWnMdExaeK30IfcSot3nEI8/86Ac
1h0C0vwmcEirpLRtcPBkQXzwwx5PEikGWYpH2sjwzPllEqeDhFVR9Q3/ErtZRHv+
Aop3/An6G+iAshu9nNeJuFZDhrTBNBuLTvb+IbFAhzPmGaEFQ1Tv+5x0Z6W6NDmd
My0Jt06ThBO/JlWcigXjuYiySEZVKda8GRS/ylAqpPnwJGTBoHa5XT7hoX44Ybui
vgEO9XjX+pmGclX40QTmZY3zrlTnMi6JJvkK7L/gMtAidmTJt77LvCHiYUGhc4I7
yk/4+5cyaK6Nx20RbulS4DKJIQYyl3c0Jdt8KhvKA7m9NV/qTAUGD8gil/1JfWov
QLK8N9QO/rSNM8hYnHdpds+oey+eSAVDWrd36GChuprA1lJXgoqnVe3LGkKFpYnC
RLxDO5/o5hnaein99cMcDO1yI5acZhTxe/+8HyTCv3SbZd3FGjJ1jLj3WCIeInnz
trco91QJ6febb8M44/F9VG3/wKUU+q7U+uDoy9ZGCxKa+VP9SmLqq+zWwzJo25/n
ToCZyhqvlXQwNa4NnYUDhNFHaBGKqHClWP2PE9/e6FlYMCj8iZsYc9servUWWy+x
qEe2R5x+J2IEkpElWvzE
=juvt
-----END PGP SIGNATURE-----
--=-=-=--