On Wed, 2006-06-21 at 10:29 +0200, Michel Stam wrote:
> Cluster actually works pretty well for us (love the failover offered by
> using multiple replicas), up until the point that the database runs
> full, actually. Thats why I chose to implement the functions to estimate
> the memory usage, as the only other way of knowing is when the database
> is about to go titsup (for some reason the management here usually
> doesn't like that ;) ). Can't wait to get my hands on a stable 5.1
> release and switch to disk-based clustering ... hopefully that'll solve
> most of the memory problems (if only for the data memory).
Great!
more monitoring is certainly on our TODO list.
> >Do you deal with redundancy? i.e. can you have more than one of your
> >processes running and still get sane SNMP stuff out?
>
> Not sure what you mean here; The subagent only connects to the mysql
> server on localhost (with a seperate parameter for the cluster
> connectstring as this does not always run on the localhost).
> If you have multiple hosts in a cluster running mysqld and wish to
> have them all export SNMP as well, sure, why not? That should work.
yeah, I mean so that if a machine goes down you still have the ability
to do SNMP stuff (like if a mysql server goes down, you've got another
one).
> Multiple agents on the same machine probably won't work as the SNMP
> OID is registered with SNMPD by the first agent only. This is
> something internal in the net-snmp library. Not sure if that'd be
> useful, in any case.
probably wouldn't be useful.
> Ok, the one thing I need some help with later on would be the Windows
> implementation; Net-SNMP does provide a windows implementation, and
> so
> does MySQL, but I do not have a Windows development platform here so
> getting this to run on Windows is fairly tricky. When I post the
> code,
> perhaps someone would be willing to check it out and see what changes
> need to be made to get that working, too?
> Since I try to strictly adhere to POSIX/ISOC99 (ANSI C is not
> entirely
> possible) I assume that most Unix platforms shouldn't pose any major
> issues.
If you want to use any of our portability libraries, that can help
things a lot. Inside NDB we have our own portlib
(ndb/src/common/portlib) as well as the MySQL mysys.
Since NDB doesn't run on Windows at the moment (it used to, and there's
a porting effort going on), you don't have to worry too much about it
yet.
> Also, I assume that the MySQL SNMP OID I mentioned in an earlier mail
> is
> currently not used by anyone in MySQL? (this to prevent any nasty
> conflicts later on). If not, I may need to register a new one (which
> takes approximately 4 weeks).
I don't know. We can chase that up a bit in the future.
> Actually its 'fairly' simple; using the mysql_fetch_fields and a SHOW
> COLUMNS FROM db.table I get all the info in terms of types,
> declarations
> and sizes I need to walk the list and apply the mysql documentation
> on
> data types and sizes. Getting the VARCHARS and BLOBS right is a pain
> (currently I use the 'length' member of the MYSQL_FIELD structure). I
> need to come up with something to get more accurate lengths there.
> Other than that its aligning (MY_ALIGN, anyone?). ENUMS and SETS
> require
> a bit of parsing on the COLUMN definitions using strstr().
>
> One thing I did see in the MySQL documentation;
> http://dev.mysql.com/doc/refman/5.0/en/mysql-cluster-db-definition.html
>
> Section IndexMemory, example;
> " For each record, there are 12 bytes of data plus 12 bytes overhead.
> Having no nullable columns saves 4 bytes of overhead. "
>
> If I read the perl script correctly, it doesn't take this 4 byte
> reduction into account, calculating 16 bytes of overhead per row in
> all
> cases. Am I correct here, or do I have an old ndb_size.pl? It doesn't
> seem like much, but with very large tables 4 bytes adds up nicely.
(from memory), no it doesn't take this into account.
> Might be a bit tricky to call out to the perl module, as parsing
> requires a lot of processing time. Not sure, but I think it could
> cost
> me more time than the solution I currently use (MYSQL_FIELD
> structures).
> It is possible, of course.
> Perhaps its an idea to export something generic through the API?
> Probably there's more uses for sizing functions.
while exporting more things through the API/SQL is great - it doesn't
fit the requirements for ndb_size.pl as it's a "what will my data take
in cluster" script - so the values returned for the (for example) MyISAM
table that currently holds your data could be very wrong when it comes
to cluster.
> >We should be able to (for cluster tables) talk to the BLOB table to
> get
> >some stats we maintain internally.
>
> Actually its calculated for all tables, even those not in a cluster
> database. This might be used for people who would like to migrate to
> a
> cluster to see how much memory they need to put in the system, for
> instance.
great - this was the purpose of ndb_size.pl too.
> If there's an internal set of functions I could call out to to get
> this
> info in a fairly fast manner that would be great, though. If not I'd
> like to either stick with using the maximum sizes possible, or (with
> a
> configuration option) allow the admin to choose for accuracy over
> speed.
In the future we'll be exporting more things in the
INFORMATION_SCHEMA.FILES table. currently we have free extents for disk
data. more, per table information is planned for sometime in the future.
having all of this from SQL IMHO is a good idea.
> I'm very open to incorporating it. Actually it was my intention to
> GPL
> it from the get-go. My employer only gets benefits if this is done as
> they get the SNMP functionality we need for monitoring the cluster
> with
> each and every new version of the database that is installed. It
> saves
> us from having to patch mysql every time we do an upgrade.
> Also, other people might be able to add functionality to the mod I'm
> not
> even aware of. Especially in the areas we don't use the database for
> (yet), like replication.
>
> Can you list any requirements in terms of code other than the ones
> put
> forth in the initial post by Sascha Pachev a few years back?
> (http://lists.mysql.com/internals/1048)
> I'm cleaning it up anyway so I could just as well take that into
> account.
sure can! Sascha's post is quite old :)
We have some people internally (right now) making the process for
accepting external contributions much easier. This is if you choose to
have it become part of the MySQL Server tree. You may want to keep it as
an external project and just GPLed - maybe at least for the first few
releases during any integration work (and until any MySQL release with
it in there).
I'll keep you informed of the relevant info as it becomes current.
Basically, we'd look for:
- is it well coded
- if it touches MySQL Server code
- does it conform to the coding guidelines/practices etc
- we'd also get people here to review it before it goes in.
- the other legal stuff
of course, having more details somewhere public would be good - i'll get
people onto it (one of those "we should have" that's been around for a
while).
--
Stewart Smith, Software Engineer
MySQL AB, www.mysql.com
Office: +14082136540 Ext: 6616
VoIP: 6616@stripped
Mobile: +61 4 3 8844 332
Jumpstart your cluster:
http://www.mysql.com/consulting/packaged/cluster.html
Attachment: [application/pgp-signature] This is a digitally signed message part signature.asc
Attachment: [application/pgp-signature] This is a digitally signed message part signature.asc
