Hey Stewart,
>If you want to use any of our portability libraries, that can help
>things a lot. Inside NDB we have our own portlib
>(ndb/src/common/portlib) as well as the MySQL mysys.
>
>Since NDB doesn't run on Windows at the moment (it used to, and there's
>a porting effort going on), you don't have to worry too much about it
>yet.
Ok I am trying to use my_sys.h as much as I can. Not sure if I can use
NDB portlib if people do not compile in ndb support?
I am probably going to need a few functions in the range of memset,
strstr and strtoll/strtoull.
>>Also, I assume that the MySQL SNMP OID I mentioned in an earlier mail
>>is
>>currently not used by anyone in MySQL? (this to prevent any nasty
>>conflicts later on). If not, I may need to register a new one (which
>>takes approximately 4 weeks).
>>
> I don't know. We can chase that up a bit in the future.
Ok, I'm using it right now, but it can be changed in under 2 minutes
(its a constant in a header file).
>>cases. Am I correct here, or do I have an old ndb_size.pl? It doesn't
>>seem like much, but with very large tables 4 bytes adds up nicely.
>>
>(from memory), no it doesn't take this into account.
It took me a little while to figure out how the calculation was done,
and I found this one in the example, I thought I'd mention it.
>while exporting more things through the API/SQL is great - it doesn't
>fit the requirements for ndb_size.pl as it's a "what will my data take
>in cluster" script - so the values returned for the (for example) MyISAM
>table that currently holds your data could be very wrong when it comes
>to cluster.
True. It would probably better fit better into support programs, not
the API.
>In the future we'll be exporting more things in the
>INFORMATION_SCHEMA.FILES table. currently we have free extents for disk
>data. more, per table information is planned for sometime in the future.
>
>having all of this from SQL IMHO is a good idea.
Ok, in that case I'll not do anything right now and assume the maximums. This can be added
later on, when the information_schema table is available
>great - this was the purpose of ndb_size.pl too.
I added it to SNMP as I thought it would be a nice feature to have.
Especially since some programs, like Zabbix, can plot very nice graphs
of SNMP values, thus showing you how fast your database runs full.
I wasn't intending on affecting ndb_size.pl in any way, mind.
>sure can! Sascha's post is quite old :)
>
>We have some people internally (right now) making the process for
>accepting external contributions much easier. This is if you choose to
>have it become part of the MySQL Server tree. You may want to keep it as
>an external project and just GPLed - maybe at least for the first few
>releases during any integration work (and until any MySQL release with
>it in there).
>
>I'll keep you informed of the relevant info as it becomes current.
>
>Basically, we'd look for:
>- is it well coded
>- if it touches MySQL Server code
> - does it conform to the coding guidelines/practices etc
>- we'd also get people here to review it before it goes in.
>- the other legal stuff
>
>of course, having more details somewhere public would be good - i'll get
>people onto it (one of those "we should have" that's been around for a
>while).
Ok, I'll await the information; Right now it isn't really important
whether its fully integrated or not. Getting a version public is.
Other than a few mysql/ndb/mgmapi/ include files which are missing by
default after a 'make install' on the base mysql source tree
(mysql/ndb/ndb_global.h and mysql/ndb/ndb_types.h if I remember
correctly), the snmp module quite happily compiles outside of the MySQL
source tree right now.
I will look into that a little later on and submit a patch, if its not
already fixed.
In my case, of course, the module code doesn't touch any of the server
code (monitoring applications should affect the systems they monitor
very little, or preferably, not at all).
I'd welcome any review of the code, of course.
Cheers,
Michel
Stewart Smith wrote:
> On Wed, 2006-06-21 at 10:29 +0200, Michel Stam wrote:
>
>> Cluster actually works pretty well for us (love the failover offered by
>> using multiple replicas), up until the point that the database runs
>> full, actually. Thats why I chose to implement the functions to estimate
>> the memory usage, as the only other way of knowing is when the database
>> is about to go titsup (for some reason the management here usually
>> doesn't like that ;) ). Can't wait to get my hands on a stable 5.1
>> release and switch to disk-based clustering ... hopefully that'll solve
>> most of the memory problems (if only for the data memory).
>>
>
> Great!
>
> more monitoring is certainly on our TODO list.
>
>
>>> Do you deal with redundancy? i.e. can you have more than one of your
>>> processes running and still get sane SNMP stuff out?
>>>
>> Not sure what you mean here; The subagent only connects to the mysql
>> server on localhost (with a seperate parameter for the cluster
>> connectstring as this does not always run on the localhost).
>> If you have multiple hosts in a cluster running mysqld and wish to
>> have them all export SNMP as well, sure, why not? That should work.
>>
>
> yeah, I mean so that if a machine goes down you still have the ability
> to do SNMP stuff (like if a mysql server goes down, you've got another
> one).
>
>
>> Multiple agents on the same machine probably won't work as the SNMP
>> OID is registered with SNMPD by the first agent only. This is
>> something internal in the net-snmp library. Not sure if that'd be
>> useful, in any case.
>>
>
> probably wouldn't be useful.
>
>
>> Ok, the one thing I need some help with later on would be the Windows
>> implementation; Net-SNMP does provide a windows implementation, and
>> so
>> does MySQL, but I do not have a Windows development platform here so
>> getting this to run on Windows is fairly tricky. When I post the
>> code,
>> perhaps someone would be willing to check it out and see what changes
>> need to be made to get that working, too?
>> Since I try to strictly adhere to POSIX/ISOC99 (ANSI C is not
>> entirely
>> possible) I assume that most Unix platforms shouldn't pose any major
>> issues.
>>
>
> If you want to use any of our portability libraries, that can help
> things a lot. Inside NDB we have our own portlib
> (ndb/src/common/portlib) as well as the MySQL mysys.
>
> Since NDB doesn't run on Windows at the moment (it used to, and there's
> a porting effort going on), you don't have to worry too much about it
> yet.
>
>
>> Also, I assume that the MySQL SNMP OID I mentioned in an earlier mail
>> is
>> currently not used by anyone in MySQL? (this to prevent any nasty
>> conflicts later on). If not, I may need to register a new one (which
>> takes approximately 4 weeks).
>>
>
> I don't know. We can chase that up a bit in the future.
>
>
>> Actually its 'fairly' simple; using the mysql_fetch_fields and a SHOW
>> COLUMNS FROM db.table I get all the info in terms of types,
>> declarations
>> and sizes I need to walk the list and apply the mysql documentation
>> on
>> data types and sizes. Getting the VARCHARS and BLOBS right is a pain
>> (currently I use the 'length' member of the MYSQL_FIELD structure). I
>> need to come up with something to get more accurate lengths there.
>> Other than that its aligning (MY_ALIGN, anyone?). ENUMS and SETS
>> require
>> a bit of parsing on the COLUMN definitions using strstr().
>>
>> One thing I did see in the MySQL documentation;
>> http://dev.mysql.com/doc/refman/5.0/en/mysql-cluster-db-definition.html
>>
>> Section IndexMemory, example;
>> " For each record, there are 12 bytes of data plus 12 bytes overhead.
>> Having no nullable columns saves 4 bytes of overhead. "
>>
>> If I read the perl script correctly, it doesn't take this 4 byte
>> reduction into account, calculating 16 bytes of overhead per row in
>> all
>> cases. Am I correct here, or do I have an old ndb_size.pl? It doesn't
>> seem like much, but with very large tables 4 bytes adds up nicely.
>>
>
> (from memory), no it doesn't take this into account.
>
>
>> Might be a bit tricky to call out to the perl module, as parsing
>> requires a lot of processing time. Not sure, but I think it could
>> cost
>> me more time than the solution I currently use (MYSQL_FIELD
>> structures).
>> It is possible, of course.
>> Perhaps its an idea to export something generic through the API?
>> Probably there's more uses for sizing functions.
>>
>
> while exporting more things through the API/SQL is great - it doesn't
> fit the requirements for ndb_size.pl as it's a "what will my data take
> in cluster" script - so the values returned for the (for example) MyISAM
> table that currently holds your data could be very wrong when it comes
> to cluster.
>
>
>>> We should be able to (for cluster tables) talk to the BLOB table to
>>>
>> get
>>
>>> some stats we maintain internally.
>>>
>> Actually its calculated for all tables, even those not in a cluster
>> database. This might be used for people who would like to migrate to
>> a
>> cluster to see how much memory they need to put in the system, for
>> instance.
>>
>
> great - this was the purpose of ndb_size.pl too.
>
>
>> If there's an internal set of functions I could call out to to get
>> this
>> info in a fairly fast manner that would be great, though. If not I'd
>> like to either stick with using the maximum sizes possible, or (with
>> a
>> configuration option) allow the admin to choose for accuracy over
>> speed.
>>
>
> In the future we'll be exporting more things in the
> INFORMATION_SCHEMA.FILES table. currently we have free extents for disk
> data. more, per table information is planned for sometime in the future.
>
> having all of this from SQL IMHO is a good idea.
>
>
>> I'm very open to incorporating it. Actually it was my intention to
>> GPL
>> it from the get-go. My employer only gets benefits if this is done as
>> they get the SNMP functionality we need for monitoring the cluster
>> with
>> each and every new version of the database that is installed. It
>> saves
>> us from having to patch mysql every time we do an upgrade.
>> Also, other people might be able to add functionality to the mod I'm
>> not
>> even aware of. Especially in the areas we don't use the database for
>> (yet), like replication.
>>
>> Can you list any requirements in terms of code other than the ones
>> put
>> forth in the initial post by Sascha Pachev a few years back?
>> (http://lists.mysql.com/internals/1048)
>> I'm cleaning it up anyway so I could just as well take that into
>> account.
>>
>
> sure can! Sascha's post is quite old :)
>
> We have some people internally (right now) making the process for
> accepting external contributions much easier. This is if you choose to
> have it become part of the MySQL Server tree. You may want to keep it as
> an external project and just GPLed - maybe at least for the first few
> releases during any integration work (and until any MySQL release with
> it in there).
>
> I'll keep you informed of the relevant info as it becomes current.
>
> Basically, we'd look for:
> - is it well coded
> - if it touches MySQL Server code
> - does it conform to the coding guidelines/practices etc
> - we'd also get people here to review it before it goes in.
> - the other legal stuff
>
> of course, having more details somewhere public would be good - i'll get
> people onto it (one of those "we should have" that's been around for a
> while).
>
>
>
--
Michel Stam ing.
Software Development Engineer
michel@stripped <mailto:michel@stripped>
Houtweg 29
1251 CS Laren
The Netherlands
t: +31 (0)35 548 98 58
f: +31 (0)35 531 65 61
www.teleconcept.nl <http://www.teleconcept.nl>
