From: Martin Gainty Date: August 23 2012 3:30pm Subject: RE: Machine Learning List-Archive: http://lists.mysql.com/mysql/228062 Message-Id: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="_215c7e8d-89a6-4943-93c3-3cb2c7b203a2_" --_215c7e8d-89a6-4943-93c3-3cb2c7b203a2_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Shawn and Garot I like the parsing capability of the lucene and its ability to stem incomin= g queries.. If you are able to start populating your data then we *should* be able to s= tart identifying which root terms we can pull for building lucene-indexes I have an upcoming stuck-in-JFK-airport-on-labor-day-waiting-for-the-next-a= m-flight-of-USscare for an interrupted timeframe(i never learned how to 'sl= eep' in airport terminal) as long as I can find a wifi connection (and an AC connection within 6 hour= s) and use a working MySQL JDBC connection string..I should be able to bang= out a quick prototype.. Let me know how i can help out=2C Martin Gainty=20 ______________________________________________=20 ..place long-winded disclaimer here.. > From: ag4ve.us@stripped > Date: Thu=2C 23 Aug 2012 10:42:55 -0400 > Subject: Re: Machine Learning > To: garotconklin@stripped > CC: webmaster@stripped=3B mgainty@stripped=3B mysql@stripped >=20 > just a few thoughts (things to look into) >=20 > if you want to populate a db with command parameters=2C i'd mine the man > pages. it's a consistent format and you should be able to find a > parser for whatever language you prefer. >=20 > if you want to see what has been entered and statistics about that > process=2C look at the audit framework. it won't tell you success or > failure=2C but you might be able to derive that based on the time=2C > memory=2C and process usage that you can get from audit. auditd also > maintains a db - you might look into the engine and schema they use. >=20 > you might also find some interesting topics looking into language > processing (after all=2C commands are a type of language). people > generally use other database engines to do this type of thing - > elastic search or some other things people have done with lucene > maybe. >=20 > do let us know where you go with this. it sounds quite interesting > (and might help me with something i've been looking into)... and > getting zsh to autocomplete options might be a bonus if you parse man > pages and shove it into a db (i might do that part for you if i get a > lazy afternoon). >=20 > On Wed=2C Aug 22=2C 2012 at 7:51 PM=2C Garot Conklin wrote: > > I am going to work up a visual diagram for this to better... visualize = (LOL) the lay of the land. I will forward it when i have it completed then = I hope we can continue the architectural discussion here! Thanks > > > > -Garot > > > > > > garotconklin@stripped > > > > > > ________________________________ > > From: Garot Conklin > > To: Garot Conklin =3B "webmaster@stripped" <= webmaster@stripped>=3B 'Martin Gainty' =3B "mysql@l= ists.mysql.com" > > Sent: Tuesday=2C August 21=2C 2012 4:40 PM > > Subject: Re: Machine Learning > > > > to elaborate: > > > > I would almost rather qualify from the OS perspective what will work in= advance=2C but take into consideration any failing condition. I may be ge= tting a bit too specific at this juncture however as I am already apply thi= s logic to issues I see in MY environment rather than total ambiguity=2C wh= ich is a best position here. > > > > I am ultimately expecting the learning algorithm to create new and exci= tingly effective ways of stringing commands together to solve issues=2C thi= s being a notable aspect in and of itself=2C then creating some centralizat= ion based on what the machine determines as a best solution. > > > > Seems a relative "snapshot of the system" pre and post execution would = be favorable as well so long as resources are not taxed as a result. This = discussion is leading me to believe that this will be a multifaceted db sol= ution without a doubt comprised of multiple layers of abstraction ultimatel= y rolling up into a single master cluster for all the higher level analytic= s and internal testing cycles to then roll out the new primary protocols=2C= which may themselves take place in a sub-instance of each possible scenari= o. > > > > I have also though of populating a db or multiple db's with actual text= in the form of .pdf's of actual technical manuals... not certain the bene= fit of the written language abstraction there but the thought process extra= ction might be an interesting end indeed. > > > > > > > > garotconklin@stripped > > > > > > ________________________________ > > From: Garot Conklin > > To: "webmaster@stripped" =3B 'Martin Gainty' = =3B "mysql@stripped" > > Sent: Tuesday=2C August 21=2C 2012 4:09 PM > > Subject: Re: Machine Learning > > > > YES! > > > > > > garotconklin@stripped > > > > > > ________________________________ > > From: "webmaster@stripped" > > To: 'Garot Conklin' =3B 'Martin Gainty' =3B mysql@stripped > > Sent: Tuesday=2C August 21=2C 2012 3:37 PM > > Subject: RE: Machine Learning > > > > > > Ah=2C > > > > Getting clearer and clearer. > > > > So these =91nodes=92 could =91learn=92 and =91teach=92 at the same time= =96 right ? For instance=2C N1 runs a command in =91domain=92 D20 which it= successful =96 it send information to node N20 that is the authority on do= main D20 and N20 records it as success=3B N5 runs a command in domain D20 w= hich goes wrong=2C and sends info to N20 and N20 records failure and sends = a correct call to N5. Is this what you have in mind ? > > > > There are a lot of variables to be considered=2C for instance: > > 1) Command A version 1 can run very happily on operating system ve= rsion 2=2C but fails in OS version 1 > > 2) Command A version 1 can run happily on OS version 2 on a 2G RAM= =2C but fail on OS version 2 on 1G RAM > > 3) etc > > > > I think the DB design issues will become straightforward once the model= is quite clear. > > > > Justin > > > > From:Garot Conklin [mailto:garotconklin@stripped] > > Sent: 21 August 2012 17:14 > > To: webmaster@stripped=3B 'Martin Gainty'=3B mysql@stripped > > Subject: Re: Machine Learning > > > > 1)Refer to it for =91knowledge=92 (for instance=2C of what the latest > > version of a command is) > > It would most likely end up being "central" in this sense: > > A distributed collection of systems=3B i.e. (possible defined i= n roles) > > DB's > > FE's > > REPL's > > Cache's > > Each DB would have its own collection of remediation's that wou= ld then be indexed to populate a central db for trending/correlation etc... > > The "Collective" itself would function as a single conceptual i= mplementation. A VIP for example might be associated with a specific role= =2C say Web FE's and remediate only/all of them=2C but only have some relat= ive access to the core db of say the Network from a Primary index perspecti= ve to make/draw associations/conclusions to issues at hand. > > > > > > 2)Send their =91knowledge=92 (for instance=2C of the latest command > > versions) to it for storage and distribution to others > > Rather than "latest command versions" I envision this to be more enc= apsulated as "latest successful invocation of the command string" and it in= verse as well (to trend/metric-ize the failures thus lending to perpetual o= ptimization). > > > > I like how this is fleshing out... This is helping me to define what I = am really trying to accomplish. Thanks very much for everyone responding he= re=2C this is wonderful=2C please keep this going... > > > > garotconklin@stripped > > > > ________________________________ > > > > From:"webmaster@stripped" > > To: 'Martin Gainty' =3B garotconklin@stripped=3B = mysql@stripped > > Sent: Tuesday=2C August 21=2C 2012 11:19 AM > > Subject: RE: Machine Learning > > > > Hi Garot=2C > > > > > > > > Ok=2C the concept is getting clearer=2C but let=92s bring this down to = earth a > > little bit more. I love DB design and problem-solving and am quite curi= ous > > about this. > > > > > > > > Is the idea that you have a central computer (not HAL J) somewhere so t= hat > > other computers can: > > > > 1) Refer to it for =91knowledge=92 (for instance=2C of what the la= test > > version of a command is) > > > > 2) Send their =91knowledge=92 (for instance=2C of the latest comma= nd > > versions) to it for storage and distribution to others > > > > > > > > If this is the model=2C then the knowledge base can build up organicall= y over > > time =96 I think. Or is this too simplistic ? > > > > > > > > Thanks=2C > > > > Justin > > > > > > > > From: Martin Gainty [mailto:mgainty@stripped] > > Sent: 21 August 2012 00:25 > > To: garotconklin@stripped=3B webmaster@stripped=3B mysql@stripped= l.com > > Subject: RE: Machine Learning > > > > > > > > When I hear 'AI' I always imagine theres a HAL 9001 behind the scenes t= hat > > is running the show constantly admonishing its creator to "take another > > stress pill" > > > > Sounds like a fun project > > > > Keep us apprised=2C > > Martin Gainty > > ______________________________________________ > > Verzicht und Vertraulichkeitanmerkung/Note de d=E9ni et de confidential= it=E9 > > > > > > Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene > > Empfaenger sein=2C so bitten wir hoeflich um eine Mitteilung. Jede unbe= fugte > > Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachric= ht > > dient lediglich dem Austausch von Informationen und entfaltet keine > > rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von > > E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen. > > > > Ce message est confidentiel et peut =EAtre privil=E9gi=E9. Si vous n'= =EAtes pas le > > destinataire pr=E9vu=2C nous te demandons avec bont=E9 que pour satisfa= ire > > informez l'exp=E9diteur. N'importe quelle diffusion non autoris=E9e ou = la copie > > de ceci est interdite. Ce message sert =E0 l'information seulement et n= 'aura > > pas n'importe quel effet l=E9galement obligatoire. =C9tant donn=E9 que = les email > > peuvent facilement =EAtre sujets =E0 la manipulation=2C nous ne pouvons= accepter > > aucune responsabilit=E9 pour le contenu fourni. > > > > > > > > > > > > _____ > > > > Date: Mon=2C 20 Aug 2012 13:50:04 -0700 > > From: garotconklin@stripped > > Subject: Re: Machine Learning > > To: webmaster@stripped=3B mgainty@stripped=3B mysql@stripped= om > > > > Ya the idea is not anything new=2C but must be apparently quit difficul= t or > > not a priority as I have yet to find it already implemented anywhere...= Far > > be it from me to not make some attempt here anyway... > > > > > > > > I am creating a fully automated framework from which a distributed > > infrastructure can be maintained. I have been writing automation > > scripts/code for some time now and the logical progression is to embark= on a > > full concept of systems health auto remediation. I have numerous > > "monitoring" solutions under my control however none that properly (in = my > > opinion) implements any real learning algorithms from which to draw eve= n a > > minimalists view of automation. I like mySQL therefor began thinking a= bout > > creating the aspects (lobes) of the "brain" as a relational database(s)= . So > > this is only one facet of what I am trying to do=2C however leveraging = a full > > command set of shell utilities/commands/programs seemed to be a good > > starting point before I get into the "hard" stuff ! > > > > > > > > -Garot > > > > > > > > garotconklin@stripped > > > > _____ > > > > From: "webmaster@stripped" > > To: 'Martin Gainty' =3B garotconklin@stripped=3B > > mysql@stripped > > Sent: Monday=2C August 20=2C 2012 3:55 PM > > Subject: RE: Machine Learning > > > > > > Hi Garot=2C > > > > > > > > You'll have to elaborate some more ... I understand you may want to pro= tect > > the idea as well=2C so if you can narrow it down to some technical spec= ifics > > then it'll help. > > > > > > > > What is the objective of this system=2C for instance ? > > > > > > > > Thanks=2C > > > > Justin > > > > > > > > From: Martin Gainty [mailto:mgainty@stripped] > > Sent: 20 August 2012 19:23 > > To: garotconklin@stripped=3B webmaster@stripped=3B mysql@stripped= l.com > > Subject: RE: Machine Learning > > > > > > > > > > > > From: garotconklin@stripped > > Subject: Re: Machine Learning > > To: webmaster@stripped=3B mgainty@stripped=3B mysql@stripped= om > > > > > > My initial thought was to propagate the db with everything and allow th= e > > algorithm to then begin to determin trends/patterns > > MG>which trends or patterns will you be modelling? > > > > and begin either an indexing methodology > > MG>which indexes are you considering: Unique index=2C primary index or = foreign > > index? > > > > additional table/db creation process or both to further optimize the ca= lls > > being made > > MG>optimize based on execution time or diskspace allocated=2C Eliminati= ngFTS > > or some other criteria? > > MG>https://dev.mysql.com/doc/refman/5.5/en/optimization.html > > > > and build in some internal levels of redundancy. > > MG>what about replication > > MG>http://dev.mysql.com/doc/refman/5.5/en/replication.html > > > > I am actually approaching this with some degree of biological conceptio= n in > > the multipathing within our own brains however until I have something u= p and > > running under some substantial load however I may not get a complete > > picture. > > > > Thanks=2C > > > > Garot > > > > > > Interesting > > Martin > > > > _____ > > > > From: webmaster@stripped =3B > > To: 'Garot Conklin' =3B 'Martin Gainty' > > =3B =3B > > Subject: RE: Machine Learning > > Sent: Mon=2C Aug 20=2C 2012 7:13:25 AM > > > > > > Hi Garot=2C > > > > This sounds an interesting idea. > > > > Are you looking to store all known commands and their options or are yo= u > > looking for a 'formula' for calling any unix command ? > > > > The reason for my question is that=2C at the end of the day=2C a unix c= ommand is > > just a program that is run in the operating system. Each program comes = with > > its own options and acceptable inputs. I don't know if there is a rule = or > > convention for structuring these commands. > > > > Are you then looking to build a system that 'knows' all commands and 'h= ow > > to' call them ? > > > > Thanks=2C > > Justin > > > > -----Original Message----- > > From: Garot Conklin [mailto:garotconklin@stripped] > > Sent: 20 August 2012 03:39 > > To: Martin Gainty=3B mysql@stripped > > Subject: Re: Machine Learning > > > > The initial goal is to provide a working framework from which to call a= ll > > UNIX shell command combinations as the underlying storage mechanism for= a > > machine learning algorithm. I would like to build a completely self aw= are > > instantiation that will maintain itself on all levels... I postulate th= at > > the first place to start would be in determining a method for maintaini= ng > > all possible remediation combinations including the unknown to eventual= ly be > > learned from and populate new knowledge into the database. Thank you f= or > > the reply=2C > > > > Garot >=20 > --=20 > MySQL General Mailing List > For list archives: http://lists.mysql.com/mysql > To unsubscribe: http://lists.mysql.com/mysql >=20 = --_215c7e8d-89a6-4943-93c3-3cb2c7b203a2_--