From: Garot Conklin Date: August 23 2012 7:14pm Subject: Re: Machine Learning List-Archive: http://lists.mysql.com/mysql/228065 Message-Id: <1345749250.10459.YahooMailNeo@web140502.mail.bf1.yahoo.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="1366632884-255714346-1345749250=:10459" --1366632884-255714346-1345749250=:10459 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable That is in fact a very interesting approach... I like it very much.=C2=A0 A= ny architectural thoughts on an auto-population attempt? Odly I have not cr= eated a db like this before, not from this large of an import anyway... Ple= ase share some thoughts there and I will get that rolling... seems that if = we can get something physical going I would like to offer it up as a sandbo= x to all those interested.=C2=A0 I have a few X336's lying around my house = just collecting dust, they would be a great testing environment for all to = work on.=0A=0A=0A=C2=A0=0Agarotconklin@stripped=0A=0A=0A__________________= ______________=0A From: Martin Gainty =0ATo: ag4ve.us@= gmail.com; garotconklin@stripped =0ACc: webmaster@stripped; "mysql@list= s.mysql.com" =0ASent: Thursday, August 23, 2012 11:= 30 AM=0ASubject: RE: Machine Learning=0A =0A=0AShawn and Garot=0A=0AI like = the parsing capability of the lucene and its ability to stem incoming queri= es..=0AIf you are able to start populating your data then we *should* be ab= le to start identifying which root terms we can pull for building lucene-in= dexes=0AI have an upcoming stuck-in-JFK-airport-on-labor-day-waiting-for-th= e-next-am-flight-of-USscare for an interrupted timeframe(i never learned ho= w to 'sleep' in airport terminal)=0Aas long as I can find a wifi connection= (and an AC connection within 6 hours) and use a working MySQL JDBC connect= ion string..I should be able to bang out a quick prototype..=0A=0ALet me kn= ow how i can help out,=0AMartin Gainty =0A_________________________________= _____________ =0A..place long-winded disclaimer here..=0A=0A=0A> From: ag4v= e.us@stripped=0A> Date: Thu, 23 Aug 2012 10:42:55 -0400=0A> Subject: Re: M= achine Learning=0A> To: garotconklin@stripped=0A> CC: webmaster@stripped= k; mgainty@stripped; mysql@stripped=0A> =0A> just a few thoughts = (things to look into)=0A> =0A> if you want to populate a db with command pa= rameters, i'd mine the man=0A> pages. it's a consistent format and you shou= ld be able to find a=0A> parser for whatever language you prefer.=0A> =0A> = if you want to see what has been entered and statistics about that=0A> proc= ess, look at the audit framework. it won't tell you success or=0A> failure,= but you might be able to derive that based on the time,=0A> memory, and pr= ocess usage that you can get from audit. auditd also=0A> maintains a db - y= ou might look into the engine and schema they use.=0A> =0A> you might also = find some interesting topics looking into language=0A> processing (after al= l, commands are a type of language). people=0A> generally use other databas= e engines to do this type of thing -=0A> elastic search or some other thing= s people have done with lucene=0A> maybe.=0A> =0A> do let us know where you= go with this. it sounds quite interesting=0A> (and might help me with some= thing i've been looking into)... and=0A> getting zsh to autocomplete option= s might be a bonus if you parse man=0A> pages and shove it into a db (i mig= ht do that part for you if i get a=0A> lazy afternoon).=0A> =0A> On Wed, Au= g 22, 2012 at 7:51 PM, Garot Conklin wrote:=0A> > = I am going to work up a visual diagram for this to better... visualize (LOL= ) the lay of the land. I will forward it when i have it completed then I ho= pe we can continue the architectural discussion here!=C2=A0 Thanks=0A> >=0A= > > -Garot=0A> >=0A> >=0A> > garotconklin@stripped=0A> >=0A> >=0A> > _____= ___________________________=0A> >=C2=A0 From: Garot Conklin =0A> > To: Garot Conklin ; "webmaster@liso= l.co.uk" ; 'Martin Gainty' ; "m= ysql@stripped" =0A> > Sent: Tuesday, August 2= 1, 2012 4:40 PM=0A> > Subject: Re: Machine Learning=0A> >=0A> > to elaborat= e:=0A> >=0A> > I would almost rather qualify from the OS perspective what w= ill work in advance, but take into consideration any failing condition.=C2= =A0 I may be getting a bit too specific at this juncture however as I am al= ready apply this logic to issues I see in MY environment rather than total = ambiguity, which is a best position here.=0A> >=0A> > I am ultimately expec= ting the learning algorithm to create new and excitingly effective ways of = stringing commands together to solve issues, this being a notable aspect in= and of itself, then creating some centralization based on what the machine= determines as a best solution.=0A> >=0A> > Seems a relative "snapshot of t= he system" pre and post execution would be favorable as well so long as res= ources are not taxed as a result.=C2=A0 This discussion is leading me to be= lieve that this will be a multifaceted db solution without a doubt comprise= d of multiple layers of abstraction ultimately rolling up into a single mas= ter cluster for all the higher level analytics and internal testing cycles = to then roll out the new primary protocols, which may themselves take place= in a sub-instance of each possible scenario.=0A> >=0A> > I have also thoug= h of populating a db or multiple db's with actual text=C2=A0 in the form of= .pdf's of actual technical manuals... not certain the benefit of the writt= en language abstraction there but the thought process extraction might be a= n interesting end indeed.=0A> >=0A> >=0A> >=0A> > garotconklin@stripped=0A= > >=0A> >=0A> > ________________________________=0A> > From: Garot Conklin = =0A> > To: "webmaster@stripped" ; 'Martin Gainty' ; "mysql@stripped" =0A> > Sent: Tuesday, August 21, 2012 4:09 PM=0A> > Sub= ject: Re: Machine Learning=0A> >=0A> > YES!=0A> >=0A> >=0A> > garotconklin@= yahoo.com=0A> >=0A> >=0A> > ________________________________=0A> > From: "w= ebmaster@stripped" =0A> > To: 'Garot Conklin' ; 'Martin Gainty' ; mysql@stripped= ysql.com=0A> > Sent: Tuesday, August 21, 2012 3:37 PM=0A> > Subject: RE: Ma= chine Learning=0A> >=0A> >=0A> > Ah,=0A> >=0A> > Getting clearer and cleare= r.=0A> >=0A> > So these =E2=80=98nodes=E2=80=99 could =E2=80=98learn=E2=80= =99 and =E2=80=98teach=E2=80=99 at the same time =E2=80=93 right ? For inst= ance, N1 runs a command in =E2=80=98domain=E2=80=99 D20 which it successful= =E2=80=93 it send information to node N20 that is the authority on domain = D20 and N20 records it as success; N5 runs a command in domain D20 which go= es wrong, and sends info to N20 and N20 records failure and sends a correct= call to N5. Is this what you have in mind ?=0A> >=0A> > There are a lot of= variables to be considered, for instance:=0A> > 1)=C2=A0 =C2=A0 =C2=A0 Com= mand A version 1 can run very happily on operating system version 2, but fa= ils in OS version 1=0A> > 2)=C2=A0 =C2=A0 =C2=A0 Command A version 1 can ru= n happily on OS version 2 on a 2G RAM, but fail on OS version 2 on 1G RAM= =0A> > 3)=C2=A0 =C2=A0 =C2=A0 etc=0A> >=0A> > I think the DB design issues = will become straightforward once the model is quite clear.=0A> >=0A> > Just= in=0A> >=0A> > From:Garot Conklin [mailto:garotconklin@stripped]=0A> > Sen= t: 21 August 2012 17:14=0A> > To: webmaster@stripped; 'Martin Gainty'; m= ysql@stripped=0A> > Subject: Re: Machine Learning=0A> >=0A> > 1)Refe= r to it for =E2=80=98knowledge=E2=80=99 (for instance, of what the latest= =0A> > version of a command is)=0A> >=C2=A0 =C2=A0 It would most likely en= d up being "central" in this sense:=0A> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 A dis= tributed collection of systems; i.e. (possible defined in roles)=0A> >=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 DB's=0A> >=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 FE's=0A> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 REP= L's=0A> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Cache's=0A> >=C2=A0 =C2= =A0 =C2=A0 =C2=A0 Each DB would have its own collection of remediation's t= hat would then be indexed to populate a central db for trending/correlation= etc...=0A> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 The "Collective" itself would fun= ction as a single conceptual implementation. A VIP for example might be ass= ociated with a specific role, say Web FE's and remediate only/all of them, = but only have some relative access to the core db of say the Network from a= Primary index perspective to make/draw associations/conclusions to issues = at hand.=0A> >=0A> >=0A> > 2)Send their =E2=80=98knowledge=E2=80=99 (for in= stance, of the latest command=0A> > versions) to it for storage and distrib= ution to others=0A> >=C2=A0 =C2=A0 Rather than "latest command versions" I = envision this to be more encapsulated as "latest successful invocation of t= he command string" and it inverse as well (to trend/metric-ize the failures= thus lending to perpetual optimization).=0A> >=0A> > I like how this is fl= eshing out... This is helping me to define what I am really trying to accom= plish. Thanks very much for everyone responding here, this is wonderful, pl= ease keep this going...=0A> >=0A> > garotconklin@stripped=0A> >=0A> > ____= ____________________________=0A> >=0A> > From:"webmaster@stripped" =0A> > To: 'Martin Gainty' ; garotco= nklin@stripped; mysql@stripped=0A> > Sent: Tuesday, August 21, 2012= 11:19 AM=0A> > Subject: RE: Machine Learning=0A> >=0A> > Hi Garot,=0A> >= =0A> >=0A> >=0A> > Ok, the concept is getting clearer, but let=E2=80=99s br= ing this down to earth a=0A> > little bit more. I love DB design and proble= m-solving and am quite curious=0A> > about this.=0A> >=0A> >=0A> >=0A> > Is= the idea that you have a central computer (not HAL J) somewhere so that=0A= > > other computers can:=0A> >=0A> > 1)=C2=A0 =C2=A0 =C2=A0 Refer to it for= =E2=80=98knowledge=E2=80=99 (for instance, of what the latest=0A> > versio= n of a command is)=0A> >=0A> > 2)=C2=A0 =C2=A0 =C2=A0 Send their =E2=80=98k= nowledge=E2=80=99 (for instance, of the latest command=0A> > versions) to i= t for storage and distribution to others=0A> >=0A> >=0A> >=0A> > If this is= the model, then the knowledge base can build up organically over=0A> > tim= e =E2=80=93 I think. Or is this too simplistic ?=0A> >=0A> >=0A> >=0A> > Th= anks,=0A> >=0A> > Justin=0A> >=0A> >=0A> >=0A> > From: Martin Gainty [mailt= o:mgainty@stripped]=0A> > Sent: 21 August 2012 00:25=0A> > To: garotconk= lin@stripped; webmaster@stripped; mysql@stripped=0A> > Subject: = RE: Machine Learning=0A> >=0A> >=0A> >=0A> > When I hear 'AI' I always imag= ine theres a HAL 9001 behind the scenes that=0A> > is running the show cons= tantly admonishing its creator to "take another=0A> > stress pill"=0A> >=0A= > > Sounds like a fun project=0A> >=0A> > Keep us apprised,=0A> > Martin Ga= inty=0A> > ______________________________________________=0A> > Verzicht un= d Vertraulichkeitanmerkung/Note de d=C3=A9ni et de confidentialit=C3=A9=0A>= >=0A> >=0A> > Diese Nachricht ist vertraulich. Sollten Sie nicht der vorge= sehene=0A> > Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Je= de unbefugte=0A> > Weiterleitung oder Fertigung einer Kopie ist unzulaessig= . Diese Nachricht=0A> > dient lediglich dem Austausch von Informationen und= entfaltet keine=0A> > rechtliche Bindungswirkung. Aufgrund der leichten Ma= nipulierbarkeit von=0A> > E-Mails koennen wir keine Haftung fuer den Inhalt= uebernehmen.=0A> >=0A> > Ce message est confidentiel et peut =C3=AAtre pri= vil=C3=A9gi=C3=A9. Si vous n'=C3=AAtes pas le=0A> > destinataire pr=C3=A9vu= , nous te demandons avec bont=C3=A9 que pour satisfaire=0A> > informez l'ex= p=C3=A9diteur. N'importe quelle diffusion non autoris=C3=A9e ou la copie=0A= > > de ceci est interdite. Ce message sert =C3=A0 l'information seulement e= t n'aura=0A> > pas n'importe quel effet l=C3=A9galement obligatoire. =C3=89= tant donn=C3=A9 que les email=0A> > peuvent facilement =C3=AAtre sujets =C3= =A0 la manipulation, nous ne pouvons accepter=0A> > aucune responsabilit=C3= =A9 pour le contenu fourni.=0A> >=0A> >=0A> >=0A> >=0A> >=0A> >=C2=A0 ____= _=0A> >=0A> > Date: Mon, 20 Aug 2012 13:50:04 -0700=0A> > From: garotconkli= n@stripped=0A> > Subject: Re: Machine Learning=0A> > To: webmaster@stripped= o.uk; mgainty@stripped; mysql@stripped=0A> >=0A> > Ya the idea is= not anything new, but must be apparently quit difficult or=0A> > not a pri= ority as I have yet to find it already implemented anywhere... Far=0A> > be= it from me to not make some attempt here anyway...=0A> >=0A> >=0A> >=0A> >= I am creating a fully automated framework from which a distributed=0A> > i= nfrastructure can be maintained.=C2=A0 I have been writing automation=0A> >= scripts/code for some time now and the logical progression is to embark on= a=0A> > full concept of systems health auto remediation.=C2=A0 I have nume= rous=0A> > "monitoring" solutions under my control however none that proper= ly (in my=0A> > opinion) implements any real learning algorithms from which= to draw even a=0A> > minimalists view of automation.=C2=A0 I like mySQL th= erefor began thinking about=0A> > creating the aspects (lobes) of the "brai= n" as a relational database(s).=C2=A0 So=0A> > this is only one facet of wh= at I am trying to do, however leveraging a full=0A> > command set of shell = utilities/commands/programs seemed to be a good=0A> > starting point before= I get into the "hard" stuff !=0A> >=0A> >=0A> >=0A> > -Garot=0A> >=0A> >= =0A> >=0A> > garotconklin@stripped=0A> >=0A> >=C2=A0 _____=0A> >=0A> > Fr= om: "webmaster@stripped" =0A> > To: 'Martin Gaint= y' ; garotconklin@stripped;=0A> > mysql@stripped= om=0A> > Sent: Monday, August 20, 2012 3:55 PM=0A> > Subject: RE: Machine L= earning=0A> >=0A> >=0A> > Hi Garot,=0A> >=0A> >=0A> >=0A> > You'll have to = elaborate some more ... I understand you may want to protect=0A> > the idea= as well, so if you can narrow it down to some technical specifics=0A> > th= en it'll help.=0A> >=0A> >=0A> >=0A> > What is the objective of this system= , for instance ?=0A> >=0A> >=0A> >=0A> > Thanks,=0A> >=0A> > Justin=0A> >= =0A> >=0A> >=0A> > From: Martin Gainty [mailto:mgainty@stripped]=0A> > S= ent: 20 August 2012 19:23=0A> > To: garotconklin@stripped; webmaster@lisol= .co.uk; mysql@stripped=0A> > Subject: RE: Machine Learning=0A> >=0A>= >=0A> >=0A> >=0A> >=0A> > From: garotconklin@stripped=0A> > Subject: Re: = Machine Learning=0A> > To: webmaster@stripped; mgainty@stripped; mysq= l@stripped=0A> >=0A> >=0A> > My initial thought was to propagate the= db with everything and allow the=0A> > algorithm to then begin to determin= trends/patterns=0A> > MG>which trends or patterns will you be modelling?= =0A> >=0A> > and begin either an indexing methodology=0A> > MG>which indexe= s are you considering: Unique index, primary index or foreign=0A> > index?= =0A> >=0A> > additional table/db creation process or both to further optimi= ze the calls=0A> > being made=0A> > MG>optimize based on execution time or = diskspace allocated, EliminatingFTS=0A> > or some other criteria?=0A> > MG>= https://dev.mysql.com/doc/refman/5.5/en/optimization.html=0A> >=0A> > and b= uild in some internal levels of redundancy.=0A> > MG>what about replication= =0A> > MG>http://dev.mysql.com/doc/refman/5.5/en/replication.html=0A> >=0A>= > I am actually approaching this with some degree of biological conception= in=0A> > the multipathing within our own brains however until I have somet= hing up and=0A> > running under some substantial load however I may not get= a complete=0A> > picture.=0A> >=0A> > Thanks,=0A> >=0A> > Garot=0A> >=0A> = >=0A> > Interesting=0A> > Martin=0A> >=0A> >=C2=A0 _____=0A> >=0A> > From:= webmaster@stripped ;=0A> > To: 'Garot Conklin' <= garotconklin@stripped>; 'Martin Gainty'=0A> > ; ;=0A> > Subject: RE: Machine Learning=0A> > Sent: Mon, Au= g 20, 2012 7:13:25 AM=0A> >=0A> >=0A> > Hi Garot,=0A> >=0A> > This sounds a= n interesting idea.=0A> >=0A> > Are you looking to store all known commands= and their options or are you=0A> > looking for a 'formula' for calling any= unix command ?=0A> >=0A> > The reason for my question is that, at the end = of the day, a unix command is=0A> > just a program that is run in the opera= ting system. Each program comes with=0A> > its own options and acceptable i= nputs. I don't know if there is a rule or=0A> > convention for structuring = these commands.=0A> >=0A> > Are you then looking to build a system that 'kn= ows' all commands and 'how=0A> > to' call them ?=0A> >=0A> > Thanks,=0A> > = Justin=0A> >=0A> > -----Original Message-----=0A> > From: Garot Conklin [ma= ilto:garotconklin@stripped]=0A> > Sent: 20 August 2012 03:39=0A> > To: Mar= tin Gainty; mysql@stripped=0A> > Subject: Re: Machine Learning=0A> >= =0A> > The initial goal is to provide a working framework from which to cal= l all=0A> > UNIX shell command combinations as the underlying storage mecha= nism for a=0A> > machine learning algorithm.=C2=A0 I would like to build a = completely self aware=0A> > instantiation that will maintain itself on all = levels... I postulate that=0A> > the first place to start would be in deter= mining a method for maintaining=0A> > all possible remediation combinations= including the unknown to eventually be=0A> > learned from and populate new= knowledge into the database.=C2=A0 Thank you for=0A> > the reply,=0A> >=0A= > > Garot=0A> =0A> -- =0A> MySQL General Mailing List=0A> For list archives= : http://lists.mysql.com/mysql=0A> To unsubscribe:=C2=A0 =C2=A0 http://list= s.mysql.com/mysql=0A> --1366632884-255714346-1345749250=:10459--