From: Martin Gainty Date: August 24 2012 1:29pm Subject: RE: Machine Learning List-Archive: http://lists.mysql.com/mysql/228077 Message-Id: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="_e99ac27a-7801-41f6-a154-c8748474fd35_" --_e99ac27a-7801-41f6-a154-c8748474fd35_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable we'll need a user-friendly architecture diagram with a hopefully one to one= mapping to DBSchema Table Elements..(run the napkin thru a scanner and pub= lish it) once the Table Elements are defined then we'll need to populate Tables with= columns..determine Primary Keys with Foreign keys..get the relationships d= efined and written down then create the database ..once the DB is up and running in an accesible lo= cation with sftp or scp running so we can import.. publish the JDBC connect= ion string Martin=20 ______________________________________________=20 ..place long winded disclaimer here.. > From: webmaster@stripped > To: garotconklin@stripped=3B mgainty@stripped=3B ag4ve.us@stripped > CC: mysql@stripped > Subject: RE: Machine Learning > Date: Fri=2C 24 Aug 2012 09:04:21 +0100 >=20 > Hi Garot=2C >=20 > How is the visual diagram coming along ? >=20 > I'd be good to have a good idea of core system specifics so that DB desig= n can be tailored to that ... of course=2C knowing that things will change = as the concept evolves. >=20 > Thanks=2C > Justin >=20 > -----Original Message----- > From: Garot Conklin [mailto:garotconklin@stripped]=20 > Sent: 23 August 2012 20:14 > To: Martin Gainty=3B ag4ve.us@stripped > Cc: webmaster@stripped=3B mysql@stripped > Subject: Re: Machine Learning >=20 > That is in fact a very interesting approach... I like it very much. Any = architectural thoughts on an auto-population attempt? Odly I have not creat= ed a db like this before=2C not from this large of an import anyway... Plea= se share some thoughts there and I will get that rolling... seems that if w= e can get something physical going I would like to offer it up as a sandbox= to all those interested. I have a few X336's lying around my house just c= ollecting dust=2C they would be a great testing environment for all to work= on. >=20 >=20 > =20 > garotconklin@stripped >=20 >=20 > ________________________________ > From: Martin Gainty > To: ag4ve.us@stripped=3B garotconklin@stripped > Cc: webmaster@stripped=3B "mysql@stripped" > Sent: Thursday=2C August 23=2C 2012 11:30 AM > Subject: RE: Machine Learning > =20 >=20 > Shawn and Garot >=20 > I like the parsing capability of the lucene and its ability to stem incom= ing queries.. > If you are able to start populating your data then we *should* be able to= start identifying which root terms we can pull for building lucene-indexes= I have an upcoming stuck-in-JFK-airport-on-labor-day-waiting-for-the-next-= am-flight-of-USscare for an interrupted timeframe(i never learned how to 's= leep' in airport terminal) as long as I can find a wifi connection (and an = AC connection within 6 hours) and use a working MySQL JDBC connection strin= g..I should be able to bang out a quick prototype.. >=20 > Let me know how i can help out=2C > Martin Gainty > ______________________________________________ > ..place long-winded disclaimer here.. >=20 >=20 > > From: ag4ve.us@stripped > > Date: Thu=2C 23 Aug 2012 10:42:55 -0400 > > Subject: Re: Machine Learning > > To: garotconklin@stripped > > CC: webmaster@stripped=3B mgainty@stripped=3B mysql@stripped= om > >=20 > > just a few thoughts (things to look into) > >=20 > > if you want to populate a db with command parameters=2C i'd mine the ma= n=20 > > pages. it's a consistent format and you should be able to find a=20 > > parser for whatever language you prefer. > >=20 > > if you want to see what has been entered and statistics about that=20 > > process=2C look at the audit framework. it won't tell you success or=20 > > failure=2C but you might be able to derive that based on the time=2C=20 > > memory=2C and process usage that you can get from audit. auditd also=20 > > maintains a db - you might look into the engine and schema they use. > >=20 > > you might also find some interesting topics looking into language=20 > > processing (after all=2C commands are a type of language). people=20 > > generally use other database engines to do this type of thing -=20 > > elastic search or some other things people have done with lucene=20 > > maybe. > >=20 > > do let us know where you go with this. it sounds quite interesting=20 > > (and might help me with something i've been looking into)... and=20 > > getting zsh to autocomplete options might be a bonus if you parse man=20 > > pages and shove it into a db (i might do that part for you if i get a=20 > > lazy afternoon). > >=20 > > On Wed=2C Aug 22=2C 2012 at 7:51 PM=2C Garot Conklin wrote: > > > I am going to work up a visual diagram for this to better...=20 > > > visualize (LOL) the lay of the land. I will forward it when i have=20 > > > it completed then I hope we can continue the architectural=20 > > > discussion here! Thanks > > > > > > -Garot > > > > > > > > > garotconklin@stripped > > > > > > > > > ________________________________ > > > From: Garot Conklin > > > To: Garot Conklin =3B "webmaster@stripped"= =20 > > >=3B 'Martin Gainty' =3B=20 > > >"mysql@stripped" > > > Sent: Tuesday=2C August 21=2C 2012 4:40 PM > > > Subject: Re: Machine Learning > > > > > > to elaborate: > > > > > > I would almost rather qualify from the OS perspective what will work = in advance=2C but take into consideration any failing condition. I may be = getting a bit too specific at this juncture however as I am already apply t= his logic to issues I see in MY environment rather than total ambiguity=2C = which is a best position here. > > > > > > I am ultimately expecting the learning algorithm to create new and ex= citingly effective ways of stringing commands together to solve issues=2C t= his being a notable aspect in and of itself=2C then creating some centraliz= ation based on what the machine determines as a best solution. > > > > > > Seems a relative "snapshot of the system" pre and post execution woul= d be favorable as well so long as resources are not taxed as a result. Thi= s discussion is leading me to believe that this will be a multifaceted db s= olution without a doubt comprised of multiple layers of abstraction ultimat= ely rolling up into a single master cluster for all the higher level analyt= ics and internal testing cycles to then roll out the new primary protocols= =2C which may themselves take place in a sub-instance of each possible scen= ario. > > > > > > I have also though of populating a db or multiple db's with actual te= xt in the form of .pdf's of actual technical manuals... not certain the be= nefit of the written language abstraction there but the thought process ext= raction might be an interesting end indeed. > > > > > > > > > > > > garotconklin@stripped > > > > > > > > > ________________________________ > > > From: Garot Conklin > > > To: "webmaster@stripped" =3B 'Martin Gainty= '=20 > > > =3B "mysql@stripped"=20 > > > > > > Sent: Tuesday=2C August 21=2C 2012 4:09 PM > > > Subject: Re: Machine Learning > > > > > > YES! > > > > > > > > > garotconklin@stripped > > > > > > > > > ________________________________ > > > From: "webmaster@stripped" > > > To: 'Garot Conklin' =3B 'Martin Gainty'=20 > > > =3B mysql@stripped > > > Sent: Tuesday=2C August 21=2C 2012 3:37 PM > > > Subject: RE: Machine Learning > > > > > > > > > Ah=2C > > > > > > Getting clearer and clearer. > > > > > > So these =91nodes=92 could =91learn=92 and =91teach=92 at the same ti= me =96 right ? For instance=2C N1 runs a command in =91domain=92 D20 which = it successful =96 it send information to node N20 that is the authority on = domain D20 and N20 records it as success=3B N5 runs a command in domain D20= which goes wrong=2C and sends info to N20 and N20 records failure and send= s a correct call to N5. Is this what you have in mind ? > > > > > > There are a lot of variables to be considered=2C for instance: > > > 1) Command A version 1 can run very happily on operating system= =20 > > > version 2=2C but fails in OS version 1 > > > 2) Command A version 1 can run happily on OS version 2 on a 2G=20 > > > RAM=2C but fail on OS version 2 on 1G RAM > > > 3) etc > > > > > > I think the DB design issues will become straightforward once the mod= el is quite clear. > > > > > > Justin > > > > > > From:Garot Conklin [mailto:garotconklin@stripped] > > > Sent: 21 August 2012 17:14 > > > To: webmaster@stripped=3B 'Martin Gainty'=3B mysql@stripped > > > Subject: Re: Machine Learning > > > > > > 1)Refer to it for =91knowledge=92 (for instance=2C of what the latest= =20 > > >version of a command is) > > > It would most likely end up being "central" in this sense: > > > A distributed collection of systems=3B i.e. (possible defined= =20 > > >in roles) > > > DB's > > > FE's > > > REPL's > > > Cache's > > > Each DB would have its own collection of remediation's that w= ould then be indexed to populate a central db for trending/correlation etc.= .. > > > The "Collective" itself would function as a single conceptual= implementation. A VIP for example might be associated with a specific role= =2C say Web FE's and remediate only/all of them=2C but only have some relat= ive access to the core db of say the Network from a Primary index perspecti= ve to make/draw associations/conclusions to issues at hand. > > > > > > > > > 2)Send their =91knowledge=92 (for instance=2C of the latest command > > > versions) to it for storage and distribution to others > > > Rather than "latest command versions" I envision this to be more e= ncapsulated as "latest successful invocation of the command string" and it = inverse as well (to trend/metric-ize the failures thus lending to perpetual= optimization). > > > > > > I like how this is fleshing out... This is helping me to define what = I am really trying to accomplish. Thanks very much for everyone responding = here=2C this is wonderful=2C please keep this going... > > > > > > garotconklin@stripped > > > > > > ________________________________ > > > > > > From:"webmaster@stripped" > > > To: 'Martin Gainty' =3B garotconklin@stripped= =3B=20 > > > mysql@stripped > > > Sent: Tuesday=2C August 21=2C 2012 11:19 AM > > > Subject: RE: Machine Learning > > > > > > Hi Garot=2C > > > > > > > > > > > > Ok=2C the concept is getting clearer=2C but let=92s bring this down t= o=20 > > > earth a little bit more. I love DB design and problem-solving and am= =20 > > > quite curious about this. > > > > > > > > > > > > Is the idea that you have a central computer (not HAL J) somewhere=20 > > > so that other computers can: > > > > > > 1) Refer to it for =91knowledge=92 (for instance=2C of what the= =20 > > > latest version of a command is) > > > > > > 2) Send their =91knowledge=92 (for instance=2C of the latest com= mand > > > versions) to it for storage and distribution to others > > > > > > > > > > > > If this is the model=2C then the knowledge base can build up=20 > > > organically over time =96 I think. Or is this too simplistic ? > > > > > > > > > > > > Thanks=2C > > > > > > Justin > > > > > > > > > > > > From: Martin Gainty [mailto:mgainty@stripped] > > > Sent: 21 August 2012 00:25 > > > To: garotconklin@stripped=3B webmaster@stripped=3B=20 > > > mysql@stripped > > > Subject: RE: Machine Learning > > > > > > > > > > > > When I hear 'AI' I always imagine theres a HAL 9001 behind the=20 > > > scenes that is running the show constantly admonishing its creator=20 > > > to "take another stress pill" > > > > > > Sounds like a fun project > > > > > > Keep us apprised=2C > > > Martin Gainty > > > ______________________________________________ > > > Verzicht und Vertraulichkeitanmerkung/Note de d=E9ni et de=20 > > > confidentialit=E9 > > > > > > > > > Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene=20 > > > Empfaenger sein=2C so bitten wir hoeflich um eine Mitteilung. Jede=20 > > > unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig.=20 > > > Diese Nachricht dient lediglich dem Austausch von Informationen und=20 > > > entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten=20 > > > Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inha= lt uebernehmen. > > > > > > Ce message est confidentiel et peut =EAtre privil=E9gi=E9. Si vous n'= =EAtes=20 > > > pas le destinataire pr=E9vu=2C nous te demandons avec bont=E9 que pou= r=20 > > > satisfaire informez l'exp=E9diteur. N'importe quelle diffusion non=20 > > > autoris=E9e ou la copie de ceci est interdite. Ce message sert =E0=20 > > > l'information seulement et n'aura pas n'importe quel effet=20 > > > l=E9galement obligatoire. =C9tant donn=E9 que les email peuvent facil= ement=20 > > > =EAtre sujets =E0 la manipulation=2C nous ne pouvons accepter aucune = responsabilit=E9 pour le contenu fourni. > > > > > > > > > > > > > > > > > > _____ > > > > > > Date: Mon=2C 20 Aug 2012 13:50:04 -0700 > > > From: garotconklin@stripped > > > Subject: Re: Machine Learning > > > To: webmaster@stripped=3B mgainty@stripped=3B=20 > > > mysql@stripped > > > > > > Ya the idea is not anything new=2C but must be apparently quit=20 > > > difficult or not a priority as I have yet to find it already=20 > > > implemented anywhere... Far be it from me to not make some attempt he= re anyway... > > > > > > > > > > > > I am creating a fully automated framework from which a distributed=20 > > > infrastructure can be maintained. I have been writing automation=20 > > > scripts/code for some time now and the logical progression is to=20 > > > embark on a full concept of systems health auto remediation. I have= =20 > > > numerous "monitoring" solutions under my control however none that=20 > > > properly (in my > > > opinion) implements any real learning algorithms from which to draw=20 > > > even a minimalists view of automation. I like mySQL therefor began=20 > > > thinking about creating the aspects (lobes) of the "brain" as a=20 > > > relational database(s). So this is only one facet of what I am=20 > > > trying to do=2C however leveraging a full command set of shell=20 > > > utilities/commands/programs seemed to be a good starting point before= I get into the "hard" stuff ! > > > > > > > > > > > > -Garot > > > > > > > > > > > > garotconklin@stripped > > > > > > _____ > > > > > > From: "webmaster@stripped" > > > To: 'Martin Gainty' =3B garotconklin@stripped= =3B=20 > > > mysql@stripped > > > Sent: Monday=2C August 20=2C 2012 3:55 PM > > > Subject: RE: Machine Learning > > > > > > > > > Hi Garot=2C > > > > > > > > > > > > You'll have to elaborate some more ... I understand you may want to=20 > > > protect the idea as well=2C so if you can narrow it down to some=20 > > > technical specifics then it'll help. > > > > > > > > > > > > What is the objective of this system=2C for instance ? > > > > > > > > > > > > Thanks=2C > > > > > > Justin > > > > > > > > > > > > From: Martin Gainty [mailto:mgainty@stripped] > > > Sent: 20 August 2012 19:23 > > > To: garotconklin@stripped=3B webmaster@stripped=3B=20 > > > mysql@stripped > > > Subject: RE: Machine Learning > > > > > > > > > > > > > > > > > > From: garotconklin@stripped > > > Subject: Re: Machine Learning > > > To: webmaster@stripped=3B mgainty@stripped=3B=20 > > > mysql@stripped > > > > > > > > > My initial thought was to propagate the db with everything and allow= =20 > > > the algorithm to then begin to determin trends/patterns > > > MG>which trends or patterns will you be modelling? > > > > > > and begin either an indexing methodology > > > MG>which indexes are you considering: Unique index=2C primary index o= r=20 > > > MG>foreign > > > index? > > > > > > additional table/db creation process or both to further optimize the= =20 > > > calls being made > > > MG>optimize based on execution time or diskspace allocated=2C=20 > > > MG>EliminatingFTS > > > or some other criteria? > > > MG>https://dev.mysql.com/doc/refman/5.5/en/optimization.html > > > > > > and build in some internal levels of redundancy. > > > MG>what about replication > > > MG>http://dev.mysql.com/doc/refman/5.5/en/replication.html > > > > > > I am actually approaching this with some degree of biological=20 > > > conception in the multipathing within our own brains however until I= =20 > > > have something up and running under some substantial load however I=20 > > > may not get a complete picture. > > > > > > Thanks=2C > > > > > > Garot > > > > > > > > > Interesting > > > Martin > > > > > > _____ > > > > > > From: webmaster@stripped =3B > > > To: 'Garot Conklin' =3B 'Martin Gainty' > > > =3B =3B > > > Subject: RE: Machine Learning > > > Sent: Mon=2C Aug 20=2C 2012 7:13:25 AM > > > > > > > > > Hi Garot=2C > > > > > > This sounds an interesting idea. > > > > > > Are you looking to store all known commands and their options or are= =20 > > > you looking for a 'formula' for calling any unix command ? > > > > > > The reason for my question is that=2C at the end of the day=2C a unix= =20 > > > command is just a program that is run in the operating system. Each=20 > > > program comes with its own options and acceptable inputs. I don't=20 > > > know if there is a rule or convention for structuring these commands. > > > > > > Are you then looking to build a system that 'knows' all commands and= =20 > > > 'how to' call them ? > > > > > > Thanks=2C > > > Justin > > > > > > -----Original Message----- > > > From: Garot Conklin [mailto:garotconklin@stripped] > > > Sent: 20 August 2012 03:39 > > > To: Martin Gainty=3B mysql@stripped > > > Subject: Re: Machine Learning > > > > > > The initial goal is to provide a working framework from which to=20 > > > call all UNIX shell command combinations as the underlying storage=20 > > > mechanism for a machine learning algorithm. I would like to build a= =20 > > > completely self aware instantiation that will maintain itself on all= =20 > > > levels... I postulate that the first place to start would be in=20 > > > determining a method for maintaining all possible remediation=20 > > > combinations including the unknown to eventually be learned from and= =20 > > > populate new knowledge into the database. Thank you for the reply=2C > > > > > > Garot > >=20 > > -- > > MySQL General Mailing List > > For list archives: http://lists.mysql.com/mysql To unsubscribe: =20 > > http://lists.mysql.com/mysql > >=20 >=20 >=20 > --=20 > MySQL General Mailing List > For list archives: http://lists.mysql.com/mysql > To unsubscribe: http://lists.mysql.com/mysql >=20 = --_e99ac27a-7801-41f6-a154-c8748474fd35_--