Hi Garot,
How is the visual diagram coming along ?
I'd be good to have a good idea of core system specifics so that DB design can be tailored
to that ... of course, knowing that things will change as the concept evolves.
Thanks,
Justin
-----Original Message-----
From: Garot Conklin [mailto:garotconklin@stripped]
Sent: 23 August 2012 20:14
To: Martin Gainty; ag4ve.us@stripped
Cc: webmaster@stripped; mysql@stripped
Subject: Re: Machine Learning
That is in fact a very interesting approach... I like it very much. Any architectural
thoughts on an auto-population attempt? Odly I have not created a db like this before,
not from this large of an import anyway... Please share some thoughts there and I will
get that rolling... seems that if we can get something physical going I would like to
offer it up as a sandbox to all those interested. I have a few X336's lying around my
house just collecting dust, they would be a great testing environment for all to work on.
garotconklin@stripped
________________________________
From: Martin Gainty <mgainty@stripped>
To: ag4ve.us@stripped; garotconklin@stripped
Cc: webmaster@stripped; "mysql@stripped" <mysql@stripped>
Sent: Thursday, August 23, 2012 11:30 AM
Subject: RE: Machine Learning
Shawn and Garot
I like the parsing capability of the lucene and its ability to stem incoming queries..
If you are able to start populating your data then we *should* be able to start
identifying which root terms we can pull for building lucene-indexes I have an upcoming
stuck-in-JFK-airport-on-labor-day-waiting-for-the-next-am-flight-of-USscare for an
interrupted timeframe(i never learned how to 'sleep' in airport terminal) as long as I
can find a wifi connection (and an AC connection within 6 hours) and use a working MySQL
JDBC connection string..I should be able to bang out a quick prototype..
Let me know how i can help out,
Martin Gainty
______________________________________________
..place long-winded disclaimer here..
> From: ag4ve.us@stripped
> Date: Thu, 23 Aug 2012 10:42:55 -0400
> Subject: Re: Machine Learning
> To: garotconklin@stripped
> CC: webmaster@stripped; mgainty@stripped; mysql@stripped
>
> just a few thoughts (things to look into)
>
> if you want to populate a db with command parameters, i'd mine the man
> pages. it's a consistent format and you should be able to find a
> parser for whatever language you prefer.
>
> if you want to see what has been entered and statistics about that
> process, look at the audit framework. it won't tell you success or
> failure, but you might be able to derive that based on the time,
> memory, and process usage that you can get from audit. auditd also
> maintains a db - you might look into the engine and schema they use.
>
> you might also find some interesting topics looking into language
> processing (after all, commands are a type of language). people
> generally use other database engines to do this type of thing -
> elastic search or some other things people have done with lucene
> maybe.
>
> do let us know where you go with this. it sounds quite interesting
> (and might help me with something i've been looking into)... and
> getting zsh to autocomplete options might be a bonus if you parse man
> pages and shove it into a db (i might do that part for you if i get a
> lazy afternoon).
>
> On Wed, Aug 22, 2012 at 7:51 PM, Garot Conklin <garotconklin@stripped> wrote:
> > I am going to work up a visual diagram for this to better...
> > visualize (LOL) the lay of the land. I will forward it when i have
> > it completed then I hope we can continue the architectural
> > discussion here! Thanks
> >
> > -Garot
> >
> >
> > garotconklin@stripped
> >
> >
> > ________________________________
> > From: Garot Conklin <garotconklin@stripped>
> > To: Garot Conklin <garotconklin@stripped>; "webmaster@stripped"
> ><webmaster@stripped>; 'Martin Gainty' <mgainty@stripped>;
> >"mysql@stripped" <mysql@stripped>
> > Sent: Tuesday, August 21, 2012 4:40 PM
> > Subject: Re: Machine Learning
> >
> > to elaborate:
> >
> > I would almost rather qualify from the OS perspective what will work in advance,
> but take into consideration any failing condition. I may be getting a bit too specific at
> this juncture however as I am already apply this logic to issues I see in MY environment
> rather than total ambiguity, which is a best position here.
> >
> > I am ultimately expecting the learning algorithm to create new and excitingly
> effective ways of stringing commands together to solve issues, this being a notable aspect
> in and of itself, then creating some centralization based on what the machine determines
> as a best solution.
> >
> > Seems a relative "snapshot of the system" pre and post execution would be
> favorable as well so long as resources are not taxed as a result. This discussion is
> leading me to believe that this will be a multifaceted db solution without a doubt
> comprised of multiple layers of abstraction ultimately rolling up into a single master
> cluster for all the higher level analytics and internal testing cycles to then roll out
> the new primary protocols, which may themselves take place in a sub-instance of each
> possible scenario.
> >
> > I have also though of populating a db or multiple db's with actual text in the
> form of .pdf's of actual technical manuals... not certain the benefit of the written
> language abstraction there but the thought process extraction might be an interesting end
> indeed.
> >
> >
> >
> > garotconklin@stripped
> >
> >
> > ________________________________
> > From: Garot Conklin <garotconklin@stripped>
> > To: "webmaster@stripped" <webmaster@stripped>; 'Martin Gainty'
> > <mgainty@stripped>; "mysql@stripped"
> > <mysql@stripped>
> > Sent: Tuesday, August 21, 2012 4:09 PM
> > Subject: Re: Machine Learning
> >
> > YES!
> >
> >
> > garotconklin@stripped
> >
> >
> > ________________________________
> > From: "webmaster@stripped" <webmaster@stripped>
> > To: 'Garot Conklin' <garotconklin@stripped>; 'Martin Gainty'
> > <mgainty@stripped>; mysql@stripped
> > Sent: Tuesday, August 21, 2012 3:37 PM
> > Subject: RE: Machine Learning
> >
> >
> > Ah,
> >
> > Getting clearer and clearer.
> >
> > So these ‘nodes’ could ‘learn’ and ‘teach’
> at the same time – right ? For instance, N1 runs a command in ‘domain’
> D20 which it successful – it send information to node N20 that is the authority on
> domain D20 and N20 records it as success; N5 runs a command in domain D20 which goes
> wrong, and sends info to N20 and N20 records failure and sends a correct call to N5. Is
> this what you have in mind ?
> >
> > There are a lot of variables to be considered, for instance:
> > 1) Command A version 1 can run very happily on operating system
> > version 2, but fails in OS version 1
> > 2) Command A version 1 can run happily on OS version 2 on a 2G
> > RAM, but fail on OS version 2 on 1G RAM
> > 3) etc
> >
> > I think the DB design issues will become straightforward once the model is quite
> clear.
> >
> > Justin
> >
> > From:Garot Conklin [mailto:garotconklin@stripped]
> > Sent: 21 August 2012 17:14
> > To: webmaster@stripped; 'Martin Gainty'; mysql@stripped
> > Subject: Re: Machine Learning
> >
> > 1)Refer to it for ‘knowledge’ (for instance, of what the latest
> >version of a command is)
> > It would most likely end up being "central" in this sense:
> > A distributed collection of systems; i.e. (possible defined
> >in roles)
> > DB's
> > FE's
> > REPL's
> > Cache's
> > Each DB would have its own collection of remediation's that would then
> be indexed to populate a central db for trending/correlation etc...
> > The "Collective" itself would function as a single conceptual
> implementation. A VIP for example might be associated with a specific role, say Web FE's
> and remediate only/all of them, but only have some relative access to the core db of say
> the Network from a Primary index perspective to make/draw associations/conclusions to
> issues at hand.
> >
> >
> > 2)Send their ‘knowledge’ (for instance, of the latest command
> > versions) to it for storage and distribution to others
> > Rather than "latest command versions" I envision this to be more encapsulated
> as "latest successful invocation of the command string" and it inverse as well (to
> trend/metric-ize the failures thus lending to perpetual optimization).
> >
> > I like how this is fleshing out... This is helping me to define what I am really
> trying to accomplish. Thanks very much for everyone responding here, this is wonderful,
> please keep this going...
> >
> > garotconklin@stripped
> >
> > ________________________________
> >
> > From:"webmaster@stripped" <webmaster@stripped>
> > To: 'Martin Gainty' <mgainty@stripped>; garotconklin@stripped;
> > mysql@stripped
> > Sent: Tuesday, August 21, 2012 11:19 AM
> > Subject: RE: Machine Learning
> >
> > Hi Garot,
> >
> >
> >
> > Ok, the concept is getting clearer, but let’s bring this down to
> > earth a little bit more. I love DB design and problem-solving and am
> > quite curious about this.
> >
> >
> >
> > Is the idea that you have a central computer (not HAL J) somewhere
> > so that other computers can:
> >
> > 1) Refer to it for ‘knowledge’ (for instance, of what the
> > latest version of a command is)
> >
> > 2) Send their ‘knowledge’ (for instance, of the latest command
> > versions) to it for storage and distribution to others
> >
> >
> >
> > If this is the model, then the knowledge base can build up
> > organically over time – I think. Or is this too simplistic ?
> >
> >
> >
> > Thanks,
> >
> > Justin
> >
> >
> >
> > From: Martin Gainty [mailto:mgainty@stripped]
> > Sent: 21 August 2012 00:25
> > To: garotconklin@stripped; webmaster@stripped;
> > mysql@stripped
> > Subject: RE: Machine Learning
> >
> >
> >
> > When I hear 'AI' I always imagine theres a HAL 9001 behind the
> > scenes that is running the show constantly admonishing its creator
> > to "take another stress pill"
> >
> > Sounds like a fun project
> >
> > Keep us apprised,
> > Martin Gainty
> > ______________________________________________
> > Verzicht und Vertraulichkeitanmerkung/Note de déni et de
> > confidentialité
> >
> >
> > Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene
> > Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede
> > unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig.
> > Diese Nachricht dient lediglich dem Austausch von Informationen und
> > entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten
> > Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt
> uebernehmen.
> >
> > Ce message est confidentiel et peut être privilégié. Si vous
> n'êtes
> > pas le destinataire prévu, nous te demandons avec bonté que pour
> > satisfaire informez l'expéditeur. N'importe quelle diffusion non
> > autorisée ou la copie de ceci est interdite. Ce message sert à
> > l'information seulement et n'aura pas n'importe quel effet
> > légalement obligatoire. Étant donné que les email peuvent
> facilement
> > être sujets à la manipulation, nous ne pouvons accepter aucune
> responsabilité pour le contenu fourni.
> >
> >
> >
> >
> >
> > _____
> >
> > Date: Mon, 20 Aug 2012 13:50:04 -0700
> > From: garotconklin@stripped
> > Subject: Re: Machine Learning
> > To: webmaster@stripped; mgainty@stripped;
> > mysql@stripped
> >
> > Ya the idea is not anything new, but must be apparently quit
> > difficult or not a priority as I have yet to find it already
> > implemented anywhere... Far be it from me to not make some attempt here
> anyway...
> >
> >
> >
> > I am creating a fully automated framework from which a distributed
> > infrastructure can be maintained. I have been writing automation
> > scripts/code for some time now and the logical progression is to
> > embark on a full concept of systems health auto remediation. I have
> > numerous "monitoring" solutions under my control however none that
> > properly (in my
> > opinion) implements any real learning algorithms from which to draw
> > even a minimalists view of automation. I like mySQL therefor began
> > thinking about creating the aspects (lobes) of the "brain" as a
> > relational database(s). So this is only one facet of what I am
> > trying to do, however leveraging a full command set of shell
> > utilities/commands/programs seemed to be a good starting point before I get into
> the "hard" stuff !
> >
> >
> >
> > -Garot
> >
> >
> >
> > garotconklin@stripped
> >
> > _____
> >
> > From: "webmaster@stripped" <webmaster@stripped>
> > To: 'Martin Gainty' <mgainty@stripped>; garotconklin@stripped;
> > mysql@stripped
> > Sent: Monday, August 20, 2012 3:55 PM
> > Subject: RE: Machine Learning
> >
> >
> > Hi Garot,
> >
> >
> >
> > You'll have to elaborate some more ... I understand you may want to
> > protect the idea as well, so if you can narrow it down to some
> > technical specifics then it'll help.
> >
> >
> >
> > What is the objective of this system, for instance ?
> >
> >
> >
> > Thanks,
> >
> > Justin
> >
> >
> >
> > From: Martin Gainty [mailto:mgainty@stripped]
> > Sent: 20 August 2012 19:23
> > To: garotconklin@stripped; webmaster@stripped;
> > mysql@stripped
> > Subject: RE: Machine Learning
> >
> >
> >
> >
> >
> > From: garotconklin@stripped
> > Subject: Re: Machine Learning
> > To: webmaster@stripped; mgainty@stripped;
> > mysql@stripped
> >
> >
> > My initial thought was to propagate the db with everything and allow
> > the algorithm to then begin to determin trends/patterns
> > MG>which trends or patterns will you be modelling?
> >
> > and begin either an indexing methodology
> > MG>which indexes are you considering: Unique index, primary index or
> > MG>foreign
> > index?
> >
> > additional table/db creation process or both to further optimize the
> > calls being made
> > MG>optimize based on execution time or diskspace allocated,
> > MG>EliminatingFTS
> > or some other criteria?
> > MG>https://dev.mysql.com/doc/refman/5.5/en/optimization.html
> >
> > and build in some internal levels of redundancy.
> > MG>what about replication
> > MG>http://dev.mysql.com/doc/refman/5.5/en/replication.html
> >
> > I am actually approaching this with some degree of biological
> > conception in the multipathing within our own brains however until I
> > have something up and running under some substantial load however I
> > may not get a complete picture.
> >
> > Thanks,
> >
> > Garot
> >
> >
> > Interesting
> > Martin
> >
> > _____
> >
> > From: webmaster@stripped <webmaster@stripped>;
> > To: 'Garot Conklin' <garotconklin@stripped>; 'Martin Gainty'
> > <mgainty@stripped>; <mysql@stripped>;
> > Subject: RE: Machine Learning
> > Sent: Mon, Aug 20, 2012 7:13:25 AM
> >
> >
> > Hi Garot,
> >
> > This sounds an interesting idea.
> >
> > Are you looking to store all known commands and their options or are
> > you looking for a 'formula' for calling any unix command ?
> >
> > The reason for my question is that, at the end of the day, a unix
> > command is just a program that is run in the operating system. Each
> > program comes with its own options and acceptable inputs. I don't
> > know if there is a rule or convention for structuring these commands.
> >
> > Are you then looking to build a system that 'knows' all commands and
> > 'how to' call them ?
> >
> > Thanks,
> > Justin
> >
> > -----Original Message-----
> > From: Garot Conklin [mailto:garotconklin@stripped]
> > Sent: 20 August 2012 03:39
> > To: Martin Gainty; mysql@stripped
> > Subject: Re: Machine Learning
> >
> > The initial goal is to provide a working framework from which to
> > call all UNIX shell command combinations as the underlying storage
> > mechanism for a machine learning algorithm. I would like to build a
> > completely self aware instantiation that will maintain itself on all
> > levels... I postulate that the first place to start would be in
> > determining a method for maintaining all possible remediation
> > combinations including the unknown to eventually be learned from and
> > populate new knowledge into the database. Thank you for the reply,
> >
> > Garot
>
> --
> MySQL General Mailing List
> For list archives: http://lists.mysql.com/mysql To unsubscribe:
> http://lists.mysql.com/mysql
>
