List:General Discussion« Previous MessageNext Message »
From:Martin Gainty Date:August 23 2012 3:30pm
Subject:RE: Machine Learning
View as plain text  
Shawn and Garot

I like the parsing capability of the lucene and its ability to stem incoming queries..
If you are able to start populating your data then we *should* be able to start
identifying which root terms we can pull for building lucene-indexes
I have an upcoming
stuck-in-JFK-airport-on-labor-day-waiting-for-the-next-am-flight-of-USscare for an
interrupted timeframe(i never learned how to 'sleep' in airport terminal)
as long as I can find a wifi connection (and an AC connection within 6 hours) and use a
working MySQL JDBC connection string..I should be able to bang out a quick prototype..

Let me know how i can help out,
Martin Gainty 
______________________________________________ 
..place long-winded disclaimer here..


> From: ag4ve.us@stripped
> Date: Thu, 23 Aug 2012 10:42:55 -0400
> Subject: Re: Machine Learning
> To: garotconklin@stripped
> CC: webmaster@stripped; mgainty@stripped; mysql@stripped
> 
> just a few thoughts (things to look into)
> 
> if you want to populate a db with command parameters, i'd mine the man
> pages. it's a consistent format and you should be able to find a
> parser for whatever language you prefer.
> 
> if you want to see what has been entered and statistics about that
> process, look at the audit framework. it won't tell you success or
> failure, but you might be able to derive that based on the time,
> memory, and process usage that you can get from audit. auditd also
> maintains a db - you might look into the engine and schema they use.
> 
> you might also find some interesting topics looking into language
> processing (after all, commands are a type of language). people
> generally use other database engines to do this type of thing -
> elastic search or some other things people have done with lucene
> maybe.
> 
> do let us know where you go with this. it sounds quite interesting
> (and might help me with something i've been looking into)... and
> getting zsh to autocomplete options might be a bonus if you parse man
> pages and shove it into a db (i might do that part for you if i get a
> lazy afternoon).
> 
> On Wed, Aug 22, 2012 at 7:51 PM, Garot Conklin <garotconklin@stripped> wrote:
> > I am going to work up a visual diagram for this to better... visualize (LOL) the
> lay of the land. I will forward it when i have it completed then I hope we can continue
> the architectural discussion here!  Thanks
> >
> > -Garot
> >
> >
> > garotconklin@stripped
> >
> >
> > ________________________________
> >  From: Garot Conklin <garotconklin@stripped>
> > To: Garot Conklin <garotconklin@stripped>; "webmaster@stripped"
> <webmaster@stripped>; 'Martin Gainty' <mgainty@stripped>;
> "mysql@stripped" <mysql@stripped>
> > Sent: Tuesday, August 21, 2012 4:40 PM
> > Subject: Re: Machine Learning
> >
> > to elaborate:
> >
> > I would almost rather qualify from the OS perspective what will work in advance,
> but take into consideration any failing condition.  I may be getting a bit too specific at
> this juncture however as I am already apply this logic to issues I see in MY environment
> rather than total ambiguity, which is a best position here.
> >
> > I am ultimately expecting the learning algorithm to create new and excitingly
> effective ways of stringing commands together to solve issues, this being a notable aspect
> in and of itself, then creating some centralization based on what the machine determines
> as a best solution.
> >
> > Seems a relative "snapshot of the system" pre and post execution would be
> favorable as well so long as resources are not taxed as a result.  This discussion is
> leading me to believe that this will be a multifaceted db solution without a doubt
> comprised of multiple layers of abstraction ultimately rolling up into a single master
> cluster for all the higher level analytics and internal testing cycles to then roll out
> the new primary protocols, which may themselves take place in a sub-instance of each
> possible scenario.
> >
> > I have also though of populating a db or multiple db's with actual text  in the
> form of .pdf's of actual technical manuals... not certain the benefit of the written
> language abstraction there but the thought process extraction might be an interesting end
> indeed.
> >
> >
> >
> > garotconklin@stripped
> >
> >
> > ________________________________
> > From: Garot Conklin <garotconklin@stripped>
> > To: "webmaster@stripped" <webmaster@stripped>; 'Martin Gainty'
> <mgainty@stripped>; "mysql@stripped" <mysql@stripped>
> > Sent: Tuesday, August 21, 2012 4:09 PM
> > Subject: Re: Machine Learning
> >
> > YES!
> >
> >
> > garotconklin@stripped
> >
> >
> > ________________________________
> > From: "webmaster@stripped" <webmaster@stripped>
> > To: 'Garot Conklin' <garotconklin@stripped>; 'Martin Gainty'
> <mgainty@stripped>; mysql@stripped
> > Sent: Tuesday, August 21, 2012 3:37 PM
> > Subject: RE: Machine Learning
> >
> >
> > Ah,
> >
> > Getting clearer and clearer.
> >
> > So these ‘nodes’ could ‘learn’ and ‘teach’
> at the same time – right ? For instance, N1 runs a command in ‘domain’
> D20 which it successful – it send information to node N20 that is the authority on
> domain D20 and N20 records it as success; N5 runs a command in domain D20 which goes
> wrong, and sends info to N20 and N20 records failure and sends a correct call to N5. Is
> this what you have in mind ?
> >
> > There are a lot of variables to be considered, for instance:
> > 1)      Command A version 1 can run very happily on operating system version 2,
> but fails in OS version 1
> > 2)      Command A version 1 can run happily on OS version 2 on a 2G RAM, but
> fail on OS version 2 on 1G RAM
> > 3)      etc
> >
> > I think the DB design issues will become straightforward once the model is quite
> clear.
> >
> > Justin
> >
> > From:Garot Conklin [mailto:garotconklin@stripped]
> > Sent: 21 August 2012 17:14
> > To: webmaster@stripped; 'Martin Gainty'; mysql@stripped
> > Subject: Re: Machine Learning
> >
> > 1)Refer to it for ‘knowledge’ (for instance, of what the latest
> > version of a command is)
> >     It would most likely end up being "central" in this sense:
> >         A distributed collection of systems; i.e. (possible defined in roles)
> >             DB's
> >             FE's
> >             REPL's
> >             Cache's
> >         Each DB would have its own collection of remediation's that would then
> be indexed to populate a central db for trending/correlation etc...
> >         The "Collective" itself would function as a single conceptual
> implementation. A VIP for example might be associated with a specific role, say Web FE's
> and remediate only/all of them, but only have some relative access to the core db of say
> the Network from a Primary index perspective to make/draw associations/conclusions to
> issues at hand.
> >
> >
> > 2)Send their ‘knowledge’ (for instance, of the latest command
> > versions) to it for storage and distribution to others
> >    Rather than "latest command versions" I envision this to be more encapsulated
> as "latest successful invocation of the command string" and it inverse as well (to
> trend/metric-ize the failures thus lending to perpetual optimization).
> >
> > I like how this is fleshing out... This is helping me to define what I am really
> trying to accomplish. Thanks very much for everyone responding here, this is wonderful,
> please keep this going...
> >
> > garotconklin@stripped
> >
> > ________________________________
> >
> > From:"webmaster@stripped" <webmaster@stripped>
> > To: 'Martin Gainty' <mgainty@stripped>; garotconklin@stripped;
> mysql@stripped
> > Sent: Tuesday, August 21, 2012 11:19 AM
> > Subject: RE: Machine Learning
> >
> > Hi Garot,
> >
> >
> >
> > Ok, the concept is getting clearer, but let’s bring this down to earth a
> > little bit more. I love DB design and problem-solving and am quite curious
> > about this.
> >
> >
> >
> > Is the idea that you have a central computer (not HAL J) somewhere so that
> > other computers can:
> >
> > 1)      Refer to it for ‘knowledge’ (for instance, of what the
> latest
> > version of a command is)
> >
> > 2)      Send their ‘knowledge’ (for instance, of the latest command
> > versions) to it for storage and distribution to others
> >
> >
> >
> > If this is the model, then the knowledge base can build up organically over
> > time – I think. Or is this too simplistic ?
> >
> >
> >
> > Thanks,
> >
> > Justin
> >
> >
> >
> > From: Martin Gainty [mailto:mgainty@stripped]
> > Sent: 21 August 2012 00:25
> > To: garotconklin@stripped; webmaster@stripped; mysql@stripped
> > Subject: RE: Machine Learning
> >
> >
> >
> > When I hear 'AI' I always imagine theres a HAL 9001 behind the scenes that
> > is running the show constantly admonishing its creator to "take another
> > stress pill"
> >
> > Sounds like a fun project
> >
> > Keep us apprised,
> > Martin Gainty
> > ______________________________________________
> > Verzicht und Vertraulichkeitanmerkung/Note de déni et de
> confidentialité
> >
> >
> > Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene
> > Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte
> > Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht
> > dient lediglich dem Austausch von Informationen und entfaltet keine
> > rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von
> > E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
> >
> > Ce message est confidentiel et peut être privilégié. Si vous
> n'êtes pas le
> > destinataire prévu, nous te demandons avec bonté que pour
> satisfaire
> > informez l'expéditeur. N'importe quelle diffusion non autorisée ou
> la copie
> > de ceci est interdite. Ce message sert à l'information seulement et
> n'aura
> > pas n'importe quel effet légalement obligatoire. Étant
> donné que les email
> > peuvent facilement être sujets à la manipulation, nous ne pouvons
> accepter
> > aucune responsabilité pour le contenu fourni.
> >
> >
> >
> >
> >
> >   _____
> >
> > Date: Mon, 20 Aug 2012 13:50:04 -0700
> > From: garotconklin@stripped
> > Subject: Re: Machine Learning
> > To: webmaster@stripped; mgainty@stripped; mysql@stripped
> >
> > Ya the idea is not anything new, but must be apparently quit difficult or
> > not a priority as I have yet to find it already implemented anywhere... Far
> > be it from me to not make some attempt here anyway...
> >
> >
> >
> > I am creating a fully automated framework from which a distributed
> > infrastructure can be maintained.  I have been writing automation
> > scripts/code for some time now and the logical progression is to embark on a
> > full concept of systems health auto remediation.  I have numerous
> > "monitoring" solutions under my control however none that properly (in my
> > opinion) implements any real learning algorithms from which to draw even a
> > minimalists view of automation.  I like mySQL therefor began thinking about
> > creating the aspects (lobes) of the "brain" as a relational database(s).  So
> > this is only one facet of what I am trying to do, however leveraging a full
> > command set of shell utilities/commands/programs seemed to be a good
> > starting point before I get into the "hard" stuff !
> >
> >
> >
> > -Garot
> >
> >
> >
> > garotconklin@stripped
> >
> >   _____
> >
> > From: "webmaster@stripped" <webmaster@stripped>
> > To: 'Martin Gainty' <mgainty@stripped>; garotconklin@stripped;
> > mysql@stripped
> > Sent: Monday, August 20, 2012 3:55 PM
> > Subject: RE: Machine Learning
> >
> >
> > Hi Garot,
> >
> >
> >
> > You'll have to elaborate some more ... I understand you may want to protect
> > the idea as well, so if you can narrow it down to some technical specifics
> > then it'll help.
> >
> >
> >
> > What is the objective of this system, for instance ?
> >
> >
> >
> > Thanks,
> >
> > Justin
> >
> >
> >
> > From: Martin Gainty [mailto:mgainty@stripped]
> > Sent: 20 August 2012 19:23
> > To: garotconklin@stripped; webmaster@stripped; mysql@stripped
> > Subject: RE: Machine Learning
> >
> >
> >
> >
> >
> > From: garotconklin@stripped
> > Subject: Re: Machine Learning
> > To: webmaster@stripped; mgainty@stripped; mysql@stripped
> >
> >
> > My initial thought was to propagate the db with everything and allow the
> > algorithm to then begin to determin trends/patterns
> > MG>which trends or patterns will you be modelling?
> >
> > and begin either an indexing methodology
> > MG>which indexes are you considering: Unique index, primary index or foreign
> > index?
> >
> > additional table/db creation process or both to further optimize the calls
> > being made
> > MG>optimize based on execution time or diskspace allocated, EliminatingFTS
> > or some other criteria?
> > MG>https://dev.mysql.com/doc/refman/5.5/en/optimization.html
> >
> > and build in some internal levels of redundancy.
> > MG>what about replication
> > MG>http://dev.mysql.com/doc/refman/5.5/en/replication.html
> >
> > I am actually approaching this with some degree of biological conception in
> > the multipathing within our own brains however until I have something up and
> > running under some substantial load however I may not get a complete
> > picture.
> >
> > Thanks,
> >
> > Garot
> >
> >
> > Interesting
> > Martin
> >
> >   _____
> >
> > From: webmaster@stripped <webmaster@stripped>;
> > To: 'Garot Conklin' <garotconklin@stripped>; 'Martin Gainty'
> > <mgainty@stripped>; <mysql@stripped>;
> > Subject: RE: Machine Learning
> > Sent: Mon, Aug 20, 2012 7:13:25 AM
> >
> >
> > Hi Garot,
> >
> > This sounds an interesting idea.
> >
> > Are you looking to store all known commands and their options or are you
> > looking for a 'formula' for calling any unix command ?
> >
> > The reason for my question is that, at the end of the day, a unix command is
> > just a program that is run in the operating system. Each program comes with
> > its own options and acceptable inputs. I don't know if there is a rule or
> > convention for structuring these commands.
> >
> > Are you then looking to build a system that 'knows' all commands and 'how
> > to' call them ?
> >
> > Thanks,
> > Justin
> >
> > -----Original Message-----
> > From: Garot Conklin [mailto:garotconklin@stripped]
> > Sent: 20 August 2012 03:39
> > To: Martin Gainty; mysql@stripped
> > Subject: Re: Machine Learning
> >
> > The initial goal is to provide a working framework from which to call all
> > UNIX shell command combinations as the underlying storage mechanism for a
> > machine learning algorithm.  I would like to build a completely self aware
> > instantiation that will maintain itself on all levels... I postulate that
> > the first place to start would be in determining a method for maintaining
> > all possible remediation combinations including the unknown to eventually be
> > learned from and populate new knowledge into the database.  Thank you for
> > the reply,
> >
> > Garot
> 
> -- 
> MySQL General Mailing List
> For list archives: http://lists.mysql.com/mysql
> To unsubscribe:    http://lists.mysql.com/mysql
> 
 		 	   		  
Thread
Machine LearningGarot Conklin19 Aug
  • RE: Machine LearningMartin Gainty19 Aug
Re: Machine LearningGarot Conklin20 Aug
  • RE: Machine Learningwebmaster20 Aug
Re: Machine LearningGarot Conklin20 Aug
  • RE: Machine LearningMartin Gainty20 Aug
    • Re: Machine LearningGarot Conklin20 Aug
    • RE: Machine Learningwebmaster20 Aug
      • Re: Machine LearningGarot Conklin20 Aug
        • RE: Machine LearningMartin Gainty20 Aug
          • Re: Machine LearningMichael21 Aug
            • RE: Machine LearningMartin Gainty21 Aug
              • Re: Machine LearningGarot Conklin21 Aug
          • RE: Machine Learningwebmaster21 Aug
            • Re: Machine LearningGarot Conklin21 Aug
              • RE: Machine Learningwebmaster21 Aug
                • RE: Machine LearningMartin Gainty21 Aug
                • RE: Machine LearningMartin Gainty21 Aug
                • RE: Machine LearningMartin Gainty21 Aug
                • Re: Machine LearningGarot Conklin21 Aug
                  • Re: Machine LearningGarot Conklin21 Aug
                    • Re: Machine LearningGarot Conklin22 Aug
                      • Re: Machine Learningshawn wilson23 Aug
RE: Machine LearningMartin Gainty23 Aug
  • Re: Machine LearningGarot Conklin23 Aug
    • RE: Machine Learningwebmaster24 Aug
      • RE: Machine LearningMartin Gainty24 Aug
      • Re: Machine LearningGarot Conklin24 Aug