List:General Discussion« Previous MessageNext Message »
From:<webmaster Date:August 24 2012 8:04am
Subject:RE: Machine Learning
View as plain text  
Hi Garot,

How is the visual diagram coming along ?

I'd be good to have a good idea of core system specifics so that DB design can be tailored
to that ... of course, knowing that things will change as the concept evolves.

Thanks,
Justin

-----Original Message-----
From: Garot Conklin [mailto:garotconklin@stripped] 
Sent: 23 August 2012 20:14
To: Martin Gainty; ag4ve.us@stripped
Cc: webmaster@stripped; mysql@stripped
Subject: Re: Machine Learning

That is in fact a very interesting approach... I like it very much.  Any architectural
thoughts on an auto-population attempt? Odly I have not created a db like this before,
not from this large of an import anyway... Please share some thoughts there and I will
get that rolling... seems that if we can get something physical going I would like to
offer it up as a sandbox to all those interested.  I have a few X336's lying around my
house just collecting dust, they would be a great testing environment for all to work on.


 
garotconklin@stripped


________________________________
 From: Martin Gainty <mgainty@stripped>
To: ag4ve.us@stripped; garotconklin@stripped
Cc: webmaster@stripped; "mysql@stripped" <mysql@stripped>
Sent: Thursday, August 23, 2012 11:30 AM
Subject: RE: Machine Learning
 

Shawn and Garot

I like the parsing capability of the lucene and its ability to stem incoming queries..
If you are able to start populating your data then we *should* be able to start
identifying which root terms we can pull for building lucene-indexes I have an upcoming
stuck-in-JFK-airport-on-labor-day-waiting-for-the-next-am-flight-of-USscare for an
interrupted timeframe(i never learned how to 'sleep' in airport terminal) as long as I
can find a wifi connection (and an AC connection within 6 hours) and use a working MySQL
JDBC connection string..I should be able to bang out a quick prototype..

Let me know how i can help out,
Martin Gainty
______________________________________________
..place long-winded disclaimer here..


> From: ag4ve.us@stripped
> Date: Thu, 23 Aug 2012 10:42:55 -0400
> Subject: Re: Machine Learning
> To: garotconklin@stripped
> CC: webmaster@stripped; mgainty@stripped; mysql@stripped
> 
> just a few thoughts (things to look into)
> 
> if you want to populate a db with command parameters, i'd mine the man 
> pages. it's a consistent format and you should be able to find a 
> parser for whatever language you prefer.
> 
> if you want to see what has been entered and statistics about that 
> process, look at the audit framework. it won't tell you success or 
> failure, but you might be able to derive that based on the time, 
> memory, and process usage that you can get from audit. auditd also 
> maintains a db - you might look into the engine and schema they use.
> 
> you might also find some interesting topics looking into language 
> processing (after all, commands are a type of language). people 
> generally use other database engines to do this type of thing - 
> elastic search or some other things people have done with lucene 
> maybe.
> 
> do let us know where you go with this. it sounds quite interesting 
> (and might help me with something i've been looking into)... and 
> getting zsh to autocomplete options might be a bonus if you parse man 
> pages and shove it into a db (i might do that part for you if i get a 
> lazy afternoon).
> 
> On Wed, Aug 22, 2012 at 7:51 PM, Garot Conklin <garotconklin@stripped> wrote:
> > I am going to work up a visual diagram for this to better... 
> > visualize (LOL) the lay of the land. I will forward it when i have 
> > it completed then I hope we can continue the architectural 
> > discussion here!  Thanks
> >
> > -Garot
> >
> >
> > garotconklin@stripped
> >
> >
> > ________________________________
> >  From: Garot Conklin <garotconklin@stripped>
> > To: Garot Conklin <garotconklin@stripped>; "webmaster@stripped" 
> ><webmaster@stripped>; 'Martin Gainty' <mgainty@stripped>; 
> >"mysql@stripped" <mysql@stripped>
> > Sent: Tuesday, August 21, 2012 4:40 PM
> > Subject: Re: Machine Learning
> >
> > to elaborate:
> >
> > I would almost rather qualify from the OS perspective what will work in advance,
> but take into consideration any failing condition.  I may be getting a bit too specific at
> this juncture however as I am already apply this logic to issues I see in MY environment
> rather than total ambiguity, which is a best position here.
> >
> > I am ultimately expecting the learning algorithm to create new and excitingly
> effective ways of stringing commands together to solve issues, this being a notable aspect
> in and of itself, then creating some centralization based on what the machine determines
> as a best solution.
> >
> > Seems a relative "snapshot of the system" pre and post execution would be
> favorable as well so long as resources are not taxed as a result.  This discussion is
> leading me to believe that this will be a multifaceted db solution without a doubt
> comprised of multiple layers of abstraction ultimately rolling up into a single master
> cluster for all the higher level analytics and internal testing cycles to then roll out
> the new primary protocols, which may themselves take place in a sub-instance of each
> possible scenario.
> >
> > I have also though of populating a db or multiple db's with actual text  in the
> form of .pdf's of actual technical manuals... not certain the benefit of the written
> language abstraction there but the thought process extraction might be an interesting end
> indeed.
> >
> >
> >
> > garotconklin@stripped
> >
> >
> > ________________________________
> > From: Garot Conklin <garotconklin@stripped>
> > To: "webmaster@stripped" <webmaster@stripped>; 'Martin Gainty' 
> > <mgainty@stripped>; "mysql@stripped" 
> > <mysql@stripped>
> > Sent: Tuesday, August 21, 2012 4:09 PM
> > Subject: Re: Machine Learning
> >
> > YES!
> >
> >
> > garotconklin@stripped
> >
> >
> > ________________________________
> > From: "webmaster@stripped" <webmaster@stripped>
> > To: 'Garot Conklin' <garotconklin@stripped>; 'Martin Gainty' 
> > <mgainty@stripped>; mysql@stripped
> > Sent: Tuesday, August 21, 2012 3:37 PM
> > Subject: RE: Machine Learning
> >
> >
> > Ah,
> >
> > Getting clearer and clearer.
> >
> > So these ‘nodes’ could ‘learn’ and ‘teach’
> at the same time – right ? For instance, N1 runs a command in ‘domain’
> D20 which it successful – it send information to node N20 that is the authority on
> domain D20 and N20 records it as success; N5 runs a command in domain D20 which goes
> wrong, and sends info to N20 and N20 records failure and sends a correct call to N5. Is
> this what you have in mind ?
> >
> > There are a lot of variables to be considered, for instance:
> > 1)      Command A version 1 can run very happily on operating system 
> > version 2, but fails in OS version 1
> > 2)      Command A version 1 can run happily on OS version 2 on a 2G 
> > RAM, but fail on OS version 2 on 1G RAM
> > 3)      etc
> >
> > I think the DB design issues will become straightforward once the model is quite
> clear.
> >
> > Justin
> >
> > From:Garot Conklin [mailto:garotconklin@stripped]
> > Sent: 21 August 2012 17:14
> > To: webmaster@stripped; 'Martin Gainty'; mysql@stripped
> > Subject: Re: Machine Learning
> >
> > 1)Refer to it for ‘knowledge’ (for instance, of what the latest  
> >version of a command is)
> >     It would most likely end up being "central" in this sense:
> >         A distributed collection of systems; i.e. (possible defined 
> >in roles)
> >             DB's
> >             FE's
> >             REPL's
> >             Cache's
> >         Each DB would have its own collection of remediation's that would then
> be indexed to populate a central db for trending/correlation etc...
> >         The "Collective" itself would function as a single conceptual
> implementation. A VIP for example might be associated with a specific role, say Web FE's
> and remediate only/all of them, but only have some relative access to the core db of say
> the Network from a Primary index perspective to make/draw associations/conclusions to
> issues at hand.
> >
> >
> > 2)Send their ‘knowledge’ (for instance, of the latest command
> > versions) to it for storage and distribution to others
> >    Rather than "latest command versions" I envision this to be more encapsulated
> as "latest successful invocation of the command string" and it inverse as well (to
> trend/metric-ize the failures thus lending to perpetual optimization).
> >
> > I like how this is fleshing out... This is helping me to define what I am really
> trying to accomplish. Thanks very much for everyone responding here, this is wonderful,
> please keep this going...
> >
> > garotconklin@stripped
> >
> > ________________________________
> >
> > From:"webmaster@stripped" <webmaster@stripped>
> > To: 'Martin Gainty' <mgainty@stripped>; garotconklin@stripped; 
> > mysql@stripped
> > Sent: Tuesday, August 21, 2012 11:19 AM
> > Subject: RE: Machine Learning
> >
> > Hi Garot,
> >
> >
> >
> > Ok, the concept is getting clearer, but let’s bring this down to 
> > earth a little bit more. I love DB design and problem-solving and am 
> > quite curious about this.
> >
> >
> >
> > Is the idea that you have a central computer (not HAL J) somewhere 
> > so that other computers can:
> >
> > 1)      Refer to it for ‘knowledge’ (for instance, of what the 
> > latest version of a command is)
> >
> > 2)      Send their ‘knowledge’ (for instance, of the latest command
> > versions) to it for storage and distribution to others
> >
> >
> >
> > If this is the model, then the knowledge base can build up 
> > organically over time – I think. Or is this too simplistic ?
> >
> >
> >
> > Thanks,
> >
> > Justin
> >
> >
> >
> > From: Martin Gainty [mailto:mgainty@stripped]
> > Sent: 21 August 2012 00:25
> > To: garotconklin@stripped; webmaster@stripped; 
> > mysql@stripped
> > Subject: RE: Machine Learning
> >
> >
> >
> > When I hear 'AI' I always imagine theres a HAL 9001 behind the 
> > scenes that is running the show constantly admonishing its creator 
> > to "take another stress pill"
> >
> > Sounds like a fun project
> >
> > Keep us apprised,
> > Martin Gainty
> > ______________________________________________
> > Verzicht und Vertraulichkeitanmerkung/Note de déni et de 
> > confidentialité
> >
> >
> > Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene 
> > Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede 
> > unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. 
> > Diese Nachricht dient lediglich dem Austausch von Informationen und 
> > entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten 
> > Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt
> uebernehmen.
> >
> > Ce message est confidentiel et peut être privilégié. Si vous
> n'êtes 
> > pas le destinataire prévu, nous te demandons avec bonté que pour 
> > satisfaire informez l'expéditeur. N'importe quelle diffusion non 
> > autorisée ou la copie de ceci est interdite. Ce message sert à 
> > l'information seulement et n'aura pas n'importe quel effet 
> > légalement obligatoire. Étant donné que les email peuvent
> facilement 
> > être sujets à la manipulation, nous ne pouvons accepter aucune
> responsabilité pour le contenu fourni.
> >
> >
> >
> >
> >
> >   _____
> >
> > Date: Mon, 20 Aug 2012 13:50:04 -0700
> > From: garotconklin@stripped
> > Subject: Re: Machine Learning
> > To: webmaster@stripped; mgainty@stripped; 
> > mysql@stripped
> >
> > Ya the idea is not anything new, but must be apparently quit 
> > difficult or not a priority as I have yet to find it already 
> > implemented anywhere... Far be it from me to not make some attempt here
> anyway...
> >
> >
> >
> > I am creating a fully automated framework from which a distributed 
> > infrastructure can be maintained.  I have been writing automation 
> > scripts/code for some time now and the logical progression is to 
> > embark on a full concept of systems health auto remediation.  I have 
> > numerous "monitoring" solutions under my control however none that 
> > properly (in my
> > opinion) implements any real learning algorithms from which to draw 
> > even a minimalists view of automation.  I like mySQL therefor began 
> > thinking about creating the aspects (lobes) of the "brain" as a 
> > relational database(s).  So this is only one facet of what I am 
> > trying to do, however leveraging a full command set of shell 
> > utilities/commands/programs seemed to be a good starting point before I get into
> the "hard" stuff !
> >
> >
> >
> > -Garot
> >
> >
> >
> > garotconklin@stripped
> >
> >   _____
> >
> > From: "webmaster@stripped" <webmaster@stripped>
> > To: 'Martin Gainty' <mgainty@stripped>; garotconklin@stripped; 
> > mysql@stripped
> > Sent: Monday, August 20, 2012 3:55 PM
> > Subject: RE: Machine Learning
> >
> >
> > Hi Garot,
> >
> >
> >
> > You'll have to elaborate some more ... I understand you may want to 
> > protect the idea as well, so if you can narrow it down to some 
> > technical specifics then it'll help.
> >
> >
> >
> > What is the objective of this system, for instance ?
> >
> >
> >
> > Thanks,
> >
> > Justin
> >
> >
> >
> > From: Martin Gainty [mailto:mgainty@stripped]
> > Sent: 20 August 2012 19:23
> > To: garotconklin@stripped; webmaster@stripped; 
> > mysql@stripped
> > Subject: RE: Machine Learning
> >
> >
> >
> >
> >
> > From: garotconklin@stripped
> > Subject: Re: Machine Learning
> > To: webmaster@stripped; mgainty@stripped; 
> > mysql@stripped
> >
> >
> > My initial thought was to propagate the db with everything and allow 
> > the algorithm to then begin to determin trends/patterns
> > MG>which trends or patterns will you be modelling?
> >
> > and begin either an indexing methodology
> > MG>which indexes are you considering: Unique index, primary index or 
> > MG>foreign
> > index?
> >
> > additional table/db creation process or both to further optimize the 
> > calls being made
> > MG>optimize based on execution time or diskspace allocated, 
> > MG>EliminatingFTS
> > or some other criteria?
> > MG>https://dev.mysql.com/doc/refman/5.5/en/optimization.html
> >
> > and build in some internal levels of redundancy.
> > MG>what about replication
> > MG>http://dev.mysql.com/doc/refman/5.5/en/replication.html
> >
> > I am actually approaching this with some degree of biological 
> > conception in the multipathing within our own brains however until I 
> > have something up and running under some substantial load however I 
> > may not get a complete picture.
> >
> > Thanks,
> >
> > Garot
> >
> >
> > Interesting
> > Martin
> >
> >   _____
> >
> > From: webmaster@stripped <webmaster@stripped>;
> > To: 'Garot Conklin' <garotconklin@stripped>; 'Martin Gainty'
> > <mgainty@stripped>; <mysql@stripped>;
> > Subject: RE: Machine Learning
> > Sent: Mon, Aug 20, 2012 7:13:25 AM
> >
> >
> > Hi Garot,
> >
> > This sounds an interesting idea.
> >
> > Are you looking to store all known commands and their options or are 
> > you looking for a 'formula' for calling any unix command ?
> >
> > The reason for my question is that, at the end of the day, a unix 
> > command is just a program that is run in the operating system. Each 
> > program comes with its own options and acceptable inputs. I don't 
> > know if there is a rule or convention for structuring these commands.
> >
> > Are you then looking to build a system that 'knows' all commands and 
> > 'how to' call them ?
> >
> > Thanks,
> > Justin
> >
> > -----Original Message-----
> > From: Garot Conklin [mailto:garotconklin@stripped]
> > Sent: 20 August 2012 03:39
> > To: Martin Gainty; mysql@stripped
> > Subject: Re: Machine Learning
> >
> > The initial goal is to provide a working framework from which to 
> > call all UNIX shell command combinations as the underlying storage 
> > mechanism for a machine learning algorithm.  I would like to build a 
> > completely self aware instantiation that will maintain itself on all 
> > levels... I postulate that the first place to start would be in 
> > determining a method for maintaining all possible remediation 
> > combinations including the unknown to eventually be learned from and 
> > populate new knowledge into the database.  Thank you for the reply,
> >
> > Garot
> 
> --
> MySQL General Mailing List
> For list archives: http://lists.mysql.com/mysql To unsubscribe:    
> http://lists.mysql.com/mysql
> 

Thread
Machine LearningGarot Conklin19 Aug
  • RE: Machine LearningMartin Gainty19 Aug
Re: Machine LearningGarot Conklin20 Aug
  • RE: Machine Learningwebmaster20 Aug
Re: Machine LearningGarot Conklin20 Aug
  • RE: Machine LearningMartin Gainty20 Aug
    • Re: Machine LearningGarot Conklin20 Aug
    • RE: Machine Learningwebmaster20 Aug
      • Re: Machine LearningGarot Conklin20 Aug
        • RE: Machine LearningMartin Gainty20 Aug
          • Re: Machine LearningMichael21 Aug
            • RE: Machine LearningMartin Gainty21 Aug
              • Re: Machine LearningGarot Conklin21 Aug
          • RE: Machine Learningwebmaster21 Aug
            • Re: Machine LearningGarot Conklin21 Aug
              • RE: Machine Learningwebmaster21 Aug
                • RE: Machine LearningMartin Gainty21 Aug
                • RE: Machine LearningMartin Gainty21 Aug
                • RE: Machine LearningMartin Gainty21 Aug
                • Re: Machine LearningGarot Conklin21 Aug
                  • Re: Machine LearningGarot Conklin21 Aug
                    • Re: Machine LearningGarot Conklin22 Aug
                      • Re: Machine Learningshawn wilson23 Aug
RE: Machine LearningMartin Gainty23 Aug
  • Re: Machine LearningGarot Conklin23 Aug
    • RE: Machine Learningwebmaster24 Aug
      • RE: Machine LearningMartin Gainty24 Aug
      • Re: Machine LearningGarot Conklin24 Aug