From Robin

Revision as of 10:42, 13 September 2019 (view source) Wonhol (Talk | contribs) ← Older edit		Revision as of 10:06, 20 September 2019 (view source) Wonhol (Talk | contribs) Newer edit →
Line 50:		Line 50:
-	=== Week 35 ===	+	=== Week 36 ===
	==== OpenAI gym setup for DyRET ====		==== OpenAI gym setup for DyRET ====
	Get used to DyRET Env		Get used to DyRET Env
Line 83:		Line 83:
	$ export LD_LIBRARY_PATH=/usr/local/lib		$ export LD_LIBRARY_PATH=/usr/local/lib
-	=== Week 36 ===	+	=== Week 37 ===
	==== OpenAI gym setup ====		==== OpenAI gym setup ====
	* Running basic examples to get familier with OpenAI Gym concepts - observation space, action space		* Running basic examples to get familier with OpenAI Gym concepts - observation space, action space
Line 92:		Line 92:
	** Was able to run example from https://gist.github.com/stefanopalmieri		** Was able to run example from https://gist.github.com/stefanopalmieri
	** Those examples are from old openai gym code, it needs appropriate fix		** Those examples are from old openai gym code, it needs appropriate fix
-	** wasn't able to create ES hyperneat object - Genome class doesn't have it!?	+	** wasn't able to create ES hyperneat object - Genome class doesn't have it!? -
		+	commented out in source code
	by this point, familir with hyperNEAT packages and chosen one for the project		by this point, familir with hyperNEAT packages and chosen one for the project
	OpenAI gym env setup for various experiments		OpenAI gym env setup for various experiments
-
-	=== Week 37 ===
-	Experiment setup
-	Implementing Risi's HyperNEAT on Dyret
	=== Week 38 ===		=== Week 38 ===
-	Experiment setup	+	==== Weekly goals ====
-	Implementing Risi's HyperNEAT on Dyret	+	* Find out how to implement CTRNN cell for HyperNEAT
		+	* Design control scheme for OpenAI gym DyRET using one of the HyperNEAT packages
		+	* Understand evolution pipeline in HyperNEAT packages together with OpenAI Gym env.
		+	* Document how to set up dev. environment - installation of openai, hyperneat
		+	packages for the larger experiment jobs later on.
		+
		+	==== Progress ====
		+
		+	* MultiNEAT has leaky integrators implemented. Following function for
		+	NeuralNetwork obj. Also parameter for time constants should be set up
		+
		+	void ActivateLeaky(double step); // activates in leaky integrator mode
		+
		+	** Substrate can also be defined to made of Leaky neurons by
		+
		+	Substrate.m_leaky = True
		+
		+	* Evolution pipeline in MultiNEAT
		+	** Define parameters - for CTRNN, define timeconstants
		+	** Define substrates
		+	** Initialize Genome with intput, hidden, output
		+	** Init population
		+	** evaluate each indivisual from pop with interacting with openAI Gym env.
		+	** pop.Epoch()
		+	** To build HyperNEAT genome, specify by function Genome::BuildHyperNEATPheonotype() when making network
		+
		+
		+	* Each of risi's neuron has 4d coordinate, (xm, ym, x, y) where xm, and ym is
		+	substrate containing sub-substrates, this can be simply set up as 4d
		+	substrate when setting up in MultiNEAT
		+
		+
		+	==== What to look closely next ====
		+
		+	* How to set up input for CPPN in MultiNEAT - considering leg length??
		+	* Need to look into docker tutorial
	=== Week 39 ===		=== Week 39 ===
Line 111:		Line 143:
	=== Week 40 ===		=== Week 40 ===
	Experiment setup		Experiment setup
-	Implementing CPG style locomotive controller on Dyret	+	Implementing Risi's HyperNEAT on Dyret
	=== Week 41 ===		=== Week 41 ===
Line 122:		Line 154:
	=== Week 43 ===		=== Week 43 ===
-	Experiment run	+	Experiment setup
		+	Implementing CPG style locomotive controller on Dyret
	=== Week 44 ===		=== Week 44 ===
Line 131:		Line 164:
	=== Week 46 ===		=== Week 46 ===
-	Experiment analysis	+	Experiment run
	=== Week 47 ===		=== Week 47 ===
Line 141:		Line 174:
	=== Week 49 ===		=== Week 49 ===
	Presenting first results		Presenting first results
		+
		+	=== Week 50 ===

---

Revision as of 10:06, 20 September 2019

Contents 1 Goals 1.1 OpenAI Gym Env for DyRET 1.2 HyperNEAT libraries 2 Plan for Fall semester 2.1 Remarks 2.2 Week 36 2.2.1 OpenAI gym setup for DyRET 2.2.2 HyperNEAT libraries 2.3 Week 37 2.3.1 OpenAI gym setup 2.3.2 HyperNEAT library test 2.4 Week 38 2.4.1 Weekly goals 2.4.2 Progress 2.4.3 What to look closely next 2.5 Week 39 2.6 Week 40 2.7 Week 41 2.8 Week 42 2.9 Week 43 2.10 Week 44 2.11 Week 45 2.12 Week 46 2.13 Week 47 2.14 Week 48 2.15 Week 49 2.16 Week 50

Goals

We are taking the start point from the paper from Risi,

Risi - Evolving flexible controller for locomotion

where locomotive controller for variable length legges were evolved with HyperNEAT approach.

The goal is to implement it on DyRet platform - where it has two actuator for each legges. Also other thing to consider will be taking account of Tegotae - where touch input is used as some kind of feedback to CPG. Risi had touch sensor as input to his substrate in the simulation.

Other aspect of it is to see if HyperNEAT approach is something plausible considering its complexity - some skeptical veiw on HyperNEAT. ie) simple CTRNN network with length of leg as one of the input.

Some of the tasks that could be done over the summer are

  * Read through DyRet doucumentation from robin wiki & github and set up a
    simulator enviorment
  * Theoretical understanding of Tegotae - is it plausible to embed it with
    CTRNN-substrate?
  * Experimenting with HyperNEAT libraries - Kyrre`s recommendation is
    "C++/Python MultiNEAT C++ with Python binding", otherwise Risi seems to

work with C# implementation -> perhaps it is a good idea to have a look.

OpenAI Gym Env for DyRET

https://github.uio.no/jorgehn/gym-dyret

HyperNEAT libraries

TODO

Plan for Fall semester

                                    2019
      September         October           November          December          
  Su  1  8 15 22 29        6 13 20 27        3 10 17 24     1  8 15 22 29   
  Mo  2  9 16 23 30        7 14 21 28        4 11 18 25     2  9 16 23 30   
  Tu  3 10 17 24        1  8 15 22 29        5 12 19 26     3 10 17 24 31   
  We  4 11 18 25        2  9 16 23 30        6 13 20 27     4 11 18 25      
  Th  5 12 19 26        3 10 17 24 31        7 14 21 28     5 12 19 26      
  Fr  6 13 20 27        4 11 18 25        1  8 15 22 29     6 13 20 27      
  Sa  7 14 21 28        5 12 19 26        2  9 16 23 30     7 14 21 28      
     35 36 37 38 39    39 40 41 42 43    43 44 45 46 47    48 49 50 51 52   

Remarks

Delivery in May 2020 Mid-term presentation in week 49

Week 36

OpenAI gym setup for DyRET

Get used to DyRET Env

• input param for step : 12 np vector for joints, 8 for extension

HyperNEAT libraries

Lists of some promising ones

• https://github.com/ukuleleplayer/pureples
  • Pure python-based HyperNEAT, ES-HyperNEAT library, based on neat-python library
  • Installed and provided example experiments runs fine

• https://github.com/peter-ch/MultiNEAT
  • MultiNEAT, implemented in C++ with python bindings
  • good review from Stanley's website
  • https://gist.github.com/stefanopalmieri This guy has some nice examples

binding multineat and gym env

• • installation
    • install boost, first bootstrap with python version 3.6 then build
    • git clone multineat then

  $ export MN_BUILD=boost
  $ python3 setup.py build_ext
  $ python3 setup.py install 

• • • in case it casts missing library link to python and numpy, make sure to

install boost with python specified and numpy is installed properly for the user

• • • if the user installing multineat has no access to write to install destination, try via venv.

• • • once installed, test by

  >>> import MultiNEAT

• • • in case it casts missing library error, explicitly set $LD_LIBRARY_PATH for

boost install location, by default

  $ export LD_LIBRARY_PATH=/usr/local/lib

Week 37

OpenAI gym setup

• Running basic examples to get familier with OpenAI Gym concepts - observation space, action space

https://towardsdatascience.com/reinforcement-learning-with-openai-d445c2c687d2

HyperNEAT library test

• MultiNEAT example running successfully with installation
  • Was able to run example from https://gist.github.com/stefanopalmieri
  • Those examples are from old openai gym code, it needs appropriate fix
  • wasn't able to create ES hyperneat object - Genome class doesn't have it!? -

commented out in source code

by this point, familir with hyperNEAT packages and chosen one for the project OpenAI gym env setup for various experiments

Week 38

Weekly goals

• Find out how to implement CTRNN cell for HyperNEAT
• Design control scheme for OpenAI gym DyRET using one of the HyperNEAT packages
• Understand evolution pipeline in HyperNEAT packages together with OpenAI Gym env.
• Document how to set up dev. environment - installation of openai, hyperneat

packages for the larger experiment jobs later on.

Progress

• MultiNEAT has leaky integrators implemented. Following function for

NeuralNetwork obj. Also parameter for time constants should be set up

void ActivateLeaky(double step); // activates in leaky integrator mode

• • Substrate can also be defined to made of Leaky neurons by

Substrate.m_leaky = True

• Evolution pipeline in MultiNEAT
  • Define parameters - for CTRNN, define timeconstants
  • Define substrates
  • Initialize Genome with intput, hidden, output
  • Init population
  • evaluate each indivisual from pop with interacting with openAI Gym env.
  • pop.Epoch()
  • To build HyperNEAT genome, specify by function Genome::BuildHyperNEATPheonotype() when making network

• Each of risi's neuron has 4d coordinate, (xm, ym, x, y) where xm, and ym is

substrate containing sub-substrates, this can be simply set up as 4d substrate when setting up in MultiNEAT

What to look closely next

• How to set up input for CPPN in MultiNEAT - considering leg length??
• Need to look into docker tutorial

Week 39

Experiment setup Implementing Risi's HyperNEAT on Dyret

Week 40

Experiment setup Implementing Risi's HyperNEAT on Dyret

Week 41

Experiment setup Implementing CPG style locomotive controller on Dyret

Week 42

Experiment setup Implementing CPG style locomotive controller on Dyret

Week 43

Experiment setup Implementing CPG style locomotive controller on Dyret

Week 44

Experiment run

Week 45

Experiment run

Week 46

Experiment run

Week 47

Experiment analysis

Week 48

Experiment analysis

Week 49

Presenting first results

Week 50