User:Wonhol

From Robin

(Difference between revisions)
Jump to: navigation, search
(Week 7)
 
(11 intermediate revisions not shown)
Line 49: Line 49:
Mid-term presentation in week 49
Mid-term presentation in week 49
-
 
-
=== Week 35 ===
 
-
OpenAI gym setup
 
-
HyperNEAT library test
 
=== Week 36 ===
=== Week 36 ===
-
OpenAI gym setup
+
==== OpenAI gym setup for DyRET ====
-
HyperNEAT library test
+
Get used to DyRET Env
 +
* input param for step : 12 np vector for joints, 8 for extension
-
by this point, familir with hyperNEAT packages and chosen one for the project
+
==== HyperNEAT libraries  ====
-
OpenAI gym env setup for various experiments
+
Lists of some promising ones
 +
* https://github.com/ukuleleplayer/pureples
 +
** Pure python-based HyperNEAT, ES-HyperNEAT library, based on neat-python library
 +
** Installed and provided example experiments runs fine
 +
 
 +
* https://github.com/peter-ch/MultiNEAT
 +
** MultiNEAT, implemented in C++ with python bindings
 +
** good review from Stanley's website
 +
** https://gist.github.com/stefanopalmieri This guy has some nice examples
 +
binding multineat and gym env  
 +
** installation
 +
*** install boost, first bootstrap with python version 3.6 then build
 +
*** git clone multineat then
 +
  $ export MN_BUILD=boost
 +
  $ python3 setup.py build_ext
 +
  $ python3 setup.py install
 +
*** in case it casts missing library link to python and numpy, make sure to
 +
install boost with python specified and numpy is installed properly for the
 +
user
 +
*** if the user installing multineat has no access to write to install destination, try via venv.
 +
 
 +
*** once installed, test by
 +
  >>> import MultiNEAT
 +
*** in case it casts missing library error, explicitly set $LD_LIBRARY_PATH for
 +
boost install location, by default
 +
  $ export LD_LIBRARY_PATH=/usr/local/lib
=== Week 37 ===
=== Week 37 ===
-
Experiment setup
+
==== OpenAI gym setup ====
-
Implementing Risi's HyperNEAT on Dyret
+
* Running basic examples to get familier with OpenAI Gym concepts - observation space, action space
 +
https://towardsdatascience.com/reinforcement-learning-with-openai-d445c2c687d2
 +
 
 +
==== HyperNEAT library test ====
 +
* MultiNEAT example running successfully with installation
 +
** Was able to run example from https://gist.github.com/stefanopalmieri
 +
** Those examples are from old openai gym code, it needs appropriate fix
 +
** wasn't able to create ES hyperneat object - Genome class doesn't have it!? -
 +
commented out in source code
 +
 
 +
 
 +
by this point, familir with hyperNEAT packages and chosen one for the project
 +
OpenAI gym env setup for various experiments
=== Week 38 ===
=== Week 38 ===
-
Experiment setup
+
==== Weekly goals ====
-
Implementing Risi's HyperNEAT on Dyret
+
* Find out how to implement CTRNN cell for HyperNEAT
 +
* Design control scheme for OpenAI gym DyRET using one of the HyperNEAT packages
 +
* Understand evolution pipeline in HyperNEAT packages together with OpenAI Gym env.
 +
* Document how to set up dev. environment - installation of openai, hyperneat
 +
packages for the larger experiment jobs later on.
 +
 
 +
==== Progress ====
 +
 
 +
* MultiNEAT has leaky integrators implemented. Following function for
 +
NeuralNetwork obj. Also parameter for time constants should be set up
 +
 
 +
void ActivateLeaky(double step); // activates in leaky integrator mode
 +
 
 +
** Substrate can also be defined to made of Leaky neurons by
 +
 
 +
Substrate.m_leaky = True
 +
 
 +
* Evolution pipeline in MultiNEAT
 +
** Define parameters - for CTRNN, define timeconstants
 +
** Define substrates
 +
** Initialize Genome with intput, hidden, output
 +
** Init population
 +
** evaluate each indivisual from pop with interacting with openAI Gym env.
 +
** pop.Epoch()
 +
** To build HyperNEAT genome, specify by function Genome::BuildHyperNEATPheonotype() when making network
 +
 
 +
 
 +
* Each of risi's neuron has 4d coordinate, (xm, ym, x, y) where xm, and ym is substrate containing sub-substrates, this can be simply set up as 4d substrate when setting up in MultiNEAT
 +
 
 +
 
 +
==== What to look closely next ====
 +
 
 +
* How to set up input for CPPN in MultiNEAT - considering leg length??
 +
* Need to look into docker tutorial
=== Week 39 ===
=== Week 39 ===
 +
==== Weekly Goals ====
 +
 +
* Find out how to customize CPPN in MultiNEAT - Risi has extra input parameter for leg length
 +
* Document setting up dev. env process
 +
** Look into Docker generation
 +
* Try simple HyperNEAT on DyRET env. - doesn't have to take account on leg length
 +
 +
==== Progress ====
 +
===== Simple HyperNEAT on DyRET environment ====
 +
 +
* what is input for Risi's network??
 +
** Current angle of hip joints
 +
* output
 +
** axis of rotation for each joint scaled to DOF of each joint. proportional
 +
controller applies torque to bridge the disparity between current and requested angle
 +
 +
* MultiNEAT throws error when 4d substrate is given.
 +
** Will need to hack MultiNEAT for 4d substrate support, perhaps it's good idea considering lack of support for custom CPPN * Fitness function need to re-adapt dyret-env's reward
 +
** highest reward for doing nothing but standing still
 +
 +
* Fully connected substrate of 4 inputs 8 hidden layer and 12 output doesn't seem to do the job well with current fitness function - extra reward based on
 +
final y axis position * 1e5
 +
* At start the best genome usually falls towards positive y axis direction
 +
 +
===== Experiment result =====
 +
* Setup
 +
** population 128
 +
** 20 generations
 +
** 2D fully connected substrate with 4 input 8 hidden 12 output
 +
** input are 4 position of hip joints
 +
** output are used directly as next position of each joint
 +
** base reward from dyret walking env with half on healty and double speed on y axis + final distance on y axis
 +
** details at ~/workspace/code/dyret_hyperneat1.py
 +
 +
[[Fil:Dyret fc g20 p128.gif]]
 +
 +
* some periodic movement on some joint observed
 +
* kinda like galloping and failing miserably
 +
 +
 +
=== Week 40/41 ===
 +
Experiment setup
Experiment setup
Implementing Risi's HyperNEAT on Dyret
Implementing Risi's HyperNEAT on Dyret
-
=== Week 40 ===
+
==== Save/load for genomes ====
-
Experiment setup
+
 
-
Implementing CPG style locomotive controller on Dyret
+
Simply applying pickle to genome, substrate and paramters to see how the best genome behaves after training
 +
* Currently saving only the best fitting genome as pickling with it's fitness value
 +
* viewer script written to re-enact each genome to see how they behave
 +
* it need to be parameterized for more general usage
 +
 
 +
==== Custom CPPN in MultiNEAT ====
 +
 
 +
* Temporary neural network is initialized in the BuildHyperNEATPhenotype(). If we want to modifiy CPPN here it should be done.
 +
** modifying this part of code indicates that I need to recompile the MultiNEAT. how is this going to work?
 +
* Reading through genome.cpp indicates that 4D substrate is supported - line 1183 genome.cpp
 +
** Why is it not working with 4d substrate? is it perhaps not connected propery?
 +
 
 +
==== Sliding behaviour in Dyret ====
 +
 
 +
* After some trial run with the same initial setup in HyperNEAT, a population evolved a sliding behaviour, using only two back legs to slide off ground.
 +
* Discussion with Joergen revealed that it has to do with the fact that the dyret env package only explicitly sets friction value on its feet -> possibly no friction value set on other parts of body or very little friction.
 +
* Joergen suggests as solution
 +
** collision detection - check if other parts of body is touching the floor
 +
*** for example on this check out observation function in abstract env class
 +
** constraints on body angle, such that huge negative reward is given if pitch and roll is over certain degrees.
 +
 
 +
===== update =====
 +
 
 +
[[Fil:dyret_crawling_slow_mo.gif]]
 +
 
 +
Setting simulation timestep up, from default 1/240 to 0.1 reveals that dyret isn't actually sliding, but crawing really low.
 +
However this revealed some problems
 +
1. Colliding leg
 +
2. changing timestep of simulator did not affect the behaviour of the robot. This is strange given that I am suppose to be using CTRNN with certain timestep at 0.01.
 +
 
 +
==== Re-defined reward for upright ====
 +
 
 +
for x and y axis, being upright returns 0 for position.
 +
* Send x and y into normal PDF with sigma = 0.2. -> output around 0 for 45 degree
 +
* Normalize the max value to 1
 +
* Map value to -1 to 1
 +
* Sum x and y output then multiply by 100
 +
 
 +
This winds up giving 100 reward if upright and negative for more than 45 degree rotation
 +
 
 +
Result after modifying the reward function: [[Fil:dyret_side_walk.gif]]
 +
 
 +
**Note** All the parameter for HyperNEAT has been same so far, including substrates
 +
 
 +
I should probably give a dedicated reward for forward motion.
 +
 
 +
==== Dockerfile Generation ====
 +
 
 +
[[https://towardsdatascience.com/build-a-docker-container-with-your-machine-learning-model-3cf906f5e07e]]
 +
 
 +
This link shows how ML dev env can be set up in Docker
 +
 
 +
list of things to be installed
 +
 
 +
* OpenAI gym
 +
* usual scientific python packages - numpy, scipy ...
 +
* dyret gym env - pull from git repo
 +
* MultiNEAT - pull and compile from git
 +
* boost lib for MultiNEAT - this will have bunch of dependancies of its own
 +
 
-
=== Week 41 ===
+
=== Week 42-43 ===
Experiment setup
Experiment setup
Implementing CPG style locomotive controller on Dyret
Implementing CPG style locomotive controller on Dyret
-
=== Week 42 ===
+
 
 +
==== Docker ====
 +
* simple docker instance with openai gym and dyret env written
 +
* manages to clone git and build pybullet sim
 +
* fails to connect to x server for GUI
 +
 
 +
note: MultiNEAT has its own dockerfile in repo
 +
 
 +
=== Week 44 ===
Experiment setup
Experiment setup
Implementing CPG style locomotive controller on Dyret
Implementing CPG style locomotive controller on Dyret
-
=== Week 43 ===
+
==== TODO ====
-
Experiment run
+
* prepare MPI script for HPC runs
 +
* try implementing PUREPLES library
 +
** this could be easier to deploy on HPC
 +
* try adjusting some parameters.
 +
 
 +
* Perhaps time to have a meeting with Kyrre and check if things are going fine?
 +
 
 +
==== PUREPLES ====
 +
* Original package written for Python 2
 +
* need to modify to get it running in Python 3, in particular
 +
** cPickle -> _pickle
 +
** range() + range() should be changed to appending lists
 +
* Running ok with the modification above, but results are not promissing
-
=== Week 44 ===
 
-
Experiment run
 
=== Week 45 ===
=== Week 45 ===
Experiment run
Experiment run
 +
 +
==== Setting up running env on ABEL ====
 +
load python3 module
 +
 +
  $ module load python3/3.7.0.gnu
 +
 +
note. default version doesn't work
 +
 +
setup venv - only need once
 +
 +
  $ python3 -m venv %PATH_TO_VENV
 +
 +
Activate venv
 +
 +
  $ source %PATH_TO_VENV/activate
 +
 +
Install gym-dyret. first pull from git repo, then
 +
 +
  $ pip install -e .
 +
 +
Install dependancies for PUREPLES
 +
 +
  $ pip install neat-python graphviz gym
 +
 +
Install PUREPLES after cloning the repo - make sure that pureples is modified
 +
for python3
 +
 +
  $ python setup.py install
 +
 +
Install MPI4Py
 +
 +
  $ pip install mpi4py
 +
 +
 +
==== Abel slurm script ====
 +
* https://www.uio.no/english/services/it/research/hpc/abel/help/user-guide/job-scripts.html
 +
 +
TODO
 +
* Test running MPI script utilizing multiple nodes
 +
 +
==== MPI4PY implementation for HyperNEAT libs ====
 +
* Easier to install & run pure python libs(PUREPLES) as it doesn't come with compiler depandancies
 +
 +
TODO
 +
* Write HyperNEAT evolution script with MPI
 +
=== Week 46 ===
=== Week 46 ===
-
Experiment analysis
+
Experiment run
=== Week 47 ===
=== Week 47 ===
-
Experiment analysis
+
Experiment run
=== Week 48 ===
=== Week 48 ===
Line 105: Line 337:
=== Week 49 ===
=== Week 49 ===
Presenting first results
Presenting first results
 +
 +
=== Week 50 ===
 +
 +
 +
=== Week 3 ===
 +
Over the new years, stable gait has been evolved with risi's multilayered hyperneat with novelty search.
 +
Note that it is different implementation due to
 +
* input is sinusoid against time - risi uses upper angle position as input
 +
* novelty search is done by calculating correlation of each joint against everyone else
 +
==== Evolved gait for static controller ====
 +
[[Fil:static1.gif]]
 +
 +
==== Evolved gait for flexible controller ====
 +
[[Fil:var_leg.gif]]
 +
 +
=== Week 7 ===
 +
[[Fil:MOEA_behaviour_descriptor_vs_fitness_duty_ampl.png]]
 +
[[Fil:MOEA_behaviour_descriptor_vs_fitness_duty_pos.png]]
 +
[[Fil:MOEA_behaviour_descriptor_vs_fitness_pos_ampl.png]]
 +
[[Fil:Duty_factor_key_44784.png]]
 +
[[Fil:MOEA_fit_vs_gen.png‎]]

Current revision as of 09:46, 13 February 2020

Contents

Goals

We are taking the start point from the paper from Risi,

Risi - Evolving flexible controller for locomotion

where locomotive controller for variable length legges were evolved with HyperNEAT approach.

The goal is to implement it on DyRet platform - where it has two actuator for each legges. Also other thing to consider will be taking account of Tegotae - where touch input is used as some kind of feedback to CPG. Risi had touch sensor as input to his substrate in the simulation.

Other aspect of it is to see if HyperNEAT approach is something plausible considering its complexity - some skeptical veiw on HyperNEAT. ie) simple CTRNN network with length of leg as one of the input.

Some of the tasks that could be done over the summer are

  * Read through DyRet doucumentation from robin wiki & github and set up a
    simulator enviorment
  * Theoretical understanding of Tegotae - is it plausible to embed it with
    CTRNN-substrate?
  * Experimenting with HyperNEAT libraries - Kyrre`s recommendation is
    "C++/Python MultiNEAT C++ with Python binding", otherwise Risi seems to

work with C# implementation -> perhaps it is a good idea to have a look.

OpenAI Gym Env for DyRET

https://github.uio.no/jorgehn/gym-dyret

HyperNEAT libraries

TODO

Plan for Fall semester

                                    2019
      September         October           November          December          
  Su  1  8 15 22 29        6 13 20 27        3 10 17 24     1  8 15 22 29   
  Mo  2  9 16 23 30        7 14 21 28        4 11 18 25     2  9 16 23 30   
  Tu  3 10 17 24        1  8 15 22 29        5 12 19 26     3 10 17 24 31   
  We  4 11 18 25        2  9 16 23 30        6 13 20 27     4 11 18 25      
  Th  5 12 19 26        3 10 17 24 31        7 14 21 28     5 12 19 26      
  Fr  6 13 20 27        4 11 18 25        1  8 15 22 29     6 13 20 27      
  Sa  7 14 21 28        5 12 19 26        2  9 16 23 30     7 14 21 28      
     35 36 37 38 39    39 40 41 42 43    43 44 45 46 47    48 49 50 51 52   

Remarks

Delivery in May 2020 Mid-term presentation in week 49


Week 36

OpenAI gym setup for DyRET

Get used to DyRET Env

  • input param for step : 12 np vector for joints, 8 for extension

HyperNEAT libraries

Lists of some promising ones

binding multineat and gym env

    • installation
      • install boost, first bootstrap with python version 3.6 then build
      • git clone multineat then
  $ export MN_BUILD=boost
  $ python3 setup.py build_ext
  $ python3 setup.py install 
      • in case it casts missing library link to python and numpy, make sure to

install boost with python specified and numpy is installed properly for the user

      • if the user installing multineat has no access to write to install destination, try via venv.
      • once installed, test by
  >>> import MultiNEAT
      • in case it casts missing library error, explicitly set $LD_LIBRARY_PATH for

boost install location, by default

  $ export LD_LIBRARY_PATH=/usr/local/lib

Week 37

OpenAI gym setup

  • Running basic examples to get familier with OpenAI Gym concepts - observation space, action space

https://towardsdatascience.com/reinforcement-learning-with-openai-d445c2c687d2

HyperNEAT library test

  • MultiNEAT example running successfully with installation
    • Was able to run example from https://gist.github.com/stefanopalmieri
    • Those examples are from old openai gym code, it needs appropriate fix
    • wasn't able to create ES hyperneat object - Genome class doesn't have it!? -

commented out in source code


by this point, familir with hyperNEAT packages and chosen one for the project OpenAI gym env setup for various experiments

Week 38

Weekly goals

  • Find out how to implement CTRNN cell for HyperNEAT
  • Design control scheme for OpenAI gym DyRET using one of the HyperNEAT packages
  • Understand evolution pipeline in HyperNEAT packages together with OpenAI Gym env.
  • Document how to set up dev. environment - installation of openai, hyperneat

packages for the larger experiment jobs later on.

Progress

  • MultiNEAT has leaky integrators implemented. Following function for

NeuralNetwork obj. Also parameter for time constants should be set up

void ActivateLeaky(double step); // activates in leaky integrator mode
    • Substrate can also be defined to made of Leaky neurons by
Substrate.m_leaky = True
  • Evolution pipeline in MultiNEAT
    • Define parameters - for CTRNN, define timeconstants
    • Define substrates
    • Initialize Genome with intput, hidden, output
    • Init population
    • evaluate each indivisual from pop with interacting with openAI Gym env.
    • pop.Epoch()
    • To build HyperNEAT genome, specify by function Genome::BuildHyperNEATPheonotype() when making network


  • Each of risi's neuron has 4d coordinate, (xm, ym, x, y) where xm, and ym is substrate containing sub-substrates, this can be simply set up as 4d substrate when setting up in MultiNEAT


What to look closely next

  • How to set up input for CPPN in MultiNEAT - considering leg length??
  • Need to look into docker tutorial

Week 39

Weekly Goals

  • Find out how to customize CPPN in MultiNEAT - Risi has extra input parameter for leg length
  • Document setting up dev. env process
    • Look into Docker generation
  • Try simple HyperNEAT on DyRET env. - doesn't have to take account on leg length

Progress

= Simple HyperNEAT on DyRET environment

  • what is input for Risi's network??
    • Current angle of hip joints
  • output
    • axis of rotation for each joint scaled to DOF of each joint. proportional

controller applies torque to bridge the disparity between current and requested angle

  • MultiNEAT throws error when 4d substrate is given.
    • Will need to hack MultiNEAT for 4d substrate support, perhaps it's good idea considering lack of support for custom CPPN * Fitness function need to re-adapt dyret-env's reward
    • highest reward for doing nothing but standing still
  • Fully connected substrate of 4 inputs 8 hidden layer and 12 output doesn't seem to do the job well with current fitness function - extra reward based on

final y axis position * 1e5

  • At start the best genome usually falls towards positive y axis direction
Experiment result
  • Setup
    • population 128
    • 20 generations
    • 2D fully connected substrate with 4 input 8 hidden 12 output
    • input are 4 position of hip joints
    • output are used directly as next position of each joint
    • base reward from dyret walking env with half on healty and double speed on y axis + final distance on y axis
    • details at ~/workspace/code/dyret_hyperneat1.py

Fil:Dyret fc g20 p128.gif

  • some periodic movement on some joint observed
  • kinda like galloping and failing miserably


Week 40/41

Experiment setup Implementing Risi's HyperNEAT on Dyret

Save/load for genomes

Simply applying pickle to genome, substrate and paramters to see how the best genome behaves after training

  • Currently saving only the best fitting genome as pickling with it's fitness value
  • viewer script written to re-enact each genome to see how they behave
  • it need to be parameterized for more general usage

Custom CPPN in MultiNEAT

  • Temporary neural network is initialized in the BuildHyperNEATPhenotype(). If we want to modifiy CPPN here it should be done.
    • modifying this part of code indicates that I need to recompile the MultiNEAT. how is this going to work?
  • Reading through genome.cpp indicates that 4D substrate is supported - line 1183 genome.cpp
    • Why is it not working with 4d substrate? is it perhaps not connected propery?

Sliding behaviour in Dyret

  • After some trial run with the same initial setup in HyperNEAT, a population evolved a sliding behaviour, using only two back legs to slide off ground.
  • Discussion with Joergen revealed that it has to do with the fact that the dyret env package only explicitly sets friction value on its feet -> possibly no friction value set on other parts of body or very little friction.
  • Joergen suggests as solution
    • collision detection - check if other parts of body is touching the floor
      • for example on this check out observation function in abstract env class
    • constraints on body angle, such that huge negative reward is given if pitch and roll is over certain degrees.
update

Fil:dyret_crawling_slow_mo.gif

Setting simulation timestep up, from default 1/240 to 0.1 reveals that dyret isn't actually sliding, but crawing really low. However this revealed some problems 1. Colliding leg 2. changing timestep of simulator did not affect the behaviour of the robot. This is strange given that I am suppose to be using CTRNN with certain timestep at 0.01.

Re-defined reward for upright

for x and y axis, being upright returns 0 for position.

  • Send x and y into normal PDF with sigma = 0.2. -> output around 0 for 45 degree
  • Normalize the max value to 1
  • Map value to -1 to 1
  • Sum x and y output then multiply by 100

This winds up giving 100 reward if upright and negative for more than 45 degree rotation

Result after modifying the reward function: Fil:dyret_side_walk.gif

    • Note** All the parameter for HyperNEAT has been same so far, including substrates

I should probably give a dedicated reward for forward motion.

Dockerfile Generation

[[1]]

This link shows how ML dev env can be set up in Docker

list of things to be installed

  • OpenAI gym
  • usual scientific python packages - numpy, scipy ...
  • dyret gym env - pull from git repo
  • MultiNEAT - pull and compile from git
  • boost lib for MultiNEAT - this will have bunch of dependancies of its own


Week 42-43

Experiment setup Implementing CPG style locomotive controller on Dyret


Docker

  • simple docker instance with openai gym and dyret env written
  • manages to clone git and build pybullet sim
  • fails to connect to x server for GUI

note: MultiNEAT has its own dockerfile in repo

Week 44

Experiment setup Implementing CPG style locomotive controller on Dyret

TODO

  • prepare MPI script for HPC runs
  • try implementing PUREPLES library
    • this could be easier to deploy on HPC
  • try adjusting some parameters.
  • Perhaps time to have a meeting with Kyrre and check if things are going fine?

PUREPLES

  • Original package written for Python 2
  • need to modify to get it running in Python 3, in particular
    • cPickle -> _pickle
    • range() + range() should be changed to appending lists
  • Running ok with the modification above, but results are not promissing


Week 45

Experiment run

Setting up running env on ABEL

load python3 module

  $ module load python3/3.7.0.gnu

note. default version doesn't work

setup venv - only need once

  $ python3 -m venv %PATH_TO_VENV

Activate venv

  $ source %PATH_TO_VENV/activate

Install gym-dyret. first pull from git repo, then

  $ pip install -e .

Install dependancies for PUREPLES

  $ pip install neat-python graphviz gym

Install PUREPLES after cloning the repo - make sure that pureples is modified for python3

  $ python setup.py install

Install MPI4Py

  $ pip install mpi4py


Abel slurm script

TODO

  • Test running MPI script utilizing multiple nodes

MPI4PY implementation for HyperNEAT libs

  • Easier to install & run pure python libs(PUREPLES) as it doesn't come with compiler depandancies

TODO

  • Write HyperNEAT evolution script with MPI


Week 46

Experiment run

Week 47

Experiment run

Week 48

Experiment analysis

Week 49

Presenting first results

Week 50

Week 3

Over the new years, stable gait has been evolved with risi's multilayered hyperneat with novelty search. Note that it is different implementation due to

  • input is sinusoid against time - risi uses upper angle position as input
  • novelty search is done by calculating correlation of each joint against everyone else

Evolved gait for static controller

Fil:static1.gif

Evolved gait for flexible controller

Fil:var_leg.gif

Week 7

Fil:MOEA_behaviour_descriptor_vs_fitness_duty_ampl.png Fil:MOEA_behaviour_descriptor_vs_fitness_duty_pos.png Fil:MOEA_behaviour_descriptor_vs_fitness_pos_ampl.png Fil:Duty_factor_key_44784.png Fil:MOEA_fit_vs_gen.png‎

Personal tools
Front page