User:Wonhol
From Robin
Contents |
Goals
We are taking the start point from the paper from Risi,
Risi - Evolving flexible controller for locomotion
where locomotive controller for variable length legges were evolved with HyperNEAT approach.
The goal is to implement it on DyRet platform - where it has two actuator for each legges. Also other thing to consider will be taking account of Tegotae - where touch input is used as some kind of feedback to CPG. Risi had touch sensor as input to his substrate in the simulation.
Other aspect of it is to see if HyperNEAT approach is something plausible considering its complexity - some skeptical veiw on HyperNEAT. ie) simple CTRNN network with length of leg as one of the input.
Some of the tasks that could be done over the summer are
* Read through DyRet doucumentation from robin wiki & github and set up a simulator enviorment * Theoretical understanding of Tegotae - is it plausible to embed it with CTRNN-substrate? * Experimenting with HyperNEAT libraries - Kyrre`s recommendation is "C++/Python MultiNEAT C++ with Python binding", otherwise Risi seems to
work with C# implementation -> perhaps it is a good idea to have a look.
OpenAI Gym Env for DyRET
https://github.uio.no/jorgehn/gym-dyret
HyperNEAT libraries
TODO
Plan for Fall semester
2019 September October November December Su 1 8 15 22 29 6 13 20 27 3 10 17 24 1 8 15 22 29 Mo 2 9 16 23 30 7 14 21 28 4 11 18 25 2 9 16 23 30 Tu 3 10 17 24 1 8 15 22 29 5 12 19 26 3 10 17 24 31 We 4 11 18 25 2 9 16 23 30 6 13 20 27 4 11 18 25 Th 5 12 19 26 3 10 17 24 31 7 14 21 28 5 12 19 26 Fr 6 13 20 27 4 11 18 25 1 8 15 22 29 6 13 20 27 Sa 7 14 21 28 5 12 19 26 2 9 16 23 30 7 14 21 28 35 36 37 38 39 39 40 41 42 43 43 44 45 46 47 48 49 50 51 52
Remarks
Delivery in May 2020 Mid-term presentation in week 49
Week 36
OpenAI gym setup for DyRET
Get used to DyRET Env
- input param for step : 12 np vector for joints, 8 for extension
HyperNEAT libraries
Lists of some promising ones
- https://github.com/ukuleleplayer/pureples
- Pure python-based HyperNEAT, ES-HyperNEAT library, based on neat-python library
- Installed and provided example experiments runs fine
- https://github.com/peter-ch/MultiNEAT
- MultiNEAT, implemented in C++ with python bindings
- good review from Stanley's website
- https://gist.github.com/stefanopalmieri This guy has some nice examples
binding multineat and gym env
- installation
- install boost, first bootstrap with python version 3.6 then build
- git clone multineat then
- installation
$ export MN_BUILD=boost $ python3 setup.py build_ext $ python3 setup.py install
- in case it casts missing library link to python and numpy, make sure to
install boost with python specified and numpy is installed properly for the user
- if the user installing multineat has no access to write to install destination, try via venv.
- once installed, test by
>>> import MultiNEAT
- in case it casts missing library error, explicitly set $LD_LIBRARY_PATH for
boost install location, by default
$ export LD_LIBRARY_PATH=/usr/local/lib
Week 37
OpenAI gym setup
- Running basic examples to get familier with OpenAI Gym concepts - observation space, action space
https://towardsdatascience.com/reinforcement-learning-with-openai-d445c2c687d2
HyperNEAT library test
- MultiNEAT example running successfully with installation
- Was able to run example from https://gist.github.com/stefanopalmieri
- Those examples are from old openai gym code, it needs appropriate fix
- wasn't able to create ES hyperneat object - Genome class doesn't have it!? -
commented out in source code
by this point, familir with hyperNEAT packages and chosen one for the project
OpenAI gym env setup for various experiments
Week 38
Weekly goals
- Find out how to implement CTRNN cell for HyperNEAT
- Design control scheme for OpenAI gym DyRET using one of the HyperNEAT packages
- Understand evolution pipeline in HyperNEAT packages together with OpenAI Gym env.
- Document how to set up dev. environment - installation of openai, hyperneat
packages for the larger experiment jobs later on.
Progress
- MultiNEAT has leaky integrators implemented. Following function for
NeuralNetwork obj. Also parameter for time constants should be set up
void ActivateLeaky(double step); // activates in leaky integrator mode
- Substrate can also be defined to made of Leaky neurons by
Substrate.m_leaky = True
- Evolution pipeline in MultiNEAT
- Define parameters - for CTRNN, define timeconstants
- Define substrates
- Initialize Genome with intput, hidden, output
- Init population
- evaluate each indivisual from pop with interacting with openAI Gym env.
- pop.Epoch()
- To build HyperNEAT genome, specify by function Genome::BuildHyperNEATPheonotype() when making network
- Each of risi's neuron has 4d coordinate, (xm, ym, x, y) where xm, and ym is substrate containing sub-substrates, this can be simply set up as 4d substrate when setting up in MultiNEAT
What to look closely next
- How to set up input for CPPN in MultiNEAT - considering leg length??
- Need to look into docker tutorial
Week 39
Weekly Goals
- Find out how to customize CPPN in MultiNEAT - Risi has extra input parameter for leg length
- Document setting up dev. env process
- Look into Docker generation
- Try simple HyperNEAT on DyRET env. - doesn't have to take account on leg length
Progress
= Simple HyperNEAT on DyRET environment
- what is input for Risi's network??
- Current angle of hip joints
- output
- axis of rotation for each joint scaled to DOF of each joint. proportional
controller applies torque to bridge the disparity between current and requested angle
- MultiNEAT throws error when 4d substrate is given.
- Will need to hack MultiNEAT for 4d substrate support, perhaps it's good idea considering lack of support for custom CPPN * Fitness function need to re-adapt dyret-env's reward
- highest reward for doing nothing but standing still
- Fully connected substrate of 4 inputs 8 hidden layer and 12 output doesn't seem to do the job well with current fitness function - extra reward based on
final y axis position * 1e5
- At start the best genome usually falls towards positive y axis direction
Experiment result
- Setup
- population 128
- 20 generations
- 2D fully connected substrate with 4 input 8 hidden 12 output
- input are 4 position of hip joints
- output are used directly as next position of each joint
- base reward from dyret walking env with half on healty and double speed on y axis + final distance on y axis
- details at ~/workspace/code/dyret_hyperneat1.py
- some periodic movement on some joint observed
- kinda like galloping and failing miserably
Week 40
Experiment setup Implementing Risi's HyperNEAT on Dyret
Week 41
Experiment setup Implementing CPG style locomotive controller on Dyret
Week 42
Experiment setup Implementing CPG style locomotive controller on Dyret
Week 43
Experiment setup Implementing CPG style locomotive controller on Dyret
Week 44
Experiment run
Week 45
Experiment run
Week 46
Experiment run
Week 47
Experiment analysis
Week 48
Experiment analysis
Week 49
Presenting first results