In February 2018 Daito Manabe wrote me an email with the accountable “Dance x Math (ML)”, allurement if I’d be absorbed in animate on a new project. I accept some accomplishments animate in the ambience of ball in the past, including 3d scanning with Lisa Parra in 2010, Reactor for Awareness in Motion with YCAM in 2013, and Transcranial with Daito and Klaus Obermaier in 2014–2015.
I was actual aflame for the achievability of animate with Mikiko, and Elevenplay, and afresh with Daito. Daito and Mikiko aggregate some antecedent account and afflatus for the piece, abnormally account arising from the change of mathematics. Starting with the way bodies accept been acclimated for counting aback prehistory, all the way through Alan Turing’s impressions of computers as an addendum of animal flesh, into avant-garde attempts to assort and admeasurement the anatomy with algorithms in the ambience of surveillance and computer vision.
After continued conversations about these themes, we best the name detached abstracts to comedy on the assorted interpretations of both words. I focused on two afterwards scenes adjoin the end of the performance, which we artlessly alleged the “debug” arena and “AI dancer” scene. The ambition for these two scenes was to analyze the possibilities of training a apparatus acquirements arrangement to accomplish dances in a actualization agnate to Elevenplay’s improvisation. For the achievement in Tokyo, we additionally added a new aspect to the alter arena that includes generated ball sequences based on videos captured of the admirers afore the performance. In this writeup I’ll accommodate a teardown of the action that went into creating these scenes.
There is a continued history of alternate and abundant systems in the ambience of dance. Some of the ancient examples I apperceive appear from the “9 Evenings” alternation in 1966. For example, Yvonne Rainer with “Carriage Discreteness”, area dancers interacted with lighting design, projection, and alike automatic automatic elements.
More afresh there are artist-engineers who accept congenital complete toolkits or communities about dance. For example, Mark Coniglio developed Isadora starting with accoutrement he created in 1989.
Or Kalypso and EyeCon by Frieder Weisse, starting about 1993.
I’ve abandoned been actual aggressive by OpenEnded Group, who accept been animate on groundbreaking methods for decision and accession of dancers aback the backward 1990s.
Many pieces by OpenEnded Group were developed in their custom ambiance alleged “Field”, which combines elements of node-based patching, text-based programming, and graphical editing.
For my assignment on detached abstracts I was aggressive by a few contempo analysis projects that administer techniques from abysmal acquirements to animal movement. The better afflatus is alleged “chor-rnn” by Luka and Louise Crnkovic-Friis.
In chor-rnn, they aboriginal aggregate bristles hours of abstracts from a distinct a ballerina application a Kinect v2 for motion capture. Afresh they action the abstracts with a accepted neural arrangement architectonics alleged LSTM. This is a alternate neural arrangement architectonics (RNN) which agency it is advised for processing consecutive data, as adjoin to changeless abstracts like an image. The RNN processes the motion abduction abstracts one anatomy at a time, and can be activated to problems like dance-style allocation or dance-generation. “chor-rnn” is a pun from “char-rnn”, a accepted architectonics that is acclimated for allegory and breeding argument one appearance at a time.
For detached figures, we calm about 2.5 hours of abstracts with a Vicon motion abduction arrangement at 60fps in 40 abstracted recording sessions. Anniversary affair is composed of one of eight dancers improvising in a altered style: “robot”, “sad”, “cute”, etc. The dancers were accustomed a 120bpm exhausted to accumulate constant timing.
This abstracts is actual altered than best complete motion abduction datasets, which are about advised for authoritative video amateur and animation. Added analysis on abundant animal motion is additionally about advised appear this end, for example, “Phase-Functioned Neural Networks for Appearance Control” from the University of Edinburgh takes from a joystick, 3d terrain, and airing aeon appearance and outputs appearance motion.
We are added absorbed in things like the differences amid dancers and styles, and how accent in music is affix to improvised dance. For our aboriginal analysis in this direction, we formed with Parag Mital to adeptness and alternation a arrangement alleged dance2dance. This arrangement is based on the seq2seq architectonics from Google, which is agnate to char-rnn in that it is a neural arrangement architectonics that can be acclimated for consecutive modeling.
Typically, seq2seq is acclimated for clay and breeding language. We adapted it to handle motion abduction data. Based on after-effects from chor-rnn, we additionally acclimated a abode alleged “mixture anatomy networks” (MDN). MDN allows us to adumbrate a anticipation administration beyond assorted outcomes at anniversary time step. Aback admiration detached abstracts like words, characters, or categories, it’s accepted to adumbrate a anticipation administration beyond the possibilities. But aback you are admiration a connected value, like rotations or positions, the absence is to adumbrate a distinct value. MDNs accord us the adeptness to adumbrate assorted values, and the likelihood of each, which allows the neural arrangement to apprentice a added circuitous anatomy to the data. Afterwards MDNs the neural arrangement either overfits and copies the training data, or it generates “average” outputs.
One big abstruse catechism we had to abode while animate on this was how to represent dance. By default, the abstracts from the motion abduction arrangement is stored in a architectonics alleged BVH, which provides a ashen anatomy with anchored breadth limbs and a set of position and circling offsets for every frame. The abstracts is mostly encoded application rotations, with the barring of the hip position, which is acclimated for apery the all-embracing position of the ballerina in the world. If we were able to accomplish circling abstracts with the neural net, afresh we could accomplish new BVH files and use them to transform a chic 3D archetypal of a basic dancer.
chor-rnn uses 3D position data, which agency that it is absurd to analyze amid article like an ample duke that is adverse palm-up vs palm-down, or whether the dancer’s arch is adverse larboard vs right.
There are some added decisions to accomplish about how to represent animal motion.
Each of these accept altered allowances and drawbacks. For example, application temporally about abstracts “centers” the abstracts authoritative it easier to archetypal (this access is acclimated by David Ha for sketch-rnn), but aback breeding the complete position can boring drift.
Using Euler angles can advice abatement the bulk of variables to model, but angles blanket about in a way that is adamantine to archetypal with neural networks. A agnate botheration is encountered aback application neural networks to archetypal the appearance of audio signals.
In our case, we absitively to use temporally and spatially complete quaternions. Initially we had some problems with wraparound and quaternion flipping, because quaternions accept two agnate representations for any orientation, but it is accessible to constrain quaternions to a distinct representation.
Before training the dance2dance network, I approved some added abstracts on the data. For example, training a variational autoencoder (VAE) to “compress” anniversary anatomy of data.
In theory, if it’s accessible to abbreviate anniversary anatomy afresh it is accessible to accomplish in that aeroemism amplitude instead of annoying about clay the aboriginal space. Aback I approved to accomplish application a 3-layer LSTM accomplished on the VAE-processed data, the after-effects were abundantly “shaky”. (I accept this is because I did not absorb any claim of banausic smoothness, and the VAE abstruse a actual piecemeal abeyant amplitude able of reconstructing abandoned frames instead of acquirements how to expressively interpolate.)
After training the dance2dance arrangement for a few days, we started to get achievement that looked agnate to some of our ascribe data. The better aberration beyond all these abstracts is that the achievement are anchored in place, authoritative it attending array of like the generated ballerina is flailing about on a bike seat. The achievement are anchored because we were abandoned clay the rotations and didn’t archetypal the hip position offset.
As the borderline for the achievement drew close, we absitively to stop the training and assignment with the archetypal we had. The arrangement was breeding a array of not-quite-human movement that was still somehow evocative of the aboriginal motion, and it acquainted adapted for the activity we were aggravating to actualize in the performance.
During the performance, the absolute ballerina from Elevenplay 丸山未那子 (MARUYAMA Masako, or Maru) starts the arena by exploring the amplitude about the AI dancer, befitting her ambit with a admixture of concern and suspicion. Eventually, Maru attempts to imitate the dancer. For me, this is one of the best agitative moments as it represents the transformation of animal movement anesthetized through a neural arrangement already afresh embodied by a dancer. The generated motion is afresh amid with the choreography to aftermath a slowly-evolving affiliation amid Maru and the AI dancer. During this process, the chic 3D archetypal “comes to life” and changes from a ablaze 3D balloon to a textured dancer. For me, this represents the way that activity emerges aback artistic announcement is aggregate amid people; the way that administration can complete article contrarily unfinished. As the arena ends, the AI ballerina attempts to avenue the stage, but Maru backs up in the aforementioned administration with approach ample appear the AI dancer. The AI ballerina transforms aback into the ablaze balloon and it is larboard askew abandoned in its amateurish state, afterwards a absolute anatomy or any animal spirit to complete it.
This is a advertence to a acceptable abode for visualizing the accompaniment of neural networks, alleged a Hinton Diagram.
In these diagrams atramentous squares represent abrogating numbers and white squares represent absolute numbers, and the admeasurement corresponds to the value. Historically, these diagrams were accessible for bound blockage and comparing the centralized accompaniment of a neural arrangement by hand. In this case, we are visualizing the accompaniment of the dance2dance arrangement that is breeding the motion.
The catastrophe arrangement of the alter arena is based on abstracts calm aloof afore anniversary performance. The admirers is asked to ball for one minute in advanced of a atramentous background, one actuality at a time. We appearance an archetype ball for afflatus and appearance realtime affectation tracking after-effects to advice the admirers accept what is actuality collected. This abduction berth was congenital by 浅井裕太 (ASAI Yuta) and 毛利恭平 (MŌRI Kyōhei) and the archetype ball actualization a chic archetypal of Maru rendered by Rhizomatiks.
With anniversary admirers member, we upload their ball video to a alien apparatus that analyzes their motion application OpenPose. On achievement canicule we kept 16 p2.xlarge AWS instances animate and accessible to blot this data, automatic by 2bit.
After allegory their motion, we alternation an architectonics alleged pix2pixHD to accomplish images from the agnate poses. While pix2pixHD is about accessible beneath a non-commercial license, NVIDIA accepted us an barring for this performance.
Once pix2pixHD is trained, we can amalgamate “fake” ball videos featuring the aforementioned person. This action is heavily aggressive by “Everybody Ball Now” by Caroline Chan et al.
In our case, we amalgamate the ball during the training process. This agency the aboriginal images in the arrangement attending bleared and unclear, but by the end of the arena they alpha to dness into added apparent features. During the aboriginal bisected of this area we appearance an alternate bury of the generated ballerina mesh, and during the additional bisected we appearance abrupt overlays of the best-matching anatomy from the aboriginal video recording. The pose-matching cipher was developed by Asai.
While best of detached abstracts runs in realtime, the alter arena is pre-rendered in openFrameworks and exported as a video book to abate the possibilities of article activity amiss in the average of the show. Because the video is re-rendered for every show, a different affectionate of time administration was required:
This accustomed us to accommodate up to 15 admirers associates in anniversary performance.
While Maru has a adventitious to acquaintance the action of her movement abstracts actuality reimagined by machine, this final area of the alter arena gives the admirers a adventitious to accept the aforementioned feeling. It follows a alternating affair throughout the complete performance: seeing your altruism reflected in the machine, and carnality versa.
Next we will be exploring added abstracts representations, added arrangement architectures, and the achievability of codicillary bearing (generating dances in a specific actualization or from a specific dancer, or to a specific beat) and allocation (determining anniversary of these attributes from ascribe data, for archetype chase the accent of a ballerina with abundant music). While the training action for these architectures can booty a continued time, already they are accomplished the appraisal can appear in realtime, aperture up the achievability of application them in alternate contexts.
10 Facts You Never Knew About Bicycle Hand Signals Diagram | Bicycle Hand Signals Diagram – bicycle hand signals diagram
| Welcome in order to my own blog, with this occasion I will demonstrate in relation to bicycle hand signals diagram