Interview: Dmytro Korolov on the results of SIGGRAPH’s AI tools for VFX panel

In this latest SIGGRAPH, Deep Learning was a hot topic. Many papers relied on it, some sessions were all about it. Dmytro Korolov, an experienced Pipeline Technical Director working at MPC, moderated a panel on the topic alongside Doug Roble (Digital Domain), Rob Pieke (MPC), Renaldas Zioma (Unity), Jeff Kember (Google), and David Luebke (Nvidia).

The panel discussed future trends of AI tools in the VFX pipeline. The session was not made public for those that did not attend the event, thankfully Dmytro was kind enough to talk to us and give his impressions on what was discussed on the panel. Here is what he had to say.

Hi Mr. Dmytro, firstly congratulations on the organization of your SIGGRAPH panel. Can you tell us a bit about your background in the VFX industry, how you first learned about Deep Learning and why you got interested in it?

I am a Technical Director, working in the VFX film industry. I have a robust engineering background (Master’s degree in Computer Science) and 20 years of artistic and TD expertise. I started my carrier in Ukraine, where I worked for a long time, then in Europe, and currently in Canada. I’m always interested in new technologies, so I started to learn about Deep Leaning in 2016. As I went deeper into it, I realized the huge possibilities of using this technology for VFX.  Because of this, in 2017 I created a LinkedIn group called “Deep Learning and Artificial Intelligence for the Visual Effects industry.” The subject attracted the curiosity of a skilled group of people. The group has reached 1600 participants of various backgrounds, and it’s the largest group on the subject of AI applied to Visual Effects.

The panel you have organized was represented by people in the VFX industry, software vendors, infrastructure providers, and academics. How do these players see the future of Deep Learning in VFX production? Which role each of them plays in this future?

The goal of the panel was to get the big picture of what we can expect in the future from Deep Learning tools for VFX. I can say that there is no consensus on this issue, but there are some common trends.

We decided to move away from the standard panel where speakers make unrelated presentations. So, we tried to build a more concise story, and focus on three themes:

1. Building VFX tools using Machine Learning (ML). What’s going on right now and examples of how it is done.
2. The problems of using machine learning in VFX and what is being developed to address these issues.
3. The future of Machine Learning in VFX. How will Machine Learning impact VFX in the next 5 years?

When I invited panelists, my goal was to cover these different areas, and it was successful.

“[Large tech companies] are the leaders of Deep Learning. VFX companies should follow AI trends and develop their own solutions based on them.”

The most interesting inventions and breakthroughs in Machine Learning come from the academic environment. They are supported by computing resources and data provided by the large technology companies. These are the leaders of Deep Learning. VFX companies should follow AI trends and develop their own solutions based on them. The VFX industry is unlikely to make foundational inventions. Instead, we need to learn how to adapt and use the inventions of these leaders.

On the other hand, the VFX industry can offer leaders ready-made pipelines for data processing and data preparation. For example, I was pleased to find out at the conference that researchers from Facebook have already used Houdini to generate a dataset of 3D renders composed of 100K images. Therefore, I am sure that the cooperation between VFX companies, scientific institutes, and technology leaders is mutually beneficial. We need to deepen this cooperation.

After hearing from all these prominent players, what do you think are some of the low-hanging fruits in Deep Learning for VFX? Where do you think we will start to see research turning into products first?

This year we have a record number of technical papers related to AI. Information about them is collected in my article http://www.aivfx.org/siggraph2018/, most of the research relates to:

1. Animation (face animation, human an animal animation)
2. Rendering
3. 2d image processing
4. 2d Image generation
5. Textures
6. 3d materials
7. 3d shape analysis
8. Dynamics simulation

Do not forget about areas that are not considered the main ones in VFX – production and render-farm scheduling and resource allocation, audio processing, video labeling… Perhaps you will be surprised, but in these areas, there is already a large number of ready-made solutions.

And in the long run? Based on what you’ve heard in the panel, can you make a prediction of what the pipeline of the future should look like?

The panel’s task was to collect all available information and provide a big picture. I still cannot say yet in all detail how the pipeline of the future will look. But I know for certain that this is exactly the area where I personally want to focus in the next years. The problem now is that machine learning is not in the VFX pipeline yet. Implementation of Deep Learning tools is rather difficult even if we have open source code from the researchers. We need to make it easier to develop AI tools and make deep learning tools more accessible to the VFX industry.

In my personal opinion, this is what we need to do:

• We need to integrate Deep Learning libraries as part of the core VFX pipeline
• We need a pipeline to automatically collect and process data for Machine Learning tools
• Computer power is important for Deep Learning. Cloud computing is one possible During the panel, Jeff Kember from Google focused on the machine learning cloud solutions that can be useful for VFX pipelines.
• AI tools themselves should be more flexible and adjustable than they currently are. In the VFX industry the “right answer” does not exist for most of the tasks. Creative tasks need to be controllable and adjustable. Tools should be ready for this.
• We need to have a core amount of technical people and artists who are prepared to use and adjust these technologies in their own workplaces.

Usually, pipelines change slowly. In the VFX industry, the average time for pipeline renewal is about 10 years. Because of this, I expect an evolution of Machine Learning tools for VFX rather than a revolution. But this is only my personal opinion.

In the VFX industry, the average time for pipeline renewal is about 10 years. Because of this, I expect an evolution of Machine Learning tools for VFX rather than a revolution

For me, it’s a very good start to be aware of the hype around Deep Learning and systematically work on its implementation. In this case, results can be much more fantastic than we can expect.

In the more mainstream discussion about AI, the loss of jobs to computers seems like a recurrent concern. Do you see this as a real threat in VFX? What jobs might be impacted most by Machine Learning, and in which way?

This was a big question that many people asked before the conference – Will AI replace artists in VFX?

I know that in some specific operations Machine Learning tools can achieve better performance than artists, but this is not a common situation. Machine Learning tools should be used by humans.

We do not expect Machine Learning tools to replace artists in the near future.

We do not expect Machine Learning tools to replace artists in the near future.  At least not while we are still making films for humans. Artists can use AI tools to reduce routine tasks to focus more on the creative tasks. This is the current situation.

But it will be a big mistake to ignore Machine Learning technologies or to treat Machine Learning as a type of hype. It already is part of VFX. For example, it is a native part of hardware accelerated Raytracing technology, nobody in the industry can ignore this. But we expect an evolution with AI tools, not a revolution. I already talked about some of the technical problems of using Machine Learning in VFX. But often the issue is non-technical, so it’s less an issue of machine learning as a field, and more an issue of practical adoption.  Artists don’t trust automatic systems; they do not like AI in a “black box” with one-button controls.

The most successful tools will be those that work together with artists, almost like an invisible extra pair of hands

Benefits from Machine Learning will be perceived by people who use this tool as a creative partner. The most successful tools will be those that work together with artists, almost like an invisible extra pair of hands. Because of this, it is good for artists who have technical skills to be more knowledgeable of Machine Learning technology. It will benefit them in the near future.

In today’s world, it is not those who know more or have more skills that are in advantage. On the contrary, it is those who can quickly and effectively change and adapt. Therefore, these are the people who will get big benefits when we have a lot of Machine Learning tools in use. The culture of using tools is slow to change.

I intend to help such people in training, and access to information and development. This is one of the goals of creating the Linkedin group and of other future projects. I hope these can help people better adapt to future change – whether it will be an evolution or a revolution.

You run the most popular Linkedin Group on Machine Learning for VFX. Who do you think are the leaders of Deep Learning amongst VFX companies and why?

I think it’s still too early to talk about leaders in this field. We will start seeing leaders when someone receives a technical Oscar for development associated with Machine Learning. Now, we are still at an early stage. I can only talk about personal preferences. I think NVIDIA is making a great contribution to Machine Learning technology. VFX companies themselves are not leaders in this field. We stand on the shoulders of technological giants like NVIDIA, Google, Microsoft, Amazon, Facebook.

We stand on the shoulders of technological giants like NVIDIA, Google, Microsoft, Amazon, Facebook.

Speakers on the panel have shown real cases including the digital human animation project “Digital Doug” from Doug Roble (Digital Domain) – which was briefly described in this FXGuide article. Dr. David Luebke (Vice President NVIDIA Research) talked about Nvidia solutions, especially about render denoising and synthetic face generation. Before the conference SIGGRAPH2018, NVIDIA together with Doug Roble conducted a webinar on the use of AI in VFX. This is a good source of information for people who could not attend the conference, and it is still available on request – http://info.nvidia.com/digital-domain-ai-media-entertainment-reg-page.html. I am impressed with the Noise2Noise solutions from NVIDIA; when using only “noisy” data, it was possible to train the neutron network to generate  “clean” pictures. The result of the network is superior to any of the pictures in the data set.

Also amongst my personal favorites were the work on Neural Networks for Quadruped Motion Control – a system that does not require labels for the phase or locomotion gaits, but can learn from unstructured motion capture data (since it is impossible to ask animals to reproduce a specific motion phase and make motion more structured), an the work on AI texture generation/expansions – which allows one to generate large non-uniform textures from a tiny sample. Again, this SIGGRAPH showed us a lot of great Machine Learning solutions, I have listed only a very small part.

Any final remarks, links or materials you would like to point people to?

For people who want to get a first contact with Deep Learning I want to suggest the “Deep Learning crash course” presented during SIGGRAPH 2018. The course does not contain math at all but talks about the basic concepts of Deep Learning. Thanks to the conference organizers we have a record of it on YouTube: https://www.youtube.com/watch?v=r0Ogt-q956I .

As a moderator I invite people who are interested in this technology to join the LinkedIn group “Deep Learning and Artificial Intelligence for Visual Effects (VFX) industry” https://www.linkedin.com/groups/13539761/. It is a continuous source of useful information about Machine Learning in VFX.

I want to say thank you to Gustavo Boehs for organizing this interview and to wish him good luck on his projects.