In this forward-looking talk, we will dive into the innovative integration of vision language or multi modal models (VLMs/MMLMs) into robotics and how these advanced AI models can significantly improve the interaction and responsiveness of robots to their environment.

The core of this talk will focus on the potential and implementation of these LMs in robotics, in particular how models such as PaLM-e, PaLI or even Azure OpenAI GPT+Vision can be used to create robots that understand and respond to visual and verbal instructions in a more natural and intuitive way. An overview of existing solutions will be given and how these technologies can be effectively integrated into robotic systems will be discussed.

Thomas will talk about the concept and theory of Vision transforming LMs. He will show how you can acquire the knowledge and implement it. To this end, he has built a robot himself and will briefly describe his experiences to give participants an introduction.

A central topic of this lecture will be the construction and application of VLMs. Looking to the future, the talk will also cover the potential of local deployment of specialized versions of VLMs, which opens up new avenues for innovation in this area.

The audience will gain insights into the practical application of Azure AI services to develop intelligent, interactive robotic systems. The goal is to inspire and equip attendees with the knowledge to take advantage of VLMs in their own projects and bridge the gap between advanced AI and practical applications.

One can see AI becoming more and more integrated into our digital lives. But also a bit under the radar, AI is taking over our physical world - Tesla, BMW, Google, and many more are developing robots that can interact with human beings.
I will explain and show how a robot understands "Go to that door
over there (pointing with the finger)" by using a Language model and combined with image understanding. This talk is not mainstream and combines actual topics changing the world.