Call 1 - Advancing Large AI Models

Call 1 - Advancing Large AI Models: Integration of New Data Modalities and Expansion of Capabilities (AI, Data and Robotics Partnership) (RIA) HORIZON-CL4-2024-HUMAN-03-01

Expected Outcome:

Projects are expected to contribute to one or more of the following outcomes:

Enhanced applicability of large AI systems to new domains through the integration of innovative data modalities, such as sensor measurements (e.g. in robotics, IoT) or remote sensing (e.g. earth observation), as input.

Improvement of current multimodal large AI systems capabilities and expansion of the number of data modalities jointly handed by one AI system, leading to broader application potential and improved AI performance.

Scope:

Large artificial intelligence (AI) models refer to a new generation of general-purpose AI models (i.e., generative AI) capable of adapting to diverse domains and tasks without significant modification. Notable examples, such as OpenAI's GPT-4V and META’s Llama 2 or DinoV2, have demonstrated a wide and growing variety of capabilities.

The swift progression of large AI models in recent years holds immense potential to revolutionize various industries, due to their ability to adapt to diverse tasks and domains. For them to achieve their potential, access to vast data repositories, significant computing resources, and skilled engineers is required. A promising avenue of research is the development of multi-modal large AI models that can seamlessly integrate multiple modalities, including text, structured data, computer code, visual or audio media, robotics or IoT sensors, and remote sensing data.

This topic centres around the development of innovative multimodal large AI models, covering both the training of foundation models and their subsequent fine-tuning. These models should show superior capabilities across a wide array of down-stream tasks. The emphasis is both on integrating new input data modalities into large AI models and on developing multimodal large AI models with either significantly higher capabilities and/or the ability to handle a greater number of modalities.

Proposals should address at least one of the following focus areas:

the integration of innovative modalities of data for large AI models during training and inference. Examples of innovative modalities include event streams, structured data and sensor measurements. The incorporation of such new modalities could potentially bring unforeseen enhancements to model performance and enable their application in new domains like weather forecasting, robotics, and manufacturing.

enhanced multimodal models that exceed the current state of the art, with either significantly improved capabilities or the ability to handle a larger number of modalities. This focus area also encompasses models capable of multi-modal output generation. Current large-scale multimodal models most commonly engage with only vision and language.

For more information on this call, please visit the Horizon Europe website: