One Language for All Roles: How Multimodal AI Understands Construction Operations
by Lisa Lenz & Marco Meinerzhagen | October 5, 2026
Artificial intelligence is increasingly finding its way into a wide variety of industries and is transforming how information is processed, linked, and made usable. While traditional applications often focus on individual data types—such as plain text or isolated image data—the construction industry faces a unique challenge: Here, extremely heterogeneous data types encounter equally heterogeneous user roles, each of which requires the same information in very different ways.
This is where multimodal AI models and user-centered design approaches come into play. The basis for this is an article by Dr.-Ing. Lisa Theresa Lenz and Marco Meinerzhagen in the book *Agile Digitalization in Construction Operations*, which specifically addresses the technological foundations of multimodal systems as well as their user-centered adaptation to different roles in construction operations.
While previous posts on our blog have already demonstrated how digital verification mechanisms can improve quality assurance in the planning phase, this post focuses on the underlying technological and design principles: How can AI systems meaningfully integrate a wide variety of data formats, and how can we ensure that the information derived from them reaches each user in the appropriate form?
Why Multimodal AI and User-Centric Design Are So Crucial in Construction Operations
A construction project involves a highly complex interplay of a wide variety of stakeholders, processes, and data sources. Plans are available as raster or vector data, bills of quantities as tabular data sets, and daily construction reports, on the other hand, as a mix of continuous text, photos, and voice memos. This fragmented data landscape, which has evolved over time, is considered one of the major obstacles to end-to-end digital analysis in the construction industry.
If these data formats are viewed in isolation from one another, valuable information is lost: A photo in a daily construction report without reference to the accompanying text remains unused, and plan content without a link to the corresponding bill of quantities can lead to miscalculations regarding quantities and costs. Only by viewing multiple data modalities together in a semantically linked manner can the actual information be fully utilized.
Added to this is the diversity of the roles involved: construction management, subcontractors, cost estimation, and project management all require the same underlying information in completely different formats—ranging from aggregated cost and schedule overviews to detailed, trade-specific plan sections. Traditional, uniform user interfaces do not do justice to this diversity of tasks and information needs.
Lenz and Meinerzhagen address precisely this dual challenge—heterogeneous data and heterogeneous user roles—to derive concrete design principles for AI systems in construction operations.
Multimodal AI and User-Centered Design as the Foundation for New Approaches
Lenz and Meinerzhagen aimed to bring together technological and design perspectives: They analyzed not only how AI models can technically process different forms of data, but also how the resulting information can be presented in a user-centered and role-appropriate manner.
This analysis yields three central building blocks, which are examined in more detail below:
- the technological foundations of multimodal AI models,
- user-centered design as an established design principle, and
- AI profiling as the technical implementation of role-specific model behavior.
Each of these building blocks addresses a crucial aspect on the path toward AI systems that truly understand and support construction operations in all their heterogeneity.
Multimodal AI Models
Unimodal AI systems process only a single modality—such as plain text or isolated image data—and convert it into an output of the same or a different modality. Multimodal AI, by contrast, takes in information from multiple, structurally distinct modalities simultaneously, relates them semantically to one another, and uses them for comprehension or generation tasks. Current large-scale multimodal language models use a powerful language model as their central processing unit and extend it with image, audio, or video encoders.
The interaction of the modalities can be described as a three-stage process: First comes “seeing”—the perceptual capture of raw visual and auditory data. This is followed by “understanding”—the cross-modal semantic linking of the perceived information. The process concludes with “action”—the translation of this understanding into concrete outputs, such as a generated text, a response to a data query, or a recommendation for action.
It is crucial that text, images, speech, and structured data do not simply coexist in an additive manner, but rather mutually compensate for information gaps in each other’s modalities.
Construction operations illustrate this heterogeneity particularly well: plans, bills of quantities, and daily construction reports are each available in completely different data formats. Initial multimodal approaches are already addressing this issue—for example, by combining facade images with accompanying text for automated building classification—and at the same time demonstrate how heavily their success depends on the availability of high-quality, multidimensional data.
Multimodal AI models thus form the technological foundation for meaningfully integrating the fragmented data landscape of construction operations.
User-Centered Design as a Design Principle
The fundamental concept of user-centered design (UCD)—developing systems consistently based on the needs and tasks of their users—has been an integral part of human-computer interaction since the 1980s and is codified in the ISO 9241-210 standard, “Human-Centered Design of Interactive Systems.” The standard derives six principles from this: an explicit understanding of users, tasks, and the work environment; the active involvement of users throughout the entire development process; continuous, user-centered evaluation and optimization; an iterative approach; consideration of the overall user experience rather than individual functions; and an interdisciplinary development team.
In the construction industry, user-centered design takes on particular relevance because construction projects involve diverse roles, each with its own tasks and information needs. The comparatively low level of digitization in the construction industry is attributed, among other things, to an insufficient systematization of digital tools for involving these diverse stakeholder groups.
The same multimodal information—such as a construction plan or a daily construction report—must therefore be presented differently depending on the role: The construction management team needs up-to-date, location-specific information on construction progress; subcontractors need only the plan and specifications excerpts relevant to their trade; the cost estimation team needs structured specifications data for determining quantities and prices; and project management needs a cross-project view of deadlines, costs, and risks.
A uniform “one-size-fits-all” interface thus contradicts the principles of ISO 9241-210—and highlights why multimodal AI systems in construction operations must explicitly understand the user profile, task, and context of each role.
AI Profiling
AI profiling refers to the targeted adaptation of a language model’s behavior to the role, task, and project context of each user. Instead of behaving identically for all users, the system is assigned a role-specific profile. This profile determines the level of aggregation, language, and presentation format in which information is output—for example, as an aggregated status report for the client or as a detailed interpretation of plans for the construction management team. In this way, profiling technically implements what was previously described as the principle of user-centered design.
Pure prompting consists of individual, situational instructions within a conversation. Profiling differs from this in its persistence and systematic nature, as a profile does not merely affect a single request but consistently structures the model’s behavior across the entire usage period and all tasks associated with a role.
Profiling can be classified as a standalone approach between pure prompting and fine-tuning. It typically uses structured, predefined contextual information—such as role, task, and project data—which is systematically fed into the model without requiring any changes to the model parameters themselves.
Conclusion
Multimodal AI models and user-centered design are intertwined: It is their interaction—technically implemented through AI profiling—that transforms heterogeneous, scattered construction data into information that is actually usable for every role in the project.
This does not yet fully resolve the challenge. Heterogeneous data and heterogeneous user roles bring with them a multitude of additional facets that have so far been only partially explored. However, the approach taken so far clearly demonstrates the immense potential inherent in the close integration of technology and the user perspective.
Building on this foundation, we can take the concept further: Multimodal models and role-specific profiling can serve as the basis for concrete AI-supported workflows in the future, making processes in construction operations noticeably more efficient and transparent.
Discover planBIC, a groundbreaking Software-as-a-Service solution that enables automated completeness and quality checks of construction project documents as well as collaborative work in the cloud. A key component of planBIC is myCheck, the new verification engine for creating custom checks. This allows project-specific requirements to be integrated into the verification process in a structured manner and ensures that recurring checks are performed consistently.