Abstract This paper presents a system for the real-time analysis of human skin temperatures using sensor fusion and thermal image recognition. The aim of this work is to introduce an open and extensible framework that supports multi-modal sensor input with a focus on merging optical data and conventional sensor input for advanced thermal comfort analysis. The goal is to obtain a more complete representation of a person in various indoor climatic conditions. Methods proposed in this paper are important for research and industrial applications with respect to the real-time analysis of thermal comfort and human physiology in indoor climates. Although this paper mainly focuses on the analysis of skin temperatures, the proposed architecture is conceived for being extendable for statistical evaluation and numerical models. Arbitrary software components can be integrated as data sources and sinks by means of a conventional TCP/IP networking interface. Main contributions of this paper are a general architecture for the fusion of multi-modal sensor input using a centralized data server structure, a method for combining depth-map based face and pose tracking with a thermal imaging device and preliminary studies demonstrating the behavior and validity of the system.