The integration of uncertain power resources is causing more challenges for traditional load modeling research. Parameter identification of load modeling is impacted by a variety of load components with time-varying characteristics. This paper develops a deep learning-based time-varying parameter identification model for composite load modeling (CLM) with ZIP load and induction motor. A multi-modal long short-term memory (M-LSTM) deep learning method is used to estimate all the time-varying parameters of CLM considering system-wide measurements. It contains a multi-modal structure that makes use of different modalities of the input data to accurately estimate time-varying load parameters. An LSTM network with a flexible number of temporal states is defined to capture powerful temporal patterns from the load parameters and measurements time series. The extracted features are further fed to a shared representation layer to capture the joint representation of input time series data. This temporal representation is used in a linear regression model to estimate time-varying load parameters at the current time. Numerical simulations on the 23- and 68-bus systems verify the effectiveness and robustness of the proposed M-LSTM method. Also, the optimal lag values of parameters and measurements as input variables are solved.