The research project and group

In this project, we have considered extensions of basic linear independent component analysis (ICA) and blind source separation (BSS) to nonlinear mixture models.

In the following, we first briefly describe in time order the most important results of the ICA and Bayes research groups of our research centre on nonlinear ICA and BSS. After that, we provide links to publications and useful web sites on nonlinear ICA and BSS. A list of these publications and web sites appears at the end of this page. Extended summaries of our results on nonlinear ICA and BSS can be found in the progress reports of our Bayes and ICA groups, which are available on the home pages [12], [13] of these groups.

Software on our Bayesian methods applicable to nonlinear BSS problems as well as more our publications related to nonlinear BSS and ICA can be found on the home pages of our Bayes group [12].

Researchers who have during the last years been active with respect to nonlinear ICA and BSS in the Adaptive Informatics Research Centre at the Department of Information and Computer Science, Aalto University Scool of Science and Technology (formerly Helsinki University of Technology), located in Espoo, Finland are

Juha Karhunen, Antti Honkela, Alexander Ilin, Tapani Raiko, and Erkki Oja.

People who have earlier carried out research on nonlinear ICA and BSS there are Harri Valpola, Aapo Hyvärinen, Markus Harva, Petteri Pajunen, Tomas Östman, and Leo Lundqvist.

Our early results

The most important early results of our ICA group deal with the existence and uniqueness of the solutions of the nonlinear ICA problem. They have been reported in the seminal theoretical paper [4], where A. Hyvärinen and P. Pajunen have shown that the nonlinear ICA problem is highly non-unique and ill-posed without additional regularization. More uniqueness results can be found in the reviews [1] and [2].

Our other early results on nonlinear ICA and BSS include a method based on the self-organizing map and a maximum likelihood based approach. They have been reported in Chapter 17 of the book [14] and the maximum likelihood method also in [8]. However, these methods are applicable to small-scale problems only because their computational load explodes with the dimensionality of the problem, and their accuracy is rather limited.

Bayesian methods

During the last years, we have successfully applied variational Bayesian learning (called in our earlier papers Bayesian ensemble learning) to nonlinear BSS and factor analysis. This methodology can be used to the selection, construction, and unsupervised (blind) learning of suitable latent variable models for nonlinear BSS. Basic methods for nonlinear factor analysis and ICA have been introduced in [5]. The simpler nonlinear factor analysis (NFA) method in [5] first finds a nonlinear PCA (principal component analysis) solution. A nonlinear BSS solution can then be obtained by applying standard linear ICA as a post-processing step to the subspace provided by NFA. The NIFA (nonlinear independent factor analysis) method is an extension of the NFA method which finds the nonlinearly mixed source signals from the NFA solution by continuing the learning process using a mixture-of-Gaussians model for the sources [5]. The paper [7] extends the method to blind identification of a nonlinear dynamic model, and our results on this line of research have been reviewed in [6] and [11].

The figure above shows 10 sources signals extracted from 30-dimensional real-world pulp data using the NFA method. This data set is clearly nonlinear, because it can approximated using the NFA method with much less components than using standard principal component analysis. See the paper [5] and Chapter 17 of our ICA book [14] for a more detailed description of these results.

Each subfigure of this figure shows the original time series (components) of the 30-dimensional pulp data on the top of each subfigure and below them their reconstruction based on the 10 source signals depicted in the previous figure.

The variational Bayesian methods [5]-[7] can be applied to considerably larger scale problems than our early methods, but even their computational load grows quite large especially for the nonlinear dynamic model introduced in [7]. To alleviate this problem, we have developed variational Bayesian methods based on standardized building blocks [10]. In these methods, all the computations can be carried out locally, resulting in linear computational complexity. A basic paper on the application this approach to the nonlinear BSS problem is [9], and the building block approach is described in detail in the long journal paper [10].

General information

A recent general review on nonlinear blind source separation and independent component analysis is the book chapter [1]. It is an updated version of the earlier review paper [2], which contains more references to older works. These two reviews [1] and [2] discuss the theoretical foundation of nonlinear ICA and BSS, especially their uniqueness issues. They also deal with the simpler problem of separating post-nonlinear mixtures and our variational Bayesian approach to nonlinear BSS. Both reviews contain an extensive list of references on nonlinear BSS and ICA. A recent short book on nonlinear blind source separation is [3]. It reviews the main methods, including in particular the MISEP method developed by the author L. Almeida.

Our book Independent Component Analysis [14] has become a standard reference in the field. It provides the necessary mathematical background to the reader, and deals extensively basic standard linear ICA as well as many of its extensions and a few applications. The handbook in [1] provides on a more advanced level for the researchers working in its fields an up-to-date reviews of currently used ICA and BSS methods as well as their most important extensions and applications.

Selected publications

[1] NEW! C. Jutten, M. Babaie-Zadeh, and J. Karhunen, "Nonlinear mixtures", Chapter 14, pp. 549-592, in P. Comon and C. Jutten (Eds.), Handbook of Blind Source Separation, Independent Component Analysis and Applications, Academic Press, 2010.
- A new review of nonlinear ICA and BSS, containing theoretical results, methods, and applications, with many references.

[2] C. Jutten and J. Karhunen, Advances in blind source separation (BSS) and independent component analysis (ICA) for nonlinear mixtures. International Journal of Neural Systems, Vol. 14, No. 5, 2004, pp. 267-292.
- Invited general review article on nonlinear blind source separation and independent component analysis containing many references.

[3] L. Almeida, Nonlinear Source Separation. Synthesis Lectures on Signal Processing, Morgan&Claypool Publishers, 2005, 114 pages.
- New concise book which reviews the main nonlinear blind separation methods, including the MISEP method developed by the author.

[4] A. Hyvärinen and P. Pajunen, Nonlinear independent component analysis: existence and uniqueness results. Neural Networks , Vol. 12, No. 3, 1999, pp. 429-439.
- Seminal paper on the existence and uniqueness of the solutions of the nonlinear ICA problem.

[5] H. Lappalainen and A. Honkela, Bayesian nonlinear independent component analysis by multilayer perceptrons. In M. Girolami (Ed.), Advances in Independent Component Analysis, pp. 93-121, Springer-Verlag, 2000.
- Basic paper on our first variational Bayesian (ensemble) learning method for nonlinear blind source separation.

[6] H. Valpola, E. Oja, A. Ilin, A. Honkela, and J. Karhunen, Nonlinear blind source separation by variational Bayesian learning. IEICE Transactions (Japan), Vol. E86-A, No. 3, March 2003, pp. 532-541.
- Invited paper which concisely reviews our static and dynamic nonlinear blind source separation methods based on variational Bayesian learning.

[7] H. Valpola and J. Karhunen, An unsupervised ensemble learning method for nonlinear dynamic state-space models. Neural Computation, Vol. 14, No. 11, 2002, pp. 2647-2692.
- Long and thorough paper which extends variational Bayesian learning to blind estimation of a nonlinear dynamic model for the source signals.

[8] J. Karhunen, Nonlinear ICA, in S. Roberts and R. Everson (Eds.) Independent Component Analysis: Principles and Practice, Cambridge Univ. Press, 2001, Chapter 4, pp. 113-134.
- A short review which discusses somewhat in more detail a maximum likelihood method and a variational Bayesian method. Chapter 17 of [14] is a longer version of this book chapter.

[9] H. Valpola, T. Östman, and J. Karhunen, Nonlinear independent factor analysis by hierarchical models, in Proc. of the 4th Int. Symp. on Independent Component Analysis and Blind Signal Separation (ICA2003), Nara, Japan, April 2003, pp. 257-262.
- A basic paper on the application of the Bayes building blocks approach to nonlinear blind source separation.

[10] T. Raiko, H. Valpola, M. Harva, and J. Karhunen, Building blocks for variational Bayesian learning of latent variable models, Journal of Machine Learning Research, Vol. 8, January 2007, pp. 155-201.
- A long journal paper where the Bayes building blocks approach is introduced and applied to the construction and variational Bayesian learning of several example structures.

[11] A. Honkela, H. Valpola, A. Ilin, and J. Karhunen, Blind Separation of Nonlinear Mixtures by Variational Bayesian Learning, Digital Signal Processing, Special issue on Bayesian Source Separation, Vol. 17, Issue 5, September 2007, pp. 914-934.
- A review paper on our Bayesian methods and results on nonlinear BSS.

Useful web sites

[12] Home page of the Bayes group in the Adaptive Informatics Research Centre of Aalto Univ. School of Science and Technology (formerly Helsinki University of Technology), Espoo, Finland. Publications and software related to nonlinear ICA and BSS.

[13] Home page of the ICA group in the Adaptive Informatics Research Centre of Aalto Univ. School of Science and Technology (formerly Helsinki University of Technology), Espoo, Finland. Useful information on many aspects of ICA and BSS and links.

[14] Home page of the book A. Hyvärinen, J. Karhunen, and E. Oja, Independent Component Analysis, Wiley 2001. Chapter 17 deals with nonlinear ICA and BSS.