Source Separation and Machine Learning

Figures......Page 3
Tables......Page 8
Foreword......Page 9
Preface......Page 11
Notation......Page 12
1.1.1 Multichannel Source Separation......Page 18
 Underdetermined System: n1.1.2 Monaural Source Separation......Page 21
1.1.3 Reverberant Source Separation......Page 23
1.2.1 Applications for Source Separation......Page 27
 Music Separation......Page 28
 Biomedical Source Separation......Page 29
 Microphone Array Signal Processing......Page 30
 Unknown Number of Sources......Page 31
1.2.3 Front-End Processing and Back-End Learning......Page 32
1.3 Overview of This Book......Page 34
2 Model-based Source Separation......Page 36
2.1 Independent Component Analysis......Page 37
2.1.1 Learning Procedure......Page 38
2.2 Nonnegative Matrix Factorization......Page 40
2.2.1 Learning Procedure......Page 41
 Squared Euclidean Distance......Page 43
 Kullback-Leibler Divergence......Page 44
 β Divergence......Page 45
2.2.3 Sparse Regularization......Page 46
2.3 Nonnegative Tensor Factorization......Page 47
2.3.1 Tucker Decomposition......Page 49
2.3.2 CP Decomposition......Page 51
2.4 Deep Neural Network......Page 52
2.4.1 Error Backpropagation Algorithm......Page 54
2.4.2 Practices in Deep Learning......Page 56
2.5.1 Backpropagation Through Time......Page 60
2.5.2 Deep Recurrent Neural Network......Page 63
2.5.3 Long Short-Term Memory......Page 64
2.6 Summary......Page 67
Regularization & Generalization......Page 68
3.2 Information-Theoretic Learning......Page 69
3.2.1 Divergence Measure......Page 70
3.3.1 Model Regularization......Page 72
3.3.2 Bayesian Source Separation......Page 74
3.4 Sparse Learning......Page 75
3.4.1 Sparse Bayesian Learning......Page 77
3.5 Online Learning......Page 80
3.5.1 Online Bayesian Learning......Page 82
3.6.1 Discriminative NMF......Page 83
 DNMF-I......Page 84
 DNMF-II......Page 85
 DNMF-III......Page 86
3.6.2 Discriminative DNN......Page 88
3.7 Approximate Inference......Page 90
 Maximum Likelihood Estimation......Page 91
 EM Algorithm......Page 92
 Alternative View of EM......Page 94
 ML Solution to NMF......Page 96
 EM Algorithm......Page 98
3.7.3 Variational Inference......Page 100
 Joint Posterior Distribution......Page 101
 Factorized Posterior Distribution......Page 102
 VB-EM Algorithm......Page 105
3.7.4 Sampling Methods......Page 106
 Markov Chain Monte Carlo......Page 107
 Metropolis-Hastings Algorithm......Page 108
 Gibbs Sampling......Page 109
3.8 Summary......Page 111
4.1 ICA for Speech Recognition......Page 112
4.1.1 Construction of Independence Space......Page 113
4.1.2 Adaptation in Independent Space......Page 115
4.1.3 System Evaluation......Page 116
4.2.1 ICA Objective Function......Page 119
 Maximum Likelihood......Page 120
 Minimum Mutual Information......Page 121
4.2.2 Hypothesis Test for Independence......Page 122
4.2.3 Nonparametric Likelihood Ratio......Page 123
4.2.4 Application and Evaluation......Page 126
 Model Clustering......Page 127
 Signal Separation......Page 130
4.3.1 Convex Divergence......Page 131
4.3.2 Divergence Measure for NMF......Page 136
4.3.3 ICA Procedure......Page 137
4.3.4 Simulated Experiments......Page 140
4.3.5 Real-World Experiments......Page 142
4.4 Nonstationary Bayesian ICA......Page 145
4.4.1 Sequential and Noisy ICA......Page 147
4.4.2 Automatic Relevance Determination......Page 149
4.4.3 Sequential and Variational Learning......Page 151
4.4.4 System Evaluation......Page 155
4.5 Online Gaussian Process ICA......Page 158
4.5.1 Temporal Structure for Separation......Page 159
4.5.2 Online Learning and Gaussian Process......Page 160
4.5.3 Sequential and Variational Learning......Page 164
4.5.4 Sequential Monte Carlo ICA......Page 166
4.5.5 System Evaluation......Page 168
4.6 Summary......Page 171
5.1 Convolutive NMF......Page 174
5.1.1 Nonnegative Matrix Factor Deconvolution......Page 175
5.1.2 Speech Dereverberation Model......Page 178
5.1.3 Bayesian Speech Dereverberation......Page 179
 Variational Inference......Page 181
 Optimization Procedure......Page 183
5.1.4 System Evaluation......Page 184
5.2.1 Probabilistic Latent Component Analysis......Page 187
5.2.2 Shift-Invariant PLCA......Page 189
 Shift-Invariant 1-D PLCA......Page 190
 Shift-Invariant 2-D PLCA......Page 191
5.2.3 PLCA Versus NMF......Page 192
5.2.4 Interpretation and Application......Page 194
5.3 Bayesian NMF......Page 195
5.3.1 Gaussian-Exponential Bayesian NMF......Page 196
 Optimization Criteria......Page 199
 Variational Bayesian Learning......Page 200
5.3.3 Poisson-Exponential Bayesian NMF......Page 203
 Variational Expectation Step......Page 204
 Variational Maximization Step......Page 206
 Properties of Bayesian NMFs......Page 208
5.3.4 Evaluation for Supervised Separation......Page 209
5.3.5 Evaluation for Unsupervised Separation......Page 212
5.4.1 Group Basis Representation......Page 215
5.4.2 Bayesian Group Sparse Learning......Page 218
 Sparse Prior......Page 219
 Learning Objective......Page 221
5.4.3 Markov Chain Monte Carlo Sampling......Page 222
5.4.4 System Evaluation......Page 229
5.5 Deep Layered NMF......Page 231
5.5.1 Deep Structural Factorization......Page 232
5.5.2 Layered NMF......Page 234
5.5.3 Discriminative Layered NMF......Page 236
5.5.4 System Evaluation......Page 240
5.6 Summary......Page 241
6 Nonnegative Tensor Factorization......Page 243
6.1.1 Modulation Spectrograms......Page 244
6.1.2 Multiresolution Spectrograms......Page 246
6.2.1 Nonnegative Tensor Factor Deconvolution......Page 249
 Squared Euclidean Distance......Page 250
 Kullback-Leibler Divergence......Page 251
 Kullback-Leibler Divergence......Page 252
6.3.1 Probabilistic Matrix Factorization......Page 254
6.3.2 Probabilistic Tensor Factorization......Page 256
 Hierarchical Bayesian Model......Page 258
 Bayesian Inference......Page 259
6.4.1 Hierarchical Bayesian Model......Page 261
6.4.2 Bayesian Learning Algorithm......Page 262
6.5.1 Positive Semidefinite Tensor Factorization......Page 263
6.5.2 Gamma Process......Page 266
6.5.3 Variational Inference......Page 267
6.5.4 System Evaluation......Page 268
6.6 Summary......Page 269
7.1 Deep Machine Learning......Page 270
7.1.1 Deep Spectral Masking......Page 271
 DNN Classifier for Separation......Page 273
 High-Resolution DNN......Page 274
7.1.2 Deep Ensemble Learning......Page 276
 Multicontext Stacking......Page 277
7.1.3 Deep Speech Segregation......Page 279
7.2.1 Deep Clustering......Page 281
 Learning for Embedding......Page 282
 System Evaluation......Page 283
7.2.2 Masking and Learning......Page 285
 Training Criterion......Page 286
 System Evaluation......Page 288
 Deep Learning and Dereverberation......Page 289
 Factorized Neural Network......Page 291
 Factorized Error Backpropagation......Page 292
 System Implementation......Page 293
7.3.1 Joint Optimization and Mask Function......Page 295
 Discriminative Training Criterion......Page 296
 Physical Interpretation......Page 297
 New Training Criterion......Page 298
 Optimization Procedure......Page 299
7.3.3 System Evaluation......Page 301
7.4.1 Physical Interpretation......Page 302
7.4.2 Speaker Generalization......Page 304
7.4.3 Bidirectional Long Short-Term Memory......Page 306
 Model Construction......Page 307
 Modern Sequential Learning......Page 309
7.5.2 Model Construction and Inference......Page 310
7.5.3 System Evaluation......Page 315
7.6.1 Memory Augmented Source Separation......Page 318
 Controller Network......Page 320
 Addressing Mechanism......Page 321
 Reading and Writing Mechanism......Page 322
7.6.3 System Evaluation......Page 323
7.7 End-to-End Memory Network......Page 324
7.7.1 Recall Neural Network......Page 325
 Encoder-Decoder Network......Page 327
 Gated Attention Mechanism......Page 328
7.7.3 System Evaluation......Page 329
7.8 Summary......Page 330
8.1 Machine Learning for Source Separation......Page 332
8.2 Potential Topics and Directions......Page 334
A.3 Gamma Function......Page 336
A.6 Derivative......Page 337
A.8 Tensor Algebra......Page 338
B.1 Multivariate Gaussian Distribution......Page 340
B.2 Laplace Distribution......Page 341
B.4 Gamma Distribution......Page 342
B.5 Wishart Distribution......Page 344
B.7 Exponential Distribution......Page 345
Biblio......Page 347
Index......Page 358