The Essential Guide To Video Processing

Cover Page
......Page 1
Copyright......Page 2
Preface......Page 3
About the Author......Page 5
1 Introduction to Digital Video Processing......Page 6
Sampled Video......Page 8
Video Transmission......Page 11
Objectives of this Guide......Page 12
Organization of the Guide......Page 13
Introduction......Page 15
Spatiotemporal Sampling Structures......Page 16
Sampling and Reconstruction of Continuous Time-Varying Imagery......Page 20
Sampling Structure Conversion......Page 24
Frame-rate Conversion......Page 26
Pure Temporal Interpolation......Page 27
Motion-compensated Interpolation......Page 28
Spatiotemporal Sampling Structure Conversion......Page 30
Deinterlacing......Page 31
References......Page 32
Further Information......Page 33
Introduction......Page 34
Binary Hypothesis Testing......Page 35
Markov Random Fields......Page 36
Variational Formulations......Page 37
Motion Detection......Page 38
Hypothesis Testing with Fixed Threshold......Page 39
Hypothesis Testing with Adaptive Threshold......Page 41
MAP MRF Formulation......Page 44
MAP Variational Formulation......Page 45
Experimental Comparison of Motion Detection Methods......Page 46
Motion Estimation......Page 47
Motion Models......Page 48
Spatial Motion Models......Page 49
Temporal Motion Models......Page 50
Region of Support......Page 51
Observation Models......Page 52
Pixel-Domain Criteria......Page 54
Frequency-Domain Criteria......Page 55
Regularization......Page 56
Search Strategies......Page 57
Global Motion Estimation......Page 59
Estimation Criterion......Page 62
Search Methods......Page 63
Phase Correlation......Page 64
Optical Flow via Regularization......Page 65
MAP Estimation of Dense Motion......Page 66
Experimental Comparison of Motion Estimation Methods......Page 67
Perspectives......Page 68
References......Page 69
Introduction......Page 71
Temporally Averaging Filters......Page 74
Temporally Recursive Filters......Page 77
Order-Statistic Filters......Page 78
Multiresolution Filters......Page 81
Coding Artifact Reduction......Page 84
Artifact Reduction in the Frequency Domain......Page 85
Blotch Detection and Removal......Page 86
Blotch Detection......Page 87
Motion Vector Repair and Interpolating Corrupted Intensities......Page 90
Video Inpainting......Page 93
Restoration in Conditions of Difficult Object Motion......Page 94
Vinegar Syndrome Removal......Page 96
Intensity Flicker Correction......Page 99
Flicker Parameter Estimation......Page 100
Estimation on Sequences with Motion......Page 101
Kinescope Moiré Removal......Page 103
Scratch Removal......Page 105
Conclusions......Page 106
References......Page 107
Introduction......Page 111
Outline......Page 112
Centering Behavior and Collision Avoidance......Page 113
Measuring Distance by Integrating Optical Flow......Page 114
Camera Model......Page 115
Effect of Camera Motion......Page 116
Image Features......Page 117
Structure from Motion......Page 118
Feature based Algorithms......Page 119
Global Flow Models......Page 120
Flow-Based Algorithm......Page 121
Stabilization and Mosaicing......Page 124
Video Mosaicing......Page 126
VIVID Metadata......Page 129
Stabilization with Metadata......Page 130
Stabilization with Inertial Measurements......Page 132
Motion Super-resolution......Page 136
Summary......Page 139
References......Page 140
Introduction......Page 143
Scene Change Detection......Page 144
Spatial Change Detection Using Two Frames......Page 146
Temporal Integration......Page 147
Motion Segmentation......Page 148
Dominant Motion Segmentation......Page 149
Segmentation Using Two Frames......Page 150
Temporal Integration......Page 151
Multiple Motion Segmentation......Page 152
Clustering in the Motion Parameter Space......Page 153
Maximum Likelihood Segmentation......Page 156
Maximum A Posteriori Probability Segmentation......Page 158
Region-Based ML Segmentation: Fusion of Color and Motion......Page 160
Simultaneous Motion Estimation and Segmentation......Page 162
Motion-Field Model and MAP Framework......Page 163
Two-Step Iteration Algorithm......Page 164
Semiautomatic Segmentation......Page 166
Examples......Page 167
References......Page 172
Introduction......Page 176
2D Rigid Object Tracking......Page 181
Region-Based Object Tracking......Page 182
Contour-Based Object Tracking......Page 189
Feature Point-Based Object Tracking......Page 191
Template-Based Object Tracking......Page 194
Bayesian Object Tracking......Page 198
Occlusion Handling......Page 204
3D Rigid Object Tracking......Page 206
3D Articulated Object Tracking......Page 210
3D Modeling......Page 213
Kinematic and Motion Constraints......Page 215
Image Cues......Page 216
Example Techniques......Page 217
2D Articulated Object Tracking......Page 220
References......Page 221
8 Basic Transform Video Coding......Page 232
Introduction to Video Compression......Page 233
Video Compression Application Requirements......Page 238
Sampling of Analog Video Signals......Page 242
Digital Video Formats......Page 244
Video Compression Techniques......Page 246
Entropy and Predictive Coding......Page 247
Block Transform Coding—The DCT......Page 249
Quantization......Page 251
MC and Estimation......Page 254
Transform Coding: Introduction to the Video Encoding Standards......Page 257
Transform Coding Standard Example: The H.261Video Encoder......Page 259
References......Page 266
Background and Structure of MPEG-1 Standards Activities......Page 268
Bidirectional Motion-Compensated Prediction......Page 269
Motion-Compensated Prediction with Half-Pixel Accuracy......Page 270
Source Input Format......Page 271
Group of Pictures and I-B-P-Pictures......Page 272
Slice, Macroblock, and Block Structures......Page 274
Summary of the Major Differences between MPEG-1 Videoand H.261......Page 275
Simulation Model......Page 276
MPEG-1 Video Bit-Stream Structures......Page 277
Background and Structure of MPEG-2 Standards Activities......Page 278
Target Applications and Requirements......Page 279
MPEG-2 Profiles and Levels......Page 280
MPEG-2 Video Input Resolutions and Formats......Page 281
Interlaced versus Progressive Video......Page 282
Interlaced Video Coding......Page 283
Quantization......Page 286
SNR Scalability......Page 287
Spatial Scalability......Page 288
Data Partitioning......Page 289
Test Model......Page 290
MPEG-2 Video and System Bit-Stream Structures......Page 291
Summary......Page 292
References......Page 293
Introduction......Page 295
Terminology......Page 296
Object-based Representation......Page 298
Video Object Coding......Page 299
Motion Texture Coding......Page 301
Shape Coding......Page 302
Mesh Object Coding......Page 304
Face Animation......Page 305
Body Animation......Page 306
Scalability......Page 307
MPEG-4 Part 2 Profiles......Page 308
H.264/AVC Video Coding Layer: Technical Overview......Page 310
Slices and Slice Groups......Page 312
Spatial Directional Intra Prediction......Page 313
Enhanced Motion Compensation Prediction Model......Page 314
In-Loop Deblocking Filter......Page 317
Transform, Quantization, and Scanning......Page 319
Frame/Field Adaptive Coding......Page 320
Profiles......Page 322
MPEG-4 Part 2......Page 323
MPEG-4 Part 10: H.264/AVC......Page 325
MPEG-4 Video Applications......Page 326
Conclusions and Outlook......Page 327
References......Page 328
Introduction......Page 331
Motion Estimation and Compensation for MCTF......Page 333
Connected and Unconnected Blocks......Page 335
Using Chroma for Motion Estimation......Page 337
Noninvertible Approach......Page 338
Lifting Implementation......Page 339
Subpixel Interpolation......Page 340
Comparison of Different Interpolation Filters......Page 341
New Haar MCTF......Page 342
Overlapped Block Motion Compensation......Page 344
Scan/spatial Prediction for Motion Vectors......Page 345
Alphabet General Partition of Motion Vector Symbols......Page 346
Layered Structure of Motion Vector Coding......Page 347
EZBC Coder......Page 348
Definitions......Page 349
Coding Steps......Page 350
Context Modeling......Page 351
Scalability......Page 352
Frequency Roll-Off......Page 353
Extension to LeGall and Tabatabai 5/3 Filtering......Page 355
Some Visual Results......Page 357
Multiple Adaptations......Page 360
Related Coders......Page 361
References......Page 363
Introduction......Page 366
Video Transcoding for Bit Rate Reduction......Page 368
Transcoding of Intracoded Frame......Page 369
Fast Video-Transcoding Architectures......Page 371
DCT Domain IMC......Page 374
Heterogeneous Video Transcoding......Page 376
MV Estimation for Spatial Resolution Reduction......Page 378
MV Estimation for Temporal Resolution Reduction......Page 379
Spatial Resolution Reduction......Page 380
Bit Rate Control in Video Transcoding......Page 382
Error-Resilient Video Transcoding......Page 383
Concluding Remarks......Page 384
References......Page 385
Introduction......Page 388
Block-Based Video Coding......Page 390
Embedded Video Codec Requirements and Constraints......Page 392
Understanding the Chip Architecture......Page 396
Understanding the Codec Algorithms......Page 399
Modularity and APIs Definitions......Page 400
Reference Codec Software Development in Golden C......Page 403
Platform-Specific Development and Porting......Page 404
Kernel Optimization and Integration......Page 405
Concurrent Processing......Page 406
Overall Optimization......Page 407
Stress and Conformance Testing......Page 409
New Trends......Page 410
Summary......Page 412
References......Page 413
Introduction......Page 416
HVS Modeling Based Methods......Page 418
VQM......Page 421
Feature Computation......Page 422
Pooling......Page 423
Motion Modeling Based Methods......Page 424
Performance......Page 430
Conclusions......Page 432
References......Page 433
Introduction......Page 436
Content Categories......Page 437
Storage and Compression......Page 438
Terminology......Page 439
Statistical Features......Page 441
Image Difference......Page 442
Video Segmentation......Page 443
Motion Analysis......Page 445
Alternative Statistical Features......Page 448
Compressed-Domain Features......Page 449
Object Detection......Page 451
Audio and Language......Page 455
Rule-Based Features......Page 456
Embedded Video Features......Page 457
Key-Frame Extraction......Page 458
Representation Based on Sequential Key Frames......Page 459
Representation Based on Video Mosaics......Page 460
Representation Based on Highlight Groups......Page 461
Video Retrieval......Page 462
Content-Based Retrieval......Page 463
Relevance Feedback......Page 464
Efficient Annotation through Active Learning......Page 465
Testing and Evaluation......Page 466
A Unified Framework for Indexing, Summarization, Browsing, and Retrieval......Page 467
Conclusions and Promising Research Directions......Page 468
References......Page 469
16 Video Communication Networks......Page 472
Introduction......Page 473
Introduction......Page 474
Overview......Page 475
MPEG-2 Video Compression Standard......Page 477
MPEG-2 Systems Standard......Page 478
MPEG-2 Elementary Stream......Page 479
MPEG-2 PES......Page 480
MPEG-2 Transport Stream......Page 481
Introduction......Page 484
Hybrid Fiber-Coax Networks......Page 485
Digital Subscriber Loop......Page 486
Wireless Networks......Page 487
Fiber Optics......Page 490
ATM Networks......Page 491
ATM Application Layer-1......Page 494
ATM Application Layer-5......Page 496
Introduction......Page 498
Multicast Backbone......Page 501
Real-Time Transport Protocol......Page 502
Real-Time Transport Control Protocol......Page 510
Real-Time Transport Streaming Protocol......Page 516
H.323......Page 517
Session Initiation Protocol......Page 519
Integrated Services—Resource Reservation Protocol......Page 520
Differentiated Services—DiffServ......Page 522
References......Page 524
Video Encryption......Page 526
Encryption Before and After Coding......Page 528
Randomized Entropy Coding......Page 529
Generalized Index Mapping with Controlled Overhead......Page 531
Constrained Shuffling......Page 533
Security Evaluation of Media Encryption......Page 535
Edge Similarity Score......Page 537
Video Encryption System Design......Page 538
Base-Layer Video Encryption......Page 539
Protecting FGS Enhancement Layer Video......Page 543
Background......Page 544
Content Level Authentication......Page 545
Stream Level Authentication......Page 550
Video Fingerprinting for Traitor Tracing......Page 552
Spread Spectrum Embedding and Fingerprinting......Page 553
Collusion Attacks......Page 554
Coded Fingerprinting......Page 555
Considerations in Code Parameters......Page 557
Analysis on Collusion Resistance......Page 558
Boosting Collusion Resistance by Permuted Subsegment Embedding......Page 560
Efficient Detection via Trimming for Large Scale Fingerprinting System......Page 561
Experimental Results of Video Fingerprinting......Page 562
Results under Nonlinear Collusion Attacks......Page 563
References......Page 565
Introduction......Page 570
Rate-Distortion Theory......Page 574
Operational Rate-Distortion Theory......Page 575
Practical Constraints in Video Communications......Page 576
Illustration......Page 577
Video Transmission System......Page 578
Video Compression Basics......Page 580
Internet......Page 583
End-to-End Distortion......Page 584
Error Resilient Source Coding......Page 586
Flexible Macroblock Ordering......Page 587
Channel Coding......Page 588
Forward Error Correction......Page 589
Retransmission......Page 591
Problem Formulation......Page 593
Joint Source Coding and FEC......Page 595
Joint Source Coding and Hybrid FEC/ARQ......Page 596
Joint Source Coding and FEC......Page 598
Joint Source Coding and Power Adaptation......Page 599
Joint Source-Channel Coding and Power Adaptation......Page 600
Joint Source Coding and Data Rate Adaptation......Page 602
Video Streaming over Multiuser Networks......Page 603
Mobile TV Standards......Page 607
MediaFLO......Page 608
Peer-to-Peer Internet Video Broadcasting......Page 609
Video Streaming over Multihop Wireless Networks......Page 610
Discussion......Page 611
References......Page 612
Introduction......Page 618
Outdoor Perimeter Security......Page 619
Automotive......Page 620
Transportation Infrastructure......Page 621
Military......Page 622
Video Surveillance Target Scenes......Page 623
(S3) Crowded/Dynamic Scene Surveillance......Page 624
Change and Motion Detection......Page 625
Shape Recognition and Categorization......Page 626
Review of Video Analytic Algorithms......Page 627
Temporal Differencing......Page 628
Adaptive Background Modeling......Page 629
Optical Flow......Page 632
Object Detection......Page 634
Shape-Based Approaches......Page 635
Object Tracking......Page 636
Region-Based Tracking......Page 637
Active Contour-Based Tracking......Page 639
Feature-Based Tracking......Page 640
Model-Based Tracking......Page 642
Hidden Markov Models......Page 643
Model-Based Approaches......Page 644
Statistical Approaches......Page 645
Language-Based Approaches......Page 646
Conclusion......Page 647
References......Page 648
Introduction......Page 651
Set of Observations......Page 653
Matrix......Page 654
PDF......Page 656
Manifold......Page 658
Temporal Continuity/Dynamics......Page 659
3D Model......Page 661
A General Framework of Probabilistic Identity Characterization......Page 663
Continuous Identity Signature......Page 666
The Effects of the Transformation......Page 667
FR from a Group of Still Images......Page 668
Subspace Identity Encoding—Invariant to Localization, Illumination, and Pose......Page 669
Experimental Results......Page 670
FR from a Video Sequence......Page 673
Simultaneous Tracking and Recognition......Page 674
Experimental Results......Page 675
The ARMA Model......Page 680
Framework for Recognition......Page 681
Experiments, Results, and Discussion......Page 682
References......Page 683
Introduction......Page 687
Analysis of Visual Signals......Page 689
Face Detection, Mouth, and Lip Tracking......Page 690
Visual Features......Page 692
Two Visual Feature Extraction Systems......Page 696
Speech Classes in Audiovisual Integration......Page 698
Classifiers in Speech Applications......Page 700
Feature and Classifier Fusion......Page 702
Audiovisual Automatic Speech Recognition......Page 705
Bimodal Corpora for ASR......Page 706
Experimental Results......Page 707
Audiovisual Speech Synthesis......Page 710
Coarticulation Modeling......Page 711
Facial Animation......Page 712
Visual Text-to-Speech......Page 715
Speech-to-Video Synthesis......Page 716
Visual Speech Synthesis Evaluation......Page 719
Audiovisual Speaker Recognition......Page 721
Summary and Discussion......Page 727
References......Page 729
B......Page 736
C......Page 737
D......Page 738
F......Page 739
H......Page 740
I......Page 741
J......Page 742
M......Page 743
P......Page 746
R......Page 747
S......Page 748
T......Page 749
V......Page 750
Z......Page 752