Emerging Technologies for 3D Video: Creation, Coding, Transmission and Rendering

Emerging Technologies for 3D Video: Creation, Coding, Transmission and Rendering......Page 1
Contents......Page 5
Preface......Page 17
List of Contributors......Page 21
Acknowledgements......Page 25
Part I: Content Creation......Page 27
1.2 Time-of-Flight Depth Camera......Page 29
1.2.1 Principle......Page 30
1.2.2 Quality of the Measured Distance......Page 32
1.3.1 Principle......Page 37
1.4 Specular and Transparent Depth......Page 38
1.5.2 Three-Dimensional Reconstruction......Page 41
References......Page 42
2.1 Introduction......Page 43
2.2.1 Light Fields, Reflectance Distribution Functions, and Optical Image Formation......Page 44
2.2.3 Synthetic Aperture Radar for Estimating Scene Reflectance......Page 46
2.3.1 Scene, Illumination, and Sensor Abstractions......Page 47
2.3.2 Scene Response Derivation......Page 48
2.4.1 Response Modeling......Page 50
2.4.2 Image Recovery using Linear Backprojection......Page 54
2.5.1 Single-Plane Response to Omnidirectional Illumination......Page 56
2.5.2 Spatially-Patterned Measurement......Page 58
2.5.3 Algorithms for Depth Map Reconstruction......Page 59
2.6 Discussion and Future Work......Page 60
References......Page 61
3.1 Introduction......Page 63
3.2.1 General Conversion Approach......Page 64
3.2.2 Depth Cues in Monoscopic Video......Page 65
3.3 Definition of Depth Structure of the Scene......Page 67
3.3.1 Depth Creation Methods......Page 68
3.3.2 Depth Recovery Methods......Page 70
3.4.1 Depth to Disparity Mapping......Page 74
3.4.2 View Synthesis and Rendering Techniques......Page 75
3.4.3 Post-Processing for Hole-Filling......Page 79
3.5 Quality of Experience of 2D-to-3D Conversion......Page 82
3.6 Conclusions......Page 83
References......Page 84
4.1 Stereoscopic Capture......Page 88
4.2 Dual-Camera Arrangements in the 1950s......Page 89
4.3 Classic “Beam-Splitter? Technology......Page 91
4.4 The Dual-Camera Form Factor and Camera Mobility......Page 92
4.5 Reduced 3D Form Factor of the Digital CCD Sensor......Page 94
4.6 Handheld Shooting with Variable Interaxial......Page 97
4.7 Single-Body Camera Solutions for Stereoscopic Cinematography......Page 99
4.9 Human Factors of Variable Interaxial......Page 102
References......Page 104
Part II: Representation, Coding and Transmission......Page 105
5.1 Introduction......Page 107
5.2.1 The Pinhole Camera Model......Page 108
5.2.2 Stereoscopic Imaging Systems......Page 111
5.3.2 Difficulties in the Stereo Matching Process......Page 114
5.3.4 Fundamental Steps Involved in Stereo Matching Algorithms......Page 115
5.4.1 Local Methods......Page 117
5.4.2 Global Methods......Page 119
References......Page 124
6.1 Introduction......Page 128
6.2.1 Stereoscopic 3D S3D Video......Page 129
6.2.2 Multiview Video MVV......Page 130
6.2.3 Video-Plus-Depth......Page 131
6.2.5 Layered Depth Video LDV......Page 133
6.3.1 Simulcast......Page 135
6.3.2 Frame-Compatible Stereo Interleaving......Page 136
6.3.5 MPEG-2 Multiview Profile MVP......Page 139
6.3.6 Multiview Video Coding MVC......Page 140
Acknowledgments......Page 144
References......Page 145
7.1 Introduction......Page 147
7.2 Depth Map Analysis and Characteristics......Page 148
7.3.1 Tools that Exploit the Inherent Characteristics of Depth Maps......Page 149
7.3.2 Tools that Exploit the Correlations with the Associated Texture......Page 153
7.3.3 Tools that Optimize Depth Map Coding for the Quality of the Synthesis......Page 155
7.4 Application Example: Depth Map Coding Using “Don’t Care? Regions......Page 158
7.4.1 Derivation of “Don’t Care? Regions......Page 159
7.4.2 Transform Domain Sparsification Using “Don’t Care? Regions......Page 160
7.4.3 Using “Don’t Care? Regions in a Hybrid Video Codec......Page 161
7.5 Concluding Remarks......Page 162
References......Page 163
8.1 Introduction......Page 165
8.1.2 Requirements for Depth-Based Format......Page 166
8.2 Depth Representation and Rendering......Page 167
8.2.1 Depth Format and Representation......Page 168
8.2.2 Depth-Image-Based Rendering......Page 169
8.3.1 AVC-Based Architecture......Page 170
8.3.3 Hybrid......Page 172
8.4 Compression Technology......Page 173
8.4.2 View Synthesis Prediction......Page 174
8.4.3 Depth Resampling and Filtering......Page 175
8.4.4 Inter-Component Parameter Prediction......Page 176
8.4.5 Depth Modelling......Page 177
8.4.6 Bit Allocation......Page 178
8.5.1 Evaluation Framework......Page 179
8.5.2 AVC-Based 3DV Coding Results......Page 181
8.5.3 HEVC-Based 3DV Coding Results......Page 182
8.6 Concluding Remarks......Page 184
References......Page 185
9.1 Introduction......Page 188
9.2.1 Challenges: Coding Efficiency and Navigation Flexibility......Page 189
9.2.2 Approaches to Interactive Media Streaming......Page 191
9.3.1 Region-of-Interest RoI Image Browsing......Page 192
9.3.2 Light-Field Streaming......Page 193
9.3.4 Video Browsing......Page 194
9.3.6 Region-of-Interest RoI Video Streaming......Page 195
9.4.1 Interactive Multiview Video Streaming IMVS......Page 198
9.4.2 IMVS with Free Viewpoint Navigation......Page 205
9.4.3 IMVS with Fixed Round-Trip Delay......Page 207
References......Page 210
10.1 Introduction......Page 213
10.2.1 Overlay Topology......Page 214
10.2.2 Sender-Driven versus Receiver-Driven P2P Video Streaming......Page 215
10.2.3 Layered versus Cross-Layer Architecture......Page 216
10.2.5 BitTorrent: A Platform for File Sharing......Page 217
10.3 Monocular Video Streaming Over P2P Networks......Page 218
10.3.2 Variable-Size Chunk Generation......Page 219
10.3.3 Time-Sensitive Chunk Scheduling Using Windowing......Page 220
10.3.5 AdaptiveWindow Size and Scheduling Restrictions......Page 221
10.3.6 Multiple Requests from Multiple Peers of a Single Chunk......Page 222
10.4.2 Rate Adaptation in Stereo Streaming: Asymmetric Coding......Page 223
10.4.3 Use Cases: Stereoscopic Video Streaming over P2P Network......Page 226
10.5.2 Rate Adaptation for MVV: View Scaling......Page 227
10.5.3 Use Cases: MVV Streaming over P2P Network......Page 228
References......Page 229
Part III: Rendering and Synthesis......Page 231
11.1 Introduction......Page 233
11.2 Background......Page 234
11.3 Image Domain Warping......Page 235
11.4.2 Disparity Range......Page 236
11.4.4 Disparity Velocity......Page 237
11.4.8 Nonlinear Operator......Page 238
11.5.1 Data Extraction......Page 239
11.5.2 Warp Calculation......Page 240
11.5.3 Applications......Page 242
11.6.1 Automatic Stereo to Multiview Conversion......Page 244
11.6.3 Warp Interpolation and Extrapolation......Page 245
11.6.4 Three-Dimensional Video Transmission Systems for Multiview Displays......Page 246
11.7 IDW for User-Driven 2D?3D Conversion......Page 247
11.7.1 Technical Challenges of 2D?3D Conversion......Page 248
11.8 Multi-Perspective Stereoscopy from Light Fields......Page 251
11.9 Conclusions and Outlook......Page 254
References......Page 255
12.1 Introduction......Page 257
12.2.1 Light Field and Surface Light Field Parameterization......Page 258
12.2.2 Epipolar Plane Image......Page 260
12.3 Uniform Sampling in a Fourier Framework......Page 261
12.3.1 Spectral Analysis of the Plenoptic Function......Page 262
12.3.2 The Plenoptic Spectrum under Realistic Conditions......Page 265
12.4 Adaptive Plenoptic Sampling......Page 268
12.4.1 Adaptive Sampling Based on Plenoptic Spectral Analysis......Page 270
12.5.1 Outlook......Page 272
References......Page 273
13.1 The Virtual Video Camera......Page 275
13.1.1 Navigation Space Embedding......Page 277
13.1.2 Space?Time Tetrahedralization......Page 278
13.1.3 Processing Pipeline......Page 281
13.1.4 Rendering......Page 282
13.1.5 Application......Page 283
13.2 Estimating Dense Image Correspondences......Page 284
13.2.1 Belief Propagation for Image Correspondences......Page 285
13.2.2 A Symmetric Extension......Page 286
13.2.4 Construction of Message-Passing Graph......Page 287
13.2.5 Data Term Compression......Page 288
13.2.8 Limitations......Page 289
13.3.1 Editing Operations......Page 290
13.4 Extending to the Third Dimension......Page 291
13.4.1 Direct Stereoscopic Virtual View Synthesis......Page 292
13.4.3 Comparison......Page 293
13.4.4 Concluding with the “Who Cares?? Post-Production Pipeline......Page 294
References......Page 296
Part IV: Display Technologies......Page 299
14.1 Introduction......Page 301
14.2.1 Automatic 2D-to-3D Image Conversion......Page 302
14.2.2 Real-Time Intermediate View Interpolation......Page 306
14.2.3 Brightness and Color Balancing in Stereopairs......Page 312
14.3.1 Ghosting Suppression for Polarized and Shuttered Stereoscopic 3D Displays......Page 313
14.3.2 Aliasing Suppression for Multiview Eyewear-Free 3D Displays......Page 315
14.4 Conclusions......Page 318
References......Page 319
15.2 Three-Dimensional Display Technologies in Cinemas......Page 321
15.2.1 Three-Dimensional Cinema Projectors Based on Light Polarization......Page 322
15.2.2 Three-Dimensional Cinema Projectors Based on Shutters......Page 325
15.2.3 Three-Dimensional Cinema Projectors Based on Interference Filters......Page 326
15.3.1 Based on Anaglyph Glasses......Page 327
15.3.2 Based on Shutter Glasses......Page 328
15.3.3 Based on Polarized Glasses......Page 330
15.3.4 Without Glasses......Page 332
15.4 Mobile 3D Display Technologies......Page 335
15.4.2 Based on Lighting Switch......Page 336
15.5 Long-Term Perspectives......Page 337
References......Page 338
16.1 Introduction......Page 339
16.2.1 Principle......Page 340
16.2.2 Integral Photography with a Concave Lens Array......Page 341
16.2.3 Holocoder Hologram......Page 343
16.2.5 Avoiding Pseudoscopic Images......Page 344
16.3.1 Orthoscopic Conversion Optics......Page 345
16.3.2 Applications of the Ultra-High-Resolution Video System......Page 348
16.4.1 Geometrical Relationship of Subject and Spatial Image......Page 351
16.4.2 Resolution......Page 352
16.4.3 Viewing Area......Page 355
16.5.1 Acquiring and Displaying Spatial Information......Page 356
16.5.2 Elemental Image Generation from 3D Object Information......Page 357
16.5.3 Three-Dimensional Measurement......Page 358
16.5.4 Hologram Conversion......Page 359
References......Page 360
17.1 Introduction......Page 362
17.2 Fundamentals of 3D Displaying......Page 363
17.3 The HoloVizio Light-Field Display System......Page 365
17.3.1 Design Principles and System Parameters......Page 366
17.3.2 Image Organization......Page 367
17.4.1 Desktop Displays......Page 368
17.4.3 Cinema Display......Page 369
17.4.5 Applications......Page 370
References......Page 371
Part V: Human Visual System and Quality Assessment......Page 373
18.2 Natural Viewing and S3D Viewing......Page 375
18.3 Perceiving 3D Structure......Page 376
18.3.1 Perceiving Depth from Binocular Disparity......Page 378
18.4 ‘Technical’ Issues in S3D Viewing......Page 380
18.4.3 Photometric Differences Between Left- and Right-Eye Images......Page 381
18.4.6 Incorrect Specular Highlights......Page 382
18.6 Motion Artefacts from Field-Sequential Stereoscopic Presentation......Page 383
18.6.2 Perception of Unsmooth or Juddering Motion......Page 385
18.6.4 Conclusions......Page 386
18.7.1 Capture Parameters......Page 387
18.7.3 Are Problems of Incorrect Geometry Unique to S3D?......Page 390
18.8 Fixating and Focusing on Stereoscopic Images......Page 392
18.8.2 Accommodation and Vergence in the Real World and in S3D......Page 393
18.8.3 Correcting Focus Cues in S3D......Page 394
18.8.4 The Stereoscopic Zone of Comfort......Page 395
18.8.5 Specifying the Zone of Comfort for Cinematography......Page 396
18.8.6 Conclusions......Page 397
References......Page 398
19.1 Introduction......Page 403
19.2 Stereoscopic Artifacts......Page 404
19.3 Subjective Quality Assessment......Page 405
19.3.1 Psycho-perceptual or Psychophysical Experiments......Page 406
19.3.3 Hybrid Approaches......Page 408
19.3.4 Open Issues......Page 409
19.4.1 Objective Quality Metrics......Page 410
19.4.2 From 2D to 3D......Page 411
19.4.3 Including Depth Information......Page 412
19.4.4 Beyond Image Quality......Page 413
19.4.5 Open Issues......Page 414
References......Page 415
Part VI: Applications and Implementation......Page 419
20.1 Introduction......Page 421
20.2 Related Work......Page 422
20.3 System Overview......Page 423
20.4.1 Camera Model......Page 424
20.4.2 Data Acquisition......Page 426
20.6 Loop and Junction Detection......Page 427
20.7.1 Notation......Page 428
20.7.2 Fusing SfM with Ground Control Points......Page 429
20.8.2 Viewer......Page 431
20.9.3 Line Extraction......Page 434
20.9.4 Line Clustering and VP Estimation......Page 435
20.10.1 Vertical Rectification......Page 436
20.10.2 Trajectory Estimation and Mapping......Page 437
References......Page 440
21.1 Introduction......Page 442
21.2.1 Pose......Page 443
21.2.3 Position......Page 445
21.3 Content Ranking......Page 447
21.3.1 Object-Centric Quality of View......Page 448
21.3.2 View-Centric Quality of View......Page 449
21.4 View Selection......Page 450
21.4.2 View Selection as an Optimization Problem......Page 451
21.5 Comparative Summary and Outlook......Page 452
References......Page 455
22.1 Mobile Ecosystem, Architecture, and Requirements......Page 458
22.2 Stereoscopic Applications on Mobile Devices......Page 459
22.2.3 3D Viewing Modalities......Page 460
22.2.6 Monoscopic 3D......Page 461
22.3.1 Stereo-Camera Design......Page 462
22.3.2 Stereo Imaging......Page 463
22.3.3 Stereo Rectification, Lens Distortion, and Camera Calibration......Page 464
22.3.4 Digital Zoom and Video Stabilization......Page 466
22.4.1 Local Auto-Stereoscopic Display......Page 468
22.4.3 Stereoscopic Rendering......Page 469
22.5.1 View Synthesis......Page 471
22.5.2 Depth Map Representation and Compression Standards......Page 472
22.5.3 Other Usages......Page 473
References......Page 474
23.1 An Interactive Composition System for 3D Displays......Page 476
23.2.1 Media Formats......Page 477
23.2.2 Multimedia Languages......Page 478
23.2.3 Multiview Displays......Page 479
23.3.1 3D Synthesis......Page 480
23.3.2 View Interleaving......Page 481
23.3.3 3D Media Rendering......Page 483
23.4.1 Quick Overview......Page 484
23.4.2 DIBR Synthesis......Page 485
23.4.3 Hardware Compositor......Page 486
23.4.4 DIBR Pre- and Post-Processing......Page 488
23.4.5 Hardware Platform......Page 490
References......Page 492
24.1 Introduction......Page 494
24.2.1 DP-Based Algorithms and Implementations......Page 495
24.2.3 BP-Based Algorithms and Implementations......Page 496
24.3 Proposed Hardware-Efficient Algorithm......Page 497
24.3.2 Hardware-Efficient Cost Diffusion Method......Page 498
24.3.3 Upsampling Disparity Maps......Page 499
24.3.4 Temporal Consistency Enhancement Methods......Page 500
24.3.5 Occlusion Handling......Page 501
24.4.1 Overview of Architecture......Page 502
24.4.2 Computational Modules......Page 503
24.4.3 External Memory Access......Page 504
24.5.1 Comparison of Disparity Quality......Page 505
24.5.2 Analysis of Sampling Factor......Page 506
24.5.3 Implementation Result......Page 507
References......Page 509
Index......Page 513