Embedded vision refers to the integration of computer vision capabilities into various devices, systems or environments; these applications often involve using cameras or sensors to capture visual data and processing that data in real-time or near-real-time in order to make decisions or provide insights to the final user.
In a later stage, the captured video might need to be sent over a suitable media to a corresponding endpoint (PC host, mobile device, others) for further analysis or visualization as shown in Figure 1.
Figure 1. Video streaming setup
Two of the most common options for sharing video to external devices are network streaming and USB video streaming, this blog will focus on describing RidgeRun’s USB-based streaming solution via LibGUVC which allows the customers to have their embedded device (SoC + carrier + camera sensor / video feed) emulate a USB webcam they can plug to a host machine and easily stream video for further processing / displaying.
Before getting into the details on how LibGUVC works, the following sections are meant to provide a reference (both historical and technical) on the evolution of USB video streaming using UVC; a good starting point would be to have a quick comparison between network and USB streaming solutions as shown in Table 1.
Features | Network Video Streaming | USB Video Streaming |
Medium | Network video streaming utilizes a network connection, which can be either wired (Ethernet) or wireless (Wi-Fi). It can be point to point or broadcast. | USB video streaming uses a USB cable to transmit the video content to a client device. It is a point to point connection. |
Protocols | It relies on network protocols such as HTTP, RTSP, RTMP, HLS, or WebRTC to transmit video data. | Utilizes USB-specific protocols for data transfer, typically USB Video Class (UVC) for video devices. |
Use case | - Typically used for streaming video content over the internet or within a local area network (LAN).- Common applications include live video broadcasting, video conferencing, online video services (like Netflix or YouTube), and streaming from network cameras (IP cameras). | - Often used for direct and immediate video input into computers or other devices.- Common applications include video capture devices, and direct video input for live streaming or recording. |
Flexibility and range | - Can stream video over long distances, including globally over the internet.- Not limited by physical proximity as long as there is network connectivity. | - Limited by the length of the USB cable, which typically maxes out at around 5 meters (16 feet) without using repeaters or active extension cables.- Requires physical proximity between the video source and the receiving device. |
Quality and latency | - The quality of the stream can be affected by network bandwidth and latency.- Potential for buffering and delays, especially over long distances or congested networks. | - Generally low latency and high-quality video transmission due to the direct connection.- Less susceptible to bandwidth fluctuations compared to network streaming. |
Application Compatibility | Requires client application to implement the required network stream and process the content accordingly. | Most operating systems already support the standard USB specifications, so almost all media applications support it without external drivers or implementations. |
USB Video Class (UVC)
USB-based video capture and streaming typically relies on the USB Video Class (UVC) protocol which is a standard for transmitting video data over a USB connection. It was developed in the early 2000s by the USB Implementers Forum (USB-IF) to standardize the way in which digital cameras could transmit video over a USB connection, making it easier for devices to connect and communicate with one another.
A historical summary of the several releases of the UVC specification up to UVC 1.5 is shown below:
UVC 1.0: The first version of the UVC specification, released in 2003.
UVC 1.0a: Released in December 2003, it featured additional descriptor subtypes for extension types.
UVC 1.0b: Include updated on protocol stall behavior and current/future payload header formats.
UVC 1.0c: Released in June 2004, it provided information on MJPEG characteristics.
UVC 1.1: Released in June 2005, it became a major update including new documents specifying streams and frame base payloads, latency optimization for stream-based formats and other technical details. H.264 format was supported as an extension unit.
UVC 1.5: Released in June 2012, it is the latest version of the specification, and officially added H.264 and VP8 payloads as well as controls for video encoders as part of the standard descriptors. It also included references for UBS3.0
With the UVC standard, the host receives structures called descriptors that describe the video device functionality and USB queries in order to control the several parameters and characteristics of the video stream; it also gives video devices the flexibility to support multiple video resolutions, formats and frame rates which is not a minor consideration as this has a big impact on the connection bandwidth between the device and the host. As mentioned earlier, all these capabilities are already supported by default by client operating systems when a UVC-capable device is attached, so there is no need for custom application development at the host side.
UVC Streaming Solution
The widespread usage of USB in the video capture and streaming field via UVC can be attributed to several key factors, some of them are summarized below:
Standardization: UVC is a standard protocol for video transmission over USB, meaning it defines how video data should be formatted and transferred. This standardization allows different devices and software to work together seamlessly.
Plug and play: Devices that adhere to the UVC standard can be easily connected to a host system (like a computer) and used immediately without the need for installing special drivers. The host system’s operating system recognizes the UVC device and uses generic drivers to enable its functionality.
Compatibility: UVC-compliant devices are compatible with a wide range of operating systems, including Windows, macOS, Linux, and Android. This broad compatibility ensures that users can use their video devices across different platforms without issues.
Applications: USB UVC streaming is commonly used in webcams, digital camcorders, video conferencing systems, medical imaging devices, frame grabbers, and other applications where video capture and streaming are required.
Quality and performance: UVC supports various video resolutions and frame rates, from standard definition (SD) to full high definition (FHD) and even higher resolutions like 4K, depending on the device’s capabilities and the USB version used (e.g., USB 2.0, USB 3.0).
Ease of use: The primary advantage of UVC streaming is its ease of use. Users can connect a UVC-compatible camera to a USB port, and the system will automatically recognize and configure the device, making it ready for immediate use with video conferencing software, recording applications, or streaming platforms.
RidgeRun’s USB Video Class Gadget Library Solution
The USB Video Class Gadget Library, commonly referred to as LibGUVC, is a platform agnostic library which is meant to abstract the complexity of developing a UVC-based gadget device by taking over the UVC communication and leaving up to the user the responsibility of performing the basic setup of the video stream. LibGUVC relies on Linux’s kernel driver and exposes a friendly interface on userspace, taking care of the communication between the user application and the Linux driver stack as shown in Figure 2.
Figure 2. Software stack description using LibGUVC
The communication between LibGUVC and the UVC kernel driver occurs through V4L2 events interface, each event is mapped to specific callbacks and methods within the library. A detailed description of how UVC driver works along with LibGUVC is provided below:
Device Detection and Initialization:
When a UVC-compliant USB device (embedded board) is connected to the USB host (PC for example), the host’s USB subsystem detects the device and identifies it as a UVC device based on its descriptors.
The USB core driver hands over control to the host’s USB Class driver, which then initializes the device. This involves parsing the device's UVC descriptors to understand its capabilities (e.g., supported video formats, resolutions, frame rates). This process is commonly known as enumeration.
2. Creating Video Device Nodes:
The host’s UVC driver registers one or more video device nodes (e.g., /dev/video0) with the Video4Linux (V4L2) framework. These nodes serve as interfaces for user-space applications to interact with the gadget (embedded board).
The host’s UVC driver sets up various V4L2 hooks and callbacks that are invoked for different operations like opening the device, querying capabilities, and starting/stopping video streams from the host side.
For each operation (start/stop the video on the host side for example) queries are sent to gadget USB device. On the gadget side, these queries appear in the form of V4L2 requests which are forwarded to the user space where LibGUVC takes care of handling and executing the corresponding logic.
3. Configuring the Device:
LibGUVC configures the video capture settings (such as resolution, pixel format, and frame rate) through the V4L2 API by installing a set of callbacks to perform the corresponding operation (changing resolution, format, frame rate, camera settings). The users take care of implementing these callbacks in their application and filling them with the corresponding logic to configure the gadget.
This configuration ensures that the device generates video in the host’s desired format.
4. Handling Video Streaming:
When a video stream is started, the host’s USB class driver issues a UVC-specific request to the USB gadget device to begin streaming.
LibGUVC manages USB transfers to continuously receive video data from the device. It uses bulk or isochronous transfers, depending on the device and USB bandwidth availability.
The incoming video frames are processed and made available to user-space applications via V4L2 in the host side.
5. Buffer Management:
LibGUVC handles buffer management, which involves allocating and queuing buffers for video data.
LibGUVC provides buffers through V4L2, which the UVC driver fills with video data from the device.
Once filled, the driver marks these buffers as ready, allowing the application to read and process the video frames.
Overall, LibGUVC delivers a library (and GStreamer reference sink element) that can be used by other applications on embedded devices to add UVC streaming capabilities by encapsulating the communication with the kernel’s UVC driver, thus, significantly reducing the complexity of this communication. LibGUVC is capable of handling more complex setups that include video streaming over one single USB link from multiple sources (several cameras, network streams) and combining with other USB class functions (audio, serial, RNDIS) by relying on a USB composite driver implementation.
Do not hesitate to contact us in case you are curious about how LibGUVC or any of the RidgeRun's products and services can help your team reaching the next level.