Introduction
As drones become increasingly integrated into our daily lives, the role of efficient video processing and streaming in harnessing their true potential cannot be overstated. In RidgeRun we recognize the potential of drone applications across diverse industries and the excellent synergy between these technologies and RidgeRun GStreamer products.
In previous blog post entries we have written about individual RidgeRun products in the context of drone use cases. We will build on from previous blog postings about drone technologies, which addressed different topics including expert software solutions, MISB-compliant metadata processing solutions, and utilizing RTSP for video network streaming, In this post, we'll delve even deeper into the world of drone technology by demonstrating a real-life and practical example that combines the use of these solutions as we continue to explore the potential of drones.
Overview of RidgeRun Products
The backbone of many of RidgeRun's products is GStreamer, an open-source multimedia platform. Numerous cutting-edge multimedia apps and libraries rely on this versatile and extendable framework as their secret component. In order to meet the specific requirements of drone technology, RidgeRun has accumulated collectively a lot of expertise in the art of leveraging GStreamer's capabilities and continues to do so.
Let's now discuss some of the essential GStreamer items RidgeRun provides, which are essential for streamlining video processing for drones.
Video and Audio Streaming
GstRtspSink: This plugin is a robust solution in the field of real-time video streaming. Using the Real-Time Streaming Protocol (RTSP), this GStreamer plugin enables drones to transmit video feeds reliably. For applications like real-time surveillance, remote monitoring or assisted teleoperation, it ensures minimal latency and high-quality streaming.
GStreamer WebRTC Wrapper: Another product in the realm of real-time video transmission is this plugin. It offers a ready to use solution between the powerful GStreamer framework and the WebRTC (Web Real-Time Communication) standard, allowing real time communication, low latency, security and extensibility.
Metadata Management
LibMISB: This solution is one of the most reliable ones for metadata transmission in drone use cases, due to its direct application from the MISB standard. There are multiple reasons why this solution is ideal for drones: MISB compliance, metadata insertion and extraction, Integration with GStreamer, data format standardization.
GstSEI: This product makes use of H264 and H265 standards to inject custom user metadata into the video stream, synchronized with each video frame, highly used in drone systems to send GPS, IMU, and general data back to the controller.
Computer Vision
GstStitcher: This plugin offers a solution when easy to implement panoramic video stitching is required. This project offers customizable configuration, support for multiple camera inputs, high quality output, out-of-the-box integration with GStreamer and scene calibration tools.
GstQTOverlay: In the category of overlay solutions this plugin is one of the most sought. It stands out as an essential product in drone products due to its unique capabilities: Dynamic On-Screen Display, Customizable Graphics and Text, Low Latency and Minimal CPU Load, Enhanced User Experience.
GstHTMLOverlay: A recent addition to the catalog of GStreamer-based overlay products is this plugin. RidgeRun's GstHTMLOverlay is a distinctive product offering a range of capabilities that enhance the utility and versatility of drones by leveraging hardware accelerated rendering of HTML-defined interfaces.
By offering these and other GStreamer products, RidgeRun empowers drone developers and operators with the tools they need to push the boundaries of what's achievable in terms of video processing and streaming. In the subsequent sections, we'll dive deeper into real-world use cases where these products shine.
Technical Deep Dive: How RidgeRun GStreamer Products Can Work For Drones Use Cases
Provide a simplified explanation of how RidgeRun GStreamer products integrate with drone systems.
Discuss the architecture and components involved, including video capture, encoding, streaming, and potential AI integration.
Use diagrams or flowcharts to illustrate the workflow.
We will utilize LibMISB, GstRTSPSink and GstQTOverlay in conjunction to demonstrate the RidgeRun solutions capabilities for a drone use case. We'll present below a straightforward network streaming example with overlay and metadata capabilities over a wireless network data link, just like a typical situation of drone usage. The general diagram of the proposed functionality is shown in figure 1.
For our proposed use case example this was our hardware and software setup:
Sender: NVIDIA Jetson Xavier NX (developer kit)
Receiver: PC. Intel Core i7, 16 GB RAM, 1 TB disk
JetPack 5.1.2 (installed in the sender system)
GStreamer 1.16.3 (sender and receiver device)
GstShark, GstPerf (optional, to measure frame rate or CPU load)
GstRtspSink (installed in the sender system)
GstD (installed in the sender system)
LibMISB (installed in the sender and receiver systems)
GstQTOverlay (installed in the sender system)
Figure 1. Video streaming and MISB metadata pipeline representation
Below it is shown the results of the video streaming (it could also include audio) and along with the MISB metadata which can be seen printed on the console. Both types of data can be processed individually once it is demuxed from the MPEG transport stream container.
Figure 2. Video streaming window screenshot
Figure 3. Video streaming and metadata result screenshot
The GStreamer pipelines run on the sender system are launched through Gstd for faster and easier prototyping. As the snippets of code show below, the simplicity of the pipeline description is evident relative to the big functionality it provides as shown in the previous screenshots. This also translates into flexibility as the pipeline can be accommodated to fit even more functionality like video stabilization or a more complex overlay.
#Sender
gst-client pipeline_create testpipe videotestsrc is-live=true pattern=0 ! "video/x-raw, width=1920, height=1080, framerate=30/1" ! queue ! qtoverlay qml=/home/nvidia/gst-qt-overlay/examples/drone/main_drone.qml ! nvvidconv ! perf ! nvv4l2h264enc idrinterval=30 iframeinterval=30 insert-aud=true insert-sps-pps=true insert-vui=true maxperf-enable=true ! h264parse name=parser ! queue ! mpegtsmux name=mux ! "video/mpegts, mapping=/stream1" ! rtspsink service=1234 misbsrc name="src" ! 'meta/x-klv' ! queue ! mux.In order to trigger sending data to the monitoring system (i.e the PC), one way is that the user changes the property json-metadata-path of the misbsrc element, this enables the receiver system to perceive fresh data every second (in this example). The JSON files in this example were prepared in anticipation and just serve to feed the pipeline example in this case, in a real world scenario this data will reflect actual telemetric data but the file format will remain the same in the end.
# Sending 1 minute of metadata
for i in {1..60}
do
echo meta$i.json
gst-client element_set testpipe src json-metadata-path "metadata/meta$i.json"
sleep 1
done
# Killing pipeline
sleep 2
gst-client pipeline_stop testpipe
gst-client pipeline_delete testpipeFor the pipeline in the receiver end in order to gather both the video streaming and the MISB metadata it is crucial to collect the RTSP streaming, depayload and demux. At this last stage is when both types of data could be processed individually, one being displayed (video) and the other print in human readable form (MISB metadata)
#Receiver
gst-launch-1.0 rtspsrc location=rtsp://10.251.101.196:1234/stream1 ! rtpmp2tdepay ! tsdemux name=mux ! h264parse ! avdec_h264 ! queue ! xvimagesink sync=false mux. ! queue ! 'meta/x-klv' ! misbsink print-metadata=true sync=false
Benefits of Using RidgeRun GStreamer Products for Drones
Performance, accuracy, and efficiency are critical in the field of drone technology. The GStreamer products from RidgeRun are designed to address these important issues and provide a wealth of advantages that enable drones to scale new heights. Here are a few significant benefits:
Improved video quality and low-latency streaming: The use of RidgeRun GStreamer products is aimed at getting the best possible video quality for drone footage. RidgeRun also offers services to make low-latency streaming possible, which is essential for real-time applications like remote piloting and monitoring.
Enhanced flexibility to work with various camera hardware: Drones can be built in many different ways and frequently use a variety of camera technologies. Unmatched flexibility is provided by RidgeRun's GStreamer solutions, which enable a variety of camera formats and resolutions. This versatility makes sure that drones can use the cameras that are best suited for their particular tasks.
Optimized performance: RidgeRun's GStreamer solutions are expertly optimized to give the best performance on hardware for drones with limited resources. By reducing processing overhead, this optimization enables drones to spare more computational power to important jobs like navigation, data analysis, and autonomous flight.
Reduced processing load on drone hardware: The key to reducing the strain on a drone's onboard hardware is effective video processing. RidgeRun GStreamer products offload time-consuming video processing duties by leveraging hardware acceleration, freeing up computing power for other crucial operations. This results in more efficient functioning and possibly longer flight times.
Streamlined integration with third-party systems: RidgeRun's GStreamer solutions provide seamless connectivity with third-party systems to guarantee that drones can quickly connect and work with other devices, ground stations, and cloud-based platforms. This interoperability improves the flexibility and capabilities of drones.
Real-Time data insights: Drones may now gain immediate insights from visual data thanks to RidgeRun's GStreamer products and services, which combine AI and computer vision. This greatly increases the overall usefulness of drones and is essential for applications like object detection, tracking, and sites research.
Reliable metadata handling: RidgeRun's LibMISB enables precise information extraction and insertion for drones used in mapping, surveillance, and defense applications. By improving data interoperability and sharing, this standardization makes it simpler to work with other systems.
Customizable and fast overlay solutions: Drones can be enabled to display dynamic, personalized overlays and information on video broadcasts thanks to GstHTMLOverlay and GstQTOverlay (overlay solution in general). Drones are more successful at carrying out numerous tasks thanks to this capacity, which improves situational awareness, operator control, and data presentation.
Future Trends and Innovations
It's critical to foresee and adjust to the changing technological and applications landscape as the drone industry seeks to reach new heights. Let's explore the upcoming developments and improvements in drone technology, as well as the potential significance of the RidgeRun GStreamer line of products:
Advanced Automation and Autonomy: Drones are bound to become even more autonomous and intelligent. Future innovations may include advanced AI algorithms, enabling drones to make real-time decisions, adapt to changing environments, and perform complex tasks with minimal human intervention.
5G Connectivity and Edge Computing: The rollout of 5G networks will revolutionize drone communication. Drones will have access to high-speed, low-latency connectivity, enabling real-time data transfer and remote control.
Extended Battery Life and Energy Efficiency: Battery technology is a significant limitation for drones. Future innovations may include breakthroughs in energy storage, leading to extended flight times.
Drone Swarms and Collaborative Missions: The use of drone swarms for collaborative missions is an emerging trend. RidgeRun's GStreamer products will facilitate seamless communication and coordination among swarm members, allowing for synchronized actions and data sharing.
Environmental Sensing and Monitoring: Drones are increasingly being used for environmental monitoring, such as wildfire detection and pollution assessment. Future drones may integrate advanced sensors for more accurate data collection.
Urban Air Mobility (UAM): UAM application is on the horizon, with drones serving as urban transport and delivery vehicles across densely populated areas.
Regulatory Compliance and Safety: As drone operations become more widespread, regulatory compliance and safety will be paramount.
Augmented Reality (AR) and Virtual Reality (VR): AR and VR technologies will find applications in drone piloting and data visualization.
Machine Learning and AI Integration: Drone applications are not immune to the current trend of AI innovation, the integration of machine learning and AI into drone systems will continue to grow and expand.
In summary, RidgeRun's GStreamer products will continuously empower drones with advanced video processing capabilities and features, reduced processing load, and enhanced compatibility. These benefits translate into improved performance, efficiency, and adaptability across a wide spectrum of drone applications, from surveillance and agriculture to defense and beyond. By integrating RidgeRun's GStreamer solutions, drone operators can unlock the full potential of their aerial platforms and accomplish missions with greater precision and confidence.
What’s next
Please refer to the following links for more information about our GStreamer products:
GstQTOverlay or GstHTMLOverlay(content overlay products in general)
GstSEI (metadata products in general)
For technical questions or any service inquiry, please send an email to support@ridgerun.com or send a message through https://www.ridgerun.com/contact.