At the Edge: AI performance Cannot be Tiny Anymore (Part 2 of 2)
By Mallik Moturi
In the first part of this series, we explored the challenges posed by the current approach to edge AI, such as scalability issues, escalating network and cloud costs, and the limitations of legacy processors. As the demand for smarter, faster, and more secure AI applications grows, it is becoming increasingly clear that edge AI cannot be tiny anymore. In this follow-up, we will discuss the reasons behind this shift and introduce Syntiant’s new NDP250 chip as a prime example of how advanced edge AI processors are addressing these challenges and making edge AI a reality.
The Evolution of Edge AI Processors
Edge AI technology has made significant strides, driven by the need for more efficient and powerful processors capable of handling complex AI tasks locally. This evolution is crucial for reducing dependency on cloud computing, minimizing latency, and enhancing data security. The trend toward more powerful edge processors is driven by one key factor: demand for natural interfaces.
The rise of conversational AI and natural interfaces driven by large language models (LLMs) and generative AI necessitates more powerful edge processors. Specifically, there is a growing need for fast, concurrent and efficient automatic speech recognition (ASR), vision processing, and sensing through motion and vibration detection to operate efficiently at the edge, enabling real-time interactions and responses.
The Drawbacks of MCUs and MPUs with AI Accelerators
Microcontroller and microprocessor units (MCUs and MPUs) with AI accelerators are being offered as a choice for edge AI applications. However, they come with significant limitations:
Limited Neural Network Capacity: MCUs typically have limited memory and computational capacity, which restricts the size and complexity of the AI models they can run. This makes them unsuitable for more demanding AI tasks that require larger neural networks.
Scalability Issues: As AI models become more complex, the scalability of MCUs becomes a significant bottleneck. They struggle to keep up with the increasing demands for higher performance and more sophisticated AI capabilities.
Energy Constraints: Although MPUs have larger processing capabilities running at GHz processor speeds, they are not designed to be energy-efficient. Most have limited AI processing power with longer processing times for AI tasks, which can negate many of the advantages of AI at the edge for creating natural interfaces.
Insights from tinyML Summit 2024 and AI Virtual Forum
A major portion of the tinyML Foundation's recent summit 2024 and the virtual forum on Generative AI and Foundation Models on the Edge was focused on TinyML 2.0 and the coming of generative AI (Gen AI). This shift marks a significant change in the landscape of tinyML (machine learning on microcontrollers) as the integration of Gen AI promises to bring even more advanced capabilities to edge devices. During this event, the presentation "Edge of Tomorrow" highlighted how next-generation AI models and processors will transform edge computing by enabling more complex and efficient AI tasks.
Introducing Syntiant's NDP250
The Syntiant NDP250 Neural Decision Processor™, unveiled at Embedded World 2024, is a 3rd generation, special-purpose chip designed for edge AI processing in battery-powered devices and other power-constrained systems. It represents a significant advancement in terms of network size and capabilities over the previous generation of Neural Decision Processors, offering several key improvements that address the challenges of modern edge AI applications.
NDP250 : Next Generation Edge AI for Always On Vision, Audio, and Sensing
Key features and benefits of the NDP250 for always-on vision, audio and sensing:
Enhanced Processing Power: Built using the Syntiant Core 3™ programmable deep learning architecture, the NDP250 delivers a 5x increase in machine learning performance compared to its predecessors. It supports dozens of application-defined imaging, speech, audio and sensor behaviors for a variety of use cases, including person presence detection, object classification, on-device ASR, motion tracking, acoustic event and scene classification, and multi-microphone full duplex voice communication.
Memory: With its increased memory capacity, the NDP250 can support more than 6 million neural parameters in 8-bit mode, making it capable of running larger AI models that were previously impractical on microcontrollers.
Energy Efficiency: The NDP250 is designed to perform high-speed AI inference while maintaining extremely low power consumption. This makes it ideal for battery-operated devices that need to be always-on. Its integrated power management unit (PMU) allows for single power rail operation, further reducing system costs. The processor's low power consumption, running powerful always-on image recognition at under 30mW, makes it suitable for always-on applications, addressing the energy constraints battery operated AI Cameras.
Neural Network Versatility: The NDP250 can run multiple deep neural networks (DNNs) on various architectures concurrently, including fully connected, convolutional and recurrent networks such as LSTM and GRU. This versatility enables it to handle a wide range of AI tasks locally.
Real-World Applications
Syntiant’s goal is to provide an end-to-end solution from sensors to software, ensuring a seamless integration of edge AI capabilities across various devices and applications. The NDP250 plays a crucial role in this ecosystem by offering advanced processing power and energy efficiency. The company’s hardware and software technologies deliver the complete journey from physical phenomena to actionable inputs for real world applications.
Syntiant hardware and software technologies deliver the complete journey from physical phenomena to actionable inputs for real world applications.
Following are some of the real-world applications of the NDP250 across different sectors:
AUTOMOTIVE
In-Cabin Monitoring for Safety and Security: The NDP250 can be used for in-cabin monitoring to enhance safety and security. It can monitor the driver for signs of fatigue or distraction and alert them accordingly. Additionally, it can monitor passengers to ensure seat belts are fastened and detect unauthorized entry.
Sound-Based Alerts: The NDP250 can process audio data to detect specific sounds such as glass breaking, triggering alerts to enhance vehicle security.
Hands-Free Camera Activation: The NDP250 can enable hands-free camera activation based on voice commands or detected sounds, allowing for seamless operation of in-cabin monitoring systems without the need for manual intervention.
Battery Monitoring: In electric vehicles, the NDP250 can monitor the performance of a large number of battery cells, potentially up to 200 cells, in real-time. This ensures optimal operation without the need for constant cloud connectivity, reducing costs and enhancing data security by keeping sensitive information local.
SMART HOME DEVICES
Smart Speakers: Devices like smart speakers can benefit from the NDP250's enhanced AI processing power to provide faster and more accurate voice recognition and responses, improving user interactions. It also supports additional processing such as noise suppression and voice quality enhancement, ensuring clear and high-quality audio.
Home Security Systems: The NDP250 can enhance home security systems with capabilities such as person detection, motion tracking and sound classification, providing comprehensive security monitoring and alerts.
Smart Appliances: Smart appliances can use the NDP250 for voice-activated controls and contextual understanding, making them more intuitive and user-friendly.
Video Doorbells: In video doorbells, the NDP250 can combine vision processing with audio capabilities to recognize and respond to triggers such as motion detection or a doorbell ring, waking up the main processing unit only when necessary.
HEALTHCARE
Wearable Devices: Wearable health monitoring devices can leverage the NDP250 to perform complex health monitoring tasks locally, such as detecting irregular heartbeats or analyzing sleep patterns, providing immediate insights to users while ensuring data privacy. Additional voice processing capabilities include noise suppression and signal enhancement to ensure clear emergency communication in various noisy environments for healthcare professionals, in addition to hands free operation.
Medical Devices: The NDP250 can also be used in medical devices to monitor vital signs in real-time, enabling early detection of potential health issues and immediate intervention.
Battery Performance Enhancement: The NDP250 can enhance battery performance in wearable and medical devices to ensure they remain operational for extended periods, enhancing user reliability and device longevity.
BATTERY-POWERED CAMERAS
Wakeup Cameras: The NDP250 can be used in security cameras as a wakeup mechanism, activating higher processing modes when specific triggers like motion or sound are detected. This ultra-low power processing capability helps conserve battery life while ensuring that important events are captured and processed efficiently.
Wildlife Cameras: For wildlife monitoring, the NDP250 can wake up the main camera system upon detecting specific movements or sounds, ensuring that battery power is conserved and only relevant data is recorded.
Battery Status: The NDP250 can continuously monitor battery status in these cameras to ensure they remain operational in remote locations for extended periods.
INDUSTRIAL
Machinery Monitoring: In industrial environments, the NDP250 can process data from multiple sensors, including motion and vibration, to detect anomalies, predict maintenance needs, and optimize performance without the need for continuous cloud connectivity.
Environmental Monitoring: The NDP250 can be used to monitor environmental conditions such as temperature, humidity, and air quality, providing real-time data to ensure safety and compliance with regulations.
Quality Control: In manufacturing, the NDP250 can enhance quality control processes by analyzing visual and sensor data to detect defects and ensure product consistency.
Battery Packs: For large battery packs in industrial energy storage systems, the NDP250 can monitor the performance of a huge number of battery cells in its entirety in real-time. This ensures long-term operational efficiency and reducing downtime.
CONSUMER DEVICES
Headsets and True Wireless Stereo (TWS) Devices: The NDP250 can enhance audio processing in headsets and TWS devices by providing features such as noise suppression, voice quality enhancement and low-latency audio recognition, improving overall user experience.
Laptops and Mobile Phones: In laptops and mobile phones, the NDP250 can enable advanced voice recognition, real-time communication and context-aware interactions, enhancing productivity and user engagement.
Smart Wearables: The NDP250 can be used in smart wearables to provide always-on voice recognition, health monitoring and contextual alerts, ensuring seamless and efficient operation without draining battery life.
Battery Enhancement: The NDP250 can manage battery charging and performance in consumer personal devices, ensuring faster charging, optimal operation and extending battery life.
These applications highlight the versatility and efficiency of the NDP250, making it a key component in the advancement of edge AI technology across various sectors.
The Future of Edge AI
The launch of the NDP250 represents a significant step forward in the evolution of edge AI. As processors continue to grow stronger and more efficient, we can expect to see an even greater shift toward local AI processing. This will enable more intelligent, responsive, and secure applications across various domains.
In conclusion, the advancements in edge AI processors, exemplified by the Syntiant NDP250, are addressing the limitations of current approaches and paving the way for a more efficient and powerful future. As these technologies continue to evolve, they will unlock new possibilities for businesses and consumers alike, driving innovation and enhancing the capabilities of edge devices.
Mallik Moturi is Chief Business Officer at Syntaint.
References
LinkedIn Article: At the Edge: AI performance cannot be tiny anymore (Part 1 of 2)
Syntiant Press Release: Syntiant Unveils NDP250 Neural Decision Processor with Next-Gen Core 3 Architecture
tinyML Foundation Event: tinyAI Virtual Forum on Generative AI and Foundation Models on the Edge
Syntiant Product Brief: Syntiant NDP250 Product Brief
Syntiant Presentation: Edge of Tomorrow Presentation at tinyAI Virtual Forum