Sonde Health API Platform Documentation

Getting Started: Sonde Edge Passive SDK

Owned by
Aman Jain
Last updated: May 07, 2024
6 min read

 

The following is an overview of Sonde Edge Passive SDK. The following is included in this product packaging:

What is Sonde Edge Passive SDK

The Sonde Edge Passive SDK is an SDK that continuously or periodically monitors your vocal biomarkers based on your voice in real-time. Understanding that these scores do not replace medical evaluations and should be regarded as just one aspect of your overall health and wellness is crucial.

Sonde offers multiple different ways to acquire audio samples:

  • Cued mode - User-facing app asks a paragraph to read or a question. Users' responses if taken as the input voice for scoring.

  • Passive mode - The user-facing app is open in the background or foreground on the Edge device (such as a mobile phone or smart speaker), and the audio samples are collected for scoring. The audio if first verified that it is only of the consented user and then scored. Passive mode offers strong adherence and requires minimal effort from the user.

Sonde Edge Passive SDK supports both Passive and Cued modes.

Why Integrate Passive

Health changes over time. Vocal biomarkers are ideal for monitoring these changes in Brain and Respiratory health. However, adherence is low when users are asked to give a daily voice sample. Users may go for days or months without give a voice sample, even when the app sends a notification. Three major reasons for integrating Passive mode are:

  • High user adherence - as the user does not need to remember to give a daily audio sample.

  • Stronger scoring - as samples are collected daily and at different times of the day.

  • Strong security and privacy - as no passively collected audio is stored on the Edge or the Cloud.

SDK Capabilities

Sonde Edge Passive SDK supports the following (when integrated with a Partner’s app):

Audio acquisition

Acquiring 16 kHz sampled audio on the Edge device such as mobile phones, smart speakers.

Audio acquisition

Acquiring 16 kHz sampled audio on the Edge device such as mobile phones, smart speakers.

Voice Activity Detection

The audio acquired must have voice and silence or audio with background noise is not acquired.

User verification

The audio in real-time is verified that it is only enrolled user’s audio. All non-user audio is immediately deleted.

Audio stitching

To generate a score, 20 seconds of enrolled user audio is required. The SDK supports stitching of users' audio collected over a limited time window (for example, over 4 hours). Once 30 seconds of stitched audio is available, it is scored, and the audio is deleted.

Scoring

The SDK supports Mental Fitness scoring and soon (Q4 2024) will support Respiratory Risk score. Mental Fitness generates aggregate scores as well as component scores (smoothness, control, liveliness, energy range, clarity, crispness, speech rate, pause duration).

User enrollment

Included with the SDK is an enrollment module. A user needs to enroll their voice such that the voice signature template can be used to verify and score only the enrolled user.

User score history

The SDK does not keep score history. Partner’s app will need to keep this for longitudinal score display and/or trend insights

Real Time

Once 30 sec. of features are collected, the scores are generated in real-time.

Offline

The SDK does not require online/internet connectivity as audio is not shared with backend/cloud.

Privacy

The SDK’s verification module is near 99% accurate in only scoring enrolled user’s voice and deleting it immediately after generating features on 3 second audio segments.

SDK size

Android SDK is 34 Mbytes; iOS SDK is 66 Mbytes

 

 

 

Sonde Edge Passive SDK Integration with Partner’s Application

Partner’s application can integrate with Sonde’s SDK. Shown in the block diagram below is the core Passive SDK functions. In lighter shade are modules that Sonde has implemented in its Demo application and offers it as source code. In addition to this, Partner’s will develop an SDK manager, UI/UX, and score display function and screens (see sample score display screens in Sonde’s MF App). Sonde’s INSIGHT product provides insights on scores when Partners share scores with Sonde’s API Platform.

 

image-20240430-162415.png

 

Sonde Edge Passive SDK Architecture

The architecture consists of three major modules:

  • User Enrollment module:

    • The Enrollment module must be integrated when a Passive implementation is done. End users will be required to follow these steps:

      Step 1: Find a quiet place

      Step 2: Speak clearly and naturally into your microphone. Articulate each word distinctly to capture the finest nuances of their voice.

      Step 3: The enrollment process involves reading aloud a series of phrases and sentences. This will enable our system to create a comprehensive voice template for the user.

       

  • Verification module:

    • The Verification module must be integrated when a Passive implementation is done. End users will be required to follow these steps:

    • The Analysis module takes 3-sec audio segments and verifies each segment using the user’s voice template file generated in the Enrollment module. If the segment is verified, features are extracted from the 3-sec audio file, and the audio file is purged. If the audio is not verified (not the user’s audio), features are not extracted, and the audio is deleted.

    • SDK stitches all the verified segments extracted features of 30 sec. user-only audio.

    • The features are scored to generate Mental Health and/or respiratory scores.

    • After this, the analysis module continues verifying the next 3 sec segments and repeats all the above steps.

  • Score module

    • To get Mental Fitness scores, you need to call this module

    • This will return 9 scores consisting of 8 acoustic scores +1 aggregated score

    • Details regarding each score are described here: Mental Fitness Voice Feature Scores

 

 

Partner’s UI/UX flow with Sonde Edge Passive SDK

Below are sample screens that may be integrated with Partner’s app. This sample's screen flow is the same as in the sample application source code for Android and iOS implementation (for integrating the SDK). Sonde’s iOS Mental Fitness app on the Apple Store has a Passive demo.

The Passive demo video is here: https://vimeo.com/853149777?share=copy
Please get in touch with Sonde technical support if you would like to try out the Passive demo.

Best Practices for Sonde SDK Integration in Partner’s App

The following are functions that a Partner may want to add to their application:

User score history

The SDK does not keep score history. Partner’s app will need to keep this for longitudinal score display and/or trend insights

Metering

For purposes of vocal biomarker usage, Partner may want to collect usage information such as # of passive sessions completed, type of health screening, time/date stamp, etc.

Version

SDK version in the backend linked to the score history

Enrollment

Users should be asked to re-enroll if the user is not being verified . Partners will need to include an option to re-enroll

Background noise

When passive mode is enabled, users should be in a quiet environment. This can be added to user’s manual.

Distance

The device which is capturing the audio should be less than 3-4 feet from the user and the audio volume should be high

Device battery level

Partner’s app should not enable the Passive mode if the device battery is low

Passive mode enable

Users should be encouraged to keep the Passive mode enabled when they are in a conference call using their computer (on Zoom, Teams, etc.).

Foreground, Background, Call handling and Notifications

Android: Developer guide - Background, Call handling, and Notifications

Integrations Details

iOS Sonde Edge Passive SDK Integration: iOS Sonde Edge Passive SDK Integration
Android Sonde Edge Passive SDK Integration: Android Sonde Edge Passive SDK Integration

Sonde Health