Skip to main content

Glass break detection demo

Introduction - VK-RA6E1​

This tutorial demonstrates the development and testing of a Glass Break Detection demo using Renesas’s VK-RA6E1 Voice User Interface Kit and Reality AI Tools (Explorer Tier). Reality AI Tools enables the creation of AI models for non-visual sensing applications.

Overview of the sections

SectionDescription
1Prerequisites
2Data Collection
3Model Generation
4Deploying & testing
NOTE

Follow the sections sequentially for best results.

Section 1: Prerequisites​

This section outlines the hardware, software, and account prerequisites for running the Glass Break Detection demo.

Hardware Requirements​

To complete this tutorial, you will need the VK-RA6E1 Voice User Interface Kit. The kit includes:

  1. RA6E1 board
  2. USB cables (included)

Kit link:
TW001-VUIA6E1POCZ - RA6E1 Voice User Demonstration Kit | Renesas

Software Requirements​

Install the following software before proceeding:

  • Renesas e² Studio IDE, version 2024-07 or newer
  • Flexible Software Package (FSP) version 5.6

Download the platform installer:
setup_fsp_v5_6_0_e2s_v2024-10.exe

After installation, take note of the e² Studio installation directory.

You will also need the pre-configured e² Studio project for this lab, which contains all required settings and files. You can download the project here.

Reality AI Tools Account Access​

To access Reality AI Tools:

  1. Navigate to the login portal: https://portal.reality.ai/login
  2. Log in using your assigned username and password.

If you do not have credentials or access to the project files, contact the Renesas AI COE Customer Success team at rai-cs@dm.renesas.com.


INFO

Keep the Reality AI Tools portal open in the background during the tutorial.

Section 2: Data Collection​

This section explains how to use the Reality AI Eclipse plugin in e² Studio to collect labeled audio data for the Glass Break Detection demo.

Import the Example Project into e² Studio​

  1. Open e² Studio IDE, and in the Launcher, click Browse to choose a workspace directory.

  2. Click Launch to proceed.

  3. Navigate to File > Import in the top menu bar.

  4. Under General, select Existing Projects into Workspace and click Next

  5. Click Select root directory, then Browse to locate the example project folder.

  6. Click Finish to import the demo project into your workspace.

Create or Select a Project in Reality AI Tools​

  1. Open the Reality AI Tools portal.

  2. Click +Add Project.

  3. Enter a project name (e.g., Glass Break Demo) and optionally provide a description.

  4. Click +Add Project in the lower-right corner.

NOTE

If you're using an Explorer Tier subscription, you can skip creating a new project and use the existing Glass Break Detection project to upload and analyze data.

Generate and Apply an API Key​

  1. In the Reality AI portal, click your username in the top-right corner and select API Key.

  2. Copy the displayed API key. This key will be used to authenticate e² Studio with Reality AI Tools.

  3. Return to e² Studio and navigate to Renesas AI > Reality AI Authentication.
    (If the Renesas AI menu is missing, refer to the appendix for troubleshooting.)

  4. Click Add, paste the API key, and click Apply and Close.

Once added, e² Studio is now connected to Reality AI Tools.

Configure the Reality AI Data Storage Tool​

  1. In e² Studio, go to Renesas AI > Show View > Reality AI Data Storage Tool.

  2. A new panel will open at the bottom. Drag the tab to the main editor area if needed.

  3. In the Reality AI Data Storage Tool, select your project from the dropdown list.

  4. Click Data Connection.

  5. In the dialog that appears:

    • Choose the correct COM port.
    • Set Baud Rate to 921600.
    • Set communication parameters to 8-N-1.
    • Click Connect, then Close.

  6. Confirm that the status in the top left shows Receiving data... with a green check mark.

NOTE

This indicates a successful connection, not actual data collection.

Build and Run the Project​

  1. Click the Hammer icon to build the project.

  2. Ensure the build completes without errors.

  3. Click the Debug icon to start a debug session.

  4. If prompted by Windows Defender, click Allow Access.

  5. If prompted to switch perspectives, check Remember my decision and click Switch.

  6. Click the Resume button twice to begin program execution.

Verify Signal in Data Storage Tool​

  1. In the Reality AI Data Storage Tool, go to the Signal view to confirm waveform activity.

  2. If the waveform is not visible:

    • Click File Review, then switch to Live View (highlighted below).
    • If the waveform is still not visible, verify all previous steps and ensure device connection.

Record Class-Specific Audio Data​

  1. In the Reality AI Data Storage Tool, enter a name in the Class Name field.

  2. Select the Unlimited checkbox to record variable-length data.

NOTE:

There are two data collection modes:

  • Unlimited: Recording continues until manually stopped.
  • Number of frames per file: Automatically stops when the set frame count is reached.

For this tutorial, select Unlimited.

tip

Setting the number of frames to 200 captures approximately 52 seconds of data.

  1. Enter the class name as glass_break.

  2. Play the provided glass break audio clips through your phone or PC speakers while recording.

  3. Once done, start a new recording session.

  4. Change the class name to everything_else. Record environmental sounds such as silence, typing, or talking for 2–3 minutes.

    The goal is to capture ambient noise for model contrast.

End Data Collection​

  1. Click the Red Square icon to terminate the debug session.

If the icon is not visible, switch to the Debug perspective.

  1. The collected data should now appear in the list of recorded files, along with metadata.

Upload Data to Reality AI Tools​

You can now upload the data to Reality AI Tools using Upload and run analysis on it. 

  1. Select all recorded data files (use Ctrl + Click to select multiple files).

  2. Scroll the tool window to reveal the Upload button.

  3. Click Upload.

  4. In the dialog box:

    • Choose the appropriate project (e.g., Glass Break Demo).
    • Set the Sampling Rate to 4096 Hz.

  5. Click OK to upload the data.

  6. After upload, a confirmation message will appear. The data and metadata.csv file will now be available in your Reality AI Tools project.

Section 3: Creating Models​

This section explains how to create an AI model using the labeled data collected in the previous step. You will format, segment, train, and finally deploy the model using Reality AI Tools.

NOTE:

A Reality AI Tools Explorer Tier account is required. If you do not have access, contact the Renesas AI COE Customer Success team at rai-cs@dm.renesas.com.

Log in to Reality AI Tools​

  1. Open a browser and go to https://portal.reality.ai/login.

  2. Log in using the credentials provided by Renesas.

Format the Collected Data​

  1. Select the Glass Break Demo project to set it as the active project.

  2. On the left sidebar, click the Data tab to expand it.

  3. Follow these steps:

    • Click Curate.
    • Expand the Source Files section.
    • Select all files by checking the box next to File Name.
    • Click the Action button.
    • Choose Format Selected.

  4. In the formatting dialog:

    • Expand the Data dropdown menu.
    • Use the scroll bar or mouse wheel to navigate.
    • Set the second class as the Target Class.
    • Assign the Sample Rate to 4096 Hz.
    • Click Confirm.

  5. Confirm that:

    • The Data Shape is 1 x (number of samples).
    • The Sample Rate is 4096 Hz.

Segment the Data for Model Training​

  1. In the Curate view, click the Action button again.

  2. Select Segment List from Selected.

NOTE:

Why Segmentation? One of the primary goals of using Reality AI Tools is to generate models that can be deployed on various Renesas MCUs. These models are expected to operate in resource-constrained environments and must process live data efficiently. For practical real-time performance, the models are typically designed to analyze short time windows—such as 1 second, 500 milliseconds, or even less—rather than extended data streams lasting several seconds, minutes, or hours. To replicate these conditions during training, the raw data is segmented into shorter windows and fed into the model generation engine, enabling the model to learn patterns similar to those it will encounter in a live production environment.

  1. In the segmentation dialog, do the following:
    • Set Window Length to 4096.
    • Click the 50% button to select a 50% offset.
    • Enter a descriptive name for the segment list.
    • Click Submit.

Note: The estimated number of samples depends on your data volume. Your sample count may differ from the screenshots shown.

Terms:​

  • Window Length: The number of samples per segment the model uses to classify an input.

  • Offset: The amount of overlap between segments.

  • 50% Overlap: A standard choice to balance coverage and data diversity.

  • Use non-overlapping segments for large datasets.

  • Use all shifts (offset = 1) to simulate a continuous stream of live data for validation.

Launch AI Explore to Train the Model​

  1. Go to AI Explore > Classes in the Reality AI portal.
NOTE

Reality AI Tools supports three types of models:

  • Classes: When there is labeled categorical data for categorical classification (used in this tutorial). You might have noticed that we uploaded perfectly labeled data in section 1. This is needed for classification models as supervised learning is being performed.  
  • Values: For regression models (e.g., predicting temperature, tire pressure). When discrete int or float values are used instead of categories. This is also supervised learning.  
  • Anomalies: For semi-supervised anomaly detection using only "normal" data samples to create a baseline model.

  1. Select the segment list you created and click Start Exploring.

  2. Once the training is complete:

    • Review the generated models.
    • Select the model with the best performance (we recommend using Spectral Magnitude as the feature space).
    • Click Create Base Tool to save the model.

Hover over each model's explanation icon to view detailed insights.

NOTE

What happens during AI Explore? Reality AI Tools automatically evaluates hundreds of feature sets and classification models using a subset of your data. The tool performs:

  • Feature selection and dimensionality reduction
  • Model training using supervised learning
  • 10-fold cross-validation to estimate accuracy and resource usage K-Fold Cross Validation Explained:
    The dataset is divided into K parts (K=10). Each fold is used once as a test set while the remaining are used for training. This repeats K times to assess performance stability.
  1. Provide a name for the Base Tool or accept the default. Click Add to save.

    The tool icon will update to reflect successful creation.

Package and Export the Trained Model​

  1. Navigate to Deploy > Embedded.

  2. In the Trained Tool Description list, select your trained model.

  3. Click + New Package.

  4. Set the deployment options:

    • Use a name like glass_break_lsp_v2 to match the function calls in the e² Studio project.
    • If you use a different name, update the code references accordingly.

  5. Click Generate Package.

The package generation process typically takes 10–15 minutes.

  1. Once ready, download the generated .zip archive.

Integrate the Model into e² Studio​

  1. Extract the downloaded archive. It should contain nine files.

  2. Copy all extracted files to the following path in your e² Studio project:
    src > realityai

Important: Overwrite any existing files.
File names may vary based on the name given during model deployment.

  1. Confirm that the model function call is declared in glass_break_lsp_v2_model.h.

You have now successfully trained, packaged, and integrated the AI model into your development environment.

Section 4: Deploying & testing the model​

This section describes how to deploy the trained model onto the RA6E1 hardware board and test its real-time inference performance using live audio input. It also includes guidance for troubleshooting and validating model accuracy using built-in tools.

Configure the Model Header and Class Reference​

  1. Open the hal_entry.c file in your e² Studio project.

  2. If you used the default deployment name (glass_break_lsp_v2), no changes are needed.

  3. If a custom name was used, go to line 37 and replace the existing include path with the name of your generated model's header file.

  4. Update the class reference in line 286 to match the name used during data recording (for example, glass_break).

Build and Run the Inference Project​

  1. Build the project using the Hammer icon in the e² Studio toolbar.

  2. After a successful build, start a debug session by clicking the Debug icon.

  3. Once the program launches, click the Resume button twice to begin real-time model inference mode.

Monitor Live Inference Results​

  1. Disconnect the Reality AI Data Storage Tool if it is still active.

  2. Navigate to Renesas AI > Show View > Reality AI Live Monitor.

  3. Click the Data Connection button.

  4. In the connection window:

    • Select the appropriate COM Port.
    • Set the Baud Rate to 921600.
    • Click Connect, then Close.

  5. Once connected, you will see live model predictions on the chart.

Interpretation:

  • Class 1 represents the everything_else label.
  • Class 2 represents the glass_break label.
  • Higher orange bars on the Y-axis indicate a stronger prediction for that class.
  1. To test the model, play a few glass break audio clips (e.g., from YouTube) near the microphone. Class 2 should become dominant in the prediction chart.

Troubleshooting Tips​

TIP

If the model does not perform as expected:

  • Collect additional training data to improve model performance under diverse conditions.
  • Ensure consistency between the data collection environment and the testing setup.
  • Retrain the model with updated samples for better accuracy and generalization.

Validate the Model Using Hardware-In-Loop Testing​

  1. Navigate to Renesas AI > Hardware In-Loop Test.

  2. In the Inference Package section:

    • Select the deployed model from your Reality AI Tools account.
    • If you are using the Explorer Tier, choose the glass_break_lsp_v1 model (a preloaded option).

  3. Ensure that the hardware is connected and the correct COM Port is selected.

  4. Click Start Test.

The plugin will:

  • Generate a test project locally.
  • Use a subset of training data (simulation data) to validate the model on actual hardware.
  1. After the test begins, you will see two key metrics displayed:
    • Inference Time
    • Overall Accuracy

You have now successfully deployed the trained AI model to hardware, verified its functionality using live input, and validated its performance using simulation data.

Section 5: Appendix​

If Reality AI options do not appear in e² Studio:

  1. Go to Help > Install Renesas IDE Features….

  1. Select Reality-AI features and click Finish.

  1. Restart e² Studio when prompted.