電子發(fā)燒友網(wǎng)>電子資料下載>電子資料>教程第2部分：帶有OV7670攝像頭模塊的TinyML

教程第2部分：帶有OV7670攝像頭模塊的TinyML

2075461 2022-10-24 | zip | 0.32 MB | 次下載 | 2積分

資料介紹

描述

這是我在 TensorFlow 下使用 Google Summer of Code (GSoC) 的第二個(gè)項(xiàng)目。互聯(lián)網(wǎng)上沒(méi)有合適的文檔來(lái)構(gòu)建自定義圖像識(shí)別 TinyML 模型，所以我的 GSoC 導(dǎo)師 Paul Ruiz 建議我應(yīng)該嘗試解決它。下面介紹了如何構(gòu)建圖像識(shí)別 TinyML 應(yīng)用程序。快樂(lè)修補(bǔ)！

項(xiàng)目背后的理念：

我想解決變量較少的問(wèn)題，因?yàn)橛嘘P(guān)如何使用相機(jī)模塊和處理其數(shù)據(jù)的文檔不是很好。我選擇構(gòu)建一個(gè) MNIST TinyML 模型，因?yàn)樵谶@種情況下，我不需要擔(dān)心訓(xùn)練數(shù)據(jù)集，它可以讓我專注于項(xiàng)目的基本部分來(lái)啟動(dòng)和運(yùn)行。但是，既然我已經(jīng)弄清楚了構(gòu)建自定義圖像識(shí)別項(xiàng)目的所有部分，我已經(jīng)記錄了如何使用相機(jī)模塊收集訓(xùn)練數(shù)據(jù)集。

博客的主題/基調(diào)？

我想警告你，這個(gè)博客可能會(huì)有點(diǎn)難以理解。對(duì)此有一個(gè)正確的解釋：使用基于加速度計(jì)的應(yīng)用程序，只需在串行監(jiān)視器或繪圖儀上打印出一個(gè)軸的加速度計(jì)值，就可以很容易地進(jìn)行完整性檢查。相比之下，對(duì)圖像識(shí)別應(yīng)用程序進(jìn)行健全性檢查至少要麻煩 10 倍，因?yàn)闊o(wú)法實(shí)時(shí)可視化檢查一段代碼是否正在執(zhí)行所需的操作。

一些評(píng)論

由于單元測(cè)試的復(fù)雜性，這個(gè)博客可能有點(diǎn)難以理解。我想通過(guò)讀者的反饋來(lái)解決解釋中的任何空白。因此，請(qǐng)?jiān)谙旅嬖u(píng)論您對(duì)嵌入式系統(tǒng)圖像識(shí)別相關(guān)的任何疑問(wèn)和問(wèn)題。

TinyML 是否有意義？

我建議您閱讀TinyML 書(shū)的作者 Pete Warden 撰寫(xiě)的這篇精彩文章，以了解為什么在微控制器上運(yùn)行機(jī)器學(xué)習(xí)模型是有意義的，并且是機(jī)器學(xué)習(xí)的未來(lái)。

即使 TinyML 有意義，圖像識(shí)別在 TinyML 上有意義嗎？

我們將在這里使用的 OV7670 相機(jī)的完整 VGA（640×480 分辨率）輸出對(duì)于當(dāng)前的 TinyML 應(yīng)用程序來(lái)說(shuō)太大了。uTensor 通過(guò)使用 28×28 圖像的 MNIST 運(yùn)行筆跡檢測(cè)。TensorFlow Lite for Microcontrollers 示例中的人員檢測(cè)示例使用 96×96，這綽綽有余。即使是最先進(jìn)的“Big ML”應(yīng)用程序通常也只使用 320×320 的圖像。總之，在微型微控制器上運(yùn)行圖像識(shí)別應(yīng)用程序很有意義

本教程簡(jiǎn)而言之：

集成攝像頭和 LCD 模塊
構(gòu)建 MNIST TinyML 模型
測(cè)試 TinyML 模型
結(jié)論

8.集成攝像頭和液晶模組

8.a TFT+OV7670：顯示測(cè)試

代碼說(shuō)明：

tft.fillScreen(ST77XX_BLACK);

這行代碼用黑色填充屏幕。

for(int i =0; i<28;i++){
    for(int j =0;j<28;j++){
      tft.drawPixel(i,j,ST77XX_GREEN);
      delay(0);
    }
  }
  delay(1000);

這些代碼行繪制了一個(gè) 28x28 的綠色塊。

草圖：

#include     // Core graphics library
#include  // Hardware-specific library for ST7735
#include 


  // For the breakout board, you can use any 2 or 3 pins.
  // These pins will also work for the 1.8" TFT shield.
  #define TFT_CS        A7
  #define TFT_RST        7 // Or set to -1 and connect to Arduino RESET pin
  #define TFT_DC         A6

// For 1.44" and 1.8" TFT with ST7735 use:
Adafruit_ST7735 tft = Adafruit_ST7735(TFT_CS, TFT_DC, TFT_RST);
#include 

uint16_t pixels[176 *144];
uint16_t color;
uint8_t red, blue, green;

void setup(void) {
  Serial.begin(9600);

  // Use this initializer if using a 1.8" TFT screen:
  tft.initR(INITR_BLACKTAB); 
  delay(1000);// Init ST7735S chip, black tab
  Serial.println(F("Initialized"));
 
  if (!Camera.begin(QCIF, RGB565, 1)) {
    Serial.println("Failed to initialize camera!");
    while (1);
  }
  Serial.println(F("Initialized"));
  


  // large block of text
  tft.fillScreen(ST77XX_BLACK);
  
  
}

void loop() {
  for(int i =0; i<28;i++){
    for(int j =0;j<28;j++){
      tft.drawPixel(i,j,ST77XX_GREEN);
      delay(0);
    }
  }
  delay(1000);
 tft.fillScreen(ST77XX_BLACK);
}

8.b TFT + OV7670：靜態(tài)圖像測(cè)試

現(xiàn)在我們已經(jīng)知道如何顯示綠色塊，讓我們從存儲(chǔ)的 HEX 值數(shù)組中顯示圖像。

代碼說(shuō)明：

uint16_t pixels[176 * 144]= {0x0D2A,0xED29,0xED29,0xED29,.....0x95B5,0xB6B5,0xB6B5};

該數(shù)組存儲(chǔ)所有像素的十六進(jìn)制值。

for(int i =0; i<50;i++){
    for(int j =0;j<50;j++){
       pixel = pixels[176*i +j];
       tft.drawPixel(i,j,pixel);
}
}

這些代碼行通過(guò)陣列循環(huán)并在 TFT LCD 顯示器上繪制圖像。

草圖：

#include     // Core graphics library
#include  // Hardware-specific library for ST7735
#include 

  #define TFT_CS        A7
  #define TFT_RST        7 // Or set to -1 and connect to Arduino RESET pin
  #define TFT_DC         A6


Adafruit_ST7735 tft = Adafruit_ST7735(TFT_CS, TFT_DC, TFT_RST);

uint16_t pixels[176 * 144]= {0x0D2A,0xED29,0xED29,0xED29,.....0x95B5,0xB6B5,0xB6B5};

uint16_t color, pixel;
uint8_t red, blue, green;

void setup() {
  Serial.begin(9600);
  while (!Serial)
    delay(10);
 // Serial.print(F("Hello! ST77xx TFT Test"));
  tft.initR(INITR_BLACKTAB); 
  delay(1000);
  // large block of text
  tft.fillScreen(ST77XX_BLACK);
  
  
}

void loop() {
  for(int i =0; i<50;i++){
    for(int j =0;j<50;j++){
       pixel = pixels[176*i +j];
        red   = ((pixel >> 11) & 0x1f) << 3;
        green = ((pixel >> 5) & 0x3f) << 2; 
       blue  = ((pixel >> 0) & 0x1f) << 3; 
      color = tft.color565(red+50, green+50, blue+50);
      Serial.print(red);
      Serial.print(", ");
      Serial.print(green);
      Serial.print(", ");
      Serial.println(blue); 
      tft.drawPixel(i,j,color);
      delay(0);
    }
  }
  delay(100);
  //tft.fillScreen(ST77XX_BLACK);
}

8.c TFT + OV7670：Liveimagetest

現(xiàn)在我們已經(jīng)知道如何從存儲(chǔ)的 HEX 值數(shù)組中顯示圖像，讓我們擴(kuò)展它以顯示實(shí)時(shí)圖像。

代碼說(shuō)明：

uint16_t pixels[176 * 144];

這行代碼聲明了一個(gè)數(shù)組來(lái)存儲(chǔ)從相機(jī)捕獲的圖像。

Camera.readFrame(pixels);

這行代碼從相機(jī)讀取一幀并將其存儲(chǔ)在像素?cái)?shù)組中。

for (int i = 0; i < 112; i++) 
{
   for(int j = 0; j < 112; j++)
   {
      uint16_t pixel = pixels[176*i +j];
      tft.drawPixel(i,j,pixel);
   }
}

這些代碼行遍歷像素陣列并在 TFT LCD 顯示器上繪制圖像。

草圖：

/*
  OV767X - Camera Test Pattern

  This sketch waits for the letter 'c' on the Serial Monitor,
  it then reads a frame from the OmniVision OV7670 camera and 
  prints the data to the Serial Monitor as a hex string.

  The website https://rawpixels.net - can be used the visualize the data:
    width: 176
    height: 144
    RGB565
    Little Endian

  Circuit:
    - Arduino Nano 33 BLE board
    - OV7670 camera module:
      - 3.3 connected to 3.3
      - GND connected GND
      - SIOC connected to A5
      - SIOD connected to A4
      - VSYNC connected to 8
      - HREF connected to A1
      - PCLK connected to A0
      - XCLK connected to 9
      - D7 connected to 4
      - D6 connected to 6
      - D5 connected to 5
      - D4 connected to 3
      - D3 connected to 2
      - D2 connected to 0 / RX
      - D1 connected to 1 / TX
      - D0 connected to 10

  This example code is in the public domain.
*/

#include 
#include     // Core graphics library
#include  // Hardware-specific library for ST7735
#include 

#define TFT_CS        A7
  #define TFT_RST        7 // Or set to -1 and connect to Arduino RESET pin
  #define TFT_DC         A6


Adafruit_ST7735 tft = Adafruit_ST7735(TFT_CS, TFT_DC, TFT_RST);

uint16_t pixels[176 * 144];
void setup() {
  Serial.begin(9600);
  while (!Serial);

  Serial.println("OV767X Camera Capture");
  Serial.println();

  tft.initR(INITR_BLACKTAB); 
  delay(1000);
  
  if (!Camera.begin(QCIF, RGB565, 1)) {
    Serial.println("Failed to initialize camera!");
    while (1);
  }

  Serial.println("Camera settings:");
  Serial.print("\twidth = ");
  Serial.println(Camera.width());
  Serial.print("\theight = ");
  Serial.println(Camera.height());
  Serial.print("\tbits per pixel = ");
  Serial.println(Camera.bitsPerPixel());
  Serial.println();

  Serial.println("Send the 'c' character to read a frame ...");
  Serial.println();
}

void loop() {
    Serial.println("Reading frame");
    Serial.println();
    Camera.readFrame(pixels);
    tft.fillScreen(ST77XX_BLACK);
    for (int i = 0; i < 112; i++) {
      for(int j = 0; j < 112; j++){
       uint16_t pixel = pixels[176*i +j];
       int red   = ((pixel >> 11) & 0x1f) << 3;
       int green = ((pixel >> 5) & 0x3f) << 2; 
       int blue  = ((pixel >> 0) & 0x1f) << 3; 
      
      Serial.println(red);
      Serial.println(green);
      Serial.println(blue);
      tft.drawPixel(i,j,pixels[176*i+j]);
      }
    }
    delay(2000);
}

9. 構(gòu)建 MNIST TinyML 模型

現(xiàn)在我們已經(jīng)讓相機(jī)和顯示器工作并集成了，讓我們構(gòu)建機(jī)器學(xué)習(xí)模型來(lái)識(shí)別數(shù)字。

9.a 探索 Colab 筆記本

代碼說(shuō)明：

import numpy as np                   # advanced math library
import matplotlib.pyplot as plt      # MATLAB like plotting routines
import random                        # for generating random numbers

from keras.datasets import mnist     # MNIST dataset is included in Keras
from keras.models import Sequential  # Model type to be used

from keras.layers.core import Dense, Dropout, Activation # Types of layers to be used in our model
from keras.utils import np_utils                         # NumPy related tools
import tensorflow as tf

tf.config.run_functions_eagerly(True)

這些代碼行導(dǎo)入必要的庫(kù)來(lái)構(gòu)建和可視化我們的模型。

(X_train, y_train), (X_test, y_test) = mnist.load_data()
print("X_train shape", X_train.shape)
print("y_train shape", y_train.shape)
print("X_test shape", X_test.shape)
print("y_test shape", y_test.shape)

>>X_train shape (60000, 28, 28)
>>y_train shape (60000,)
>>X_test shape (10000, 28, 28)
>>y_test shape (10000,)

這些代碼行加載 MNIST 測(cè)試并將圖像訓(xùn)練到正確的變量中。

plt.rcParams['figure.figsize'] = (9,9) # Make the figures a bit bigger

for i in range(9):
    plt.subplot(3,3,i+1)
    num = random.randint(0, len(X_train))
    plt.imshow(X_train[num], cmap='gray', interpolation='none')
    plt.title("Class {}".format(y_train[num]))
    
plt.tight_layout()

來(lái)自 MNIST 數(shù)據(jù)集的一些圖像

這些代碼行從 MNIST 數(shù)據(jù)集的訓(xùn)練數(shù)據(jù)中可視化了九個(gè)不同的圖像。

def matprint(mat, fmt="g"):
    col_maxes = [max([len(("{:"+fmt+"}").format(x)) for x in col]) for col in mat.T]
    for x in mat:
        for i, y in enumerate(x):
            print(("{:"+str(col_maxes[i])+fmt+"}").format(y), end=",")
        print("")
     
matprint(X_train[num])

這些代碼行將訓(xùn)練數(shù)據(jù)中的隨機(jī)圖像顯示為值數(shù)組。

from keras.preprocessing.image import ImageDataGenerator
from keras.layers import Conv2D, MaxPooling2D, ZeroPadding2D, GlobalAveragePooling2D, Flatten
from keras.layers import BatchNormalization

這些代碼行導(dǎo)入必要的層來(lái)構(gòu)建機(jī)器學(xué)習(xí)模型。

# Again, do some formatting
# Except we do not flatten each image into a 784-length vector because we want to perform convolutions first

X_train = X_train.reshape(60000, 28, 28, 1) #add an additional dimension to represent the single-channel
X_test = X_test.reshape(10000, 28, 28, 1)

X_train = X_train.astype('float32')         # change integers to 32-bit floating point numbers
X_test = X_test.astype('float32')

#X_train /= 255                              # normalize each value for each pixel for the entire vector for each input
#X_test /= 255

print("Training matrix shape", X_train.shape)
print("Testing matrix shape", X_test.shape)

這些代碼行對(duì)訓(xùn)練和測(cè)試數(shù)據(jù)進(jìn)行預(yù)處理以使其正常工作，例如：標(biāo)準(zhǔn)化、Float64 到 Float32 的轉(zhuǎn)換和重塑。

# one-hot format classes

nb_classes = 10 # number of unique digits

Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)

這些代碼行 one-hot 對(duì)訓(xùn)練和測(cè)試圖像的標(biāo)簽進(jìn)行編碼。

from keras.layers.convolutional import DepthwiseConv2D
from keras.backend import relu
from keras.activations import softmax
model = Sequential()                                 # Linear stacking of layers

model.add(DepthwiseConv2D((3,3),input_shape=(28,28,1)))
# Convolution Layer 1
model.add(Conv2D(2, (3, 3)))      # 2 different 3x3 kernels -- so 2 feature maps
model.add(BatchNormalization(axis=-1))               # normalize each feature map before activation
convLayer1 = Activation('relu')                     # activation
model.add(convLayer1)

# Convolution Layer 2
model.add(Conv2D(2, (3, 3)))                        # 2 different 3x3 kernels -- so 2 feature maps
model.add(BatchNormalization(axis=-1))               # normalize each feature map before activation
model.add(Activation('relu'))                        # activation
convLayer2 = MaxPooling2D(pool_size=(2,2))          # Pool the max values over a 2x2 kernel
model.add(convLayer2)

# Convolution Layer 3
model.add(Conv2D(4,(3, 3)))                         # 4 different 3x3 kernels -- so 4 feature maps
model.add(BatchNormalization(axis=-1))               # normalize each feature map before activation
convLayer3 = Activation('relu')                     # activation
model.add(convLayer3)

# Convolution Layer 4
model.add(Conv2D(4, (3, 3)))                        # 4 different 3x3 kernels -- so 64 feature maps
model.add(BatchNormalization(axis=-1))               # normalize each feature map before activation
model.add(Activation('relu'))                        # activation
convLayer4 = MaxPooling2D(pool_size=(2,2))          # Pool the max values over a 2x2 kernel
model.add(convLayer4)

model.add(Flatten())                                 
model.add(Dense(5,activation = relu))                           
model.add(Dense(10, activation = softmax))

這些代碼行定義了機(jī)器學(xué)習(xí)模型中的實(shí)際層。

model.summary()

這行代碼向用戶顯示有關(guān)模型架構(gòu)的信息。

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

這行代碼定義了機(jī)器學(xué)習(xí)模型在訓(xùn)練時(shí)要使用的損失、優(yōu)化器和其他指標(biāo)。

history = model.fit(X_train,Y_train, steps_per_epoch=60000//128, epochs=3, verbose=1, 
                    validation_data=(X_test,Y_test))

這行代碼訓(xùn)練機(jī)器學(xué)習(xí)模型。

score = model.evaluate(X_test, Y_test)
print('Test score:', score[0])
print('Test accuracy:', score[1])

這些代碼行評(píng)估機(jī)器學(xué)習(xí)模型并將準(zhǔn)確性和分?jǐn)?shù)打印給用戶。

!apt-get -qq install xxd
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()

# Save the model to disk
open("gesture_model.tflite", "wb").write(tflite_model)
  
import os
basic_model_size = os.path.getsize("gesture_model.tflite")
print("Model is %d bytes" % basic_model_size)

這些代碼行將 TensorFlow 模型轉(zhuǎn)換為 TensorFlow Lite 模型。

!echo "const unsigned char model[] = {" > /content/model.h
!cat gesture_model.tflite | xxd -i      >> /content/model.h
!echo "};"                              >> /content/model.h

import os
model_h_size = os.path.getsize("model.h")
print(f"Header file, model.h, is {model_h_size:,} bytes.")
print("\nOpen the side panel (refresh if needed). Double click model.h to download the file.")

這些代碼行將 TensorFlow Lite 模型轉(zhuǎn)換為 C 文件以供微控制器使用。

現(xiàn)在我們已經(jīng)構(gòu)建并訓(xùn)練了我們的模型，我們現(xiàn)在需要弄清楚如何將數(shù)據(jù)從相機(jī)發(fā)送到 TinyML 模型。

相機(jī)的輸出尺寸為 176x144，模型的輸入尺寸為 28x28。我們可以嘗試兩種方法：

1. 從相機(jī)的輸出中裁剪出 28x28 的圖像。
2. 從相機(jī)的輸出中裁剪出 128x128 的圖像并將其重新整形為 28x28

我們將在以下部分嘗試這兩種方法。

9.b OV7670：作物測(cè)試

代碼說(shuō)明：

uint16_t pixels[176 * 144]= {0x0D2A,0xED29,0xED29,0xED29,....,0xB5AD,0x95B5,0xB6B5,0xB6B5};

這行代碼將 HEX 值存儲(chǔ)在大小為 176X144 的數(shù)組中。

for(int a = 0; a< 112; a++)
   {
    for(int b = 0; b< 112; b++)
    {
      Serial.print( pixels[176*a +b]);
      Serial.print(", ");
    }
    Serial.println("");
   }

這些代碼行遍歷數(shù)組并打印出圖像的前 28x28 像素。

草圖：

/*
  OV767X - Camera Test Pattern

  This sketch waits for the letter 'c' on the Serial Monitor,
  it then reads a frame from the OmniVision OV7670 camera and 
  prints the data to the Serial Monitor as a hex string.

  The website https://rawpixels.net - can be used the visualize the data:
    width: 176
    height: 144
    RGB565
    Little Endian

  Circuit:
    - Arduino Nano 33 BLE board
    - OV7670 camera module:
      - 3.3 connected to 3.3
      - GND connected GND
      - SIOC connected to A5
      - SIOD connected to A4
      - VSYNC connected to 8
      - HREF connected to A1
      - PCLK connected to A0
      - XCLK connected to 9
      - D7 connected to 4
      - D6 connected to 6
      - D5 connected to 5
      - D4 connected to 3
      - D3 connected to 2
      - D2 connected to 0 / RX
      - D1 connected to 1 / TX
      - D0 connected to 10

  This example code is in the public domain.
*/

#include <Arduino_OV767X.h>

uint16_t pixels[176 * 144]= {0x0D2A,0xED29,0xED29,0xED29,....,0xB5AD,0x95B5,0xB6B5,0xB6B5};
int arr1[28*28];
void setup() {
  Serial.begin(9600);
  while (!Serial);

  Serial.println("OV767X Camera Capture");
  Serial.println();
  Serial.println("Send the 'c' character to read a frame ...");
  Serial.println();
}

void loop() {
  if (Serial.read() == 'c') {
    
    for(int a =0; a< 112; a++)
   {
    for(int b =0; b< 112; b++)
    {
      Serial.print( pixels[176*a +b]);
      Serial.print(", ");
    }
    Serial.println("");
   }
   Serial.println("");
  }
}

原始圖像

28x28 裁剪圖像

9.c OV7670：重塑測(cè)試

代碼說(shuō)明：

for(int i=0; i < 28; i++){
    for(int j=0; j< 28; j++){
      int sum =0;

     for(int k =0; k<4;k++){
      for(int l =0; l<4; l++){
        sum += arr[4*(112*i+j) + 112 * k + l];
      }
     }
      sum = sum /16;
      arr1[i*28+j] = sum;
      Serial.print(sum);
     Serial.print(", ");
    }
    Serial.println("");
  }

Serial.println("");

這些代碼行使用 4x4 池化內(nèi)核，步長(zhǎng)為 1，遍歷 112x112 2D 數(shù)組以輸出 28x28 圖像。

草圖：

#include "num.h"
float arr1[28 * 28];
int filterWidth = 4;
int filterheight = 4;
void setup() {
  Serial.begin(9600);
  
  
}

void loop() {
  // put your main code here, to run repeatedly:
  for(int i=0; i < 28; i++){
    for(int j=0; j< 28; j++){
      int sum =0;

     for(int k =0; k<4;k++){
      for(int l =0; l<4; l++){
        sum += arr[4*(112*i+j) + 112 * k + l];
      }
     }
      sum = sum /16;
      arr1[i*28+j] = sum;
      Serial.print(sum);
     Serial.print(", ");
    }
    Serial.println("");
  }

Serial.println("");
 
}

原始圖像：28x28

用 python 放大：112x112

使用 Arduino 縮小尺寸：28x28

9.d 探索 reshape.ipynb Colab 筆記本

代碼說(shuō)明：

from skimage.transform import resize
t = number28.reshape(28,28)
print(t.shape)
number112 = resize(t, (112, 112))
print(number112.dtype)
"""
for i in range(0,112):
  for j in range(0,112):
    if number112[i][j] < 10e-20:
      number112[i][j] = 0
      /
"""
for i in range(0,112):
  for j in range(0,112):
      number112[i][j] = number112[i][j] * 10e+19
      if number112[i][j] < 10:
       number112[i][j] = 0
np.amax(number112)
number112 = number112 /12
np.amax(number112)
plt.imshow(number112, cmap='gray', interpolation='none')
plt.imshow(t, cmap='gray', interpolation='none')

這些代碼行將 28X28 MNIST 圖像放大為 112X112 圖像。

def matprint(mat, fmt="g"):
    col_maxes = [max([len(("{:"+fmt+"}").format(x)) for x in col]) for col in mat.T]
    for x in mat:
        for i, y in enumerate(x):
            print(("{:"+str(col_maxes[i])+fmt+"}").format(y), end=",")
        print("")
matprint(number112)

這些代碼行打印放大的 112X112 MNIST 圖像。

number28new = number28new.reshape(28,28)
for i in range(0,28):
  for j in range(0,28):
      if number28new[i][j] < 35:
       number28new[i][j] = 0
plt.imshow(number28new, cmap='gray', interpolation='none')

這些代碼行打印重新整形的 28x28 MNIST 圖像。

原圖：112x112

縮小圖像：28x28

10. 測(cè)試 TinyML 模型

現(xiàn)在我們已經(jīng)構(gòu)建了一個(gè) TinyML 模型并測(cè)試了將數(shù)據(jù)輸入模型的兩種方法，是時(shí)候?qū)?TinyML 模型集成到主應(yīng)用程序中了。

10.a TinyML 模型：輸入測(cè)試

我們?nèi)绾伟l(fā)送輸入數(shù)據(jù)？

對(duì)于基于時(shí)間序列的模型，很明顯發(fā)送輸入數(shù)據(jù)是因?yàn)檩斎霃埩渴且痪S數(shù)組。

//1D input tensor
tflInputTensor->data.f[i * 3 + 0] = (ax + 8.0) / 16.0;
tflInputTensor->data.f[i * 3 + 1] = (ay + 8.0) / 16.0;
tflInputTensor->data.f[i * 3 + 2] = (az + 8.0) / 16.0;

我不知道如何將數(shù)據(jù)發(fā)送到這個(gè)特定的 TinyML 模型，所以我設(shè)計(jì)了一個(gè)測(cè)試。

我將 MNIST 訓(xùn)練數(shù)據(jù)的數(shù)組值存儲(chǔ)到 num.h 文件中。

數(shù)小時(shí)：

float num[784] = {0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0, 49,143,223,196,149, 73,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,126,228,252,257,252,248,242,193, 67,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,176,247,254,213,156,149,175,236,256,204, 53,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,119,246,248,156,  0,  0,  0,  0, 69,216,259,221, 50,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,166,246,160,  0,  0,  0,  0,  0,  0,107,225,259,177,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,115,229,234, 86,  0,  0,  0,  0,  0,  0,  0,142,252,209,0,0,0,0,0,0,
0,0,0,0,0,0,107,223,230,214,237,192, 50,  0,  0,  0,  0,  0,  0,124,245,186,0,0,0,0,0,0,
0,0,0,0,0,0,201,251,147, 44, 95,154,127,  0,  0,  0,  0,  0,116,224,235, 91,0,0,0,0,0,0,
0,0,0,0,0,0,192,254,178, 89,  0,  0,  0,  0,  0,  0,  0, 80,224,242,149,  0,0,0,0,0,0,0,
0,0,0,0,0,0, 67,227,256,244,190, 94,  0,  0,  0,  0, 82,218,248,163,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0, 91,210,245,244,227,184, 90,  0, 84,219,256,188, 38,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0, 76,114,127,201,234,231,244,263,218, 74,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0, 47,205,266,273,250, 92,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0, 44,227,264,260,253,145,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,173,251,208,158,218,239,163,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,120,245,236, 92,  0, 52,201,227, 98,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,184,256,192,  0,  0,  0, 65,205,213, 40,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,192,250,146,  0,  0,  0, 70,206,225, 42,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,173,247,232,186,181,199,240,249,178,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0, 39,140,201,226,230,232,233,184, 65,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0};

模型.h：

//MPU6050_model.ino
#include 
#include "tensorflow/lite/micro/micro_error_reporter.h"
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
#include "tensorflow/lite/schema/schema_generated.h"
#include "tensorflow/lite/version.h"
#include "model.h"
#include "num.h"


const tflite::Model*  tflModel = nullptr; 
tflite::ErrorReporter*  tflErrorReporter = nullptr; 
TfLiteTensor* tflInputTensor = nullptr;  
TfLiteTensor* tflOutputTensor = nullptr; 
tflite::MicroInterpreter* tflInterpreter = nullptr; 

constexpr int tensorArenaSize = 140 * 1024; 
uint8_t tensorArena[tensorArenaSize];
float out[10];

void setup() {

  Serial.begin(115200);
  while (!Serial)
    delay(10);
     
  
  static tflite::MicroErrorReporter micro_error_reporter; 
  tflErrorReporter = µ_error_reporter;

   tflModel = tflite::GetModel(model);
   if (tflModel->version() != TFLITE_SCHEMA_VERSION) {
   TF_LITE_REPORT_ERROR(tflErrorReporter,
        "Model provided is schema version %d not equal "
        "to supported version %d.",
        tflModel->version(), TFLITE_SCHEMA_VERSION);
    return;
  }

  static tflite::MicroMutableOpResolver<6> micro_op_resolver;
  micro_op_resolver.AddMaxPool2D();
  micro_op_resolver.AddConv2D();
  micro_op_resolver.AddDepthwiseConv2D();
  micro_op_resolver.AddFullyConnected();
  micro_op_resolver.AddReshape();
  micro_op_resolver.AddSoftmax();

  static tflite::MicroInterpreter static_interpreter(tflModel, micro_op_resolver, tensorArena, tensorArenaSize, tflErrorReporter);
  tflInterpreter = &static_interpreter;

  TfLiteStatus allocate_status = tflInterpreter->AllocateTensors();
  if (allocate_status != kTfLiteOk) {
    TF_LITE_REPORT_ERROR(tflErrorReporter, "AllocateTensors() failed");
    return;
  }
  tflInputTensor = tflInterpreter->input(0);
  

}

void test_func(){
  for(int i = 0; i < 28; i++){
    for(int j =0; j < 28; j++){
      tflInterpreter->input(0)->data.f[28*i+j] = num[28*i+j] / 255;
    } 
  }
   for(int i = 0; i < 28; i++){
    for(int j =0; j < 28; j++){
      Serial.print(tflInterpreter->input(0)->data.f[28*i+j]);
      Serial.print(", ");
    } 
      Serial.println("");
  }
 Serial.println("");
  for(int i = 0; i < 28; i++){
    for(int j =0; j < 28; j++){
      Serial.print(num[28*i+j]);
      Serial.print(", ");
    } 
      Serial.println("");
  }
  Serial.println("");
  for(int i = 0; i < 28; i++){
    for(int j =0; j < 28; j++){
      Serial.print(tflInterpreter->input(0)->data.f[28*i+j]-num[28*i+j]);
      Serial.print(", ");
    } 
      Serial.println("");
  }
  */
  
}
void loop() {
  
}

代碼說(shuō)明：輸入張量：

for(int i = 0; i < 28; i++){
  for(int j =0; j < 28; j++){
      tflInterpreter->input(0)->data.f[28*i+j] = num[28*i+j] / 255;
    } 
  }

為了測(cè)試輸入張量，我使用 2D 循環(huán)發(fā)送輸入數(shù)據(jù)。

for(int i = 0; i < 28; i++){
  for(int j =0; j < 28; j++){
      Serial.print(tflInterpreter->input(0)->data.f[28*i+j]);
      Serial.print(", ");
    } 
      Serial.println("");
  }
Serial.println("");

然后，我打印出存儲(chǔ)在 TinyML 模型的輸入張量中的值。

for(int i = 0; i < 28; i++){
  for(int j =0; j < 28; j++){
      Serial.print(num[28*i+j]);
      Serial.print(", ");
    } 
      Serial.println("");
  }
Serial.println("");

然后我打印出存儲(chǔ)在循環(huán)中的實(shí)際值。

for(int i = 0; i < 28; i++){
  for(int j =0; j < 28; j++){
      Serial.print(tflInterpreter->input(0)->data.f[28*i+j]-num[28*i+j]);
      Serial.print(", ");
    } 
      Serial.println("");
  }

最后，我打印出存儲(chǔ)在輸入張量中的值與存儲(chǔ)在數(shù)組中的值之間的差異。如果所有差異都為零，則輸入已按照我們想要的方式正確存儲(chǔ)在輸入張量中。

10.b TinyML 模型：模型測(cè)試

現(xiàn)在我們已經(jīng)弄清楚了如何發(fā)送輸入數(shù)據(jù)，是時(shí)候測(cè)試模型了。這段代碼與上面的代碼相同，只是調(diào)用了推理并打印了輸出張量。

num.h 文件：

float num[784] = {0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0, 49,143,223,196,149, 73,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,126,228,252,257,252,248,242,193, 67,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,176,247,254,213,156,149,175,236,256,204, 53,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,119,246,248,156,  0,  0,  0,  0, 69,216,259,221, 50,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,166,246,160,  0,  0,  0,  0,  0,  0,107,225,259,177,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,115,229,234, 86,  0,  0,  0,  0,  0,  0,  0,142,252,209,0,0,0,0,0,0,
0,0,0,0,0,0,107,223,230,214,237,192, 50,  0,  0,  0,  0,  0,  0,124,245,186,0,0,0,0,0,0,
0,0,0,0,0,0,201,251,147, 44, 95,154,127,  0,  0,  0,  0,  0,116,224,235, 91,0,0,0,0,0,0,
0,0,0,0,0,0,192,254,178, 89,  0,  0,  0,  0,  0,  0,  0, 80,224,242,149,  0,0,0,0,0,0,0,
0,0,0,0,0,0, 67,227,256,244,190, 94,  0,  0,  0,  0, 82,218,248,163,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0, 91,210,245,244,227,184, 90,  0, 84,219,256,188, 38,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0, 76,114,127,201,234,231,244,263,218, 74,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0, 47,205,266,273,250, 92,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0, 44,227,264,260,253,145,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,173,251,208,158,218,239,163,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,120,245,236, 92,  0, 52,201,227, 98,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,184,256,192,  0,  0,  0, 65,205,213, 40,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,192,250,146,  0,  0,  0, 70,206,225, 42,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,173,247,232,186,181,199,240,249,178,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0, 39,140,201,226,230,232,233,184, 65,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0

草圖：

//MPU6050_model.ino
#include 
#include "tensorflow/lite/micro/micro_error_reporter.h"
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
#include "tensorflow/lite/schema/schema_generated.h"
#include "tensorflow/lite/version.h"
#include "model.h"
#include "num.h"


const tflite::Model*  tflModel = nullptr; 
tflite::ErrorReporter*  tflErrorReporter = nullptr; 
TfLiteTensor* tflInputTensor = nullptr;  
TfLiteTensor* tflOutputTensor = nullptr; 
tflite::MicroInterpreter* tflInterpreter = nullptr; 

constexpr int tensorArenaSize = 140 * 1024; 
uint8_t tensorArena[tensorArenaSize];
float out[10];

void setup() {

  Serial.begin(115200);
  while (!Serial)
    delay(10);
     
  
  static tflite::MicroErrorReporter micro_error_reporter; 
  tflErrorReporter = µ_error_reporter;

   tflModel = tflite::GetModel(model);
   if (tflModel->version() != TFLITE_SCHEMA_VERSION) {
   TF_LITE_REPORT_ERROR(tflErrorReporter,
        "Model provided is schema version %d not equal "
        "to supported version %d.",
        tflModel->version(), TFLITE_SCHEMA_VERSION);
    return;
  }

  static tflite::MicroMutableOpResolver<6> micro_op_resolver;
  micro_op_resolver.AddMaxPool2D();
  micro_op_resolver.AddConv2D();
  micro_op_resolver.AddDepthwiseConv2D();
  micro_op_resolver.AddFullyConnected();
  micro_op_resolver.AddReshape();
  micro_op_resolver.AddSoftmax();

  static tflite::MicroInterpreter static_interpreter(tflModel, micro_op_resolver, tensorArena, tensorArenaSize, tflErrorReporter);
  tflInterpreter = &static_interpreter;

  TfLiteStatus allocate_status = tflInterpreter->AllocateTensors();
  if (allocate_status != kTfLiteOk) {
    TF_LITE_REPORT_ERROR(tflErrorReporter, "AllocateTensors() failed");
    return;
  }
  tflInputTensor = tflInterpreter->input(0);
  

}


void loop() {
  
    
   for(int i = 0; i < 28; i++){
    for(int j =0; j < 28; j++){
      tflInterpreter->input(0)->data.f[28*i+j] = num[28*i+j]/255.0;
    } 
  }
  /*
   for(int i = 0; i < 1; i++){
    for(int j =0; j < 1; j++){
      Serial.print(tflInterpreter->input(0)->data.f[28*i+j]);
      Serial.print(", ");
    } 

  }
  */
  
  TfLiteStatus invokeStatus = tflInterpreter->Invoke();
  out[0] = tflInterpreter->output(0)->data.f[0];
  out[1] = tflInterpreter->output(0)->data.f[1];
  out[2] = tflInterpreter->output(0)->data.f[2];
  out[3] = tflInterpreter->output(0)->data.f[3];
  out[4] = tflInterpreter->output(0)->data.f[4];
  out[5] = tflInterpreter->output(0)->data.f[5];
  out[6] = tflInterpreter->output(0)->data.f[6];
  out[7] = tflInterpreter->output(0)->data.f[7];
  out[8] = tflInterpreter->output(0)->data.f[8];
  out[9] = tflInterpreter->output(0)->data.f[9];
  
  float maxVal = out[0];
  int maxIndex = 0;
  for(int k =0; k < 10;k++){
    if (out[k] > maxVal) {
         maxVal = out[k];
         maxIndex = k;
      } 
  }
  Serial.print("Number ");
  Serial.print(maxIndex);
  Serial.println(" detected");
  Serial.print("Confidence: ");
  Serial.println(maxVal);
  
  Serial.print(out[0]);
  Serial.print(",");
  Serial.print(out[1]);
  Serial.print(",");
  Serial.print(out[2]);
  Serial.print(",");
  Serial.print(out[3]);
  Serial.print(",");
  Serial.print(out[4]);
  Serial.print(",");
  Serial.print(out[5]);
  Serial.print(",");
  Serial.print(out[6]);
  Serial.print(",");
  Serial.print(out[7]);
  Serial.print(",");
  Serial.print(out[8]);
  Serial.print(",");
  Serial.println(out[9]);
  
}

代碼說(shuō)明：

TfLiteStatus invokeStatus = tflInterpreter->Invoke();

這些代碼行對(duì)存儲(chǔ)在輸入張量中的數(shù)據(jù)調(diào)用推理。

out[0] = tflInterpreter->output(0)->data.f[0];
  out[1] = tflInterpreter->output(0)->data.f[1];
  out[2] = tflInterpreter->output(0)->data.f[2];
  out[3] = tflInterpreter->output(0)->data.f[3];
  out[4] = tflInterpreter->output(0)->data.f[4];
  out[5] = tflInterpreter->output(0)->data.f[5];
  out[6] = tflInterpreter->output(0)->data.f[6];
  out[7] = tflInterpreter->output(0)->data.f[7];
  out[8] = tflInterpreter->output(0)->data.f[8];
  out[9] = tflInterpreter->output(0)->data.f[9];

這些代碼行訪問(wèn)輸出張量并將它們存儲(chǔ)在一個(gè)數(shù)組中以供以后使用。

float maxVal = out[0];
int maxIndex = 0;
for(int k =0; k < 10;k++){
  if (out[k] > maxVal) {
      maxVal = out[k];
      maxIndex = k;
    } 
  }
Serial.print("Number ");
Serial.print(maxIndex);
Serial.println(" detected");
Serial.print("Confidence: ");
Serial.println(maxVal);

這幾行代碼打印出輸出值最高的類的類索引和置信度值。

Serial.print(out[0]);
  Serial.print(",");
  Serial.print(out[1]);
  Serial.print(",");
  Serial.print(out[2]);
  Serial.print(",");
  Serial.print(out[3]);
  Serial.print(",");
  Serial.print(out[4]);
  Serial.print(",");
  Serial.print(out[5]);
  Serial.print(",");
  Serial.print(out[6]);
  Serial.print(",");
  Serial.print(out[7]);
  Serial.print(",");
  Serial.print(out[8]);
  Serial.print(",");
  Serial.println(out[9]);