Hello, community! As you may be aware if you follow the OpenCV AI Kit Kickstarter campaign (and why wouldn’t you?) last week we announced a huge new resource for Artificial Intelligence models: The OpenCV AI Marketplace. Marketplace is currently in invite-only mode, but we’ll be opening it up more in due time.
In the meantime, we’re excited to show you some of the new site’s features, and how to build a simple but elegant product using OAK and the OpenCV AI Marketplace. First we’ll explain a bit about the tools we are using and our goal for today’s project. Let’s get started.
Prerequisites To Build This Project
You will need the following in order to create this project along with us, or try the final code out yourself, but you don’t have to build along in order to enjoy the article. I promise!
• 1x OpenCV AI Kit (OAK-1 or OAK-D),
• Python 3
• FFmpeg
What is OpenCV AI Kit (OAK)?
According to the hugely-successful Kickstarter Campaign:
OAK is a modular, open-source ecosystem composed of MIT-licensed hardware, software, and AI training - that allows you to embed the super-power of spatial AI plus accelerated computer vision functions into your product. OAK provides in a single, cohesive solution what would otherwise require cobbling together disparate hardware and software components.
We built modeplace.ai with OAK in mind, and many of the models on it run through a custom-built continuous integration system on real OAK devices to ensure compatibility and performance each time they are updated.
If you don’t already have one and are feeling left out you can pick up an OAK-1 or OAK-D on the store today.
You’ll want to make sure you’ve followed the handy setup instructions, too. One of the coolest things about OAK is the easy setup — it really only takes a minute.
Our Idea: Using A Retail Emotion Estimator For Movie & TV Trailers
Like many great ideas, today’s project was partially inspired by a classic episode of The Simpsons. As a video producer myself I love the art of creating great trailers and It seems like every few days a hot new one drops which becomes all the internet can talk about, with the most recent being the Disney+ Original Series “The Falcon and the Winter Soldier.” We thought it would be fun to try and see how these made us feel, using our own facial reactions as a guide.
So, where does the Simpsons come in? In the episode “Itchy & Scratchy & Poochie” (Prod. Code. 4F12) a group of Springfield youths are brought together to watch some cartoons in what’s called a focus group. The job of the focus group is to take in some media and tell you what they think. In the show they represent this by using physical dials that the kids turn left and right to register their happiness or unhappiness at what is on screen.
We’re going to take it one step beyond and use the AI + Camera capabilities of OAK to run a real-time emotion estimator we’ll download from modelplace.ai, and use it to automatically detect and display the feelings of viewers who are watching a specific trailer. Because OAK is Edge AI, we don’t need to run any cloud services and all this emotion data is kept locally.
We’re going to try it out on some classic film trailers and that new Marvel series in just a bit. If you don’t want to know how we built it, skip on down to the results.
Choosing An AI Model
We already knew what model we had planned to use, which is the emotion_recognition_retail model from Intel’s Open Model Zoo, which has been converted for use on OAK by the team. When working on your projects, maybe you’ll need to browse for ideas.
Browsing on the OpenCV AI Marketplace is free and you should check it out! Even if you have not been invited yet, you can browse the models available and see sample results from them, which are generated from real videos each time a model is updated. You can also view the code and download models without needing an account. Testing models live in the browser is still an invite-only feature, but can sign up to be first to know when more people are let in.
Our chosen model is listed on the marketplace, so we’ll click on the link labeled “OAK-ready model” to find the code on Github and clone it locally:
git clone https://github.com/opencv-ai/oak-model-samples.git
Setting Up The Emotional Recognizer Model
To run any of the models from the oak-models-samples repository we just need to follow a few steps described in the instruction. In this post, we are using Emotional Recognizer Model which can be easily installed using the following command:
git clone https://github.com/opencv-ai/oak-model-samples.git && \
cd oak-model-samples && \
git checkout 54fa4fe8bd2bb42d6bae49213dbbe6b2f1ce5c5e && \
cd emotion_recognition_retail && \
pip3 install wheel && python3 setup.py bdist_wheel && rm -R build/ *.egg-info && \
pip3 install dist/*.whl -f https://artifacts.luxonis.com/artifactory/luxonis-python-snapshot-local/depthai/ && rm -R dist/ && \
export PATH_TO_EMOTION_RECOGNITION_MODEL=$(pwd)There are several requirements to install on a fresh system, so it may take a few minutes. Once the pip3 install command finishes, you should be all set. Be sure to follow the instructions above in the correct order, or you might have a bad time.
How It Works
Now, when we are all set, it’s time to build a simple project using the chosen model. We want to show how easy it is to start working with OAK models from Modelplace.
Let’s get started! First of all, we need to build our emotion recognition model that can be easily done just in two lines:
We use InferenceModel class from the emotion_recognition_retail package to initialize the model and load the weights
# build model
model = InferenceModel(
model_path=model_path,
) # initialize the emotion recognition model
model.model_load() # load weightsThe main idea of our simple product is to collect user emotion statistics. So, we need to create an Emotion Analyzer object, which will aggregate model results and display aggregated statistics on progress bars.
emotion_analyzer = EmotionAnalyzer()Now, we are ready to run inference. For this step we need to implement process_cam function which will take frames from the OAK camera, run inference on its accelerator and aggregate emotions during device operation.
# run inference
original_images, results, fps = process_cam(
model, emotion_analyzer, show=args.visualize,
)The function returns a list with frames captured from OAK, a list with model inference results, and calculated fps.
When inference is done, we need to provide the summary of user emotions. We can do it in several ways:
We can create a result emotion bar that will show the user's most popular emotion.
# create report with emotion bar statistics
result_emotion_bar = emotion_analyzer.create_result_emotion_bar(
save=args.save_statistics,
)
result_emotion_bar.show()
Also, we can create a user emotion statistic pie chart.
result_emotion_pie_chart = emotion_analyzer.create_statistics_pie_chart(
save=args.save_statistics,
)
result_emotion_pie_chart.show()
It would be nice to share with friends your reaction to a new trailer. So we can save visualization results into video.
# save result video
if args.save_video:
save_results_into_video(
original_images, results, fps=fps, size=args.visualization_size,
)Running The Project Yourself
Running this project requires 3 steps:
1) Run the script we’ve created,
2) Push Play on a video you’d like to capture reactions to
3) Stop the script
To capture your own reactions, you can simply load up any video (for example, the classic 1960s horror movie A BUCKET OF BLOOD) then run the following command:
python main.py --model-path $PATH_TO_EMOTION_RECOGNITION_MODELThis will spawn a new video window with your face, and OAK will begin estimating your emotions. Push play on the video and watch it! At the end push CTRL+C to stop the recognizer and save your reaction video.
Using the models provided by modelplace.ai along with the powerful OpenCV AI Kit allows you to create new, customized, AI workflows in hours instead of days, weeks instead of months. Want to join the OAK community? Head over to the store and get your own OAK-1 or OAK-D. We can’t wait to see what you build with it.
Bonus: Emotion Recognition .vs. The Falcon and the Winter Soldier
Of course, you don’t have to use this new tool on dusty old movies from the 1960s! We decided to have the team watch the trailer for the new Marvel series, The Falcon and the Winter Soldier. If the below embed doesn’t work for you watch on YouTube:
Alex S, Alex L, and Me (Phil) watched it separately and recorded our reactions from watching it the first time. Our results are embedded below! Apparently I stone-faced my way through it, Alex S was shocked, and Alex L was happiest.