Enhancing Emotion Data Collection with Selenium Pt.1

László Zabb and the data collection team

At Realeyes, we build emotion recognition software, using facial coding via webcams directly through the browser. Our technology enables brands and content creators to see how their audience feels in response to their video, so from a technological perspective we believe that that’s quite special - it’s not something that many software companies do everyday. Webcams are everwhere, but the devices they're on pose many dimensions which makes testing these applications an interesting challenge.

With such a variety of browsers out there, browser plugins, different operating systems, webpage layouts, multiply that with different webcams, webcam setups and two very different technologies to access these webcams: Flash and WebRTC, means creating a robust solution requires some serious thought and teamwork.

We thrive on such challenges and so I felt quite excited to share some of the findings with this particular experience – I’m sure there will be other software engineers who’ll have come across some of the difficulties we found along the way.

It started with a peculiar problem that we were puzzled with for quite some time. We had a long and a quite nasty bug, internally codenamed as "User cannot click camera allow button". When a flash application attempts to access webcam a camera access prompt window is displayed to the user (see image).

> View Data Collection Demo

Our application needs it to measure user’s emotional response to videos that they watch through our platform. The problem was when occasionally user was unable to successfully grant access to their webcam because pressing “Allow” button was not possible.

It was as if clicks on the button were not getting registered by the browser. Unfortunately, our logs showed that this problem occurred more often than it would be acceptable, we were loosing quite a lot of users because of it. But it did not occur frequently enough for us to reproduce it on any of our office machines to be able to analyse what was really going on there.

 

It was as if the clicks on the Allow button were not getting registered by the browser.

 

To tackle the issue, we decided to narrow down the scope of possible options by conducting a big number of different test experiments. Obviously, it would have been impossible to do them manually, so we started to look for automated solutions. It was the day, when we took a deep dive into Selenium.

Pretty quickly we faced many different challenges. For example,

  • External Selenium providers did not support working with a real webcam. But some of our tests did rely on working with a real webcam and a real face in front of the webcam.
  • They also did not have support for a virtual webcam. Chrome and Firefox do offer fake webcam testing capability for WebRTC, but for Flash no such support exists. There is no support for IE at all.
  • Selenium cannot natively interact with Flash, so there was no simple way for us to automate pressing of Allow button via it.
  • Selenium cannot allow webcam access when it is requested for WebRTC. There is only an option to automatically allow webcam request for WebRTC for Firefox and Chrome.
  • We also needed real mouse clicks on the Allow button in case of Flash to make sure we fully reproduce the environment of our users, which was not possible with Selenium.
  • Selenium does not allow to retrieve a full-screen screenshot, only a screenshot of the content of the browser window. It makes it really hard to detect infrastructural issues, for example if an unexpected popup window covers the browser.

I still remember clearly when we drew the envisioned architecture of our own Selenium Grid and automated testing framework to drive it. It was an ugly drawing that engineers typically draw, but there was an innate beauty in the idea of being able to run all those different scenarios in complete automation. It was something that kept us awake at late night hours as our hands were itchy to implement this experimental idea.


Selenium_Blueprint-2.png

Pretty quickly into the development of the automation framework we started to use it. Yes, using the half ready fragile prototype to reproduce the “Allow button” issue. And it helped us a lot in guiding the development process towards the features that could have the biggest impact towards the ultimate resolution of the problem.

The framework was evolving day by day, with new features, support for new automated testing scenarios, new browser support, stabilisation and bug fixes. In the end not only did we solve this particular challenge, but we also ended up with a rapidly evolving and feature reach automation testing framework customised for our needs that since then helped us resolve many other difficult bugs.

The framework was evolving day by day, with new features, support for new automated testing scenarios, new browser support, stabilisation and bug fixes.

In the next post I will share some more details about the architecture of the actual framework and how it helped us resolve our tricky problem. 

Special thanks to all contributors, who took part in building and shaping the framework, namely: Tamás Varga, László Vass, Máté Sárdi, Vladislav Stolyarov, Imre Kleman, László Dunai and Lajos Marton.

READ PART 2