Last night, BrandsEye launched a crowdsourced component to their Online Reputation Management software at an event in Johannesburg; a global first! With huge social media uptake, the announcement immediately trended in South Africa and has driven conversation across the globe.
BrandsEye lead engineer, David Tinker, took a little time out to answer some questions about the technology his team has developed.
BrandsEye lead engineer, David Tinker.
What’s your role on this project?
I'm lead engineer and my responsiblilty is to provide strategic direction to the software development, manage supporting infrastructure and manage the team of engineers who collectively made it happen.
As lead engineer, tell us about the BrandsEye Crowd platform
The BrandsEye Crowd is a crowdsourcing platform where mentions from our BrandsEye accounts (tweets, Facebook status updates, press content, blogs etc.) are fed to a community who rate each of the mentions according to criteria such as sentiment and language. By using crowdsourcing, we can provide a far higher quality (upward of 90%) and larger volume of data very quickly through the parallel processing ability of the crowd.
How do you make sure that the data is accurate?
There are a number of steps taken to ensure that the data is accurate. Firstly, we use intercoder reliability (peer review); comparing the ratings of multiple members of the crowd for a particular piece of data. Once there is agreement from two or more people, the data is considered correct. Secondly, we have a system of control mentions that we already know the answers for. These are worked out by the BrandsEye team and very experienced members of our crowd.
The control mentions are then fed to the rest of the crowd periodically, to check they are being rated correctly. This provides them with on-going training. Thirdly, there is an appeals process where the crowd are able to contest the result of any mention they rated. This ensures that special edge-case inaccuracies are identified and dealt with in a completely scalable way.
Underlying all of this is the rule that the crowd will only get paid for mentions that are correct. The amount they’re paid is also calculated on a sliding scale based on their most recent accuracy.
How big is the team behind this development?
The BrandsEye engineering team behind this project is made up of four members, namely: Ben Steenhuisen, Lyndsay Lawrence, Rudy Neeser and me. Considerable consulting is channelled from our BrandsEye Chief Technical Officer, Craig Raw.
How long has the team been working on this?
5 months, from initial discussions to the platform going live. In reality, we had the system up and running in Beta from November and we’ve spent the last couple of months tweaking it to increase the overall accuracy and efficiency of the crowd.
How does this interface with the original BrandsEye software?
On the BrandsEye side, the data in the account is pushed through to the crowd for verification. Once completed, the data is returned to BrandsEye. From a client’s point of view this is entirely seamless and perfect for their reporting.
So, what were the key challenges in building the BrandsEye Crowd?
Besides the team having to learn an entirely new language (Grails) with which to build the platform, the biggest challenge during the development process was around crowd behaviour management (in particular weeding out poor behaviour). We spent a huge amount of time identifying what we could tweak to modify the crowd behaviour in order to help improve the accuracy.
These changes ranged from adapting the ELO ranking system (used to rank chess masters) through to tweaking how the appeals process functioned, what statistics to display to drive behavioural changes. Sounds very exciting and it’s great to see truly innovative tech coming out of South Africa!
Learn more about BrandsEye Crowd here.
