March Madness — Analyze video to detect players, teams, and who attempted the basket
Mar 21
1_QVyHGpFxmDbwMISXOYNNAQ

March Madness — Analyze video to detect players, teams, and who attempted the basket

Doing cool things with data!

Introduction

Its March madness month! And what an exciting season it has been. For the data scientist within you lets use this opportunity to do some analysis on basketball clips. With the use of deep learning and opencv we can extract interesting insights from video clips. See example gif below of the game b/w UCF and Duke where we can identify all the players + referees, label players into teams based on their jersey colour. Later in the blog I show how we can also identify which player is attempting to shoot the basket . And all of this can be done real time.

image

You can find the code on my Github repo

So let’s start.

Detecting Players

I have used a pretrained detection model like the Faster RCNN to detect players. It is easy to download a Faster RCNN trained on the COCO data set from the Tensorflow Object Detection API and test it. The API takes as input each frame of image and detects among 80 different classes. If you are new to Tensorflow Object Detection and want to learn more, please checkout this blog. The model does quite well in detecting persons but there are many detections in this video due to the large number of people in the crowds. See sample detection below. I suppressed detections that were too big to more cleanly segment out players. You can also play with the score threshold in the API to filter out low confidence detections. Checkout the code on Github for tips on how to suppress boxes with low scores and multiple false detections.

image

 

Detection output from Pretrained Tensorflow model

 

Detecting Teams

Now comes the interesting part. How do we detect which players are UCF vs Duke? We can use OpenCV to do that.

If you are new to OpenCV please see the tutorial below:

OpenCV Tutorial

OpenCV allows us to identify masks of specific colours and we can use that to identify white and black players. The main steps are:

  • Convert the image from BGR to HSV colour space.
  • pace specify colour ranges for white and black. This takes a bit of experimentation and you can visualize the impact of different thresholds in the notebook.
  • Use OpenCV to mask(colour) pixels that are in the threshold range.
  • OpenCV Bitwise_and to colour black any pixels not in the mask

See output below for white colour. They are masked as “pink” with everything else in the background in black

image

 

Detecting white colour pixels

 

To identify team for each individual player we extracted bounding box from tensorflow object detection and count the percent of pixels in that bounding box that are non black to decide the team for that player.

Overall code works quite well. However this is a hard coded logic for identifying black and white jersey players. It can be made more general by using clustering to find similar players

Detecting pose and who is shooting

OpenPose is a real-time multi person pose detection library. It detects persons in an image and outputs keypoints for the main joints for every person — can total up to 25 keypoints per person. The code is open sourced. You have to install as suggested in README here. Once installed you can run images through it and get keypoints for all the players in a scene as shown below.

img

 

Open pose output

 

So how do we identify players attempting to shoot a basket?

We can look for players with their wrist keypoints above their head. Implying hands raised. This could indicate ready to shoot as in the scene above or could be defensive. Further the coordinates of the ball along with those of the wrist keypoint can be used to identify which player with raised hands has ball close to them.

Conclusion

Deep learning has made is possible to do really cool analysis by chaining different ideas. There is a lot of open source code and pretrained models that you can use on your data to get started. Above is just the starting point. Other cool things that can be done include:

  • Using OCR to read the game scores to allow your system to understand which team is winning
  • Tracking the ball to predict when a shot can score points
  • Tracking players to get stats for each individual players
  • Detecting events like a slam dunk, 3 point basketball etc.

Hope you pull the code and try it yourself.

I have my own deep learning consultancy and love to work on interesting problems. I have helped many startups deploy innovative AI based solutions. Check us out at — https://deeplearninganalytics.org/.

You can also see my other writings at: https://medium.com/@priya.dwivedi

If you have a project that we can collaborate on, then please contact me through my website or at info@deeplearninganalytics.org

 

References