To help the audience better understand the meaning of the referee's gesture when watching a live broadcast or a video of a basketball game, or to help video analysts analyze the game video, a real-time basketball referee gesture detection and recognition system, Yolov5-Basketball Referee (Yolov5-BR), was designed. The Yolov5 target detection algorithm was used as the basic model, and the intersection over union (IoU) and complete IoU (CIoU) loss functions of its boundary box were weighted to enhance the robustness of the prediction box. After the C3 module, an attention mechanism was added to generate more distinguishing feature representation and improve the network recognition performance. In addition, an adaptive feature fusion mechanism was incorporated into the head of the detection layer to make full use of the high-level semantic information of the image. Finally, the target confidence loss function was weighted unequally to improve the robustness of small-target detection. On a self-made referee gesture dataset, Yolov5-BR achieved a 95.4% mAP value, with a local video detection rate of 55.5 frame/s, an external camera resolution of $1 280 \times 960$, and a detection rate of 25 frame/s. Experimental results show that, compared with the original model, Yolov5-BR can effectively improve the performance of judging gestures while maintaining high accuracy, stability, and real-time response.