In the field operation, crossing the fence is a common illegal behavior, which needs to be paid attention to. Especially, the live part of the power station site is mixed with the power outage part, and some construction workers cross the fence to enter the live area, which can easily cause safety problems. The power station has a wide range of operations, and the manual monitoring method is inefficient. With the popularization of video monitoring devices in power stations, this paper proposes a detection and identification method for fence crossing violations based on video monitoring. The method extracts video frames as input, uses convolution to extract temporal and spatial features, and classifies and regresses the features fused in time and space, which can effectively identify fence crossing behaviors. Finally, a video processing platform is built to process alarms for illegal operations. Engineering practice shows that the method shown in the article can effectively predict the illegal crossing of the fence in the power station and improve the intelligent monitoring level of the power station.