Deep learning-based semantic communications (DLSC) significantly improve communication efficiency by only transmitting the meaning of the data rather than a raw message. Such a novel paradigm can brace the high-demand applications with massive data transmission and connectivities, such as automatic driving and internet-of-things. However, DLSC are also highly vulnerable to various attacks, such as eavesdropping, surveillance, and spoofing, due to the openness of wireless channels and the fragility of neural models. To tackle this problem, we present SemKey, a novel physical layer key generation (PKG) scheme that aims to secure the DLSC by exploring the underlying randomness of deep learning-based semantic communication systems. To boost the generation rate of the secret key, we introduce a reconfigurable intelligent surface (RIS) and tune its elements with the randomness of semantic drifts between a transmitter and a receiver. Precisely, we first extract the random features of the semantic communication system to form the randomly varying switch sequence of the RIS-assisted channel and then employ the parallel factor-based channel detection method to perform the channel detection under RIS assistance. Experimental results show that our proposed SemKey significantly improves the secret key generation rate, potentially paving the way for physical layer security for DLSC.