With the development of Internet of Things (IoT) technology, the digital pill has been employed as an IoT system for emerging remote health monitoring to detect the impact of medicine intake on patients’ biological index. The medical data is then used for model training with federated learning. An adversary can launch poisoning attacks by tampering with patients’ medical data, which will lead to misdiagnosis of the patients’ conditions. Lots of studies have been conducted to defend against poisoning attacks based on blockchain or hardware. However, 1) Blockchain-based schemes can only exploit on-chain data to deal with poisoning attacks due to the lack of off-chain trusted entities. 2) Typical hardware-based schemes have the bottleneck of single point of failure. To overcome these defects, we propose a defense scheme via multiple Trusted Platform Modules (TPMs) and blockchain oracle. Benefitting from multiple TPMs verification results, a distributed blockchain oracle is proposed to obtain off-chain verification results for smart contracts. Then, the smart contracts could utilize the off-chain verification result to identify poisoning attacks and store the unique identifiers of the non-threatening IoT device immutably on the blockchain as a whitelist of federated learning participants. Finally, we analyze the security features and evaluate the performance of our scheme, which shows the robustness and efficiency of the proposed work.