Video surveillance enabled by Internet of Things (IoT) devices, such as smart cameras, has become a popular set of applications recently with the trend of adopting IoT in multimedia signal processing and smart home use cases. Despite its intelligence and convenience, the video motion detection module deployed on the IoT devices poses security challenges due to the sensitive nature of the captured surveillance video and the motion detection operation. In this paper, we investigate the security vulnerabilities of IoT video surveillance from the hardware system point of view. We First develop a proof-of-concept prototype demonstrating video replay attacks, in which the compromised surveillance device hides the chosen suspicious motion by overwriting the corresponding frames with pre-recorded normal frames under the control of the attacker. To address the security concerns, we develop a hardware-based IoT security framework that creates a trusted execution environment and physically isolates the security sensitive components, such as the motion detection module, from the rest of the system. We implement the security framework on an ARM system on chip (SoC). Our evaluations on the real hardware reveal superior security and low performance/power overhead in IoT video surveillance applications.