Given the increasing popularity of mobile and wearable devices, this paper explores the potential use of inertial sensors that are widely available on mobile and wearable devices for vehicle and driver tracking. Such a capability would enable novel classes of mobile safety and assisted driving applications without relying on information or sensors in the vehicle. Although inertial sensors have been widely used in motion tracking, existing approaches cannot distinguish the motion of the vehicle and the device's motion in the vehicle. Additionally, the noise exerted from the electronic components in the vehicle and the ferromagnetic frame of the vehicle distorts the inertial sensor readings. This paper introduces a method to separately estimate the orientation of both the vehicle and the sensor by tracking the earth's magnetic field and the electromagnetic distortion from the vehicle, as measured by a magnetometer in addition to a gyroscope and an accelerometer. Specifically, the vehicle noise is used to estimate the orientation of the sensor within the vehicle while the earth's magnetic field combined with vehicle noise is used to estimate the vehicle's heading. Our on-road experiments show that the technique is able to estimate the sensor orientation with a mean error of 5.61° for the yaw angle and 3.73° for the pitch angle, as well as able to estimate the vehicle heading with a mean error of 4.12°.