Tool abrasion and material adhesion lead to tool flank wear and flank build-up respectively in machining magnesium alloys, which adversely affect surface integrity due to higher cutting forces and temperatures. In this paper, a novel on-line optical tool monitoring system integrated with CNC machining center is designed and applied to examining these two phenomena. Flank wear evolution on PVD coated inserts in end milling of AISI H13 tool steel and material adhesion on flank face of PCD inserts in face milling of lightweight magnesium alloys were on-line inspected to assess the function of the designed optical measurement system. Tool wear failure criterion for the PVD milling inserts is set to be VB max = 250 μm and flank wear evolution with cutting time is presented. Tool/work adhesion between the PCD inserts and magnesium alloy were observed under different combinations of cutting parameters including speed, feed, and depth-of-cut. The advantages of the on-line optical tool inspection system are discussed.