We demonstrate a single-photon vibrometer for material recognition, combining time-gated photon counting with machine learning. It captures unique vibrational characteristics of materials under acoustic excitation, by measuring oscillating flux of the reflected photons projected onto a single spatial mode. The gating window is electronically swept to temporally locate the target while minimizing the background photon counts, thereby enabling faithful measurements under photon-starved operations. Analyzed using various machine learning models, including deep learning architectures like fully connected networks and convolutional neural networks, the system achieves high classification accuracy. Our results suggest new tools for non-destructive testing, remote sensing, and structural health monitoring, providing robust material recognition in challenging environments.