This paper presents Confidence, a tool for identifying and addressing faults in wireless sensing systems. Confidence pinpoints potential sensor and network faults in real time, allowing users to validate unexpected data and address any failures in the field. By introducing a well defined, low-dimension feature space, and functions to map sensor data into this space, we are able to achieve fault detection and diagnosis with relatively simple mechanisms such as outlier detection. Users can directly modify system outcomes by altering a classification label in instances when Confidence's automated algorithm draws the wrong inference. This label is applied to all similar points in the feature space, enabling Confidence to learn from user interaction in the field. This abstraction for incorporating user knowledge provides a lightweight and easy-to-understand interface for the user, while limiting user burden and reducing the required a priori environmental knowledge. Confidence has performed well on real-world deployments, including one deployment of 130 sensors, replayed datasets, and network simulations. Confidence accurately detects and diagnoses at least 90% of all data, and user interaction improves it's performance.