Lab 5: Classification Accuracy Assessment

Goals and Background:

The goal of this lab was for students to practice important and necessary evaluation of classification results. Skills practiced in this lab included collecting ground reference samples to perform accuracy assessment with, using those samples to perform accuracy assessment, and then interpreting the results. The processes practiced were performed on two images, the result of the unsupervised classification performed in Lab 3, and the result of the supervised classification performed in Lab 4. An error matrix, accuracy totals, and Kappa statistics were generated.

Methods:

Part 1: Creating testing samples

Figure 1: Setting up ground reference samples

Part 1 involved bringing in the recoded unsupervised classified image from Lab 3 and collecting ground reference sample data for comparison with the classification of those points. Under the raster tab, and then the supervised menu, the Accuracy Assessment tool was opened, the image was opened in the tool, and then a viewer with fine spatial resolution color imagery was selected as the reference. 125 stratified random points were then created selecting the five informational classes that were created by merging the multitude of classes in Lab 3 with the recode tool (Figure 1). 

Part 2 Section 1: Evaluation of reference sample points

Part 2 consisted of assigning the reference samples their real classifications. After highlighting the first 10 samples that appeared in the Accuracy Assessment window, the Show Current Selection option was chosen under the view menu. After zooming into each of these points and entering in the real classification using the table and the reference image supplied, the next 10 samples were selected, shown, and assigned their reference class values. This process was continued until all 125 points were processed. The table was then saved for future use.

Part 2 Section 2: Generating an accuracy assessment report

After clicking on report and then checking all three options (error matrix, accuracy totals, and Kappa statistics) in the Accuracy Assessment window, the Accuracy Report was selected to run the report. A .txt file was the result, and the results were copied over to a word document and were put in an easier to digest format. A segment of the original accuracy assessment report is shown below in Figure 2. 

Figure 2: Accuracy Assessment Report

Part 3:

Part 3 consisted of all of the same procedures as parts 1 and 2, but utilized the supervised classified image and a different reference image matching the time of that image.

Results: 

Unsupervised Classification Accuracy

Figure 3 shows the error matrix for the unsupervised classification performed in Lab 3. Going across a row shows you what pixels actually are (what you identified them as when viewing the reference sample). For example, there were two sample pixels identified in truth as agriculture and bare soil that were classified by the classifier as water.

	Ground Test Reference Information
Classification	Class	Water	Forest	Agriculture	Urban/Built-up	Bare Soil	Row Total
	Water	13	0	1	0	1	15
	Forest	0	32	3	1	2	38
	Agriculture	0	9	14	1	6	30
	Urban/Built-up	0	1	3	6	6	16
	Bare Soil	0	0	8	0	13	21
	Column Total	13	42	29	8	28	120

Figure 3: Unsupervised Classification Error Matrix

The overall classification accuracy was 65.00%, and the overall kappa statistics are 0.5428, both moderate statistics that indicate the data created is not ready for use. Figure 4 shows the producer's and user's accuracy. These indicate per class accuracy. 

Producer’s accuracy (Omission error)		User’s accuracy (Commission error)
Water	100%	Water	87%
Forest	76%	Forest	84%
Agriculture	48%	Agriculture	47%
Urban/Built-up	75%	Urban/Built-up	38%
Bare Soul	46%	Bare Soul	62%
Average	69.00%	Average	63.60%

Figure 4: Per Class Producer's and User's Accuracy

Supervised Classification Accuracy

Figure 5 shows the error matrix for the supervised classification performed in Lab 4.

	Ground Test Reference Information
Classification	Class	Water	Forest	Agriculture	Urban/Built-up	Bare Soil	Row Total
	Water	14	1	0	0	0	15
	Forest	0	26	1	0	0	27
	Agriculture	0	16	6	0	2	24
	Urban/Built-up	0	3	11	3	7	24
	Bare Soil	0	8	19	1	9	37
	Column Total	14	54	37	4	18	127

Figure 5: Supervised Classification Error Matrix

The overall classification this time was a poor 45.67%, and the overall kappa statistics were 0.3160, also very poor. The supervised classification created inaccurate results. Figure 6 shows the producer's and user's accuracy.

Producer’s accuracy (Omission error)		User’s accuracy (Commission error)
Water	100%	Water	93%
Forest	48%	Forest	96%
Agriculture	16%	Agriculture	25%
Urban/Built-up	75%	Urban/Built-up	13%
Bare Soul	50%	Bare Soul	24%
Average	57.80%	Average	50.20%

Figure 6: Per Class Producer's and User's Accuracy

Comparison

The unsupervised classification method produced more highly accurate resultant data. A few reasons may have caused the unsupervised classification to perform better than the supervised classification. The first reason is that the unsupervised classification split the image into many more initial classes before a recode was performed, leading to more minute differences between spectral profiles being picked up. The second reason may have to do with user error in picking training samples for the supervised classifier. The training samples may have needed to be picked in order to create more varying average spectral profiles for each class. It seems that the unsupervised classifier better picked up differences in spectral profiles of areas due to the introduction of user error in creating mean average signatures through the collection of samples which is a nuanced and tricky process.

The average producer’s and user’s accuracies were greater for the unsupervised classification accuracy assessment than the supervised classification accuracy assessment. The lowest accuracies for the unsupervised classifications were in the agriculture, urban, and bare soil classes. This makes sense because the agriculture and bare soils often were mixed up by this classifier, or the classifier was confused between urban and bare soil lands. Also, user error in identifying reference samples could have been at play in lowering the accuracy for the urban class. With the coarser pixel density of classification image compared to the reference image, a pixel’s spectral profile of the classified image could have been a mix of multiple smaller pixels in the reference image. Whichever identified pixel on the reference image could have been trees which I would have identified as forest even through the average of that pixel and the pixels around it (all corresponding to one larger overlying classification raster pixel) would have been classified as urban. The same classes from the unsupervised classification had lower accuracies, and this may have been due to the same reasons specified.

Reflection on Supervised and Unsupervised Classification

I would not trust these classification methods, and if at all possible would shoot for something else. When asked if the outputs from these two methods could be used by local government for land use planning this is how I responded:

"I would not submit either map to the counties as both of the maps’ accuracy levels were below the level you would want when LU/LC policy decisions or other government decisions such as having to do with emergency management would be at stake. Policy decisions could affect people negatively due to the inaccurate data, and other government decisions could affect people even more negatively. I would recommend that the counties look to other counties who have already created maps in assistance with creation of a LU/LC map. Information learned from these counties could be applied to creating new data, or helping to edit the data from the classifications performed and assessed in these labs. Object based classification is a much better alternative to supervised and unsupervised raster classification. For example, Dane County has polygon vector data available and their procedure has been published here."

Sources:

Lab instruction is from Dr. Cyril Wilson. Landsat imagery is from the Earth Resources Observation and Science Center of the US Geological Survey. High resolution reference imagery is from the United States Department of Agriculture (USDA) National Agriculture Imagery Program.