Older
labs / tiddlers / content / labs / lab01 / _Labs_01_Computing Entropy.md
```As mentioned in lectures, entropy is measure of how much information content (“surprise”) is present in a system.

Given a set of N symbols, and the probability of each symbol occurring, we can compute the entropy (in bits) as:

{{/Labs/01/Images/entropy.svg}}

where p<sub>i</sub> is the probability of encountering a given symbol. A worked example of computing entropy was given in lectures. For another example, consider a system with five symbols: A; B; C; D; and E, each occurring with probabilities: 0.0625; 0.25; 0.5; 0.0625; and 0.125. The entropy of this system is 1.875 bits, the computation of which is outlined in the following table:

<table>
<tr>
<th>Symbol</th>
<th>i</th>
<th>p<sub>i</sub></th>
<th>log<sub>2</sub>p<sub>i</sub></th>
<th>-pi x log<sub>2</sub>p<sub>i</sub></th>
</tr>
<tr>
<td>A</td><td>1</td><td>0.2</td><td>-2.322</td><td>0.464</td>
</tr>
<tr>
<td>B</td><td>2</td><td>0.1</td><td>-3.322</td><td>0.332</td>
</tr>
<tr>
<td>C</td><td>3</td><td>0.3</td><td>-1.737</td><td>0.521</td>
</tr>
<tr>
<td>D</td><td>4</td><td>0.3</td><td>-1.737</td><td>0.521</td>
</tr>
<tr>
<td>E</td><td>5</td><td>0.1</td><td>-3.322</td><td>0.332</td>
</tr>
<tr>
<td></td><td></td><td></td><td></td><td>s=2.171</td>
</tr>
</table>
### Exercise
Use the table below to help you compute the entropy of a system with five symbols (A, B, C, D, E) with the probabilities 0.0625, 0.25, 0.5, 0.0625 and 0.125 (respectively):
Note:There is an editable worksheet doc on Blackboard you can use for your workings in this lab.
<table>
<tr>
<th>Symbol</th>    <th>i</th>	 <th>p<sub>i</sub></th>	 <th>log<sub>2</sub>p<sub>i</sub></th>	 <th>-p<sub>i</sub> x log<sub>2</sub>p<sub>i</sub></th>
</tr>
<tr>
<td>A</td><td>1</td><td>0.0625</td><td></td><td></td>
</tr>
<tr>
<td>B</td><td>2</td><td>0.25</td><td></td><td></td>
</tr>
<tr>
<td>C</td><td>3</td><td>0.5</td><td></td><td></td>
</tr>
<tr>
<td>D</td><td>4</td><td>0.0625</td><td></td><td></td>
</tr>
<tr>
<td>E</td><td>5</td><td>0.125</td><td></td><td></td>
</tr>
<tr>
<td></td><td></td><td></td><td></td><td>s= ?</td>
</tr>
</table>```