When in the Columns area while configuring an Agile board, there is a delete link toward the top of each column. Clicking this should be the only thing that causes the column to delete. However, clicking anywhere in the column header below the editable title will cause the column to delete.

For example, click on the word "Drag" to the left of the "delete" link. You column is removed.