On the contrary introduces a statement that says the opposite of the last one:
It must have been difficult. On the contrary it was easy.
To the contrary is when you prove the opposite:
Show me some evidence to the contrary (which proves this is wrong).