You can enable or disable piece rotation when selecting a new puzzle by using the switch under the puzzle. If piece rotation is enabled, you'll be able to rotate puzzle pieces by tapping them.
The option to rotate pieces becomes available after you've solved 3 puzzles.