Tessellation is a DX11 feature and requires hardware that supports DX11. What it does is splits triangles into smaller triangles at runtime to increase the surface detail of a mesh. Effect can be very slow to render and should be used within reason.
The Flat setting simply splits up the triangles. While this would not be of much use by itself, adjusting the world position of the new vertices using the normal map and displacement map, can create new surface details.
PN Triangles smoothes the object. It does require the mesh to have at least one smoothing group to make the smoothing work. As with the Flat setting, textures can be used to displace the tessellated vertices giving it a very detailed look.