Transparent materials such as glass and water have the ability to let light through as if they were made of nothing. Yet some transparent materials, such as glass, are very dense. They are full of atoms.
For light to pass through such material without scatter, every photon has to meander through the material in a fashion identical to every other photon.
A way to envision this is to think of photons as slalom skiers, and the glass as a slope full of evenly spaced poles in all directions, with the poles being atoms.
When a photon enters such a material, it starts with a half roll past the first atom. Then it continues with a full roll past every subsequent atom until it makes a final half roll past the last atom before exiting.
The photon may make a first half roll to the left or the right. It does not matter. However, the next roll has to be in the other direction, and the next roll after that has to be opposite to the previous, and so on all the way through the material.
For photons entering the material at an angle, the first half roll will either be larger or smaller than average, depending on the angle of entry and which side of the first atom they enter. However, this is perfectly balanced on exit with a corresponding deviation from the average.
This will result in all photons leaving the material in the exact same direction that they entered it, provided the first row of atoms are parallel to the last row of atoms.
Since photons travel at the exact same speed regardless of their size. They always travel at the speed of light. However, the length of the path travelled by a small photon and a big photon will not be identical.
Small photons roll past atoms with their geometrical centre closer to the atom than the bigger photons, so even when large photons and small photons take the exact same path through a transparent medium, the smaller ones end up travelling a shorter distance.
Send a red photon and a blue photon through a piece of glass at the exact same time, and the red one ends up exiting the glass ahead of the blue one. The red one has less energy than the blue one. It is smaller, and is therefore rolling past the atoms in the glass at a shorter distance from the atoms’ centre than the blue one.
This explains why blue light takes more time to travel through transparent media than red light.
It also explains why blue light refracts more through a prism than red light. It explains why a mix of various size photons, known to us as white light, get split into all the colours of the rainbow, with blue light always at the most acute angle from the prism, and red light at the least acute angle.
Being larger than red photons, blue photons take more time rolling past the first atom. This makes the initial half roll more acute for blue photons than red photons. It also makes the full rolls and the final half roll more acute.
The initial and final half roll of photons are precisely defined by the photons’ size compared to the atoms in the medium. The bigger the photons, the more acute are their half rolls into and out of the prism.
Note that the photons do not divert from each other in their overall change in direction on entering a medium. Photons of different colours race through the medium in parallel.
It is not until the final half roll that diffraction happens. If the final half roll is back into the original direction, as is the case with plain glass sheets, the difference in original half roll entering the glass is cancelled out by the difference in half roll on exiting the glass.
However, if the final half roll is to the same side as the original half roll on entering the glass, as is the case in a prism, the original angle does not cancel out. It gets added to, and there is diffraction.
This is why white light remain sharp and focused even through the thickest of glass sheets, while the smallest of prisms split white light just as well as big ones.
Note also that this has nothing to do with wavelength. All that matters is the size of the photons.