Generally, Rallisport is a mixture of different driving/racing games, all with the Rally style. So you have self-challenges against the clock (but it's never some 10-15 second thing), which is real Rally. This is where you tackle long stretches of land, for minutes at a time, with the help of a navigator; to try to post the best times, for the course. (Different weather patterns are simulated on certain courses, requiring you to compensate for that)
There's hillclimb, in which you do something similar, with the aid of a limited-view map, but with the idea that you'll be going up and up, putting more emphasis on the power needed to consistantly go uphill. (Now with downhill challenges, which is the exact opposite. Akin to the Initial D downhill, but with Rally and not street racers) Both uphill and downhill are always mountain courses, with cliffs and the like and the dangers that ensue(but in this case, only dangerous to your time). Another possible case you find in certain Hillclimb events is the chance of weather changing on you during the course of doing the task.
You also have Rallycross, which is lap-based competition racing against other cars on courses that usually have different terrain features over various parts of the track.
Iceracing, is the same as Rallycross but on snow/ice-covered tracks.
Another type of racing is the Cross-over challenges. These typically involve 2 cars (offline) and what you do is race the other person, on the same track that has two different starting positions and they cross-over at some point. The tracks are kind of like a mobius thing (the idea of two tracks that link into each other). You race against each other, but are given indicators and times for how well you are doing for each section. The general case is that you wont meet each other, but it's possible. You'll see the other car zip by at times, because of the various places the two different sides cross-over or closely meet.
The two modes that you see in Colin McRae 04 are the regular Rally events and the Cross-over challenges. The other modes are specific to Rallisport.