Is the Koya line from Namba a direct line?Limited express trains travel directly to Gokurakubashi Station, where you transfer to the cable car. By express trains, a transfer at Hashimoto Station is usually required.
Limited express trains are faster and require a surcharge and seat reservation, while seat reservations are not possible on express trains.
Should I book the ticket in Tokyo due to the busy period?
I have the feeling that except on very busy days, the limited express trains between Namba and Koyasan don't get booked out. And even if they do, then you can still ride an express train. I personally would not make reservation for the Nankai trains until arriving at Namba (or Shin-Imamiya Station, which is the better transfer station for Japan Rail Pass holders than Namba).
As for reservations for the shinkansen, I recommend to make them in advance, if possible.
From Koyasan I will go to Nara: should I go back to Osaka and then get a train from there or there is a different and faster alternative?
The faster way is via Osaka. But you can also take Nankai Railways only as far as Hashimoto Station and transfer to the JR Wakayama Line there to Nara. It takes more time, but is cheaper for Japan Rail Pass holders, and the scenery is quite nice.