The hosts were 173 for four at tea, trailing the West Indies by 138 runs. Ajinkya Rahane on 19 and Rishabh Pant on 9 were batting at the break in Hyderabad.
Holder, who scored 52 in the West Indies' first innings score of 311, trapped Kohli lbw for 45.
The wicket ended a 60-run fourth-wicket stand with Rahane.
Kohli reviewed the decision but replays suggested the ball would have hit the leg stump and Holder was ecstatic. He also bowled opener Lokesh Rahul for four in the first session.
The West Indies, who were bowled out in the morning session, struck back after lunch with two quick wickets of teenage opener Prithvi Shaw and then Cheteshwar Pujara.
Shaw hit a quickfire 70 off 53 deliveries before falling to left-arm spinner Jomel Warrican and Pujara went for 10 in the next over off paceman Shannon Gabriel.
Kohli and Rahane then batted with a mix of caution and aggression as Holder juggled through his bowling options.
The West Indies were hit by injury to wicketkeeper Shane Dowrich early in the Indian innings, and replaced him with substitute Jahmar Hamilton behind the stumps.
Earlier Indian fast bowler Umesh Yadav returned career-best figures of 6-88 to help dismiss the West Indies early in the morning session after the tourists started the day on 295-7.
Roston Chase top-scored with 106 after West Indies elected to bat first. Chase, who started the day on 98, was eventually bowled by Yadav.