That’s a very interesting question. The journal impact factor and h index are different in their fundamental design: The former is used to measure journal prestige, while the latter is used to measure researcher impact. Therefore, the two cannot be compared. Let me explain.
The journal impact factor measures the average number of citations received by articles published within a journal over a two-year period. It can be a reliable measure of journal reputation but does not measure the impact of individual articles or researchers.
On the other hand, the h index is designed to measure the scientific output of a researcher by considering a combination of the number of papers the researcher has published and the number of citations those papers have received. Thus it measures both your publication record and its impact.
Do you now see how the impact factor and h index cannot be compared because they serve different purposes?
Your colleagues are right though. As a researcher, it is more useful for you to calculate your h index than to use the journal impact factor as a measure of prestige. The h index can especially work to your advantage if you have published many papers. I understand that it is troublesome and time consuming to calculate your h index because it requires you to maintain a record of all your published papers and the citations they have received through timely searches on multiple databases. But this would be time and effort well spent because the h index is gaining popularity and is increasingly being considered by grant and tenure committees.
However, like the journal impact factor, even the h index has its limitations, and it is advisable for you to mention a combination of citation metrics—the impact factor of journals in which you have published, your h index, and other article-level metrics—on your grant applications or statements of purpose, so as to give people a holistic view of your impact as a researcher.