Kilmacduagh and Kiltartan are both alternate names for the Catholic parish of Gort. Such alternate names occur for various reasons. In this case, the Catholic parish of Gort contains wthin it several (or at least parts of several) administrative districts. Kilmacduagh and Kiltartan are the names of two different civil parishes, while Gort is the name of an electoral division (in addition to being the name of the Catholic parish as a whole). The electoral division is presumaby located within one (or perhaps partly in each) of those civil parishes. The Tithe Applotment records would likely have used the civil parish divisions, as you found. You may be dealing with two different James's there, or just one, who held tenancies on either side of the civil parish boundary.
Kiltartan is also the name of the local barony, an older division which is of nothing but historical interest at this point (and never was of all that much importance).
At this link you can see a list of the various districts in Galway, including the three mentioned above: https://www.townlands.ie/galway/
There are further links there to info about each of the places listed. At the townland level, there are further links to census records from 1901 and 1911, and also Griffith's Valuation (mid-1800's), the last of which may have a record for the father, James. Of course, you'd have to go townland by townland, but you might start with the townland of Gort, which is in the electoral division of the same name. I was going to check it out for you, but the records are pretty lengthy, so you can check them out yourself here (they appear to be for the town of Gort itself):
The Catholic parish records for Gort only go back to 1848, so they may not be much help, but families sometimes had children over a long period of time, so you could look at the baptisms in the earliest available records to see whether James and Margaret had any more children. If so, that may tell you in what townland they lived. The records are available here:
In case you're interested, the Dolan surname originated northeast of Gort, along what is now the Galway-Roscommon border. The Irish versions are usualy gven as Ó Dubhláin, or in modern times Ó Dúláin, but MacLysaght says, in his Surnames of ireand, that the original form was Ó Dobhailen, and that a family of that name has records in the area as far back as the twelfth century. He also say that the name spread far to the northeaast, and can be found now as far away as southern Ulster.