The 1831 census just lists heads of household who are invariably going to be adults. Since your Martha would have been about 12 in 1831, I’d safely assume the one in the census is not yours.
However I did find a baptism for Sarah J. Duggan on 3rd July 1842 in St Columb’s/ St Columba’s (aka as Long Tower), in the city of Derry. Sponsors were Gerard (?) Cassidy & Catherine Donaghy.
So that places your couple in that year. I did not see baptisms for the other 2 children nor the couple’s marriage. So it’s possible they came from outside the city, possibly from a parish which does not have records for the 1830s or 1840s. On the other hand St Columb’s has a gap in it’s marriage records for 1837 – 1841 so they could have married there, and there’s just no surviving record. But the baptism records are complete so that wouldn’t account for the other 2 children not being found. Hard to know.
Assuming James & Martha both died in the US, have you found their death certificates, to get parents names from that?