Are you certain that your John Magee’s mother was named Sarah Stuart ?
There’s a baptism for St. Peter’s Church of Ireland parish in May 1859 of a John Magee to parents John Magee and Elizabeth - residence is familiar, although for the Skelly side.. 31 Upper Mercer Street, and John’s occupation is Shoemaker as per John jnr’s marriage… there are also at least two earlier children baptised to the same couple also in St. Peters.
Other than Sarah I did not locate any definite marriage for any additional siblings of John jnr - marriages are often the best evidence as they civil marriages and Church of Ireland records after 1845 include fathers names and occupations.
Gorey is a town in Co. Wexford located a little inland from the coast, and the good news is that if your Stuart line came from the town then there does appear to be at least some early records held locally for the Church of Ireland parish of Corey, with records of baptisms and marriages going back to 1801. As far as I know there are no records for the parish online at present - the RCB Library is undertaking a register transcription project, but this will take quite some time to complete. Some earlier records for the parish were lost in the civil war. The RCB Library in Dublin hold some transcripts - baptisms 1878 onward, marriages 1818 to 1973.
A search for possible siblings to Catherine Skelly shows a very promising marriage in St. Peter’s CofI parish - William Joseph Elmes & Martha Mary Skelly on 24 January 1884, Martha’s residence is shown as 50 York Street which is right beside Upper Mercer Street, and her father is Sylvester Skelly with occupation Printer. Martha is shown as a minor, so under 21, which if accurate woudl put her year of birth around 1863 or 1864 - right at the start of civil registration, but I found no sign of a birth record for her, or a baptism in the Dublin church records online.
William and Martha had a number of children and had moved to the suburban North strand area of Dublin by the time of the 1901 at Hawthorne Terrace and 1911 census at Northbrook Ave. Lwr. The 1901 census listing for the Elmes household was filled in by William, and he recorded a ‘Harriet Skilly’ age 64 as ‘Other Relative’, William died in 1909 and the 1911 census form was filled in by Martha snr, and this time ‘Harriet Skelly’ age 78 is shown as Martha’s mother. Maybe one or more of these children stayed in Ireland and could be the people your Aunt Sadie/Sarah visited with...
Based on the name of Martha’s mother we can also confirm a probable marriage for her parents, Sylvester Skelly & Harriet Stocker on the 8th June 1851 at St Audeon’s Church of Ireland in Dublin city, this parish is located to the north of St. Peter’s. Sylvester’s residence is shown as 42 Back Lane, which is adjacent to High Street, his occupation appears to be painter, Harriet gives the same residence. Their father’s are Sylvester Skelly an upholsterer, and Edward Stocker a lightkeeper. The witnesses are a Daniel Twomey and Julia Hayden.
At the time of the marriage number 42 Back lane is listed as W. Scully brassfounder, so appears Sylvster & Harriet lived over this business.
Harriet Skelly nee Stocker died in 1915, at 8 Northbrook Avenue (same address as 1911 census) with reported age of 86, the informant is Alice Elmes (her granddaughter).
Martha Mary Elmes (nee Skelly) died at St. Stephen's Hospital Dublin City on the 21st July 1928, age given as 65, her occupation is shown as sorters widow (William is shown as G.P.O clerk on the 1901 census)
(all the links included above are to free websites - IrishGenealogy or the National Archives of Ireland)
