Terrestrial-sized moons can exist around gas giant planets. This is especially interesting for gas giant planets that occupy the habitable zone around their host stars because terrestrial-sized moons around such planets can support Earth-like conditions. A terrestrial-sized moon requires a minimum mass of at least 0.1 to 0.2 Earth mass in order to hold on to an atmosphere for billions of years. This is about 4 to 5 times more massive than the largest moons in our Solar System and is comparable to the mass of Mars.
The largest moons in our Solar System presumably formed through accretion of material in the circumplanetary disks around Jupiter and Saturn. This poses a problem for forming a terrestrial-sized moon because there is not enough material and/or accretion efficiency in the circumplanetary disk around Jupiter or Saturn to form anything more massive than ~ 0.025 Earth mass. The moons Callisto and Ganymede around Jupiter, and Titan around Saturn show that a formation process via accretion in a circumplanetary disk is unlikely to form a moon larger than 1/10,000th the mass of its host planet.
Williams (2013) proposes that terrestrial-sized moons can exist around gas giant planets through a formation process known as binary-exchange capture. This process occurs when a binary comprising of two terrestrial-sized objects is tidally disrupted during a close encounter with a gas giant planet. One member of the binary gets ejected (escaping mass) while the other remains behind as a moon (captured mass) around the gas giant planet. Binary-exchange capture can only take place if the ratio of captured mass to escaping mass is not too large.
An example of a binary exchange capture involves a Jupiter-mass gas giant planet in orbit around a Sun-like star at a distance of 1 AU and a binary comprising of one object with the mass of Mars and one object with the mass of Mercury. If the binary approaches as close as ~ 5 planetary radii from the gas giant planet, the object with the mass of Mars remains behind as a moon (captured mass) while the object with the mass of Mercury gets ejected (escaping mass). In this scenario, the ratio of captured mass to escaping mass is ~ 2:1.
However, if the binary is comprised of one object with the mass of Earth and one object with the mass of Mars, then capture of the Earth-mass object as moon around the gas giant planet is not possible because the binary’s mass ratio of ~ 10:1 is simply too large. Nevertheless, if the distance of the Jupiter-mass gas giant planet is doubled to 2 AU, binary exchange capture of the Earth-mass object as moon around the gas giant planet becomes possible.
In our Solar System, Neptune’s moon Triton is believed to have been placed into orbit around Neptune via binary exchange capture. This is because Triton’s orbit around Neptune is retrograde, which means it orbits in the opposite direction to the planet’s spin and could not have accreted out from a circumplanetary disk around Neptune. Although Triton in nowhere as massive as a terrestrial-sized moon, its existence does supports binary exchange capture as a viable mechanism.
Williams (April 2013), “Capture of Terrestrial-Sized Moons by Gas Giant Planets”, Astrobiology, vol. 13, issue 4, pp. 315-323