In 1951, his family moved to Chapel Hill, North Carolina, when Isaac took a job as an assistant professor of medicine at the University of North Carolina School of Medicine. They built a house in the Morgan Creek area off the present Morgan Creek Road, which was sparsely populated. James would later say, "Chapel Hill, the Piedmont, the outlying hills, were tranquil, rural, beautiful, but quiet. Thinking of the red soil, the seasons, the way things smelled down there, I feel as though my experience of coming of age there was more a matter of landscape and climate than people. " James attended a public primary school in Chapel Hill. Isaac's career prospered, but he was frequently away from home, on military service at Bethesda Naval Hospital in Maryland, or as part of Operation Deep Freeze in Antarctica in 1955 and 1956. Isaac Taylor later rose to become dean of the UNC School of Medicine from 1964 to 1971. Beginning in 1953, the Taylors spent summers on Martha's Vineyard.