Charles W. Joy lived in Ellsworth, Maine his whole life. He started working at a grocery store owned by his father, Austin Joy, after his sophomore year of high school, took over the business in 1923, and retired in 1960. He was also part of the fire department.