Wylie was born in Somerville, New Jersey. Her grandfather, Henry M. Hoyt, was a governor of Pennsylvania. She was raised in this socially prominent family in Washington, D.C. Her aunt was Helen Hoyt, a minor poet. She graduated in 1912 from the Holton-Arms School in Bethesda, Maryland. She eloped with Harvard graduate Philip Simmons Hichborn (September 30, 1882 in Philadelphia, Pennsylvania - committed suicide 1912) son of a rear-admiral Philip Hichborn (March 4, 1839 in Charlestown, Massachusetts - 1910), they were married on December 13, 1906. She later eloped with Horace Wylie while still married to Hichborn. She married three times and had a son Philip Simmons Hichborn, Jr. (September 22, 1907 in Washington, D.C. - ?) by her first husband. Her last marriage (in 1923) was to William Rose Benét (February 2, 1886 in Brooklyn, New York - May 4, 1950) who was part of her literary circle and brother of Stephen Vincent Benet.Her parents were Henry Martyn Hoyt (December 1856 in Wilkes-Barre, Pennsylvania - ?) and Anne Morton McMichael (July 31, 1861 in Pa. -?). Their other children were:
- Henry Martyn Hoyt (May 8, 1887 in Pa. - 1920 in New York City) who married Alice Gordon Parker (January 27, 1885 in Newark, New Jersey - 1951)
- Constance A. Hoyt (May 20, 1889 in Pa. - 1923 in Bavaria, Germany) who married Ferdinand von Stumm-Halberg on March 30, 1910 in Washington, D.C.
- Morton McMichael Hoyt (April 4, 1899 in Washington, D.C. - ?), three times married and divorced Eugenia Bankhead, known as "Sister" and sister of Tallulah Bankhead
- Nancy McMichael Hoyt (October 01, 1902 in Washington, D.C - ?) romance novelist who wrote Elinor Wylie: The Portrait of an Unknown Woman (1935). She married Edward Davison Curtis, they divorced in 1932.