A son of Revolutionary War officer Henry "Light Horse Harry" Lee III, Lee was a top graduate of the United States Military Academy and an exceptional officer and military engineer in the United States Army for 32 years. During this time, he served throughout the United States, distinguished himself during the Mexican–American War, and served as Superintendent of the United States Military Academy.

Fitzhugh Lee (1835 - 1905) was a Confederate cavalry general in the American Civil War, the 40th Governor of Virginia, diplomat, and United States Army general in the Spanish–American War. He was the son of Sydney Smith Lee, a captain in the Confederate States Navy, and the nephew of General Robert E. Lee.