Rayburn married Anne Gray Houchens Smith, daughter of William {Bill} Houchens and Elsie Florence Stables Houchens, on Apr 9, 1971 in Richmond, Va. (Anne Gray Houchens Smith was born on Jul 12, 1945 in Richmond, Virginia.) The cause of her death was natural causes.
