© Getty Images

Dolley Madison -

Back when her husband James Madison was serving as Thomas Jefferson's secretary of state, Dolley Madison was said to have helped host White House social events because President Jefferson's wife, Martha, had died.

© Getty Images

Dolley Madison -

But rumors quickly escalated pointing to Madison being Thomas Jefferson's mistress. Not only that, she was also accused of helping her husband win the election of 1808 by becoming romantically involved with certain members of the electorate. The allegations have never been proven.

© Getty Images

Rachel Jackson -

Rachel Jackson was accused of being a bigamist after marrying future president Andrew Jackson while still married to a man named Lewis Robards. The couple wed in March 1791, but her divorce was not finalized until 1793.

© Getty Images

Rachel Jackson -

When Jackson ran for president his opponent, John Quincy Adams, tried to use this information to his advantage but to no avail—he ultimately lost to Jackson. And Rachel Jackson never got to be First Lady. She died before Andrew Jackson's inauguration.

© Getty Images

Julia Tyler -

Julia Gardiner married President John Tyler and became First Lady in 1844. Before tying the knot though, she posed for a picture with a man for an advertisement for a dry-goods store called Bogert & Mecamly (pictured).

© Public Domain

Julia Tyler -

This may seem innocent by today's standards but at the time it caused quite a stir, especially for someone from such a high-class background. Her family was so embarrassed by the scandal that they sent her away to Europe. When she returned, Julia went ahead and tied the knot with Tyler.

© Getty Images

Mary Todd Lincoln -

The wife of President Abraham Lincoln had a reputation for being a big spender. Mary Todd Lincoln was accused of spending federal money on personal luxuries, and stealing White House property.

© Getty Images

Lucy Hayes -

The wife of President Rutherford Hayes rocked the boat in Washington, D.C. by supporting the Temperance movement, which opposed the consumption of alcohol. The First Lady went as far as banning booze from all White House functions. Lucy Hayes was so unpopular among White House staff that she became known as "Lemonade Lucy."

© Getty Images

Frances Folsom -

Frances Folsom and President Grover Cleveland married in 1886, in what was (so far) the only wedding conducted in the White House. Frances was 21 and the president 49. But the age gap wasn't the only thing that raised eyebrows at the time. Frances was the daughter of a friend of Grover's, and the president had known her since her birth.

© Getty Images

Edith Wilson -

Edith Wilson's husband, President Woodrow Wilson, had a stroke in 1919 and as First Lady, Edith took over the running of the country, albeit unofficially. By doing so she became the first women to head the United States, even if it was just briefly. 

© Getty Images

Florence Kling Harding -

Warren Harding's wife Florence became known as the "Duchess" for her dedication in furthering the president's career. Florence was hit by a fair share of personal scandals  before marrying Harding, but none came close to what she was accused of after her husband passed away.

© Getty Images

Florence Kling Harding -

Florence Kling Harding was accused of poisoning the president! The posthumous allegation is present in Justice Department official Gaston Means' 1930 memoir, 'The Strange Death of President Harding.' While this accusation has long been dismissed as unfounded, another allegation in the book turned out to be true: Warren Harding had an illegitimate child with his mistress Nan Britton.

© Getty Images

Grace Coolidge -

The wife of President Calvin Coolidge was accused of adultery. Apparently a fling took place when the First Lady and her secret service escort, James Haley, went on a hike. 

© Getty Images

Grace Coolidge -

Haley was eventually removed from the Secret Service, a fact that only strengthened the allegation. The press at the time picked up on it, with papers like the Boston Herald running headlines such as "Wife's Long Hike Vexes Coolidge."

© Getty Images

Lou Hoover -

In 1929, President Herbert Hoover's wife sparked a race-related row after inviting Jessie DePriest, wife of the only black congressman, Oscar DePriest, for tea at the White House. The issue here was not the presence of color in the White House, but that Southern voters were concerned that civil rights leaders would use DePriest's visit to promote their cause (Oscar DePriest was a civil rights advocate).

© Getty Images

Eleanor Roosevelt -

It has been speculated that the wife of President Franklin D. Roosevelt may have had an affair with a female journalist called Lorena Hickok (pictured left, next to Eleanor Roosevelt).

© Getty Images

Eleanor Roosevelt -

Hickok was very close to Eleanor Roosevelt. Letters between the two, published by The New York Times in 1979, contained sentences such as "Hick darling ... Oh, I want to put my arms around you ... I ache to hold you close."

© Getty Images

Eleanor Roosevelt -

The Roosevelts' son, Franklin Roosevelt Jr., didn't interpret the correspondence as being sexual in nature. He said that modern audiences "don't understand that kind of love, which occurred between people who needed each other and gave to each other." Apparently, his mother was a fan of authors such as Jane Austen and the Brontë sisters, who wrote in a similar style, and she liked to use "that effusive form of writing."

© Getty Images

Bess Truman -

Back in 1945, the Daughters of the American Revolution (DAR) barred the black pianist Hazel Scott (pictured) from the society's Constitution Hall because it was reserved for "white artists only." Bess Truman was an honorary member of the DAR and was shocked. She asked her husband, Harry S. Truman, to intervene, but there was nothing he could do because the DAR was a private entity.

© Getty Images

Bess Truman -

Bess Truman telegraphed Scott's husband, Congressman Adam Clayton Powell, saying she "deplore[d] any action which denie[d] artistic talent an opportunity to express itself because of prejudice against race or origin." Powell called Bess Truman the "last lady of the land."

© Getty Images

Jacqueline Kennedy -

Unlike her husband, President John F. Kennedy, Jackie remained pretty scandal-free as First Lady. Things however did get a bit tricky for her during her second marriage to Aristotle Onassis. This had to do with the fact that Onassis had been divorced, but the Catholic Church didn't recognize the annulment. Religious differences aside, the two did eventually marry and Jackie wasn't excommunicated.

© Getty Images

Betty Ford -

This photo of the First Lady dancing on the Cabinet Room table in the White House in 1977 caused a stir, but probably not for the reasons you think. It was not the fact that Gerald Ford's wife was standing barefoot on the table, but rather the fact she was wearing pants.

© Getty Images

Betty Ford -

Apparently, wearing pants in public as the First Lady was frowned upon. Ford was considered to be too liberal to be a Republican, which put his wife in the spotlight too.

© Getty Images

Nancy Reagan -

The wife of Ronald Reagan famously consulted an astrologist to help plan the president's day. Apparently she wouldn't do anything without first consulting what the stars and planets had to say.

© Getty Images

Nancy Reagan -

But bringing astrology into the White House wasn't the only thing that rocked the boat for the First Lady. Kitty Kelley's 1991 book, 'Nancy Reagan: The Unauthorized Biography,' included some serious allegations about the First Lady, including running the White House and smoking marijuana.

© Getty Images

Hillary Clinton -

Hillary Clinton is no stranger to scandals, including accusations of financial misconduct in the Cattlegate and Whitewater scandals. Another distraction came about when she was campaigning with Bill Clinton in 1992. When asked why, Hillary famously remarked that she could have "stayed home and baked cookies." This comment did not sit well with homemakers.

© Getty Images

Hillary Clinton -

She made the headlines again during what became known as 'Hillarycare.' She chaired the task force behind the Clinton health care plan but ended up embroiled in a lawsuit brought by a number of healthcare lobbies. The Senate eventually blocked the legislation. 

© Getty Images

Hillary Clinton -

And then of course there was the cheating scandal involving her husband, Bill Clinton, with whom she stood by throughout the whole affair but was criticized for doing so. Hillary Clinton continued to be hit by a number of scandals throughout her career, including accusations of using a private email server for official communications.

© Getty Images

Laura Bush -

Laura Bush's past came back to haunt her in 2000 when it surfaced that George W. Bush's wife was involved in a car accident when she was 17. The future First Lady drove through a stop sign and hit another car. The 17-year-old driver of the other vehicle died as a result.

© Getty Images

Melania Trump -

Donald Trump's wife was accused of running a fake charity called Fostering the Future. A statement issued read: "Mrs. Trump does not operate a 501(c)(3) charitable organization. 'Fostering the Future' is the platform's name and part of a Be Best initiative."

© Getty Images

Melania Trump -

The First Lady was also accused of plagiarizing speeches, at least twice. One of them was taken from former First Lady Michelle Obama. Melania Trump's fashion choices were also controversial at times, including the "I don’t really care. Do u?" jacket she wore while visiting migrant children.

Sources: (Grunge) (Insider) (SheKnows) (Time) (The New York Times)

See also: First Ladies' inauguration day fashion throughout history

© Getty Images