The protesters wanted to show solidarity with the victims of last week's terror attacks in Paris.

German Muslim groups had organized the event with the support of Germany's Jewish community, representatives of other churches, as well as all parties that are represented in the German Parliament, the Bundestag.

German President Joachim Gauck held a speech at the rally in which he said that the fight against religious extremism should be the responsibility of all Germans -- and not only of Muslims or Jews. Directly addressing religious extremists, Gauck said: "We don't fear you. Your hatred motivates us. We stand by our country and our values. ... The terrorists (in Paris) wanted to divide us, but they have achieved the opposite: They have united us."

Chancellor Angela Merkel and members of her Cabinet attended the demonstration without making public remarks. Merkel is scheduled to comment on the Parisian attacks on Thursday in a governmental address in the German Bundestag.

The rally started at 5:45 p.m. local time under the slogan "Standing together -- Showing support." Representatives of Germany's Muslim community paid tribute to the victims of last week's attacks at the French Embassy in Berlin, which is located right next to the Brandenburg Gate. Shortly afterward, Quran verses that emphasize the peaceful nature of Islam were recited on stage, followed by a minute of silence for the Parisian terror victims.

Aiman Mazyek, the chairman of Germany's Central Council of Muslims said: "Today, we send a strong signal from Berlin to the world and to Germany: We protect the freedom of speech, and we represent a liberal-minded and friendly Germany."

Some protesters held "Je suis Charlie" posters, or waved French flags. Daniela Mayer, 55, from Berlin held a sign saying: "Terror has no god." She told CNN: "Right-wing Germans are trying to abuse the horrific terror attacks in Paris for their own purposes. That's why I'm here today. We need to emphasize that Muslims are no terrorists." Twenty-one-year-old French citizen Julie, who is spending her vacation in Berlin with friends, was surprised to see the large turnout for the vigil. "These attacks could have happened everywhere -- that's why it's great to see so much sympathy here in Germany."

Rising number of anti-Islam protesters

While the rally was organized in reaction to last week's attacks in Paris, its organizers also hope to make an impact in Germany. On Monday, a record 25,000 protested against what they call an "Islamization of German society and culture" in the city of Dresden, according to local police estimates. At least 80,000 people rallied against the anti-Muslim protesters in various cities across Germany, according to German press agency DPA.

"I'm not scared by the rise of those anti-Islam protests," Bekir Yilmaz, the president of the Turkish Community in Berlin, or TCB, told CNN on Tuesday. "But we as Muslims will have to become more active to counter such prejudices. We need to send clear signals that we want to build a peaceful future in Germany to convince those anti-Islam protesters."

Yimaz also said German Muslims "are opposed to violence and hatred of all kinds." "We want to show the world that we are peaceful and would never sympathize with terrorists," he said.

Most German politicians have condemned the anti-Muslim marches, but some agree that Germany must do more to protect itself against potential terror attacks.

Speaking to CNN's Christiane Amanpour, German Interior Minister Thomas de Maiziere paraphrased some of the anti-Muslim protesters' most prominent arguments last week: " 'How dangerous is Islam for our society?' 'What about the fight against (ISIS)?' 'Can this be successful?' "

De Maiziere went on to say that "these critical questions have to be faced, have to be answered."

Some conservative politicians have urged Chancellor Merkel to push for tougher laws that would give authorities more leeway to track suspects and prevent terror attacks.

Merkel: Islam belongs to Germany

In her New Year's speech, Merkel addressed the issue of xenophobia, saying anti-Muslim demonstrations encourage the exclusion of people because of their skin color or religion.

Merkel urged people not to attend such rallies, where people have "hatred in their hearts," she said.

"We know the value of unity in our country," she said. "It is the foundation of our success."

On Monday, Merkel re-emphasized her opinion by saying that Islam belonged to Germany -- a remark which had already caused criticism in the predominantly Christian country when former President Christian Wulff took a similar stance in 2010.

Some representatives of Germany's Muslim community have voiced their agreement with Merkel for trying to bridge divides between Muslims, Christians and Jews.

(CNN)