The attacker, who has not been named, was shot dead by police.
Witnesses described scenes of panic, with many people fleeing the building in tears. The area in the île de la Cité has been sealed off.
The attack comes a day after police went on strike across France over increasing violence towards officers.
The Paris prosecutor said a murder investigation had been launched.
A motive for the attack remains unclear. However, police union officials have suggested the attacker may have been involved in a workplace dispute.
At about 13:00 local time (11:00 GMT; 12:00 BST), the attacker is said to have gone into the building and straight to his office where he began attacking colleagues with a knife.
He stabbed three people inside two offices and two women on a stairway, before he was shot dead by an officer inside the building’s courtyard, according to French media.
Three men and one woman were killed, Paris prosecutor Remy Heitz told reporters.
A fifth person was critically injured in the attack. The building is near major tourist sites including the Notre-Dame cathedral.
“Police were running around in panic,” a witness who was inside the courtyard at the time of the attack told Le Parisien newspaper.
“I was surprised to hear shooting because this is not a place where you hear that kind of thing. I first thought it was a suicide because there are a lot of those at the moment,” he added.
President Emmanuel Macron, Prime Minister Edouard Philippe and Interior Minister Christophe Castaner have all visited the site.
Who was the attacker?
According to the interior minister, he was a 45-year-old IT specialist who had worked for the Paris police force for 16 years.
Officials said he had been working in the force’s intelligence division.
Mr Castaner said there had been no warning signs about the attacker, adding: “On the face of it, he looked a model employee.” A search of his home in the north of Paris is currently under way.
There were tensions between the knifeman and his supervisor, according to police union official Christophe Crepin.
“I do not think this is a terrorist act,” Mr Crepin, who knew the attacker, told Franceinfo Radio.
Police union leader Jean-Marc Bailleul described it as a criminal act, telling BFMTV: “It was a moment of madness.”
The attack follows a rare nationwide police strike on Wednesday, which saw thousands of officers demonstrating in Paris over working hours, shortages in resources and controversial pension reforms.
Police unions say there have been more than 50 suicides by police officers since the start of the year.
They blame the rising numbers on difficult working conditions and increasing violence towards police.