PM Shinzo Abe's ruling bloc pushed it through the upper house early on Thursday, despite vocal opposition.
The government argues the law is needed to improve security ahead of the 2020 Olympics, and to comply with a UN agreement Japan has signed.
But critics say it weakens civil liberties and could be abused to monitor and target innocent citizens.
The law has sparked protests, including one on Wednesday night outside the parliament building which was attended by thousands of demonstrators.
What changes with the new law?
The law, which criminalises the plotting and committing of 277 acts, amends an existing law against organised crime syndicates.
It bans the procurement of funds or supplies and the surveying of a location in preparation of any of these offences.
An entire group - defined as two or more people - can be charged if at least one member is found to have been plotting the crime.
It also bans the expansion or maintenance of illicit interests of organised crime groups.
Japan has signed a UN convention against transnational organised crime, but not yet ratified it. The government said the new law was needed for ratification to go ahead.
Mr Abe told reporters the law would allow Japan to "firmly cooperate with international society to prevent terrorism".
What kind of crimes are on the list?
The ruling bloc has been attempting to push through the legislation for months. An earlier draft had listed 676 crimes, but it was pared down to 277.
The law bans the plotting of serious crimes such as terrorism but also lesser offences such as;
- Copying music
- Conducting sit-ins to protest against the construction of apartment buildings
- Using forged stamps
- Competing in a motor boat race without a licence
- Mushroom picking in conservation forests
- Avoiding paying consumption tax
Why are critics objecting?
Though the government has promised that the law will not be used unfairly, critics remain unconvinced.
They say the law is too broadly worded and gives the authorities sweeping powers.
They have also questioned the inclusion of certain acts and asked how they could be linked to terrorism and organised crime.
The government argues some could be used in association with criminal operations - for example, a gang or terror cell could fund its operations from the sale of illegally picked mushrooms.
But an editorial by newspaper The Mainichi said this argument was "unconvincing", as many other possible illegal sources of revenue such as marine poaching are not included on the list.
"While the sale of such seafood may also bring profits, that is not subject to the anti-conspiracy bill. What sets seafood apart from blessings from the mountains?" it said.
Critics have also taken issue with the way the bill was pushed through, as the ruling bloc took the unusual step of bypassing certain formalities to ensure it would be speedily passed.
They have accused the government of steamrolling the opposition, and questioned whether this was aimed at protecting Mr Abe from being grilled on a brewing political scandal.
An opposition party recently accused the prime minister of influencing a government decision to fund and approve a veterinary school at a university owned by Mr Abe's friend.
Mr Abe has repeatedly denied any wrongdoing.