A panel of three judges at the Court of Session found in favour of a cross-party group of politicians who were challenging the prime minister’s move.
The judges said the PM was attempting to prevent Parliament holding the government to account ahead of Brexit.
The UK government said it will appeal against the ruling to the Supreme Court in London.
The Court of Session decision overturns an earlier ruling from the court, which said last week that Mr Johnson had not broken the law.
But it is currently unclear what impact the judgement will have on the current five week suspension of Parliament – a process known as proroguing – which started in the early hours of Tuesday.
MPs are not scheduled to return to Parliament until 14 October, when there will be a Queen’s Speech outlining Mr Johnson’s legislative plans. The UK is due to leave the EU on 31 October.
Mr Johnson has previously insisted that it was normal practice for a new government to prorogue Parliament, and that it was “nonsense” to suggest he was attempting to undermine democracy.
But the Court of Session judges said they were unanimous in their belief that Mr Johnson was motivated by the “improper purpose of stymieing Parliament”, and that he had therefore effectively misled the Queen in advising her to suspend Parliament.
They added: “The Court will accordingly make an Order declaring that the prime minister’s advice to HM the Queen and the prorogation which followed thereon was unlawful and is thus null and of no effect.”
The group of more than 70 largely pro-Remain MPs and peers behind the legal challenge were headed by SNP MP Joanna Cherry, who called for Parliament to be immediately reconvened following the ruling.
She added: “We feel utterly vindicated and I would be confident that the UK Supreme Court will uphold this decision.”
The parliamentarians appealed to the Inner House of the Court of Session after their original challenge to the suspension of Parliament was dismissed by judge Lord Doherty last week.
Lord Doherty said Mr Johnson had not broken the law by proroguing Parliament, and that it was for MPs and the electorate to judge the prime minister’s actions rather than the courts.
But the three Inner House judges said they disagreed with Lord Doherty’s ruling because this particular prorogation had been a “tactic to frustrate Parliament” rather than a legitimate use of the power.
One of the three judges, Lord Brodie, said: “This was an egregious case of a clear failure to comply with generally accepted standards of behaviour of public authorities.
“It was to be inferred that the principal reasons for the prorogation were to prevent or impede Parliament holding the executive to account and legislating with regard to Brexit, and to allow the executive to pursue a policy of a no-deal Brexit without further Parliamentary interference.”
And Lord Drummond Young said that the UK government had failed to show a valid reason for the prorogation, adding: “The circumstances, particularly the length of the prorogation, showed that the purpose was to prevent such scrutiny.
“The only inference that could be drawn was that the UK government and the prime minister wished to restrict Parliament.”
The judges will release their full findings on Friday.
The ruling goes to the heart of the advice the prime minister gave to the Queen. In effect, that advice has been ruled unlawful because it was given for an improper purpose – the purpose of stymieing Parliament, reducing the amount of time MPs had to debate the various Brexit options.
This was the basis of the legal challenge, which was really looking at the prime minister’s intent in giving that advice to the Queen. The government, of course, argued it was done entirely lawfully but the argument from not only the Scottish case, but also the Gina Miller case, was really about the intent.
What did Boris Johnson really intend to do? What was his real intention behind giving his advice to the Queen to prorogue? Intention is a difficult thing to tie down but we seemingly have a really strong ruling from the Scottish court about that.
A spokesman for Number 10 said it was “disappointed” by the decision, and would appeal to the Supreme Court.
He added: “The UK government needs to bring forward a strong domestic legislative agenda. Proroguing Parliament is the legal and necessary way of delivering this.”
Scotland’s first minister, Nicola Sturgeon, said the ruling was of “enormous constitutional significance”, and that Parliament should be recalled immediately to allow it to do the “real and substantive work of scrutiny”.
She added: “The prime minister’s behaviour has been outrageous and reckless, and has shown a complete disregard for constitutional rules and norms.”
Labour’s Shadow Brexit secretary Sir Keir Starmer said Parliament should be recalled as early as this afternoon.
He told the BBC: “Most people didn’t believe Boris Johnson, but for the courts to find he has unlawfully shut down Parliament and that his motive wasn’t the one he said it was? That’s very powerful.
“I call on him to recall Parliament. Let’s get it back open, and sitting this afternoon and tomorrow, so we can debate what happens next and we can debate this judgement.”
And Dominic Grieve, the former Conservative MP and attorney general who now sits as an independent, said the prime minister should “resign very swiftly” if he has misled the Queen.
Mr Grieve said: “He will find himself in an untenable position in Parliament. Every member that believes in our constitution would simply say it’s over.”
It emerged during last week’s hearings that Mr Johnson appeared to have approved a plan to shut down Parliament two weeks before publicly announcing it.
The court heard the prime minister was sent a note on 15 August asking if he wanted to prorogue parliament from mid-September. A tick and the word “yes” were written on the document. He announced the plan on 28 August.
The court later agreed to release the documents to the media.
In a separate case brought by anti-Brexit campaigner Gina Miller, the High Court in London also ruled last week that Mr Johnson had acted lawfully. Ms Miller is appealing that decision in the Supreme Court.
Meanwhile, a hearing at the High Court in Belfast into the implications of a no-deal exit is continuing, with a campaigner for victims of the Troubles arguing that it could jeopardise the Northern Ireland peace process