At least 18 people were injured in a residential area. Two of them, an 11-year-old girl and a 39-year-old man, have died.
A suspect, a man in his 50s, stabbed himself in the neck after his rampage and later died in hospital.
Violent crime is rare in Japan and the motive for the attack is unknown.
The suspect was holding knives in both hands as he attacked the victims - 16 of whom were schoolgirls.
Police later named the victims of the fatal attack as Hanako Kuribayashi, who was in the sixth grade, and foreign ministry official Satoshi Oyama, who is believed to be the parent of another child.