When Japanese say Japanese is difficult to learn

Very often Japanese say Japanese language is hard to learn.
I guess the one reason is that you have to learn three sets of characteres, which are Hiragana, Katakana and Kanji.
And the other is that the subtleness of usage of particles and auxiliary verbs may be confusing, even for Japanese.

But then, Hiragana and Katakan are easy to learn as Alphabets,
And even much easier to use them
as you have to learn spelling to use Alphabets,
on the other hand you do not have to when you use
Hiragana and Katakana as they are 100% phonetics.

Of course, Kanji, which is an ideograph, is very difficult learn.
However, it is not from Japan, it is from China.
And Hiragana and Katakana were derived from Kanji
as it was too difficult to learn for Japanese.

When Japanese say Japanese is difficult to learn,
what they really mean is that Japanese are smart enough
to speak such a complexed language.

Then, I wonder why no Japanese speak English after studying for six years at their middle schools and high schools.
Also many Japanese still speak crappy English
after they have lived here for sveral years.
If they are so language-smart,
why can't they learn English easily?
Especially many of them think English is
easier language to learn than Japanese.

Hmmmmmm?