What does this poem mean "the apology"???

THINK me not unkind and rude


That I walk alone in grove and glen;


I go to the god of the wood


To fetch his word to men.


Tax not my sloth that I


Fold my arms beside the brook;


Each cloud that floated in the sky


Writes a letter in my book.


Chide me not, laborious band,


For the idle flowers I brought;


Every aster in my hand


Goes home loaded with a thought.


There was never mystery


But't is figured in the flowers;


Was never secret history


But birds tell it in the bowers.


One harvest from thy field


Homeward brought the oxen strong;


A second crop thine acres yield,


Which I gather in a song.

What does this poem mean "the apology"???
to me , it sounds like the person is apologizing for wandering off contemplating deep thoughts, while others are toiling away at physical labor,,,,,, he/she is saying that each thing they see/hear/touch, is bringing revelations to them,,,, it seems like the person has been doing this for a long time,,,,, threw two harvests