After you meet with Vagner on the cliff and he gets captured and you get a song. Give this song to Ms. Charm. She will want to meet Vagner, but, he is in the Fort.
Then Speak to the Astrologer and let him know that there is a way he can help Ms. Charm.
You need to give him a Silver Book at first...then you have wait a week...after a week give him a Gold Book ...then wait another week... speak to him again. Now the Astrologer wants Aristocrat Papers before they can go to the fort under the guise of a University Expedition.