Jackson, who joined the Air Force when he was 21 years old and now resides in St. Cloud, Minnesota, says he hadn't spoken to Prince in 15 years.
"He was so busy on the road because he had so many engagements with his music," Jackson says. "So, he had to travel, all around the world. I always saw him on television, read the magazines, and said, 'There's my brother, Prince.' I was so happy he was making it for himself, I really was."
He and his mother would visit Prince at his Paisley Park estate, where the icon would spend most of his time.
"My mother liked looking at his records on the wall -- she wishes she had that many herself," Jackson jokes.
When it comes to childhood memories of Prince, he notes that the "Purple Rain" singer was incredibly gifted from an early age. In the memoir Possessed, written by Prince's former drummer, Charles Smith, Jackson was said to have influenced Prince's music when he was developing his sound at an early age. Jackson recalls to ET playing the piano in their mother's basement, and Prince being able to copy everything he did, only "faster" and "better."
"He was a genius," Jackson says. "Wherever he heard music, he could copy it just like that -- like he had a photographic memory or something."
Jackson also recalls Prince's fondness for singer James Brown.
"When he was younger, my father and mother would put him on stage, and he was dancing on stage and you should have seen him out there -- he wasn't but five years old," Jackson remembers. "He would just copy him and do everything James Brown was doing."
One memory that stands out is Prince's well-known love of basketball.
"I think he wanted to be taller because he wanted to be a basketball player," Jackson smiles, referring to Prince's 5'2" height. "We'd talk about basketball ... this was when we were kids. I took him downstairs in the basement, you know, I said, 'Prince, let's play basketball.' So we brought up all the paper in the house and put the trash can over there in the corner and threw paper into the basket ... After I did him in that way, he went to school ... and they kind of did him a favor because he was the smallest on the team. He did better than some of the rest of them."
Jackson has been named as one of Prince's potential heirs in court documents the singer's sister, Tyka Nelson, filed in Minnesota on Tuesday asking for a special administrator of Prince's estate since he didn't leave a will. On Wednesday, the court appointed Bremer Trust, National Association to oversee Prince's estate.
Other potential heirs of Prince's estimated $300 million fortune include Prince's four other half-siblings: John Nelson, Norrine Nelson, Sharon Nelson, and Omarr Baker. According to Minnesota state code, Prince's estate will be divided among his siblings, since he didn't leave behind a living spouse, children, parents, or grandparents.
Jackson's lawyer, Frank Wheaton, tells ET that he and his client are going to a meeting on Thursday to discuss how Prince's estate will be handled.
"We are about to head into a meeting with all the beneficiaries and the heirs -- that very subject will be discussed shortly ... how everything will be played out ... how the heirs will be treated," Wheaton says.