In 1984, Poland's bandmate from "The New Yorkers," Gar Samuelson, became the drummer of thrash metal band Megadeth. After seeing Samuelson perform live with Megadeth, Poland went backstage and suggested an impromptu audition as a lead guitarist. Poland thereafter became a full member of the band, along with Samuelson, Mustaine, and bass player David Ellefson.

In 1985, Megadeth released their first album Killing Is My Business... And Business Is Good!, which showcased Poland's dynamic style of playing. Poland left the band during the tour to support the album, and was replaced temporarily by guitarist Mike Albert. Poland later rejoined the band to record their second album "Peace Sells... But Who's Buying?" in 1986. The album was highly acclaimed critically for both its instrumental and lyrical complexity, and is today considered widely to be one of the defining records of heavy metal.

Ater leaving Megadeth, Poland joined punk rock legends The Circle Jerks as a bass player. In 1990, he released Return to Metalopolis a solo heavy metal/jazz fusion album. Shortly beforehand, he agreed to record demos for Megadeth's fourth album, Rust in Peace, but declined what would be the first of two offers to re-join the band permanently (in this instance partly because of his concentration on his solo album, and the fact that, since most of the album's music and arrangements had already been written, Poland felt that the scope for his input would be limited). Guitarist Marty Friedman joined Megadeth later that year, and adapted some of Poland's guitar solos for the release version of Rust in Peace, which would go on to be Megadeth's most critically praised album.

In a surprise turn in 2004, Dave Mustaine requested Poland to record guitar solos for the Megadeth record The System Has Failed, the first Megadeth album not to feature original bass player and founding member David Ellefson. Poland recorded the solos on a contractual basis, but did not re-join the band. With Poland's contribution, The System Has Failed is, at present, the last Megadeth album to feature any of the band's original recording line-up other than Dave Mustaine with Ellefson no longer a member of Megadeth, but now has returned to replace James LoMenzo, Mustaine was then the sole remaining member of the band that recorded Killing Is My Business... And Business Is Good!

In 2009, Poland formed OHMPHREY, a side-project with Robby Pagliari of OHM, and Jake Cinninger, Kris Myers, and Joel Cummins of progressive rock band Umphrey's McGee. OHMPHREY released debut album, OMPHREY (on Magna Carta Records) on May 19, 2009.