Martin appeared wearing a police ankle tag device, which he said was, 'The latest celebrity thing.'
'I'm going to return it to the jewellery shop,' he joked. Host Letterman then claimed he had some security footage of Martin's crime.
'I went in to buy a ham,' said Martin as they showed the hilarious video.
The 65-year-old actor and musician just released his second album with The Steep Canyon Rangers, a banjo compilation called Rare Bird Alert. The album was produced by Tony Trischka and released on the Rounder Record label. It has 13 tracks, all of which are original songs written by Martin. He collaborated with Paul McCartney and The Dixie Chicks for the record.
'The idea of having Paul McCartney sing a song that I wrote is one of the most exciting moments of his life,' joked Martin.
He also said Lorne Michaels, creator and producer of Saturday Night Live, was the person who encouraged him to team up with other artists for the album.
'He said to me one day, 'You don't use your friends enough,'' Martin quoted Michaels as saying.
'I then began a most convoluted search for a contact number for Sir Paul, whom I had met briefly only twice, but then conveniently counted among my closest friends. Great thanks to him for showing up and throwing his heart into the song,' he writes in the CD's sleeve notes.