| Year | Title | Role | Notes |
| 1941 | Pot o' Gold | Band member / Radio Announcer | Uncredited |
| 1949-50 | Morey Amsterdam Show | | |
| 1950 | PM Picnic | The Narrator | |
| 1955–56 | The Honeymooners | Ed Norton | Primetime Emmy Award for Best Actor in a Supporting Role (1956) |
| 1958 | Alfred Hitchcock Presents | Cyril T. Jones | Episode: "Safety for the Witness" |
| 1960 | The Twilight Zone | Henry Corwin | Episode: "The Night of the Meek" |
| 1964 | The Yellow Rolls-Royce | Joey Friedlander | |
| 1967 | A Guide for the Married Man | "Joe X", Married Man | |
| 1972 | The Snoop Sisters: 'Female Instinct' | Barney | with Helen Hayes and Mildred Natwick |
| 1974 | Harry and Tonto | Harry Coombes | Academy Award for Best Actor Golden Globe Award for Best Actor in a Motion Picture – Musical or Comedy |
| 1975 | and the Dixie Dancekings | Deacon John Wesley Gore | |
| Death Scream | Mr. Jacobs | TV movie (aka Street Kill) |
| Katherine | Thornton Alman | TV movie Nominated—Primetime Emmy Award for Outstanding Single Performance by a Supporting Actor in Comedy or Drama Special |
| 1976 | Won Ton Ton, the Dog Who Saved Hollywood | Fromberg | |
| 1977 | The Late Show | Ira Wells | National Society of Film Critics Award for Best Actor |
| Scott Joplin | John Stark | |
| 1978 | House Calls | Dr. Amos Willoughby | |
| Movie Movie | Dr. Blaine / Dr. Bowers | Segments: "Dynamite Hands" and "Baxter's Beauties of 1933" |
| Star Wars Holiday Special | Trader Saun Dann | |
| 1979 | Ravagers | Sergeant | |
| You Can't Take It With You | Grandpa Martin Vanderhof | |
| Steel | "Pignose" Moran | |
| Sunburn | Marcus | |
| Going in Style | Al | Pasinetti Award for Best Actor |
| 1980 | Defiance | Abe | |
| Roadie | Corpus C. Redfish | |
| Alcatraz: The Whole Shocking Story | Robert Stroud | TV movie |
| Fighting Back: The Rocky Bleier Story | Art Rooney | TV movie |
| 1981 | Bitter Harvest | Walter Peary | TV movie |
| Take This Job and Shove It | Charlie Pickett | |
| St. Helens | Harry Truman | |
| The Leprechauns' Christmas Gold | The Narrator / Blarney Kilakilarney | TV movie, Voice |
| 1982 | Better Late Than Never | Charley Dunbar | |
| 1983 | The Last Leaf | Mr. Behrman | |
| 1984 | Terrible Joe Moran | Tony | TV movie Primetime Emmy Award for Outstanding Supporting Actor in a Limited Series or a Special |
| Firestarter | Irv Manders | |
| The Naked Face | Morgens | |
| The Muppets Take Manhattan | Bernard Crawford | |
| The Night They Saved Christmas | Santa Claus | TV movie |
| 1985 | The Undergrads | Mel Adler | |
| Izzy and Moe | Moe Smith | TV movie |
| The Blue Yonder | Henry Coogan | TV movie |
| 1986 | Miracle of the Heart: A Boys Town Story | Father Michael T. O'Halloran | |
| 1987 | Night Friend | Monsignor O'Brien | |
| 1990 | Where Pigeons Go to Die | Da | Nominated—Primetime Emmy Award for Outstanding Lead Actor in a Miniseries or a Special |
| 1993 | Last Action Hero | Frank | Final film role |