Born Sylvester Thompson in Holly Springs, Mississippi, Johnson sang and played with blues artists Magic Sam, Billy Boy Arnold, Junior Wells and Howlin' Wolf in the 1950s, before recording with Jimmy Reed for Vee-Jay in 1959. He made his solo debut that same year with Federal, a subsidiary of King Records of Cincinnati, backed by Freddie King on guitar.
He then began recording for Twinight Records of Chicago in the mid 1960s. Beginning with his first hit, Come On Sock It to Me in 1967, Johnson dominated the label as both a hitmaker and producer. His song Different Strokes, also from 1967, featured recently on the Ultimate Breaks and Beats breakbeat compilation.
Like other black songwriters of the period, several of his records at this time explored themes of African-American identity and social problems in songs including "Is It Because I'm Black", which reached Number 11 in the R&B charts in 1969.
In 1971, Willie Mitchell brought Johnson to Hi Records, the two recording three albums which spawned a number of singles. Produced in Memphis with the Hi house band, these yielded music of power and enduring value, including the hits We Did It, Back for a Taste of Your Love and Take Me to the River, his biggest success, reaching Number 7 on the R&B charts in 1975. However, at Hi Johnson was always to some extent in Al Green's shadow commercially, if not artistically. Mitchell also chose to use mainly in-house material rather than Johnson originals.
After the Hi years ended, Johnson produced two LPs for his own Shama label, the latter of which (Ms. Fine Brown Frame, 1982) was picked up for distribution by Boardwalk Records and produced Johnson's last hit record, the title cut.
Around the mid-1980s, Johnson started a fast-food fish restaurant business, and became semi-retired from performing, only making occasional appearances at blues club gigs.
In 1992, Johnson found out that his classic song "Different Strokes" had been sampled by number of rappers including Wu-Tang Clan, Kool G Rap, Hammer, and the Geto Boys. Stimulated by this fact, he decided to make a come back in the music business. In 1994, he released the album Back in the Game on Delmark Records. The album featured the Hi rhythm section and his youngest daughter Syleena Johnson.