Kueng, the 21-year-old world individual pursuit champion, was part of an early five-man breakaway to the 168.9km ride from Neuveville to Fribourg.
He finished 38 seconds clear of Belgian Jan Bakelants and Dutch rider Bert-Jan Lindeman (at 45s).
Albasini will have to work hard to keep the yellow jersey on Saturday’s mountainous fifth stage.
Briton’s Chris Froome is lurking in third in the overall standings, only 20sec off the pacesetter.
Reigning Tour de France champion Vincenzo Nibali (37s adrift) and Nairo Quintana (1min) are also expected to go on the attack ahead of Sunday’s final stage.
Saturday’s 162.7 km ride will take the peloton from Fribourg to Champex-Lac with three climbs, including the category one Col des Mosses.
Results of the fourth stage of the Tour de Romandie, a 169.8 kilometer race from La Neuveville to Fribourg on Friday:
1. Stefan Kueng (SUI/BMC) 4hr 35min 10sec, 2. Jan Bakelants (BEL/AG2R) at 38sec, 3. Bertjan Lindeman (NED/Team Lotto) at 39sec, 4. Tony Martin (GER/Etixx) at 45sec, 5. Gianni Meersman (BEL/Etixx) at 52sec, 6. Tosh van der Sande (BEL/Lotto Soudal) s.t., 7. Johannes Fröhlinger (GER/Giant) s.t., 8. Michael Albasini (SUI/Orica) s.t., 9. Simon Yates (GBR/Orica) s.t., 10. Ivan Santaromita (ITA/Orica) s.t.
1. Michael Albasini (SUI/Orica) 13hr 33min 40sec, 2. Ivan Santaromita (ITA/Orica) at 20sec, 3. Christopher Froome (GBR/Sky) s.t., 4. Simon Yates (GBR/Orica) m.t., 5. Ilnur Zakarin (RUS/Katusha) at 25sec, 6. Egor Silin (RUS/Katusha) s.t., 7. Yuri Trofimov (RUS/Katusha) s.t., 8. Simon Spilak (SLO/Katusha) s.t., 9. Tony Martin (GER/Etixx) at 27sec, 10. Rigoberto Uran (COL/Etixx) at 34sec.
12. Vincenzo Nibali (ITA/Astana) at 37sec, 25. Nairo Quintana (COL/Movistar) at 1min 41sec
– Agence France-Presse