Sequential Invariant Information Bottleneck

Yichen Zhang; Shujian Yu; Badong Chen

doi:10.1109/ICASSP49357.2023.10094563

Sequential Invariant Information Bottleneck

Yichen Zhang, Shujian Yu^*, Badong Chen^*

^*Corresponding author for this work

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

9 Downloads (Pure)

Abstract

Previous approaches to the problem of generalization for out-of-distribution (OOD) data usually assume that data from each environment is available simultaneously, which is unrealistic in real-world applications. In this paper, we develop a new framework termed the sequential invariant information bottleneck (seq-IIB) to improve the generalization ability of learning agents in sequential environments. Our main idea is to combine the merits of the famed Information Bottleneck (IB) principle with the Invariant Risk Minimization (IRM), such that the learning agent can gradually remove spurious features and remain invariant and compact task-relevant information in a sequential manner. Experimental results on three MNIST-like datasets show the effectiveness of our method.

Original language	English
Title of host publication	ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Subtitle of host publication	[Proceedings]
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1-5
Number of pages	5
ISBN (Electronic)	9781728163277
ISBN (Print)	9781728163284
DOIs	https://doi.org/10.1109/ICASSP49357.2023.10094563
Publication status	Published - 2023
Event	48th IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2023 - Rhodes Island, Greece Duration: 4 Jun 2023 → 10 Jun 2023

Publication series

Name	ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
Volume	2023-June
ISSN (Print)	1520-6149

Conference

Conference	48th IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2023
Country/Territory	Greece
City	Rhodes Island
Period	4/06/23 → 10/06/23

Bibliographical note

Publisher Copyright:
© 2023 IEEE.

Funding

∗To whom correspondence should be ([email protected]; [email protected]). †This study was funded by the National Natural Science Foundation of China with grant numbers (U21A20485, 61976175)

Funders	Funder number
National Natural Science Foundation of China	U21A20485, 61976175

Keywords

Information Bottleneck
IRM
Out-of-distribution generalization
sequential environments

Access to Document

10.1109/ICASSP49357.2023.10094563

Sequential_Invariant_Information_BottleneckFinal published version, 2.65 MBLicence: Article 25fa Dutch Copyright Act

Persistent URL (handle)

Persistent link

Cite this

Zhang, Y., Yu, S., & Chen, B. (2023). Sequential Invariant Information Bottleneck. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP): [Proceedings] (pp. 1-5). (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings; Vol. 2023-June). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICASSP49357.2023.10094563

Zhang, Yichen ; Yu, Shujian ; Chen, Badong. / Sequential Invariant Information Bottleneck. ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP): [Proceedings]. Institute of Electrical and Electronics Engineers Inc., 2023. pp. 1-5 (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings).

@inproceedings{65b7e09c1588432588d5e522836e5df5,

title = "Sequential Invariant Information Bottleneck",

abstract = "Previous approaches to the problem of generalization for out-of-distribution (OOD) data usually assume that data from each environment is available simultaneously, which is unrealistic in real-world applications. In this paper, we develop a new framework termed the sequential invariant information bottleneck (seq-IIB) to improve the generalization ability of learning agents in sequential environments. Our main idea is to combine the merits of the famed Information Bottleneck (IB) principle with the Invariant Risk Minimization (IRM), such that the learning agent can gradually remove spurious features and remain invariant and compact task-relevant information in a sequential manner. Experimental results on three MNIST-like datasets show the effectiveness of our method.",

keywords = "Information Bottleneck, IRM, Out-of-distribution generalization, sequential environments",

author = "Yichen Zhang and Shujian Yu and Badong Chen",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 48th IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2023 ; Conference date: 04-06-2023 Through 10-06-2023",

year = "2023",

doi = "10.1109/ICASSP49357.2023.10094563",

language = "English",

isbn = "9781728163284",

series = "ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1--5",

booktitle = "ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)",

address = "United States",

}

Zhang, Y, Yu, S & Chen, B 2023, Sequential Invariant Information Bottleneck. in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP): [Proceedings]. ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, vol. 2023-June, Institute of Electrical and Electronics Engineers Inc., pp. 1-5, 48th IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2023, Rhodes Island, Greece, 4/06/23. https://doi.org/10.1109/ICASSP49357.2023.10094563

Sequential Invariant Information Bottleneck. / Zhang, Yichen; Yu, Shujian; Chen, Badong.
ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP): [Proceedings]. Institute of Electrical and Electronics Engineers Inc., 2023. p. 1-5 (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings; Vol. 2023-June).

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Sequential Invariant Information Bottleneck

AU - Zhang, Yichen

AU - Yu, Shujian

AU - Chen, Badong

PY - 2023

Y1 - 2023

N2 - Previous approaches to the problem of generalization for out-of-distribution (OOD) data usually assume that data from each environment is available simultaneously, which is unrealistic in real-world applications. In this paper, we develop a new framework termed the sequential invariant information bottleneck (seq-IIB) to improve the generalization ability of learning agents in sequential environments. Our main idea is to combine the merits of the famed Information Bottleneck (IB) principle with the Invariant Risk Minimization (IRM), such that the learning agent can gradually remove spurious features and remain invariant and compact task-relevant information in a sequential manner. Experimental results on three MNIST-like datasets show the effectiveness of our method.

AB - Previous approaches to the problem of generalization for out-of-distribution (OOD) data usually assume that data from each environment is available simultaneously, which is unrealistic in real-world applications. In this paper, we develop a new framework termed the sequential invariant information bottleneck (seq-IIB) to improve the generalization ability of learning agents in sequential environments. Our main idea is to combine the merits of the famed Information Bottleneck (IB) principle with the Invariant Risk Minimization (IRM), such that the learning agent can gradually remove spurious features and remain invariant and compact task-relevant information in a sequential manner. Experimental results on three MNIST-like datasets show the effectiveness of our method.

KW - Information Bottleneck

KW - IRM

KW - Out-of-distribution generalization

KW - sequential environments

UR - http://www.scopus.com/inward/record.url?scp=85177603687&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85177603687&partnerID=8YFLogxK

U2 - 10.1109/ICASSP49357.2023.10094563

DO - 10.1109/ICASSP49357.2023.10094563

M3 - Conference contribution

AN - SCOPUS:85177603687

SN - 9781728163284

T3 - ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

SP - 1

EP - 5

BT - ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 48th IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2023

Y2 - 4 June 2023 through 10 June 2023

ER -

Zhang Y, Yu S, Chen B. Sequential Invariant Information Bottleneck. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP): [Proceedings]. Institute of Electrical and Electronics Engineers Inc. 2023. p. 1-5. (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings). Epub 2023 May 5. doi: 10.1109/ICASSP49357.2023.10094563

Sequential Invariant Information Bottleneck

Abstract

Publication series

Conference

Bibliographical note

Funding

Keywords

Access to Document

Persistent URL (handle)

Other files and links

Fingerprint

Cite this