Self supervised vision transformer github
Code: github. Abstract: One paradigm for learning from few labeled self supervised vision transformer github while making best use of a large amount of unlabeled data is unsupervised pretraining followed by supervised fine-tuning. Although this paradigm uses unlabeled data in a task-agnostic way, in contrast to most previous approaches to semi-supervised learning for computer vision, we show that it is surprisingly effective for semi-supervised learning on ImageNet.
A key ingredient of our approach is the use of a big deep and wide network during pretraining and fine-tuning. We find that, the fewer the labels, the more this approach task-agnostic use of unlabeled data benefits from a bigger network. After fine-tuning, the big network can be further improved and distilled into a much smaller one with little loss in classification accuracy by using the unlabeled examples for a second time, but in a task-specific way.
The proposed semi-supervised learning algorithm can be summarized in three steps: unsupervised pretraining of a big ResNet model using SimCLRv2 a modification of SimCLRsupervised fine-tuning on a few labeled examples, and distillation with unlabeled examples for refining and transferring the task-specific knowledge. This procedure achieves