I am a computer science graduate student at University of Oregon, and am advised by Prof. Humphrey Shi. I work at SHI Lab @ UO as a graduate researcher.
My research interests are attention-based models, data efficiency and self-supervision. My first project, Compact Transformers, looked into data efficent and lightweight vision transformers, directly applicable to small-data regimes.
During my undergrad, I did research under the supervision of Prof. Abbas Salemi at the Mahani Mathematical Research Center.
Top undergraduate science major researcher in the Province of Kerman in 2020
Top undergraduate researcher in the College of Math. and CS & the CS Dept., University of Kerman in 2020
Distinguished undergraduate researcher in the College of Math. and CS, University of Kerman in 2019
CIS 322 (Introduction to Software Engineering) teaching assistant.
Coordinated and held the workshop online, gave lectures and wrote programming examples on data representation and feature engineering, introduction to PyTorch, classical learning algorithms, support vector machines, neural networks, convolutional networks and recurrent networks.
https://arxiv.org/abs/2104.05704
https://raw.githubusercontent.com/SHI-Labs/Compact-Transformers/main/images/model_sym.png
With the rise of Transformers as the standard for language processing, and their advancements in computer vision, along with their unprecedented size and amounts of training data, many have come to believe that they are not suitable for small sets of data. This trend leads to great concerns, including but not limited to: limited availability of data in certain scientific domains and the exclusion of those with limited resource from research in the field. In this paper, we dispel the myth that transformers are "data hungry" and therefore can only be applied to large sets of data. We show for the first time that with the right size and tokenization, transformers can perform head-to-head with state-of-the-art CNNs on small datasets, often with better accuracy and fewer parameters. Our model eliminates the requirement for class token and positional embeddings through a novel sequence pooling strategy and the use of convolution/s. It is flexible in terms of model size, and can have as little as 0.28M parameters while achieving good results. Our model can reach 98.00% accuracy when training from scratch on CIFAR-10, which is a significant improvement over previous Transformer based models. It also outperforms many modern CNN based approaches, such as ResNet, and even some recent NAS-based approaches, such as Proxyless-NAS. Our simple and compact design democratizes transformers by making them accessible to those with limited computing resources and/or dealing with small datasets. Our method also works on larger datasets, such as ImageNet (82.71% accuracy with 29% parameters of ViT), and NLP tasks as well.
arXiv GitHubhttps://arxiv.org/abs/2109.04454
https://raw.githubusercontent.com/SHI-Labs/Convolutional-MLPs/master/images/comparison.png
MLP-based architectures, which consist of a sequence of consecutive multi-layer perceptron blocks, have recently been found to reach comparable results to convolutional and transformer-based methods. However, most adopt spatial MLPs which take fixed dimension inputs, therefore making it difficult to apply them to downstream tasks, such as object detection and semantic segmentation. Moreover, single-stage designs further limit performance in other computer vision tasks and fully connected layers bear heavy computation. To tackle these problems, we propose ConvMLP: a hierarchical Convolutional MLP for visual recognition, which is a light-weight, stage-wise, co-design of convolution layers, and MLPs. In particular, ConvMLP-S achieves 76.8% top-1 accuracy on ImageNet-1k with 9M parameters and 2.4 GMACs (15% and 19% of MLP-Mixer-B/16, respectively). Experiments on object detection and semantic segmentation further show that visual representation learned by ConvMLP can be seamlessly transferred and achieve competitive results with fewer parameters.
arXiv GitHub