Research

Learning Probabilistic Models from Generator Latent Spaces with Hat EBM

view publication in Neurips 2023

• Turn any generative model (GAN, VAE etc) into a Probabilistic model.

• Make these models better at image synthesis/sampling

• Performs incredibly well at ODD detection.

Collaboration with Mitch Hill, Erik Nijkamp, Bo Pang and Song-Chun Zhu

Stochastic Security: Adversarial Defense Using Long-Run Dynamics of Energy-Based Models

view publication

• Preprocessing based Adversarial Defense architecture that uses long run MCMC to erase adversarial signals and replace them with class features

• Background supported in generative modeling and non-linear dynamics

• Achieves 54% robustness on BPDA attack, 78% on PGD for CIFAR-10

Collaboration with Mitch Hill and Song-Chun Zhu

Bounding Box Embedding for Single Shot Person Instance Segmentation

view publication

• Wrote and designed the training pipeline, loss functions, data generation, evaluation. Trained and benchmarked on the COCO dataset. Implemented using Keras and Tensorflow.

• Implemented several different network architectures during training to find the best overall structure.

• Experimented with several encoders / decoder combinations:

Encoders/Backbones:
• ResNet50
• ResNet101
• MobileNet
• MobileNetV2
Decoders:
• U-Net
• DeepLabV3+

• Built Atrous Spatial Pyramid Pooling (ASPP) module in order to increase semantic information flow while maintaining receptive field view between our encoder and decoder. Achieved 0.374 AP on the Person class compared to PersonLab’s 0.382 AP, and our network is mobile ready

Instance and Pose Detection network

• Designed a neural network that is able to identity individual person instance masks and human pose given a random image

• Added a human pose detection branch to an existing Bounding Box Embedding network in order to predict human pose and instance masks from a random image.

• Constructed data generation in order to include human keypoints, wrote entire code base from scratch; designed, trained, and tested network on Microsoft's COCO dataset.

• Network is mobile-ready and can run in real time on an iPhoneX’s GPU.

Depth foreground inference

• Design a neural network to produce depth maps from indoor scenes in order to reject background noise when performing human pose detection

• Implement a multi-scale neural network that uses global context, gradient estimation, and refined feature smoothing for depth estimation in Keras

Bounding Box Embedding for Single Shot Person Instance Segmentation

• We present a bottom-up approach for the task of object instance segmentation using a single-shot model. The proposed model employs a fully convolutional network which is trained to predict class-wise segmentation masks as well as the bounding boxes of the object instances to which each pixel belongs.

• Our network architecture is based on the DeepLabv3+ model, and requires only minimal extra computation to achieve pixel-wise instance assignments. See paper

Autonomous Vehicle Lane and Object detector

Detects vehicles and lane lines from a video stream using a support vector machine and computer vision techniques

Computer Vision Lane Line Detection

Basic lane line detection

Machine Learning from 0 to 10 Lecture Slides

Computer Build x 4 GPU

The Dracarys Build | Deep Learning Rig

Computer Build x 8 GPU

The Odin Build | Deep Learning Rig