Feijiang Han

Feijiang Han

NLPer @ UPenn | Developing Efficient, Effective, and Explainable Methods for (M)LLMs | Aspiring PhD and Entrepreneur

❤️ Future Research Directions

While Transformer-based architectures have dominated recent years, I see substantial room for progress across several directions:

1. Fundamental Model Enhancement

Despite extensive progress in both academia and industry, key opportunities remain for improving foundation models:

  • Capability Gaps: Create targeted benchmarks to expose human–AI gaps, then close them with explainable methods, such as: (1) Optimize layer and head interactions (Information Flow); (2) Introduce interpretable decoding control (Token Generation); (3) Enable small models to compete with larger ones (Reasoning); (4) Orchestrate efficient interactions between reasoning and non‑reasoning modules

  • Training vs. Inference: Inference-time adaptations are effective, but scaling during training will finally surpass and substitute them. As resources allow, I will shift emphasis from inference-time tweaks to training-time optimization.

  • Interpretability for Innovation: Use interpretability not only to explain but to improve training. For example, insights from the attention‑sink mechanism (since 2022) have informed KV‑cache optimization, extensions to VLMs, and quantization-aware training.

2. Multimodal Evolution

As LLM research has outpaced multimodal progress, I am especially interested in:

  • Identifying and addressing limitations in current MLLM architectures
  • Developing more efficient architectures for processing multimodal information, considering visual redundancy and modality alignment challenges
  • Exploring novel architectural paradigms beyond current conventions

3. AI for Scientific Discovery

The next frontier is applying LLMs to scientific discovery, with focus on:

  • Problem Identification: Discover valuable new application areas as LLM capabilities expand
  • Targeted Solutions: Adapt and optimize models for specific scientific domains
  • Evaluation Framework: Tackle problems through multiple lenses:
    • Unknown problems (benchmark construction)
    • Known problems with: (1) Simple evaluation but challenging solutions (effective methods) or high‑cost evaluation (efficiency); (2) Easy solutions but complex evaluation requirements (e.g., RLVR)